fix(alloy): drop docker log lines older than Loki accepts

loki.source.docker resumes from a one-second position and Docker's since
is inclusive, so each restart re-reads a container's last logged second.
For idle containers those lines are older than 7 days and Grafana Cloud
Loki rejects the batch with 400 timestamp too old. Drop them before
loki.write; Loki would not have stored them anyway.
This commit is contained in:
tiennm99 committed 2026-10-06 16:56:46 +07:00
1 parent feb6f035c8
commit d889338306
4 files changed
+55 -2

No files matched your search

@@ -0,0 +1,37 @@
# Docker logs re-sent on restart
Applies to `alloy/compose.yml`. Checked against Alloy v1.20.1.
## Symptom
Every time Alloy restarts, `loki.write` logs one rejected batch:
```
final error sending batch, no retries left, dropping data ... status=400
13 errors like: entry for stream '{container="coolify-realtime", ...}'
has timestamp too old: 2026-09-18T17:00:51Z, oldest acceptable timestamp is: ...
```
The same containers, the same timestamps and the same counts show up on every
restart.
## Cause
`loki.source.docker` saves each container's read position as a Unix timestamp
in **seconds** (`internal/component/loki/source/docker/tailer.go`,
`t.positions.Put(..., ts.Unix())`). On start it asks the Docker API for logs
`since` that second, and Docker includes the second itself. So every restart
re-reads every line written in the container's last logged second.
For a busy container those lines are recent, and Loki silently discards them
as exact duplicates. For an idle container whose last line is more than 7 days
old, Grafana Cloud Loki rejects them with `timestamp too old` instead.
Positions are kept in the `alloy-data` volume, so they are not being lost.
Re-reading that last second is simply how the component works.
## Fix
`loki.process "logs_integrations_docker"` sits between `loki.source.docker` and
`loki.write` and drops lines older than `168h` with `stage.drop`. Those lines
would be rejected by Loki anyway, so nothing that would have been stored is
lost. The drops are counted under `loki_process_dropped_lines_total{reason="too_old"}`.
+1
View File
@@ -51,6 +51,7 @@ Where official mixin dashboards exist they're tier-2 corroboration:
| `prometheus.exporter.cadvisor` (`docker_only = true`) | Docker integration page |
| `prometheus.relabel "integrations_cadvisor"` (`keep` allowlist of 16 metrics) | Docker integration page → [Metrics](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-docker/#metrics) section, verbatim |
| `discovery.docker` + `loki.source.docker` (job/instance/container/stream) | Docker integration page → log scraping |
| `loki.process "logs_integrations_docker"` (drop lines older than 168h) | Local addition, not upstream. Drops only what Loki rejects anyway; see [docker-logs-resent-on-restart.md](docker-logs-resent-on-restart.md) |
## What "follow upstream" means in practice