skip to content

Before Promtail ships a line to Loki, what must it decide about labels, multi-line entries and timestamps?

level: seniorimportance: nice to knowfreq 32%

answer

  1. the agent fixes things storage cannot undo
  2. discovery metadata is not a label set
  3. one event can span many lines
  4. read time is not event time
  5. entries per stream arrive broadly in order

basics

~20 s

The agent picks the stream labels through discovery and relabelling, joins continuation lines into one entry, and sets each entry's timestamp. All three are expensive to revisit: labels and timestamps are baked into chunks that are immutable once flushed.

solid answer

~50 s

Promtail's `scrape_configs` discover targets and `relabel_configs` turn the discovered metadata into the final stream labels — this is where you drop the meta labels that would otherwise become high-cardinality streams. Its `pipeline_stages` then process each line: a `cri` or `docker` stage strips the container runtime's framing, a `multiline` stage with a `firstline` regex joins a stack trace into one entry rather than forty, and a `timestamp` stage parses the real event time out of the line instead of letting the read time stand in. A positions file records how far each source has been read, so a restart resumes rather than replaying. Ordering matters too: entries within one stream should arrive in timestamp order, and a recent Loki accepts out-of-order writes only inside a bounded window. Get any of it wrong and the damage is already in immutable chunks.

code

yaml · 19 lines
yaml
scrape_configs:
  - job_name: planner
    static_configs:
      - targets: [localhost]
        labels:
          app: planner
          env: prod
          __path__: /var/log/planner/*.log
    pipeline_stages:
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
          max_wait_time: 3s
      - regex:
          expression: '^(?P<ts>\S+)\s+(?P<level>\w+)\s+(?P<msg>.*)$'
      - labels:
          level:
      - timestamp:
          source: ts
          format: RFC3339

go deeper

for a junior

Know that something has to tail the logs and push them, that it attaches the labels, and that a stack trace needs help to be treated as one entry rather than many.

for a middle

Explain the pieces: scrape configs and relabelling produce the label set, a multiline stage joins continuation lines using a first-line pattern, and a timestamp stage replaces read time with event time.

for a senior

Show the consequences. Chunks are immutable, so a bad label or a split entry persists for the whole retention period, and out-of-order writes only survive inside a bounded window — which is why backfill and merged sources fail.

for a principal

Treat the agent configuration as the estate's contract. Decide the standard label set every service ships, who reviews changes to it, and how you migrate a fleet's collection config without a retention period of mixed shapes.

## Three decisions, all of them one-way doors A log-shipping agent — Promtail, or Grafana Alloy, which Grafana has moved this role into — is not a dumb pipe. Before a line reaches Loki's push endpoint, the agent has fixed which stream it belongs to, where the entry begins and ends, and when it happened. Chunks are immutable once flushed, so all three decisions are written into storage permanently. Query-time cleverness can compensate for none of them. ## Decision one: which labels to attach `scrape_configs` discover sources — files matched by a `__path__`, a systemd journal, syslog, or containers found through Kubernetes service discovery — and `relabel_configs` rewrite the discovered metadata into the final label set. This is the leverage point for stream cardinality, because discovery hands the agent far more metadata than belongs in an index. Pod names with generated suffixes, container ids, image digests and node identifiers all arrive as meta labels, and every one of them churns the stream population if it survives to the push. Relabelling is where you keep application, environment, namespace and severity and drop the rest. | Decision | Where it is made | What it costs to change later | |---|---|---| | Final stream labels | `relabel_configs` in a scrape config | Old chunks keep their old label sets until retention expires them | | Entry boundaries | A `multiline` or `cri` pipeline stage | Split lines are stored split; no query can rejoin them | | Entry timestamp | A `timestamp` pipeline stage | Entries are already placed in chunks by that timestamp | | Read position | The positions file | Only affects replay and gaps on restart | ## Decision two: where an entry begins and ends A source emits lines; an application emits events, and a Java stack trace is one event across forty lines. Whoever decides which is which has to be the agent, because Loki stores exactly what it is given. - A `multiline` stage joins lines using a `firstline` regular expression that identifies the start of a new entry — typically a timestamp or a severity token at the beginning of the line. Everything not matching it is appended to the entry in progress. - Its `max_wait_time` bounds how long a partial entry is held before being flushed anyway, which matters when the last line of a trace is also the last line before the process goes quiet. `max_lines` bounds how large a joined entry can grow, so one pathological trace cannot consume unbounded memory. - A `cri` or `docker` stage handles the container runtime's own framing first, including the runtime's own splitting of long lines into partial fragments. The consequence of skipping this is not cosmetic. A stack trace stored as forty entries means a line filter matches the one line containing the exception class and shows you nothing around it, and any per-event counting is inflated by the number of continuation lines. ## Decision three: the timestamp, and the order Every entry carries a nanosecond timestamp. Without a `timestamp` stage, that is the moment the agent read the line, which drifts from the event time whenever the agent was restarted, was backlogged, or is replaying a file from its recorded position. Parsing the real time out of the line fixes that, and immediately introduces the ordering problem. Loki accumulates entries into per-stream chunks and expects them broadly in increasing timestamp order for a given stream. Assume a recent release: older Loki rejected any entry for a stream that was not newer than the last one accepted, while current versions accept out-of-order entries inside a bounded window, configurable per tenant and limited in practice by how long a chunk stays open. Streams are independent of one another, so lateness in one does not affect another. That makes two real failure modes worth naming: 1. **Backfill.** Replaying yesterday's files into a stream that is still receiving today's lines pushes entries far outside the window, and they are rejected rather than reordered. 2. **Merging sources.** Attaching one label set to several hosts' files merges them into a single stream whose entries interleave from clocks that do not agree. Keeping a host or instance label separates them into independent streams and removes the problem entirely. ## Why the ordering of decisions matters operationally For a maintenance-planning platform on 41 hosts running 12 containers each, with an on-call rotation of three people, the agent configuration is the only place where the whole estate's log shape is decided at once. A bad label survives in immutable chunks for the length of retention. A missing multiline stage makes every incident's stack traces unreadable in exactly the moment somebody needs them. A missing timestamp stage puts the incident's logs minutes away from the metrics that led you there. None of the three is recoverable at query time, which is why review of an agent configuration change deserves more care than its size suggests.

  • Why does Promtail keep a positions file, and what breaks if it is lost?
    It records how far each source has been read, so a restart resumes at the right offset instead of replaying the file from the start. Lose it and the agent re-reads whatever is still on disk, pushing duplicate entries whose timestamps are far behind the stream's current position — which is both a duplication problem and an out-of-order problem at once. Storing it on a volume that survives the agent's own restarts is a small configuration detail with a large blast radius.
  • A stack trace shows up in Loki as forty separate entries. Can you fix it at query time?
    Not properly. LogQL can format and filter what is stored, but the entries are forty rows with forty timestamps and no marker saying they belong together, so nothing reliably rejoins them. The fix is a multiline stage at the agent with a firstline pattern matching the start of a real entry, and it only helps from the moment it is deployed — the traces already in flushed chunks stay split for the rest of their retention.
  • Which is safer for a merged log source: one shared label set, or a per-host label?
    Per-host, in almost every case. Merging several hosts under one label set puts their entries into a single stream, so clocks that disagree by even a second produce out-of-order writes that can be rejected. A host or instance label splits them into independent streams whose ordering constraints are independent too. It costs you one more label of bounded cardinality, which is exactly the kind of label the stream index is designed for.

saying these in an interview costs you the question

  • Treats the shipping agent as a dumb pipe with no decisions
  • Passes every discovered meta label straight through as a stream label
  • Expects a query to rejoin a stack trace split across lines
  • Uses read time as event time and never notices the drift
  • Thinks out-of-order entries are silently reordered by the store
  • Backfills old files into a stream that is still receiving live writes