A log shipping agent cannot reach its destination for 40 minutes — what are its options, and what does a disk-backed buffer change over an in-memory one?

When the destination goes away, the log pipeline does not stop producing. A parking-permit service still handles renewals at 1,240 requests a minute, still writes a line each, and the agent on each node has to do something with them for the next 40 minutes. What it does was decided by configuration nobody revisits until this moment. ## The four options 1. **Buffer in memory.** Records accumulate in the agent's process. This is fast and simple, is bounded by whatever memory limit the agent runs under, and evaporates completely if the agent is restarted or killed for exceeding that limit. On a busy node, memory buffers measure their capacity in minutes. 2. **Buffer to disk.** Records are written to files on the node and drained when the destination returns. Capacity goes from minutes to hours, and the buffer survives an agent restart or upgrade. 3. **Stop reading (backpressure).** The agent simply stops consuming, leaving the data where it is. Where the source is a file, this means not advancing the read offset. 4. **Drop.** Discard records once the buffer is full, choosing which end deliberately. ## Memory buffer versus disk buffer | | In-memory buffer | Disk-backed buffer | | --- | --- | --- | | Capacity | minutes of traffic, bounded by RAM | hours, bounded by a disk allowance | | Survives agent restart | no | yes | | Cost when idle | none | file space, plus write I/O on every batch | | Failure mode | agent killed for memory, everything lost | node disk fills, harming co-located workloads | | Effect on latency | none | small, from writing before sending | The disk buffer's failure mode is the one people miss: the buffer is on a resource shared with everything else on the node. An unbounded queue that fills the node's disk turns a destination outage into a node outage, so its allowance must be capped explicitly, and what happens at that cap is the real design decision. ## Backpressure has a destination too Backpressure is not a way of avoiding a decision; it moves the problem to whoever is upstream. - **With file sources**, refusing to advance the read offset leaves the lines in the container's log files. Those files rotate on size and old segments are deleted, so a long enough stall means the agent's next read finds that the data it was waiting on has been removed. This is the classic **silent** loss: nothing errored, records simply never existed downstream. - **When the application writes directly into the pipeline** — over a socket, or through a shipping library in-process — backpressure reaches the service. A synchronous logging call that blocks turns a log-destination outage into a latency incident, and then into an availability incident when the request threads are all parked in a logging call. Logging must degrade before the service does: a non-blocking appender with a bounded queue that drops, and a counter for what it dropped. ## At least once, and what it means to the reader The read offset — or the acknowledgement of a batch — only advances when the destination confirms receipt. If the destination processed a batch and the acknowledgement was lost, the agent retries and the records land twice. That is **at-least-once** delivery, and it is the honest default: the alternative, advancing before confirmation, silently loses data instead. For whoever reads the logs later, that has consequences: - **Counting log lines is not counting events.** Seeing three copies of an error may mean one failure delivered three times. - **Deduplication needs identity the emitter supplied.** Timestamp plus text is not identity, because two identical retries of the same real operation look the same as one duplicated record. If exact counts matter — evidence a regulator will ask for — the emitter has to write a unique event identifier. - **Order is not preserved across a retry.** A retried batch arrives after records produced later, so the store's arrival time diverges from the event time. Query on the emitted timestamp, and be careful with anything that assumes monotonic ingestion. ## Draining without causing the second outage When the destination comes back, an agent holding 40 minutes of records will try to ship them as fast as it can, and a fleet of agents doing that simultaneously is a thundering herd aimed at a service that has just recovered. Two habits prevent the follow-on incident: cap the drain rate so recovery traffic is a bounded multiple of normal ingest, and jitter retry backoff so the fleet does not reconnect in lockstep. ## What to have decided in advance 1. Buffer type and an explicit **size cap**, on a disk allowance that is not the same space the workloads need. 2. **What happens at the cap** — drop oldest, drop newest, or block — decided per source rather than left to a default. 3. A **dropped-records counter**, exported as a metric and alerted on, so a gap is visible without reading the logs you have just lost. 4. A **drain rate limit** and jittered backoff. 5. An agreement with readers that the pipeline is at-least-once, so nobody builds a count that assumes otherwise.