What do Logstash's persistent queue and dead-letter queue each protect, and how does backpressure reach Filebeat?

## Two queues, two entirely different jobs Logstash has two on-disk structures whose names sound similar and whose purposes do not overlap at all. Confusing them is the single most common mistake on this subject. | | Persistent queue | Dead-letter queue | |---|---|---| | Sits | between the input and the filter stage | after an output that rejected an event | | Holds | events accepted but not yet fully processed | events an output was told it can never deliver | | Protects against | process crash, restart, transient output outage | permanently malformed or unindexable documents | | Turned on with | `queue.type: persisted` in `logstash.yml` | `dead_letter_queue.enable: true` | | Drains by | the pipeline catching up | a human, via a pipeline reading the queue | ## The persistent queue: what it changes over an in-memory one By default a pipeline's queue lives in memory. It holds a bounded number of in-flight batches, and if the process is killed — an out-of-memory kill, a node reboot, a bad deploy — every event sitting in it is gone. There is no copy anywhere else, because the input already told the sender it had the data. Setting `queue.type: persisted` writes each incoming batch to an on-disk queue under `path.queue` *before* the filter stage runs, and only acknowledges an event once it has been processed and handed to the outputs. Three consequences follow: 1. **In-flight events survive a restart.** On startup the pipeline replays what was never acknowledged. Duplicates are possible on the replayed batch; loss largely is not. 2. **The queue absorbs an outage.** If the destination is unavailable, events accumulate on disk up to `queue.max_bytes` instead of immediately pushing back on the shipper. That converts a short outage into a delay rather than an incident. 3. **You pay for it.** Every event is now a disk write with periodic checkpointing (tunable through `queue.checkpoint.writes`), so throughput drops and the queue directory becomes a disk you must size and monitor. A full persistent queue does not silently discard — it blocks, which is the correct behaviour and is what makes backpressure work. The queue is per pipeline, not per node, and it lives on that node's local disk. It is durability against a process dying, not against the machine being destroyed. ## The dead-letter queue: what actually lands there The dead-letter queue exists for the opposite problem: an event that will *never* succeed no matter how many times it is retried. In practice it is fed by the Elasticsearch output. When the cluster rejects a document with a response the plugin treats as non-retryable — a type conflict, a malformed value, a rejected field — the event is written to the dead-letter queue with the failure reason attached instead of being retried forever or dropped. Responses that *are* retryable, such as a rejection caused by back-pressure or a server error, are retried rather than dead-lettered. Its size is bounded by `dead_letter_queue.max_bytes` and a policy that decides what happens when it fills, so it is not an unbounded archive of your mistakes. The part interviews actually probe is the last line of the definition: **the dead-letter queue is only useful if someone reads it.** Nothing drains it automatically. You read it back with a pipeline whose input is the `dead_letter_queue` plugin, fix the offending field, and re-index. Teams that enable the setting, never build that pipeline and never alert on the queue's size have converted silent data loss into silent data loss with extra disk usage. On a district-heating billing platform, "we still have the events, they are in a queue nobody opened for four months" is not an answer that survives a regulator asking for six months of evidence. ## How backpressure reaches all the way back to the file The chain is worth being able to recite, because it is what makes the Elastic stack lose data or not: 1. Elasticsearch rejects or slows writes — usually because indexing threads are saturated. 2. The Logstash `elasticsearch` output retries with backoff and stops draining its batch. 3. Filter workers finish what they hold and cannot hand anything on, so the queue stops draining. 4. The queue fills — to its memory bound, or to `queue.max_bytes` on disk — and the input stops accepting. 5. The `beats` input stops acknowledging, so Filebeat's protocol, which waits for acknowledgement, stops sending and stops reading new lines. 6. Filebeat holds its position in its registry, and the log **file on disk** becomes the buffer. Step 6 is where the real risk lives. The file is a good buffer right up until log rotation removes a file Filebeat has not finished reading, at which point those lines are gone and no queue anywhere downstream ever saw them. That is why a persistent queue sized for the realistic outage window, and retention on the application hosts sized for the same window, are the same decision made in two places. On an 18 GB-a-day stream, a two-hour cluster outage is roughly 1.5 GB to hold somewhere — trivial on disk, fatal if the answer is "in memory".