In self-hosted Langfuse v4, what does the worker container do that the web container does not?

## The shape of the pipeline An LLM application produces a lot of small writes, in bursts, from processes that must not be slowed down by observability. Langfuse v4 answers that with an asynchronous pipeline and a two-container topology, both stateless and both horizontally scalable. The path of a single event: 1. The SDK batches events in the background and posts them to the ingestion endpoint on the **web** container. 2. Web authenticates the project keys, writes the raw event bodies into the S3-compatible bucket, and enqueues references on Redis or Valkey. 3. Web returns to the SDK. From the application's point of view, the trace is delivered. 4. The **worker** consumes the queue, reads the raw payloads, and writes rows into the ClickHouse trace, observation and score tables. 5. The UI, served by web, reads back from ClickHouse and Postgres. Step 3 is why the split exists. The instrumented application never waits on an OLAP insert, and a burst of traffic becomes queue depth rather than request latency or dropped data. ## What else lives on the worker The worker is not only an ingestion consumer. Background work that must not run in a request goes there: server-side evaluator runs, which call an LLM per sampled trace and therefore take seconds each; batch exports and integrations; and the retention job that deletes aged data from ClickHouse and blob storage. That matters for sizing, because an evaluator sampling a busy project competes for the same worker capacity as ingestion. ## The failure mode you must recognise This architecture produces one signature incident, and it is the reason the question is asked at senior level. When the worker is stopped, crash-looping, or simply out-scaled by traffic: - The SDK is happy. Ingestion returned success, so the application logs nothing. - The UI shows no new traces, or traces that lag by minutes and then hours. - Redis queue depth climbs steadily. - The blob bucket keeps growing, because raw events are still being accepted and stored. Nothing in the instrumented application surfaces this. You find it by monitoring queue depth and worker throughput, and by alerting on ingestion-to-visible lag rather than on API errors. "Traces are missing" from a developer is far more often a worker problem than an SDK problem, and the second thing to check, after confirming the SDK flushed, is the queue. ## Scaling and sizing Because both containers are stateless, both scale out. They scale on different signals: - **web** scales with UI users and ingestion request rate. - **worker** scales with event volume and background job load. A common misconfiguration is running one replica of each and assuming symmetry. A chatty agent application generates many observations per user request, so worker capacity is usually the binding constraint long before web is. Equally, a short-lived job that exits immediately after work can lose events that never left the SDK buffer, which is a client-side flush problem rather than a worker one, and the two get confused constantly. ## Interview framing Say the split explicitly: web is the synchronous accept path plus the read path, worker is the asynchronous write path plus background jobs, Redis is the seam, blob storage is the durable landing zone that makes the fast accept safe. Then name the silent-failure consequence. That combination, mechanism plus operational signature, is what distinguishes a real answer from a diagram recital.