How would you tune decoding differently for a throughput-bound batch job versus a tail-latency-bound request path?

## One workload optimises the mean; the other optimises the worst case Both workloads decode the same bytes, but the number each is graded on is different, and that difference inverts several decisions. - **Throughput-bound batch decoding** is graded on total work per unit of input: records per second, or processor time per gigabyte. Individual record latency is invisible as long as the job finishes. - **Latency-bound request decoding** is graded on a high percentile of individual requests. A change that improves the average while adding a rare hundred-millisecond stall is a regression. The practical consequence is that batch tuning is an **amortisation** problem and request-path tuning is a **variance** problem. ## What batch tuning looks like 1. **Amortise setup.** Reader construction, schema resolution and buffer acquisition happen once per batch, not once per record. 2. **Prefer large, sequential buffers.** Big reads and linear scans exploit prefetching; the memory ceiling is whatever the machine has, and nothing is waiting. 3. **Parallelise across records.** Records are usually independent, so partition the input and decode with as many workers as there are cores. 4. **Accept pauses.** A collector configuration favouring throughput over pause time is the right choice when no one is waiting on an individual record. 5. **Decode thoroughly.** If the job reads most fields anyway, eager materialisation is simpler and often faster than deferral. ## What request-path tuning looks like Here every source of *bimodal* cost is a target, because the tail is where those land: - **Growth and resizing.** A buffer that doubles, or a container that rehashes, makes one message in a few hundred far more expensive. Pre-size the destination from the message's own framing where it exposes one. - **Pauses.** Allocation per request is the lever that removes work rather than rescheduling it; materialising only what the handler reads is usually the largest single reduction available. - **Warm-up and first-call effects.** The first request after a deploy, or after a lull, pays costs no steady-state benchmark shows. Warm the decode path before taking traffic. - **Shared state.** A single global pool guarded by one lock becomes the tail under concurrency. Per-worker resources trade a little memory for the absence of contention. - **Unbounded per-message work.** The cost of a request must not be dominated by how large a caller chose to make its payload; bound the work per message so one large input cannot set the percentile for everyone. | Decision | Batch, throughput-bound | Request path, latency-bound | |---|---|---| | Graded on | records per second, processor time per byte | high percentile per request | | Buffers | large, amortised over the batch | pre-sized per message, reused per worker | | Parallelism | across records, all cores | across requests; per-message parallelism rarely pays | | Pauses | acceptable if the average improves | the thing being eliminated | | Field materialisation | eager is fine if most are read | only what the handler reads | | Pooling | global pools are fine | per-worker, to avoid contention | | Benchmark | total wall-clock over a real corpus | percentiles under sustained load, after warm-up | ## The measurement mistake that causes most of the damage The single most common failure is using the wrong instrument. A microbenchmark that reports mean operations per second is a **throughput** instrument. Applied to a request path it will happily endorse a change that raised the ninety-ninth percentile, because rare events barely move an average. Latency work needs a load generator that holds an arrival rate independent of how fast the system responds, runs long enough for the collector to reach steady state, and reports percentiles — not a loop that measures its own best case. ## Where the judgment actually sits At the level where someone owns the budget, two further options outrank any reader tuning. - **Move the work.** If a request path decodes a large payload to use a fraction of it, the fix may be a narrower message on that hop rather than a faster reader — which is a contract change, and therefore an organisational one. - **Split the workloads.** If the same service serves both a batch consumer and a request path from one code path, it is being tuned for two objectives at once and will satisfy neither. Separating them, at the cost of a second deployment to operate, is often the honest answer. And state the ceiling plainly: decoding is rarely the only cost in a request. Before funding a decode optimisation, establish what share of the percentile it actually owns, because a fifty-percent improvement to fifteen percent of the budget is a seven-and-a-half-percent improvement overall.