skip to content

An upload endpoint accepts compressed request bodies. How do you bound what a small compressed body can expand to?

level: seniorimportance: nice to knowfreq 34%

answer

  1. the input size stops predicting the cost
  2. count the output, not the input
  3. abort inside the loop, mid-stream
  4. absolute cap plus a ratio with a floor
  5. nested layers multiply the expansion

basics

~20 s

Count decompressed bytes as they are produced and abort the moment the running total crosses an absolute cap, with a secondary check on the output-to-input ratio so a bomb dies early. Measuring the size after decompression has already paid for it.

solid answer

~50 s

General-purpose compression is built to turn repetitive input into very few bytes, and that property runs backwards: a highly repetitive stream can expand by three orders of magnitude or more under algorithms such as DEFLATE, Zstandard or LZ4. So a body that passes any sane request-size limit can still produce gigabytes. Two ceilings are needed, not one. An **absolute output cap** bounds total decompressed bytes and is what actually protects memory or disk. An **expansion-ratio cap** — bytes produced over bytes consumed, checked once output passes a small floor — kills an obvious bomb in the first few kilobytes instead of letting it run to the absolute cap. Both are tested inside the decompression loop, before the block is handed on, and both abort and discard. Also cap how many compression layers you will unwrap, because nested layers multiply.

code

pseudocode · 17 lines
pseudocode
produced = 0
consumed = 0
while true:
    block = decompress_next_block()      // consumes input, yields output
    if is_end_of_stream(block):
        break
    produced = produced + length(block)
    consumed = bytes_consumed_so_far()

    if produced > MAX_OUTPUT_BYTES:
        abort("output cap exceeded")
    if produced > RATIO_FLOOR_BYTES and produced > MAX_RATIO * consumed:
        abort("expansion ratio exceeded")

    write(block, to = sink)

// abort() discards the sink, including any partial temporary file

go deeper

for a junior

Remember that a compressed upload can be far larger once expanded, so the size of the request that arrived is not a measure of the work it causes.

for a middle

Explain the counter inside the decompression loop and why the check must abort mid-stream rather than measure the result once it exists.

for a senior

Show the pair of ceilings and where each fails alone, handle nested layers, and say what the service returns, deletes and records when one trips.

for a principal

Decide where expansion is allowed to happen at all across the fleet, what the output budget is worth against the traffic it serves, and who reviews that number.

## Why compressed input is a special case Every other decode ceiling can be expressed against the bytes the sender transmitted. Compression breaks that link on purpose: its job is to make the transmitted size much smaller than the represented size. A highly repetitive stream — the same byte, or the same short pattern, repeated — compresses spectacularly well under the usual algorithms, and a body comfortably under any request-size limit can therefore describe an output measured in gigabytes. This is not a flaw in the algorithms; it is the property that makes them useful. It simply means the request-size limit at your edge bounds the **cheap** half of the transaction and says nothing about the expensive half. ## Two ceilings, and what each one buys | Ceiling | What it bounds | What it misses on its own | |---|---|---| | Absolute output bytes | Total memory or disk one decode can consume | A body that stays just under it on every request, repeatedly | | Expansion ratio (output over input) | How suspicious the stream is, detectable early | A large input at an ordinary ratio, which is scale-free and passes | | Number of compression layers | Nested wrappers that multiply the expansion | Nothing on its own; it exists because layers compose | The absolute cap is the one that actually protects the machine, because it is denominated in the resource you are trying to preserve. The ratio cap is an early detector: a bomb reaches a wild ratio within the first few kilobytes of output, so the ratio check ends it long before the absolute cap would. ## The ratio check needs a floor A running ratio is meaningless at the start of a stream. Headers, dictionaries and the first block make the quotient jump around, and a perfectly legitimate short body can briefly look like a bomb. So the ratio branch only applies once output has passed a floor — a few tens of kilobytes is a reasonable starting point — and below that floor only the absolute cap governs. A ratio check without a floor produces false rejections on small ordinary uploads, which is how the whole defence gets switched off in production. ## Where the output goes matters too - **Into memory**: the failure is an out-of-memory condition affecting every request the process is serving, not just this one. - **Into a temporary file**: the failure is a full disk, which is often worse, because it outlives the request and can take down logging, the database and anything else sharing that volume. - **Straight into a downstream call**: you become the amplifier, and the ceiling you did not enforce is now someone else's outage. In all three cases the counter goes in the same place: the loop that produces output, checked before the block is written anywhere. ## Layers multiply If a container holds a compressed member that is itself compressed, the expansion factors multiply rather than add. Two layers at 1000:1 is 1,000,000:1. Two defences follow: cap the number of layers you will unwrap — for most endpoints that number is one — and charge **all** layers against a single output budget, rather than resetting the counter for each nested decode. ## Never trust a declared uncompressed size Many containers carry a field stating how large the decompressed data will be. It is written by whoever produced the container, so it is a claim, not a bound. Use it, if at all, to reject early — a declaration above your cap is an immediate rejection — never to size a buffer and never as a reason to skip the running count. ## What the endpoint does when a ceiling trips Abort the decompression, discard everything produced so far, delete the partial temporary file if there is one, and return a payload-too-large status. Do not return the number of bytes produced before the abort or the observed ratio; keep both in the log, alongside the caller and the request id, and count breaches as a metric. As with every other decode ceiling, the metric is what distinguishes an attack from a limit set below the traffic you actually serve. ## Stating the principle The rule generalises past compression: **the resources a decode may consume must be bounded by constants you chose, and checked as they are consumed.** Compression is simply the case where the sender's byte count stops being a usable proxy for that consumption, which is why an independent output counter is not optional here.

  • Why is an expansion-ratio cap on its own not enough?
    Because a ratio is scale-free. A 100 MiB upload at an unremarkable 20:1 produces roughly 2 GiB of output and breaks no ratio rule at all, while every individual block looks ordinary. The absolute cap is the one denominated in the resource you are protecting; the ratio cap only makes an obvious bomb fail sooner.
  • What changes when the payload is compressed twice?
    The expansion factors multiply rather than add, so two layers at 1000:1 each is a million to one. Cap how many layers you will unwrap — one is the right answer for most endpoints — and charge every layer against a single output budget instead of restarting the counter for each nested decode.
  • Should the decompressed output go to memory or to a temporary file?
    Either is defensible, and both need the same counter. A file moves the failure from out-of-memory to a full volume, which outlives the request and can take down logging and anything else sharing the disk. If you spool to a file, cap it, delete the partial file on abort, and keep the spool area on a volume nothing critical shares.

saying these in an interview costs you the question

  • Decompresses to a temporary file and checks its size afterwards
  • Trusts a declared uncompressed size in the container header
  • Believes a ratio cap alone bounds absolute memory use
  • Assumes the compression library imposes a default output limit
  • Thinks a small request-size limit makes expansion harmless
  • Resets the output counter for each nested compression layer