Why does an archival tier re-compressing already-compressed or encrypted blocks get output slightly larger than the input?
answer
- nothing left to model
- skew and repeats already consumed
- ciphertext looks uniform by design
- framing costs bytes either way
- compress first, then encrypt
basics
~20 sAlready-compressed and encrypted blocks have no repeated sequences and a near-uniform byte distribution, so the coder finds nothing to exploit and emits at least as much as it read, plus headers — a few bytes of growth per block.
solid answer
~40 sA lossless coder wins in exactly two ways: it spends short codes on frequent symbols, and it replaces repeats with references. A well-compressed payload has had both consumed already, and a cipher's output is required to look uniform to anyone without the key, so both kinds of structure are absent by design rather than by accident. With nothing to convert into savings, the coded form is at least as long as the input, and the container still writes its per-block header, its stream preamble and its integrity check. The counting bound predicts exactly this: some inputs must pay, and every sensible format arranges for the ones that look like noise to be the payers. The growth is therefore a few bytes per block, not a defect and not something a higher effort level removes.
go deeper
Remember that a coder saves bytes only by exploiting repeated sequences and uneven symbol frequencies, and that compressed or encrypted bytes contain neither.
Explain both halves: why the structure is absent by design in each case, and why the container's per-block header and stream framing still cost bytes when nothing is coded.
Recognise the pattern from the size delta alone, place compression before encryption in the pipeline, and decide which input classes are worth attempting at all.
Weigh the processor cost of attempting compression across a whole corpus against a saving that counting says cannot exist for part of it, and set the policy per input class.
## What a lossless coder actually has to work with Every lossless coder converts one of two kinds of structure into fewer bits, and it has nothing else: - **Skew.** Some symbols occur far more often than others, so the coder can spend short codes on the frequent ones and long codes on the rare ones. When every byte value is about equally likely, every code ends up about the same length and the trade is a wash. - **Repetition.** A run of identical bytes, or a sequence that appeared earlier within the coder's search window, can be replaced by a short reference. When no sequence ever repeats in that window, there is nothing to point back at. A block with neither is not *difficult* input. It is input that is empty of the raw material the coder turns into savings, and no amount of search finds something that is not there. ## Why compressed output and ciphertext both look that way - The output of a good compressor has, by construction, had its skew and its repeats consumed. Structure left behind would mean the first coder missed savings it was built to take. - A cipher's output is required to be indistinguishable from uniform random bytes to anyone without the key. Any statistical structure a compressor could exploit would be exactly a distinguisher, which is a break of the cipher. A sound construction leaves none, deliberately. - Neither statement is about a particular tool. Both follow from what the two mechanisms are for, which is why the effect is the same across every ecosystem. ## Where the extra bytes come from The growth is framing, not coding. Even when the writer gives up and stores the block's original bytes, the container still describes what it wrote. | Source of growth | Rough size | Written even for a raw-stored block? | |---|---|---| | Stream preamble and trailer | tens of bytes, once per stream | yes | | Per-block header: type and length | a few bytes per block | yes | | Integrity check over the stream | a few bytes | yes | | Model or code-table description | tens to hundreds of bytes per coded block | no | So the floor for a wholly incompressible payload is `input + (number of blocks x header) + a constant`. That is why the answer is 'a few bytes larger', not 'twice the size'. ## The counting bound predicted the victim 1. Counting guarantees that some inputs must fail to shrink, and that a scheme which shrinks anything must grow something. It does not say which inputs. 2. The designer picks. Every practical format arranges for the loss to land on inputs that look like uniform noise, because those are the inputs nobody expects to save on. 3. Already-compressed and encrypted blocks are precisely that class. They are not unlucky; they are the ones the format chose to charge, and the charge was capped at a header. ## The ordering consequence - **Compress, then encrypt.** The coder sees the structured plaintext and wins; the cipher then hides the smaller result. - **Encrypt, then compress.** The coder sees uniform bytes and loses, permanently, no matter which scheme is chosen. - One caveat belongs on the record: when a secret and attacker-influenced data are compressed in the *same* context, the observable compressed length depends on how well the two match, which leaks information about the secret. Where that risk applies, the answer is to keep them in separate compression contexts, or not to compress, never to reverse the order in the hope of a saving that does not exist. ## Three things this is not - **Not corruption.** A round trip still returns the original bytes exactly; only the stored size moved. - **Not a tuning failure.** Raising the effort level makes the coder search harder over a space with nothing in it: more processor time, the same output. Where ecosystems differ is in how loudly the tooling reports the fallback, not in whether the fallback happens. - **Not fixed by another pass.** Pass two sees the near-uniform output of pass one and adds a second layer of framing on top of the first. ## What to say in review Name the class of the input, not the tool. 'These blocks are already compressed, so the coder has no skew and no repeats to trade on; counting says something has to pay and this is the class that pays. The growth is one header per block. The decision in front of us is whether to attempt coding on this class at all, not which coder to attempt it with.' That sentence ends the thread, and it costs no benchmark.
- Why does a cipher's output resist compression even when the plaintext compressed beautifully?Because indistinguishability is the security requirement. Any skew or repetition a coder could exploit is also a statistical test that separates ciphertext from random bytes, which is a break. A sound construction therefore leaves no exploitable structure, and the coder finds exactly nothing.
- If compressing before encrypting saves space, when is that ordering still unsafe?When a secret and attacker-influenced data share one compression context. The compressed length shrinks when the attacker's guess matches the secret, and that length is visible even though the bytes are not. The fix is separate contexts, or no compression on that path.
- How big is the growth in practice, and what controls it?Roughly one block header per block, plus a fixed stream overhead. It is controlled by block size rather than by the coder: doubling the block size halves the number of headers. With blocks around 64 KiB and headers of a few bytes, the total is under a hundredth of a percent.
saying these in an interview costs you the question
- Blames the tool and proposes a stronger coder as the fix
- Thinks encrypted data compresses if the effort level is raised
- Believes encrypting first and compressing after still saves space
- Assumes an incompressible input can come back twice its size
- Treats high entropy as a property of the coder, not the data