A nightly report job Base64-encodes a 2.4 GB artifact into one JSON field. What goes wrong, and how would you fix it?

### The arithmetic first Base64 reads three input bytes and emits four characters, padding the final group. The encoded size is therefore `4 * ceil(n / 3)` — a growth factor of 4/3, about 33%, plus at most two padding characters. For a 2.4 GB artifact that is 3,200,000,000 characters. Since the Base64 alphabet contains no JSON metacharacters, `json.dumps` adds only the two quotes and escapes nothing, so the field really is 3.2 GB on the wire. A candidate who says "a bit of overhead" has not sized it. A candidate who says "it doubles" is describing hex. The one-third figure is the number to have. ### The memory profile is worse than the wire size Wire size is the part people estimate; peak resident memory is the part that pages the job out. `base64.b64encode(data)` is not a generator. It allocates one contiguous 3.2 GB `bytes` object while `data` is still referenced by the caller, so you are already at 5.6 GB. Then `.decode('ascii')` or `json.dumps` builds a `str` — another 3.2 GB, since a pure-ASCII `str` is one byte per character in CPython's compact representation. Then serialising the response body back to UTF-8 makes another. A nightly job that reads its artifact into memory and hands it to `json.dumps` can touch four times the artifact size at peak, and the allocations are large contiguous blocks, which is the shape most likely to fail even when the total looks affordable. The receiver has the mirror problem: JSON has no framing inside a string value, so a parser cannot hand you the field until it has read the closing quote. The whole 3.2 GB is buffered before a single byte is decoded. ### The fix that actually belongs in the design Do not put a 2.4 GB blob in a JSON field. Write the artifact to a blob store or a file endpoint and put a URL, a size and a digest in the JSON. That removes the encoding overhead entirely (binary goes as binary), makes the transfer resumable and cacheable, and lets the report metadata stay a small document that anything can parse. Base64 in JSON is a good answer for kilobytes — a thumbnail, a signature, an encrypted token — and a bad one for gigabytes. ### If it must be inline: the three-byte boundary Sometimes the pipeline is fixed and you must stream the encoding. The rule is that **every chunk you encode independently must be a multiple of three bytes**, except the last. The reason is padding. Three bytes map to four characters with no `=`; any other length produces padding, and padding means end-of-message. Encode in 4096-byte chunks — 4096 is one more than 4095, which is a multiple of three — and every chunk ends in `=`, so the concatenated text is not one Base64 message but many jammed together. Here is what CPython 3.14 does with that, and it is the reason this question is a senior one: ```python data = bytes(range(256)) * 40 # 10240 bytes blob = b"".join(base64.b64encode(data[i:i + 4096]) for i in range(0, len(data), 4096)) len(base64.b64decode(blob)) # 4096, not 10240 — no exception ``` The decoder stops at the first padding run and returns only the first chunk. No error, no warning, a truncated artifact. Change the chunk to 4095 — or any multiple of three — and the round trip is exact. Passing `validate=True` converts the silent truncation into `binascii.Error: Excess data after padding`, and since Python 3.11 `binascii.a2b_base64(..., strict_mode=True)` rejects trailing data at the lower level too. Both are worth having on any decode path that consumes something a producer assembled in pieces. The practical shape is a chunk size that is a comfortable multiple of three — `3 * 1024 * 1024` is a common choice — written straight to the output stream, so neither the encoded text nor the input is ever fully resident. `base64.encodebytes` exists as a convenience but wraps its output in newlines every 76 characters, which is right for MIME and wrong for a JSON string, where each newline must then be escaped. ### What to say in the interview Lead with the number (4/3, so 3.2 GB), then the memory multiplication that the wire size hides, then the design fix (a reference, not an embed), and keep the chunking rule as the answer to "suppose you cannot change the format". Mentioning that the chunk-boundary bug fails silently rather than loudly is the part that shows you have debugged it rather than read about it.