A framed stream starts yielding garbage messages hours into a connection — how do you detect and recover from framing desynchronisation?

## What desynchronisation actually is A framed reader holds one crucial piece of state: the offset at which the next message begins. Desynchronisation is that offset being wrong. The consequences are not local — once the boundary shifts by k bytes, every later message is sliced from the wrong place, so the connection produces an unbounded run of decode failures, or, far worse, values that decode successfully and are wrong. The symptom in production is distinctive: a connection that worked for hours begins failing continuously, restarting the client fixes it immediately, and other connections to the same peer are unaffected. That per-connection signature is the tell. ## How a reader loses the boundary - **The writer's declared length disagrees with the bytes it wrote** — the most common cause, usually from a body computed twice, or one measured in characters and written in bytes. - **Two writers share one connection without serialising the write path**, so their frames interleave and neither is intact. - **A missed escape in delimiter framing** — an unescaped reserved byte ends a frame early, and the remainder becomes the head of a phantom message. - **A crash mid-frame**, leaving a partial frame that the next reader treats as a whole one. - **A reader that consumes a different count than it declared** — an off-by-one in the slice is enough. ## Why a length-prefixed stream does not self-heal A length prefix has no distinguishable form. Read four arbitrary bytes and you get a number; it looks exactly like a length. The reader confidently skips that many bytes, lands somewhere else arbitrary, and repeats. There is nothing to search for, which is the single structural weakness of pure length framing and the reason production protocols add something recognisable. ## Detecting it | Signal | What it usually means | |---|---| | Marker absent at the expected offset | The boundary is wrong right now | | Checksum fails over the declared payload | Wrong bytes, wrong boundary, or corruption | | Declared length wildly implausible | The count was read from payload bytes | | Unknown frame type on a stable protocol | The type byte came from the middle of a body | | Decode failures clustered on one connection | Desync rather than a bad producer version | The invariants matter more than any single one of them: a marker tells you the boundary is wrong, a checksum tells you the bytes are wrong, and together they catch both the shifted case and the corrupted case. **None of them can be checked if the frame carries no redundancy at all** — which is the argument for spending a few bytes per frame on it. ## Recovering 1. **Stop immediately.** Never hand a frame that failed its invariant to a decoder or to business logic; a garbage message that parses is worse than one that fails. 2. **Resynchronise if the format allows it.** With a fixed marker, scan forward byte by byte for the next occurrence, verify the frame that follows it against its length and checksum, and resume there. Accept that a marker value can also occur inside a payload, so the confirmation step is what makes the resync trustworthy. 3. **Otherwise drop the connection.** Framing state is per connection, so closing and reconnecting resets the offset to a known good zero. This is the honest answer for most systems, and it is why framing errors are usually modelled as fatal to the connection rather than to the message. 4. **Preserve evidence.** Log the offset, the declared length, the surrounding bytes and the producer identity before you discard, or the cause is unrecoverable after the fact. ## Preventing it - Give every frame a **marker, a frame type and a checksum**, so any shift is caught at the next frame rather than fifty frames later. - **Assert on the writer** that the bytes emitted equal the declared length, and fail the frame rather than send a lie. - **Serialise writes** to one connection, or give each writer its own. - **Test with adversarial chunking**: feed the same stream split at every byte offset, and include a truncated final frame, and assert the reader reports rather than delivers. - Fail **loudly** — a framing error is a correctness incident, not a warning to be sampled. ## Why it is a differentiator Most engineers have never seen this because a hardened protocol implementation hides it. The ones who have can name the per-connection signature, know why a bare length prefix cannot be searched, and reach for connection teardown rather than a clever partial recovery. That combination is what the question is really testing.