skip to content

How does DTLS carry a handshake message larger than one datagram, and how does the receiver reassemble it?

level: middleimportance: should knowfreq 42%

answer

  1. a datagram has a size limit
  2. the stream used to reassemble for you
  3. three fields in the handshake header
  4. offset plus length, per message
  5. a retransmission may re-split

basics

~20 s

DTLS adds message_seq, fragment_offset and fragment_length to the handshake header. A sender splits one message into fragments that each fit the path MTU, every fragment repeats the message's type, total length and message_seq, and the receiver reassembles by offset.

solid answer

~40 s

A datagram has to fit the path MTU, and a `Certificate` message carrying a chain does not. In stream mode this never arises, because the record layer splits messages across records and the stream glues them back. DTLS instead grows the handshake header from 4 bytes to 12: `msg_type` and the 3-byte `length` of the **whole** message stay, and `message_seq`, `fragment_offset` and `fragment_length` are added. Each fragment travels in its own record, repeating `msg_type`, `length` and `message_seq` while carrying its own offset and length. The receiver buffers fragments per `message_seq` and copies each into place at its offset until the whole `length` is covered. A retransmission may be re-fragmented differently — typically because the path MTU has dropped — so a receiver must tolerate overlapping fragments rather than expecting the same split twice.

code

pseudocode · 11 lines
pseudocode
DTLS handshake message header   (12 bytes; stream-mode TLS uses 4)
  msg_type          1 byte
  length            3 bytes   length of the WHOLE message
  message_seq       2 bytes   which handshake message this is
  fragment_offset   3 bytes   where this piece starts in that message
  fragment_length   3 bytes   how much of it is in this record

one 2400-byte certificate(11) message over a path that fits 1000:
  fragment A:  message_seq N, length 2400, offset    0, fragment_length 1000
  fragment B:  message_seq N, length 2400, offset 1000, fragment_length 1000
  fragment C:  message_seq N, length 2400, offset 2000, fragment_length  400

go deeper

for a junior

Remember that a handshake message can be larger than a datagram, and that DTLS adds offset and length fields to the handshake header so one message can be spread across several packets.

for a middle

Explain the reassembly rule: every fragment repeats the message type, the whole message's length and its message_seq, and carries its own offset and length, so a receiver copies bytes into place until the range is covered.

for a senior

Bring the failure mode. A path that silently drops oversized datagrams gives you a flight that retransmits forever while small messages pass, and a re-fragmented retransmission must be tolerated rather than assumed identical.

for a principal

Treat fragment count as a reliability budget. Every extra fragment multiplies the chance of losing a flight on a constrained link, which turns chain length and maximum fragment size into fleet-level configuration decisions.

## The problem the stream never had A TLS handshake message can be any size the peers agree to, and in stream mode nobody cares: the record layer chops the message into records however it likes, the transport glues the bytes back together in order, and the handshake layer sees one contiguous message. Over datagrams every one of those helpers is gone. A record must fit inside one datagram, a datagram must fit the path MTU or risk fragmentation and loss at the IP layer, and there is no stream to reassemble across records. So DTLS moves reassembly into the handshake header itself. ## The three fields DTLS adds | Field | Size | Meaning | |---|---|---| | `msg_type` | 1 byte | the handshake message type, unchanged from TLS | | `length` | 3 bytes | the length of the **complete** message, repeated in every fragment | | `message_seq` | 2 bytes | which handshake message this is, one number per message | | `fragment_offset` | 3 bytes | where this piece starts inside the complete message | | `fragment_length` | 3 bytes | how many bytes of the message are in this record | That is a 12-byte header against TLS's 4. An unfragmented message is simply the degenerate case: `fragment_offset` is 0 and `fragment_length` equals `length`. ## How a receiver puts it together 1. **Read `message_seq`** to find (or create) the buffer for this message. A message with a *higher* sequence than the one expected is a future message and is queued rather than processed — the handshake is processed in `message_seq` order regardless of arrival order. 2. **Allocate against `length`.** Every fragment states the size of the whole message, so a receiver knows how much it is committing to before the pieces arrive. 3. **Copy `fragment_length` bytes to `fragment_offset`** and record which byte ranges are now present. 4. **Deliver the message once the ranges cover it**, then move to the next `message_seq`. ## Re-fragmentation, and why overlaps are legal The tempting assumption is that a retransmission is byte-identical to the first attempt. It is not required to be. A sender that has learned of a smaller path MTU may split the same message into more, smaller fragments on the second attempt, and the offsets will not line up with the first attempt's. A receiver that keeps a byte-range map handles this naturally; one that keys a buffer on 'fragment number' does not, and this is a classic interoperability bug in constrained implementations. A second consequence matters on a lossy uplink: because each fragment is an independent record, **losing any one fragment loses the whole message** until it is retransmitted. A three-kilobyte chain over a link with a one-kilobyte MTU is three chances to lose the flight rather than one, and the loss probability of a flight compounds with the fragment count. That is the mechanism behind an observation operators report constantly — that handshakes over a constrained link fail far more often than steady-state traffic does, even though the steady-state records are tiny. ## The practical failure mode The unpleasant version of this is a path that silently drops datagrams above a certain size instead of signalling it. The small messages of a flight get through, the large one never does, and the handshake retries on its timer forever while every diagnostic says the link is up. The tells are worth knowing: - the same flight retransmitted repeatedly with exponentially growing gaps; - small handshake messages arriving while one large one never completes; - success once the chain is shortened, or once the sender's fragment size is lowered. The fixes belong to the deployment rather than the protocol: send a shorter chain, lower the sender's maximum fragment size so every fragment clears the worst hop, and make sure the reassembly buffer is bounded so a peer cannot make a receiver hold an incomplete multi-kilobyte message indefinitely. ## What is not in scope here Fragmentation answers *how a message crosses a datagram boundary*. It does not answer *what happens when a fragment is lost* — that is the flight retransmission timer's job — and it does not change which messages the handshake contains or in what order, which is plain TLS. The fields are also unchanged in DTLS 1.3; what 1.3 changes is the recovery around them, by letting a receiver acknowledge the records it did get so the sender can resend less.

  • May a retransmitted handshake message be fragmented differently from the first attempt?
    Yes, and a receiver must cope with it. A sender that has learned of a smaller path MTU will re-split the message, so the second attempt's offsets need not match the first's and fragments may overlap. Reassembly must therefore track byte ranges rather than counting fragments, which is a frequent source of interoperability failures in small implementations.
  • What does message_seq give a receiver that fragment_offset does not?
    Ordering across messages rather than within one. `fragment_offset` places bytes inside a single message; `message_seq` says which message it is, letting a receiver queue a message that arrives before its predecessor and process the handshake in the specified order regardless of datagram arrival order.

saying these in an interview costs you the question

  • Says the record layer reassembles handshake messages for you
  • Assumes a retransmission is always fragmented identically
  • Thinks length in a fragment means that fragment's size
  • Treats message_seq and the record sequence_number as the same counter
  • Claims one lost fragment costs only that fragment