skip to content

Why can a BSON string value hold a NUL byte when a BSON key cannot?

level: middleimportance: should knowfreq 20%

answer

  1. two encodings of text
  2. what marks where each one ends
  3. cstring against string in the grammar
  4. an empty string's int32 is 1

basics

~20 s

A BSON key is a cstring with no length field, so its first NUL is its end and its bytes cannot contain one. A string value carries an int32 length, so NUL bytes inside it are just data.

solid answer

~40 s

BSON has two encodings of text. A key, the `e_name`, is a `cstring`: bytes, then `0x00`, with no length. The terminator is the only thing marking its end, so the specification says those bytes MUST NOT contain `0x00`. A string value is `int32 (byte*) 0x00`, where the `int32` is the byte count plus one for the trailing NUL, not counting its own four bytes. The length marks the end, so a NUL inside is ordinary data, and the corpus has a valid vector with embedded NULs. Drivers must refuse a NUL in a key at encode time, because a reader would end the key at that byte and misread what follows as the value.

code

plaintext · 6 lines
plaintext
0D 00 00 00  02  61 00  01 00 00 00  00  00
             |   |      |            |   root terminator
             |   |      |            trailing NUL of the empty string
             |   |      int32 = 1
             |   e_name "a" and its NUL
             type: string

go deeper

for a junior

Remember that keys end at their first zero byte and have no length field, while string values start with a length, so only string values can contain a zero byte.

for a middle

Name the grammar terms, cstring and string, and state what each int32 counts. Be able to contrast an empty string's int32 of 1 with an empty document's length of 5.

for a senior

Expect NUL handling to surface at boundaries: a JSON key spelling the NUL escape has no BSON form, and code treating a string value as a C string truncates it. Say which layer rejects what.

for a principal

The tradeoff angle is small but real: keys with no length are compact and cannot hold arbitrary bytes. If teams pass user-supplied keys into documents, the rule needs handling at the boundary.

## Two encodings of text The BSON grammar defines two text forms, and they differ in one way that decides the answer: how the end is marked. | | `cstring` | `string` | |---|---|---| | Grammar | `(byte*) unsigned_byte(0)` | `int32 (byte*) unsigned_byte(0)` | | Used for | keys (`e_name`), regex pattern and options | string values, JavaScript code, symbols | | End marked by | the first `0x00` | the `int32` | | May contain `0x00` | no | yes | | Empty form | `00` | `01 00 00 00 00` | The specification words the rule strongly: the bytes of a `cstring` **MUST NOT** contain `unsigned_byte(0)`, which is why it describes them as modified UTF-8 rather than full UTF-8. The `0x00` is the one byte value this rule bans; the format itself has no objection to dots or dollar signs in a key, and the corpus lists valid vectors for both. ## Why a NUL in a key breaks things The reader finds a key's end only by looking for the first `0x00`. It has no second opinion. So a NUL inside a key does not merely shorten the key; it misaligns the value. The corpus lists this vector as a decode error, for the key `x` plus a NUL with `int32` value 1: ```plaintext offset bytes what the writer meant what a reader sees 0x00 0D 00 00 00 length 13 length 13 0x04 10 type int32 type int32 0x05 78 key byte "x" key byte "x" 0x06 00 NUL inside the key end of the key 0x07 00 end of the key first byte of the value 0x08 01 00 00 00 value 1 rest of the value, then a 00 it takes as the terminator 0x0C 00 terminator left over, never reached ``` Step through what the reader does: 1. It takes `00 01 00 00` at `0x07` as the `int32`, which is 256. 2. It finds `0x00` at offset `0x0B` and treats it as the terminator. 3. Its position is now 12, while the prefix says 13, and that mismatch is why the vector fails. A reader that skipped the check would have returned a wrong key and a wrong value. ## Who enforces the rule - **The specification** states the `MUST NOT` for `cstring`. - **Drivers.** The BSON corpus document requires them to prohibit null bytes when encoding BSON in four places: a field name in a root document, a field name in a sub-document, a regex pattern and regex options. They may raise the error when a type is constructed or when it is encoded. - **Extended JSON parsers.** The corpus lists `{"a\u0000": 1 }` as a parse error, along with the same key in a sub-document. That last point catches real data. JSON text can spell `\u0000` inside a key, so a JSON document can be fine as JSON and still have no BSON form. Stripping the byte silently would change the key, so failing is the honest outcome. ## Why a string can carry one A string's end is found by arithmetic, not by searching, so its contents do not matter to the reader. The corpus vector for embedded NULs is: ```plaintext 19 00 00 00 02 61 00 0D 00 00 00 61 62 00 62 61 62 00 62 61 62 61 62 00 00 ``` The `int32` is 13: twelve bytes of text (`ab`, NUL, `bab`, NUL, `babab`) plus the trailing NUL. Total: 4 + 1 + 2 + 4 + 13 + 1 = 25, or `0x19`. The grammar requires the trailing `0x00` even though the length already marks the end, and the corpus lists a string with no such NUL as a decode error. A consumer that treats the value as a C string, reading to the first NUL, returns `ab` from this vector, which is a truncation bug rather than a format problem. ## The length rule differs by type Strings and documents both start with an `int32`, and a reader has to keep their rules apart: - **Document or array:** counts itself, the elements and the trailing `0x00`. An empty one is 5. - **String:** counts the text and the trailing NUL, not itself. An empty one is 1. - **Binary:** counts the payload only, not itself and not the subtype byte. An empty one is 0. A string declaring 0 is invalid, since at least the trailing NUL must be counted, and the corpus lists a length of 0 and a length of -1 as decode errors. So the boundary for a string's `int32` is 1, and the boundary for a document's is 5.

  • What should happen when an application gives a BSON driver a key containing a NUL byte?
    The driver should raise an error, either when the document type is constructed or when it is encoded. The BSON corpus document says drivers must prohibit it for field names at any level and for regex pattern and options. Silently stripping the byte would change the key.
  • Do BSON's cstring rules also forbid a dot or a dollar sign in a key?
    No. The format bans only the 0x00 byte in a key, and the BSON corpus has valid vectors with a dotted key and a dollar-prefixed key. Any further rule on such names comes from whatever consumes the document, not from the format.

  • A BSON key has an int32 length, just as a string value does.
  • A BSON string value cannot contain a NUL byte.
  • A string's int32 includes its own four bytes, as a document's does.
  • Stripping a NUL from a key is the correct way to encode it.
  • The BSON format bans dots and dollar signs in keys.
  • A string declaring length 0 is the valid form of an empty string.