skip to content

How does the output of json.NewEncoder(w).Encode(v) differ from json.Marshal(v)?

level: middleimportance: should knowfreq 48%

answer

  1. one returns bytes, one writes to a writer
  2. look at the very last byte
  3. the writer form has knobs the function lacks
  4. who can stop < turning into \u003c

basics

~20 s

Encode writes the value to the io.Writer and appends a trailing newline, which json.Marshal never does. The encoder also carries settings json.Marshal has no access to: SetIndent for formatting and SetEscapeHTML(false) to stop < > & being escaped.

solid answer

~40 s

`json.Marshal` returns a `[]byte` with no trailing newline; `(*json.Encoder).Encode` writes to the `io.Writer` you gave `json.NewEncoder` and always terminates the value with `\n`. That single byte is why byte-for-byte comparisons against `json.Marshal` output fail, and equally why an encoder is the natural way to emit records one JSON value per line. The encoder is also configurable in ways the package-level function is not: `SetIndent(prefix, indent)` is the `json.MarshalIndent` equivalent and applies to every subsequent `Encode`, and `SetEscapeHTML(false)` turns off the default escaping of `<`, `>` and `&` into `\u003c`, `\u003e` and `\u0026` — there is no way to disable that in `json.Marshal`. Finally, `Encode` returns the writer's error, and a write failure is sticky: later `Encode` calls on that encoder return it too.

code

go · 9 lines
go
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
enc.SetEscapeHTML(false)
enc.SetIndent("", "  ")
if err := enc.Encode(map[string]string{"q": "a<b"}); err != nil {
	return err
}
// buf holds the object indented over three lines, the raw "a<b",
// and a final newline. json.Marshal gives one line, \u003c, no newline.

go deeper

for a junior

Remember that Encode writes to a writer and adds a newline, while json.Marshal hands back bytes with none. That newline is the usual reason two outputs that look the same do not compare equal.

for a middle

Explain the mechanics of each knob: SetIndent as the MarshalIndent equivalent that persists across calls, and SetEscapeHTML as a switch that exists only on the encoder because the default protects HTML embedding.

for a senior

Show that you know where the actual saving is: per-value encoding, not per-byte streaming, plus a sticky write error that must be checked so a dead connection does not silently consume a whole result set.

for a principal

Own the output convention. Whether an export is a wrapped array or newline-delimited records, and whether it is indented, is a published contract that consumers build tooling around, so pick one deliberately rather than letting each endpoint choose.

## Two writers, two shapes of output `json.Marshal(v any) ([]byte, error)` gives you bytes. `json.NewEncoder(w io.Writer)` gives you a `*json.Encoder` whose `Encode(v any) error` pushes the encoded value into `w`. The encoding rules — struct tags, exported fields only, the type mapping — are identical. Three things about the *output and the API surface* are not. ## 1. The trailing newline `Encode` terminates each value with a `\n`. The package documents it, and it exists partly so that consecutive values are unambiguously separated even when one ends in a number. Two consequences follow, one annoying and one useful. The annoying one: a golden-file test that compares `json.Marshal(v)` against what a handler produced with an encoder fails on one byte at the end. The fix is to compare after trimming, or to produce both sides the same way. The useful one: newline-delimited output falls out for free. An export endpoint that streams a report writes one `Encode` per row and the result is a stream of records, one per line, with no enclosing array and no comma bookkeeping. A consumer reads it back with a `json.Decoder` looping until `io.EOF`, and can start processing the first row before the last has been produced — which a single wrapping array does not allow as cleanly. ## 2. Formatting: SetIndent `(*json.Encoder).SetIndent(prefix, indent string)` makes every *subsequent* encoded value formatted the way `json.MarshalIndent(v, prefix, indent)` would be. It is not a per-call argument; you set it on the encoder and it stays set. The trailing newline still follows the indented value. This interacts directly with the newline-delimited use above: the moment you call `SetIndent`, records span multiple lines and the one-record-per-line framing is gone. For a human-facing config dump, indent. For a machine-consumed export, do not. ## 3. Escaping: SetEscapeHTML By default `encoding/json` escapes `<`, `>` and `&` as `\u003c`, `\u003e` and `\u0026`, so that the output can be embedded inside an HTML `<script>` block without terminating it early. Both `json.Marshal` and an encoder do this. Only the encoder can be told to stop: `SetEscapeHTML(false)`. There is no equivalent flag on `json.Marshal`, which is a common reason to reach for `json.NewEncoder(&buf)` over a `bytes.Buffer` even when you only wanted bytes. Turn it off when the consumer is not a browser and the escaped form is noise; leave it on when the output may end up inline in a page. ## What "streaming" does and does not mean on the encode side It is worth being exact, because the name misleads. A `json.Decoder` genuinely consumes input incrementally. An `Encoder` does not emit a value incrementally: internally each `Encode` call marshals the whole value into a buffer and then hands it to the writer, in current implementations in a single `Write`. So encoding one gigantic slice through an encoder is no cheaper in peak memory than `json.Marshal` on the same slice. The win comes from *granularity*. Encoding ten thousand rows one at a time keeps only one row's worth of encoded bytes alive at a time, and the encoded bytes leave for the writer as you go. A benchmark run with `-benchmem` shows the difference plainly: bytes per operation and allocations per operation drop sharply when a single `json.Marshal` of the whole result set is replaced by a per-row `Encode`, with no change in what the client receives. ## Errors are sticky `Encode` returns whatever the underlying writer returned. If a write fails — a client disconnected mid-download, a disk filled — the encoder remembers it and every later `Encode` on that encoder returns the same error without attempting more work. That is useful in a row loop: you do not need to guard each call separately to avoid hammering a dead connection, but you *do* need to check the error and stop, otherwise the loop spins through the whole result set producing nothing. One thing the encoder does not do is flush. It writes to the `io.Writer` it was given; if that writer is a `bufio.Writer`, buffered bytes stay buffered until you flush it, and if it is an HTTP response writer the bytes may sit until the response is large enough or the handler returns.

  • Why does a test comparing json.Marshal output with json.Encoder output fail?
    Because `Encode` appends a `\n` after every value and `json.Marshal` does not, so the two byte slices differ by that final byte even when the JSON is identical. Either trim the trailing newline before comparing, or generate both sides through the same API.
  • Can you turn off the escaping of < and > in json.Marshal?
    No. HTML escaping is on by default so output is safe to embed in a page, and `json.Marshal` exposes no switch. The usual workaround is to encode into a `bytes.Buffer` through `json.NewEncoder` with `SetEscapeHTML(false)` and take the buffer's bytes — accepting the trailing newline that comes with it.
  • Does using an encoder mean a huge value is never fully held in memory?
    No. Each `Encode` call marshals the whole value into an internal buffer before writing it, so one enormous slice costs the same as `json.Marshal`. The saving comes from encoding many small values instead of one big one — per-row `Encode` keeps a single row's bytes alive at a time, which a `-benchmem` benchmark shows as lower B/op and allocs/op.
  • What happens to a json.Encoder after a write to its io.Writer fails?
    The error is recorded on the encoder and every subsequent `Encode` returns it immediately without encoding anything further. So a row loop that ignores the error will run to completion producing no output at all; check it and break out on the first failure.

saying these in an interview costs you the question

  • Says Encode and json.Marshal produce byte-identical output
  • Thinks SetEscapeHTML can be passed to json.Marshal
  • Believes the trailing newline appears only when indenting
  • Assumes SetIndent affects just the next Encode call
  • Claims an encoder never holds a whole value in memory