What does each call to xml.Decoder.Token return while streaming a large XML file in Go?

## Two ways into encoding/xml `xml.Unmarshal` and `Decoder.Decode` build a Go value out of a whole element: they read the element and everything under it, allocate, and hand you a struct. That is fine for a configuration file. For a multi-gigabyte export it is not — the peak footprint is the document. `Decoder.Token` is the other door. It is a *pull parser*: each call advances the decoder by one syntactic piece and returns it, and nothing accumulates. Memory stays flat no matter how big the file is, because the only thing alive at any moment is the current token plus whatever you chose to keep. ## The signature and the concrete types ```go func (d *Decoder) Token() (xml.Token, error) ``` `xml.Token` is declared as `type Token any` — an empty interface — so the value carries no useful methods and you must type-switch on it. The concrete types the decoder can hand back are: - **`xml.StartElement`** — `struct { Name xml.Name; Attr []xml.Attr }`. An opening tag, with its attributes already parsed. - **`xml.EndElement`** — `struct { Name xml.Name }`. A closing tag. - **`xml.CharData`** — a `[]byte` of character data between tags, with entity references such as `&amp;` already expanded. - **`xml.Comment`** — a `[]byte` holding the body of a `<!-- … -->`. - **`xml.ProcInst`** — `struct { Target string; Inst []byte }`, a processing instruction; the leading `<?xml version="1.0"?>` declaration arrives this way. - **`xml.Directive`** — a `[]byte` for a `<!DOCTYPE …>` and friends. A self-closing element such as `<br/>` is not a special case: the decoder synthesises a `StartElement` followed immediately by the matching `EndElement`, so a loop that tracks depth by counting starts and ends stays balanced. ## Ending the loop correctly When the input is exhausted, `Token` returns `nil, io.EOF`. That is the normal exit and must not be reported as a failure. Anything else is a genuine error — most often an `*xml.SyntaxError`, which is what you get for mismatched tags, an unexpected end of input in the middle of an element, or bytes that are not valid UTF-8. The idiomatic shape: ```go for { tok, err := dec.Token() if errors.Is(err, io.EOF) { break } if err != nil { return err } // use tok } ``` `Decoder.InputOffset()` returns the byte offset just past the most recently returned token, which is the single most useful thing to attach to an error message when the failure is 4 GB into a file. ## Whitespace is character data A pretty-printed document is mostly indentation, and every run of it comes back as an `xml.CharData` token. A loop that assumes it only sees elements will silently miscount, and code that concatenates every `CharData` it meets will collect newlines and tabs. Filter with `bytes.TrimSpace` when you only want real text. ## The borrowed-bytes rule This is the trap that turns a working prototype into a corrupt output file. The byte slices inside `CharData`, `Comment` and `Directive`, and the `Attr` slice inside `StartElement`, refer to the decoder's internal buffer. They stay valid only until the next call to `Token`. If you append tokens to a slice, stash them in a map, or hand them to another goroutine, the contents change underneath you. The fix is in the package: `xml.CopyToken(t)` returns a token that owns its memory, and each of the token types has its own `Copy` method (`StartElement.Copy`, `CharData.Copy`, and so on). Converting to a `string` also copies, because a Go string conversion of a `[]byte` allocates. ## Skipping and raw tokens When you are positioned on a `StartElement` you do not care about, `Decoder.Skip()` consumes tokens until the matching `EndElement` without materialising anything — cheaper and less error-prone than counting depth yourself. `Decoder.RawToken()` is the lower-level sibling: it returns the same token types but does not check that start and end elements match and does not translate namespace prefixes into URIs. You reach for it only when you deliberately want the document as written rather than as resolved. ## Why this matters in an interview The token loop is the answer to "how would you process an XML feed you cannot fit in RAM", and the three details that separate a confident answer from a hesitant one are: the return type is an interface, `io.EOF` is the terminator, and the bytes are borrowed.