In a compiler front end, what is a token, and what does the scanner throw away to produce one?
answer
- three parts, not one
- class, lexeme, span
- separators carry no grammar
- trivia skipped or attached
- positions recorded, never recomputed
basics
~20 sA token is one lexical unit: a class such as identifier, number, keyword, operator or punctuator, together with its text and the source span it came from. The scanner throws away whitespace and comments but keeps the spans.
solid answer
~40 sThe scanner reads characters and emits tokens. A token carries three things: a **class** (the terminal symbol the grammar talks about, such as `identifier`, `number`, `keyword`, `operator`), the **lexeme** (the exact characters matched), and a **span** (where in the source those characters were). Whitespace and comments are trivia: they separate tokens but carry no grammatical meaning, so the scanner skips them rather than emitting them. What it must not skip is position information - every later diagnostic, squiggle and jump-to-definition is a span, and spans are cheap to record while scanning and expensive to reconstruct afterwards. The parser then works over token classes, so its rules never mention a space or a comment.
go deeper
Be able to say the three parts of a token out loud - class, lexeme, span - and give one example of each. Know that whitespace and comments separate tokens rather than becoming them.
Explain why the class set is chosen from the grammar's needs, and what changes when trivia is attached to tokens instead of dropped. Say why spans are recorded during the scan rather than reconstructed.
Show what depends on spans in a real tool chain: diagnostics, editor decorations, source maps. Explain how a layout-sensitive language turns whitespace back into tokens without changing the stage's job.
Frame the token stream as an interface between front-end halves, and weigh what to put in it - decoded values, attached trivia, absolute offsets - against the cost paid by every consumer downstream.
## What a token actually is A **token** is the smallest unit the grammar of a language is written in terms of. The scanner (also called the lexer or tokenizer) reads the source as a flat sequence of characters and produces a flat sequence of tokens. Each token normally carries three parts: - a **class** (or kind, or type): the terminal symbol the parser matches against - `identifier`, `number`, `string`, `keyword-if`, `operator-plus`, `left-paren`; - the **lexeme**: the exact characters that were matched, `count` or `42` or `+=`; - the **span**: where those characters sit in the source, usually a byte or character offset plus a length, from which line and column are derived on demand. Some scanners add a fourth part, a **decoded value** - the numeric value of `42`, or the string contents with escape sequences already resolved - so that later stages do not re-parse the lexeme. | Class | Example lexeme | Why the parser cares | |---|---|---| | identifier | `count` | a name to be resolved later | | number | `42` | a literal operand | | keyword | `while` | fixes which rule applies | | operator | `+=` | drives precedence and shape | | punctuator | `(` | delimits a construct | ## Token classes are a design decision The set of classes is not handed down; it is chosen to be exactly the terminal alphabet the grammar needs. If the grammar never distinguishes an integer literal from a floating-point literal, one `number` class is enough and the distinction can live in the decoded value. If it does distinguish them, they are two classes. The working rule is: **split a class exactly when a parser rule needs to tell the two apart**, because every extra class is another symbol the grammar and its tables must carry. ## What the scanner throws away Between tokens the source usually holds material that separates but does not mean: - runs of spaces, tabs and newlines; - line comments and block comments; - line-continuation markers and other purely typographic constructs. This material is collectively called **trivia**. Discarding it is what lets the grammar stay readable: without a scanning stage, almost every grammar rule would have to allow optional whitespace between every pair of symbols. Discarding is not universal, and this is where ecosystems genuinely differ. A pipeline that only needs to run the code drops trivia outright. A pipeline that also has to reprint the source - a formatter, a refactoring tool, a syntax highlighter - usually **attaches** trivia to the token that follows it, so the original text can be reconstructed byte for byte. Both are called tokenizing; they differ only in whether the thrown-away characters are recorded on the side. Whitespace is not always trivia, either. In a layout-sensitive language, a newline or a change of indentation is itself meaningful, and the scanner emits explicit tokens for it rather than skipping it. That is still the same stage doing the same job: it is deciding, per character run, whether this is a token or a separator. ## What survives: the span The one thing the scanner must not discard is position. The reason is practical: 1. Every error message points at characters, not at tokens. 2. Editor features - underlining, hovering, renaming - address the buffer by offset. 3. Once the character stream is gone, recovering the offset of a token means scanning the file again, and the mapping is not free because trivia has already been skipped. So spans are recorded while the position counter is already in hand, and everything downstream inherits them. A parse-tree node's span is usually just the span from its first token's start to its last token's end. ## Why the stream matters downstream The token stream is the contract between the two halves of the front end: - the parser's rules are written over **classes**, so `while ( x )` and `while(x)` produce identical token streams and one rule handles both; - the parser never needs to look at raw characters, which keeps its lookahead small and its tables finite; - tooling that does not need a tree at all - a highlighter, a simple linter, a brace matcher - can consume the token stream directly and stop there. That is the whole payoff of the stage: it converts an unstructured character buffer into a short, classified, position-carrying sequence that every later stage can rely on.
- If whitespace is discarded, how does a tool reprint the original source exactly?By keeping the discarded characters instead of dropping them. A scanner aimed at formatters and editors attaches each run of trivia to the token that follows it, so the token stream still holds every character of the file. A compile-only pipeline has no such need and drops trivia, which is why the same stage is described both ways.
- What decides how many token classes a language has?The grammar. A class exists so a parser rule can match it, so two lexemes belong to different classes exactly when some rule treats them differently. If no rule distinguishes integer from floating-point literals, one `number` class carrying a decoded value is enough; every extra class adds a symbol to the grammar and its tables for no benefit.
- Is a token's span a line and column pair?Usually not as stored. Scanners record an offset and a length because the counter is already there and the pair is cheap to compare and shift. Line and column are derived on demand from a table of line-start offsets built during the same pass, which keeps the common case fast and the human-facing case correct.
saying these in an interview costs you the question
- Says a token is just the matched text, with no class attached
- Thinks the scanner builds a tree rather than a flat sequence
- Believes positions can be recovered later by rescanning for free
- Assumes whitespace is always meaningless in every language
- Expects the scanner to resolve a name to its declaration