skip to content

questions

28

In a compiler front end, what is a token, and what does the scanner throw away to produce one?

level: juniorimportance: must knowfreq 65%

answer

  1. three parts, not one
  2. class, lexeme, span
  3. separators carry no grammar
  4. trivia skipped or attached
  5. positions recorded, never recomputed

basics

~20 s

A token is one lexical unit: a class such as identifier, number, keyword, operator or punctuator, together with its text and the source span it came from. The scanner throws away whitespace and comments but keeps the spans.

solid answer

~40 s

The scanner reads characters and emits tokens. A token carries three things: a **class** (the terminal symbol the grammar talks about, such as `identifier`, `number`, `keyword`, `operator`), the **lexeme** (the exact characters matched), and a **span** (where in the source those characters were). Whitespace and comments are trivia: they separate tokens but carry no grammatical meaning, so the scanner skips them rather than emitting them. What it must not skip is position information - every later diagnostic, squiggle and jump-to-definition is a span, and spans are cheap to record while scanning and expensive to reconstruct afterwards. The parser then works over token classes, so its rules never mention a space or a comment.

go deeper

for a junior

Be able to say the three parts of a token out loud - class, lexeme, span - and give one example of each. Know that whitespace and comments separate tokens rather than becoming them.

for a middle

Explain why the class set is chosen from the grammar's needs, and what changes when trivia is attached to tokens instead of dropped. Say why spans are recorded during the scan rather than reconstructed.

for a senior

Show what depends on spans in a real tool chain: diagnostics, editor decorations, source maps. Explain how a layout-sensitive language turns whitespace back into tokens without changing the stage's job.

for a principal

Frame the token stream as an interface between front-end halves, and weigh what to put in it - decoded values, attached trivia, absolute offsets - against the cost paid by every consumer downstream.

## What a token actually is A **token** is the smallest unit the grammar of a language is written in terms of. The scanner (also called the lexer or tokenizer) reads the source as a flat sequence of characters and produces a flat sequence of tokens. Each token normally carries three parts: - a **class** (or kind, or type): the terminal symbol the parser matches against - `identifier`, `number`, `string`, `keyword-if`, `operator-plus`, `left-paren`; - the **lexeme**: the exact characters that were matched, `count` or `42` or `+=`; - the **span**: where those characters sit in the source, usually a byte or character offset plus a length, from which line and column are derived on demand. Some scanners add a fourth part, a **decoded value** - the numeric value of `42`, or the string contents with escape sequences already resolved - so that later stages do not re-parse the lexeme. | Class | Example lexeme | Why the parser cares | |---|---|---| | identifier | `count` | a name to be resolved later | | number | `42` | a literal operand | | keyword | `while` | fixes which rule applies | | operator | `+=` | drives precedence and shape | | punctuator | `(` | delimits a construct | ## Token classes are a design decision The set of classes is not handed down; it is chosen to be exactly the terminal alphabet the grammar needs. If the grammar never distinguishes an integer literal from a floating-point literal, one `number` class is enough and the distinction can live in the decoded value. If it does distinguish them, they are two classes. The working rule is: **split a class exactly when a parser rule needs to tell the two apart**, because every extra class is another symbol the grammar and its tables must carry. ## What the scanner throws away Between tokens the source usually holds material that separates but does not mean: - runs of spaces, tabs and newlines; - line comments and block comments; - line-continuation markers and other purely typographic constructs. This material is collectively called **trivia**. Discarding it is what lets the grammar stay readable: without a scanning stage, almost every grammar rule would have to allow optional whitespace between every pair of symbols. Discarding is not universal, and this is where ecosystems genuinely differ. A pipeline that only needs to run the code drops trivia outright. A pipeline that also has to reprint the source - a formatter, a refactoring tool, a syntax highlighter - usually **attaches** trivia to the token that follows it, so the original text can be reconstructed byte for byte. Both are called tokenizing; they differ only in whether the thrown-away characters are recorded on the side. Whitespace is not always trivia, either. In a layout-sensitive language, a newline or a change of indentation is itself meaningful, and the scanner emits explicit tokens for it rather than skipping it. That is still the same stage doing the same job: it is deciding, per character run, whether this is a token or a separator. ## What survives: the span The one thing the scanner must not discard is position. The reason is practical: 1. Every error message points at characters, not at tokens. 2. Editor features - underlining, hovering, renaming - address the buffer by offset. 3. Once the character stream is gone, recovering the offset of a token means scanning the file again, and the mapping is not free because trivia has already been skipped. So spans are recorded while the position counter is already in hand, and everything downstream inherits them. A parse-tree node's span is usually just the span from its first token's start to its last token's end. ## Why the stream matters downstream The token stream is the contract between the two halves of the front end: - the parser's rules are written over **classes**, so `while ( x )` and `while(x)` produce identical token streams and one rule handles both; - the parser never needs to look at raw characters, which keeps its lookahead small and its tables finite; - tooling that does not need a tree at all - a highlighter, a simple linter, a brace matcher - can consume the token stream directly and stop there. That is the whole payoff of the stage: it converts an unstructured character buffer into a short, classified, position-carrying sequence that every later stage can rely on.

  • If whitespace is discarded, how does a tool reprint the original source exactly?
    By keeping the discarded characters instead of dropping them. A scanner aimed at formatters and editors attaches each run of trivia to the token that follows it, so the token stream still holds every character of the file. A compile-only pipeline has no such need and drops trivia, which is why the same stage is described both ways.
  • What decides how many token classes a language has?
    The grammar. A class exists so a parser rule can match it, so two lexemes belong to different classes exactly when some rule treats them differently. If no rule distinguishes integer from floating-point literals, one `number` class carrying a decoded value is enough; every extra class adds a symbol to the grammar and its tables for no benefit.
  • Is a token's span a line and column pair?
    Usually not as stored. Scanners record an offset and a length because the counter is already there and the pair is cheap to compare and shift. Line and column are derived on demand from a table of line-start offsets built during the same pass, which keeps the common case fast and the human-facing case correct.

saying these in an interview costs you the question

  • Says a token is just the matched text, with no class attached
  • Thinks the scanner builds a tree rather than a flat sequence
  • Believes positions can be recovered later by rescanning for free
  • Assumes whitespace is always meaningless in every language
  • Expects the scanner to resolve a name to its declaration
open as a page

Why does a tiered runtime start a method in an interpreter and compile it only after invocation and loop counters cross a threshold?

level: middleimportance: must knowfreq 64%

basics

~20 s

Compilation costs time, CPU and memory, and most methods run only a handful of times. Counting invocations and loop iterations finds the small hot fraction where optimisation will be repaid, and the interpreted phase supplies the profile the compiler then optimises against.

open as a page

What does a compiler drop when it converts a parse tree into an abstract syntax tree?

level: middleimportance: must knowfreq 66%

basics

~20 s

An abstract syntax tree keeps operators, operands and structure, and drops what only guided the parser: grouping parentheses, separators and terminators, layout, and single-child chain nodes from precedence rules. The tree's shape already encodes the grouping those tokens expressed.

open as a page

What does flattening expressions into three-address instructions with temporaries expose that an expression tree leaves implicit?

level: middleimportance: must knowfreq 52%

basics

~20 s

Three-address code gives every intermediate value a name and every operation its own instruction in a fixed order. Evaluation order and the producer of each value become explicit data rather than facts implied by a tree walk.

open as a page

When several token rules match at one position, how does a maximal-munch scanner pick the token it emits?

level: middleimportance: must knowfreq 58%

basics

~10 s

It takes the longest prefix starting at that position that any rule accepts, not the first rule that matches. If two rules accept the same longest prefix, the rule declared earlier wins.

open as a page

What do constant folding and constant propagation each do to a block of straight-line code?

level: middleimportance: must knowfreq 64%

basics

~20 s

Constant folding evaluates an operation whose operands are all literals and replaces the expression with the result. Constant propagation replaces a use of a variable with the literal every reaching definition assigns it. Each creates work for the other, so pipelines iterate them.

open as a page

Which conditions must hold before a compiler hoists a computation out of a loop body?

level: middleimportance: must knowfreq 56%

basics

~20 s

The expression must be invariant — every operand defined outside the loop and unchanged by it — and moving it must not change behaviour. That means it is effect-free and safe to evaluate even when the loop body would have run zero times, or the hoist is guarded.

open as a page

In a language with nested blocks, how does the name-resolution pass decide which declaration an identifier refers to?

level: middleimportance: must knowfreq 65%

basics

~20 s

Name resolution keeps one symbol table per open scope, each linked to the table of the scope around it. An identifier binds to the first matching entry found walking that chain outward, so an inner declaration hides an outer one of the same name.

open as a page

A load test measures a long-lived request server during its first thirty seconds - why does that figure understate steady-state throughput?

level: seniorimportance: must knowfreq 58%

basics

~20 s

Early requests execute interpreted or lightly optimised code while counters climb, and compilation itself takes CPU away from request handling. Peak arrives only after the hot paths have been compiled, so an early measurement times warmup rather than the system that was shipped.

open as a page

Why can a script that parses with no syntax error still be rejected by a later compiler pass?

level: seniorimportance: must knowfreq 55%

basics

~20 s

Because parsing only proves the text has a legal shape. The pass after it checks meaning against a symbol table — that every name resolves, that it is used as the kind of thing it was declared, and that operand types agree — and those depend on declarations arbitrarily far away.

open as a page

What does desugaring a surface construct into a smaller core of constructs buy the rest of a compiler?

level: middleimportance: should knowfreq 40%

basics

~20 s

Desugaring rewrites convenient surface forms into a small core the compiler already handles, so every later pass sees one shape instead of many. The cost is that diagnostics and debug output now describe code the programmer did not write.

open as a page

Why do scanners often match an identifier first and then consult a reserved-word table rather than writing one rule per keyword?

level: middleimportance: should knowfreq 38%

basics

~20 s

Because every keyword also matches the identifier pattern. Matching the identifier once and looking the text up in a table keeps the recogniser small, makes the keyword set a data edit, and leaves room for words that are keywords only in some positions.

open as a page

Why do compiler front ends run a separate scanning stage instead of parsing the raw characters directly?

level: middleimportance: should knowfreq 47%

basics

~20 s

Because the two jobs need different machine power and different rules. Token shapes are regular, so a finite-state pass handles them in one linear sweep, which keeps whitespace, comments and longest-match logic out of every grammar rule.

open as a page

What must a dead-code elimination pass establish before it deletes a computation from a program?

level: middleimportance: should knowfreq 48%

basics

~20 s

Two things, both required: nothing later reads the result, and the computation has no observable effect. Liveness analysis settles the first; effect reasoning settles the second. A statement with an effect stays even when its result is unused.

open as a page

Why does a resolver collect a scope's declarations before it resolves the references inside that scope?

level: middleimportance: should knowfreq 48%

basics

~20 s

So that forward references resolve. Collecting every declaration of a scope into its symbol table first means a reference can name something declared later in the same scope — mutually recursive definitions being the case a single left-to-right walk cannot handle at all.

open as a page

A just-in-time compiler specialises a call site that has only ever seen one receiver type - what happens when a second type arrives?

level: seniorimportance: should knowfreq 44%

basics

~20 s

The specialised code carries a guard that tests the assumption on every execution. When the guard fails, the runtime deoptimises: it reconstructs interpreter state for the running frame, resumes there, and later recompiles the method against the widened profile.

open as a page

In an SSA intermediate form where every name is assigned exactly once, what does a phi node at a control-flow merge do?

level: seniorimportance: should knowfreq 46%

basics

~20 s

A phi node defines a new version of a value at a merge point, selecting among the versions that arrive on each incoming edge. It preserves the single-assignment rule where two branches would otherwise both assign the same name.

open as a page

A scanner must handle block comments that nest to any depth; what does a pure finite-state pass lack for that job?

level: seniorimportance: should knowfreq 33%

basics

~20 s

It lacks a counter. A finite-state pass has a fixed number of states, so it cannot track how many openings are still unclosed when the depth is unbounded. Scanners add a depth counter, and a stack of modes for constructs that nest.

open as a page

When a graph-colouring register allocator cannot colour the interference graph with the registers it has, what does it do?

level: seniorimportance: should knowfreq 40%

basics

~20 s

It spills. The allocator picks a value by cost, rewrites it to live in the stack frame with a store after its definition and a load before each use, which breaks one long live range into several short ones, then rebuilds the graph and tries to colour again.

open as a page

The same program ships as a two-second batch job and as a long-lived request server - how do you choose ahead-of-time compilation or a tiered just-in-time approach for each?

level: principalimportance: should knowfreq 38%

basics

~20 s

Let process lifetime and restart frequency decide. A two-second job never runs long enough for in-process profiling to repay itself, so compile ahead of time; a server that runs for days can afford warmup once to buy the peak that profile-driven speculation reaches.

open as a page

How do you decide how many intermediate representation levels a small language's compiler should carry between its syntax tree and target code?

level: principalimportance: should knowfreq 33%

basics

~20 s

Add a level only when a whole class of work is natural there and awkward at both neighbours — because each level costs another lowering step, printer, verifier and position mapping to keep correct forever. Start with two and split under pressure.

open as a page

How do you decide the order of an ahead-of-time compiler's optimisation passes when no single order is best for every program?

level: principalimportance: should knowfreq 36%

basics

~20 s

Order passes by what they enable: transforms that expose facts run before the passes that consume them, cheap cleanups run after every heavyweight transform, and lossy lowering runs last. Then cap the repetition with a compile-time budget, because searching for a per-program optimal order costs a full compile per candidate.

open as a page

When you set the scoping rules for a new automation scripting language, should an inner block be allowed to shadow an outer name?

level: principalimportance: should knowfreq 32%

basics

~20 s

There are three defensible policies — permit silently, permit with a warning, or reject — and the choice trades authoring freedom against the reader's ability to predict what a name means. A common landing point is permit with a warning, and reject only where the collision is across kinds.

open as a page

Two interpreters run the same program - one walks the syntax tree, one executes a bytecode stream: where does the tree-walker lose time?

level: middleimportance: nice to knowfreq 30%

basics

~20 s

A tree-walking interpreter pays a node-kind test, an indirect call and several pointer loads for every node it visits, re-deciding the program's shape on each visit. A bytecode interpreter settles that shape once and then runs a flat instruction loop.

open as a page

Why is instruction selection usually described as tiling the intermediate representation with machine instructions?

level: middleimportance: nice to knowfreq 24%

basics

~20 s

Because one machine instruction can implement several operations of the intermediate form at once. Selection covers the operation graph with patterns, each pattern standing for one instruction and carrying a cost, and looks for a cheap cover rather than a one-to-one translation.

open as a page

How does an intermediate bytecode change if it is stack-based rather than register-based?

level: seniorimportance: nice to knowfreq 30%

basics

~20 s

Stack-based code leaves operands implicit on an operand stack: instructions are short and trivial to emit from a tree walk, but the stream is longer and data flow must be reconstructed. Register-based code names operands explicitly, costing wider instructions and a naming step.

open as a page

In an editor that re-scans a buffer after every keystroke, why can typing one character re-tokenize the rest of the file?

level: seniorimportance: nice to knowfreq 26%

basics

~20 s

Because the scanner carries state forward. One character can open a comment or a string, and every token after it is then classified under a different rule set, so the change propagates to the end of the buffer.

open as a page

Why does a resolver record each reference's binding at compile time instead of searching the scopes again at run time?

level: seniorimportance: nice to knowfreq 30%

basics

~20 s

So that a name's meaning is fixed by where it is written, not by who is running. Recording the binding turns each access into a direct slot reference and makes shadowing and capture predictable; searching by name at run time makes the same text mean different things on different call paths.

open as a page