Why do scanners often match an identifier first and then consult a reserved-word table rather than writing one rule per keyword?

## Keywords are a subset of the identifier pattern In almost every language, a keyword is spelled exactly like an identifier: letters, maybe digits and an underscore. So the character run `if` is accepted by the keyword pattern and by the identifier pattern at the same time and at the same length. The scanner must decide which token class to emit, and there are two standard designs. ## Design one: a rule per keyword Each reserved word gets its own pattern, declared before the general identifier rule. This is correct, and it relies on two rules working together: 1. **Longest match** stops the keyword rule from firing on a prefix. Without it, `iffy` would be tokenized as the keyword `if` followed by the identifier `fy`, because the keyword pattern does accept a prefix of the input. 2. **Declaration order** settles the genuine tie. On the exact text `if`, both rules accept the same two characters, so the earlier-declared keyword rule wins. Get either wrong and the bug is subtle: reorder the rules and every keyword silently becomes an identifier; drop longest match and every identifier that starts with a keyword breaks. ## Design two: identifier, then a lookup The scanner has one pattern for the identifier shape. When it matches, it takes the matched text and looks it up in a table of reserved words; a hit means the token class is that keyword, a miss means it is an identifier. | Aspect | Rule per keyword | Identifier plus table | |---|---|---| | Recogniser size | grows with every keyword | one identifier pattern | | Adding a keyword | edits the pattern specification | edits a data table | | Prefix hazard (`iffy`) | handled by longest match | cannot arise - the identifier already took the whole run | | Tie-break needed | yes, declaration order | none | | Soft keywords | awkward - the class is fixed too early | easy - emit identifier, decide later | The prefix row is the quietly important one. With the table design the hazard **disappears by construction**: the identifier pattern consumes the entire run `iffy` before any lookup happens, so there is nothing for a keyword to steal. ## Why the lookup is cheap - It happens once per identifier-shaped token, not per character. - The table is small, fixed at build time, and often a perfect hash or a length-bucketed comparison, so a hit or miss costs work proportional to the word's length. - The keyword set becomes introspectable data: tooling can list it, a highlighter can consume it, a diagnostic can suggest a near-miss spelling. Compare that with the rule-per-keyword design, where the same information is baked into transition tables and cannot be recovered without decompiling them. ## Contextual and soft keywords Some languages reserve a word only in one syntactic position - it names a construct there and is a perfectly legal variable name everywhere else. A scanner cannot know which position it is in, because positions are a parser notion. The table design handles this gracefully: - keep such words **out** of the reserved table; - emit them as ordinary identifiers with their lexeme intact; - let the parser match on the lexeme text in the one place the word is special. With a rule per keyword the class is fixed at scan time, so the parser receives a keyword token in positions where a variable name was intended, and the grammar has to accept that keyword token as a name everywhere - a rule that must then be repeated for each soft keyword. ## What to say in an interview The honest framing is not *one design is wrong*. It is: both work, the table design keeps the recogniser independent of the size of the keyword set, removes the need for a tie-break at all, and leaves the keyword-or-not decision late enough that a parser can still change its mind. That last property is what makes it the default in languages that keep adding syntax without wanting to break existing code that used the new word as a name.