In a scannerless grammar with no token stage, why must every rule that may be followed by spacing say so explicitly?

## One stage, so no free gaps In a design with a separate first stage, that stage groups characters into units and discards the runs between them, so the rules that follow never see a space. A **scannerless** grammar has no such stage: its rules match characters directly. Whitespace is therefore ordinary input that some rule must consume, exactly like a bracket or a name. The symptom is unmistakable once you know it. A grammar reading ``` Annotation <- "@" Name "(" Arg ")" ``` accepts `@note(x)` and rejects `@note (x)`, because between `Name` and `"("` there is nothing that can consume the space. The rule is not wrong about the syntax; it is silent about the gaps, and silence means *no gap allowed*. ## The discipline that keeps it manageable ``` Spacing <- (" " / "\t" / "\r" / "\n")* token(e) <- e Spacing # match e, then absorb any trailing gap ``` 1. Define one `Spacing` rule and nothing else that consumes gaps, so the definition of "blank" lives in one place. 2. Apply it in **one consistent position** — conventionally after each element rather than before — so that every rule can assume the input starts at a non-blank character. 3. Consume leading spacing exactly once, at the entry rule, since the after-each-element convention never handles the very beginning. 4. Wrap literals in a helper so adding a new literal cannot forget the spacing; a grammar that spells `Spacing` out by hand at fifty call sites will miss some. Because `Spacing` uses repetition, it succeeds on zero characters too, so writing it in a place where no gap appears costs nothing. ## Boundaries are your job as well The same absence explains a second duty. With a separate stage, a run of letters is grouped before any comparison happens, so a keyword can never match the front of a longer word. Here it can, and the fix is a predicate: ``` Keyword <- "note" !Letter Spacing ``` `!Letter` succeeds only when the next character is not a letter and consumes nothing, so `notebook` no longer matches the keyword. Any place where one literal is a prefix of possible input needs that guard written by hand. ## What you get in exchange | Concern | Separate stage first | Scannerless | |---|---|---| | Whitespace between elements | dropped for you | consumed by a rule you write | | Word boundaries | implied by grouping | stated with a predicate | | Layout-significant syntax | awkward: the stage must report it | natural: the rules see the characters | | Foreign syntax nested inside text | two components must agree | one grammar covers both | | Where a spacing bug shows up | in the grouping stage | as an unexpected parse failure | For the frame this leaf lives in — annotations embedded in ordinary prose — the last two rows are the whole reason to be scannerless. The document is mostly free text, and the ordinary-text rule can be written directly: ``` Text <- (!Annotation .)+ ``` This consumes characters one at a time for as long as an annotation does not start here; `!Annotation` looks ahead and consumes nothing, and `.` takes exactly one character. There is no way to express that cleanly when an earlier stage has already decided what counts as a unit, because "everything up to the next thing that would parse as an annotation" is not a category that stage knows about. ## Failure modes worth recognising - **A grammar that works only on compact input**, because spacing was added where the examples needed it and nowhere else. - **Spacing consumed in two places**, before and after an element, which is harmless for correctness but doubles the work and confuses later readers about the convention. - **A gap allowed where it must not be**, such as between the `@` marker and the name, because the helper was applied uniformly without asking where a break is meaningful. - **Comments forgotten.** If the syntax has them, they belong inside `Spacing`; otherwise a comment is a parse error everywhere except the one place someone handled it. - **An error message pointing at the wrong place**, since a missing spacing rule surfaces at the next literal rather than at the gap. The reflex to build: when a rule sequences two visible elements, ask what may legally sit between them, and write that answer into the grammar. Nothing else will.