Your component test suite runs an automated accessibility rule check on every component and reports zero violations. What can you honestly claim from that, and which accessibility problems does it leave undetected?
answer
- presence is checkable, meaning is not
- a string exists is not a string is good
- order, focus and timing go unchecked
- no layout means no contrast verdict
- roles pass, behaviour still missing
basics
~20 sZero violations only means no enabled rule found a machine-decidable markup defect. Automated rules cannot judge whether names and alt text are meaningful, whether focus order and visibility work, or whether announcements arrive usefully — those need manual keyboard and screen-reader checks.
solid answer
~50 sThe honest claim is narrow: for each component, no enabled rule found a defect it can decide from the markup — no missing `alt`, no unlabelled input, no illegal `aria-*` value, no nameless button. What it leaves untested is everything requiring judgment or real rendering: whether the alternative text says something useful, whether a control's name matches its visible label, whether tab order follows the visual layout, whether focus is visible and gets moved and restored around dialogs, whether a status change is announced at a moment the user can use, and whether a widget that has the right roles actually responds to the keyboard. Published estimates put automated coverage somewhere between a third and half of real issues. So I treat the suite as a regression floor and pair it with a manual keyboard pass, a screen-reader spot check on the critical flows, and a real-browser check for anything involving colour or layout.
go deeper
Be able to say that automated checks look at markup facts like missing labels or alt text, and that they cannot tell whether the text is meaningful — so a green run is a starting point, not proof.
Explain the categories the rules structurally cannot reach: meaning of names, focus order and visibility, timing of announcements, keyboard behaviour behind correct roles, and anything needing real layout. Name the compensating manual checks.
Demonstrate that you run a real process — a keyboard pass, screen-reader spot checks on high-value flows, a browser check for rendering-dependent rules — and that you convert each manual finding into a permanent assertion where it is machine-checkable.
Own the definition of done and the metric. Be ready to argue why violation count is a misleading target, how much manual audit budget the product warrants, and how you keep the automated floor from crowding out judgment-based review.
## What a rule engine can decide An automated accessibility check answers questions that are decidable by inspecting the DOM. Those questions fall into a few families: - **Presence**: does this image expose alternative text, does this input have an associated label, does this button end up with a non-empty name. - **Validity**: is this ARIA attribute one that exists, is its value legal, is it allowed on this element's role. - **Structural relationships**: does a role that requires particular children have them, do referenced ids resolve. All of these are objective. A machine reads the markup, applies a fixed rule, and returns a verdict that a reasonable human would agree with. That is genuinely valuable — these are also the mistakes people make most often and fix most cheaply, and once a rule is in the suite the defect can never silently return. ## Where the ceiling is The ceiling is not arbitrary; it follows from the same property that makes the rules automatable. A rule can check that a string exists. It cannot check that the string is *good*. **Meaning.** `alt="image"`, `alt="IMG_2043.png"` and `aria-label="button"` all satisfy the presence rules and all fail the user. Same for a label left behind after the copy changed: the control now reads "Save" but announces "Submit form", so a voice-control user saying "click Save" gets nothing, and the check stays green. **Order and focus.** Tab order is a function of DOM order, tabindex and layout. A rule can flag a positive tabindex as suspicious, but it cannot tell you that a visually two-column form tabs down the left column and then jumps back up to the right, which is disorienting. Nor can it tell you that the focus indicator is invisible against the component's background, that a dialog opens without moving focus into it, or that closing the dialog drops focus back to the top of the document instead of the control that opened it. **Announcements over time.** Accessibility is partly temporal. Did the error message reach the user when validation failed? Did the results count announce once, or on every keystroke until it was unusable? Was the loading state communicated? A single-snapshot rule pass over a rendered tree has no notion of what happened when. **Interaction behaviour.** Correct roles do not imply correct keyboard behaviour. A custom widget can carry exactly the right roles and states, pass every rule, and still not respond to arrow keys, Home/End, or Escape — so it announces itself as something the user cannot operate. Roles are markup; behaviour is code. **Anything requiring rendering.** Contrast, target size, overlap, reflow at zoom, and content that becomes unreachable at a narrow viewport all depend on layout and painting. In a simulated DOM there is no layout and no paint, so these rules cannot even produce a verdict — they return undecided results, which do not fail the test. **Cognition and flow.** Whether an interface is understandable, whether an error tells the user how to recover, whether a multi-step task can be completed without sighted assistance — nothing automatable touches these. This is why the widely quoted figures for automated coverage sit between roughly a third and half of real-world issues, depending on the site and who is counting. The exact number matters less than the shape of the gap: automation covers the objective floor, humans cover the judgment. ## What to say you will do instead A good answer does not stop at "automation is limited" — it names the compensating checks: - **A keyboard pass.** Tab through the component or flow with no mouse. Can you reach everything interactive, in an order that matches the visual layout, with a visible focus indicator, and get back out without a trap? - **A screen-reader spot check** on the flows that carry the product's value — sign-up, checkout, search — rather than on every component. You are listening for whether the announced name matches what is on screen and whether state changes are communicated. - **A real-browser check** for the rendering-dependent rules the simulated environment could not decide, so contrast and layout problems are caught somewhere rather than nowhere. - **Turning findings back into tests.** Once a manual pass finds a real defect, encode the machine-checkable part of it: assert the dialog moves focus, assert the error message is associated with the input, assert the control exposes the exact expected name. This is how the manual budget shrinks over time instead of repeating the same discoveries. ## The organisational failure mode The dangerous outcome is not a missed rule; it is a team that adopts the green suite as its definition of done. Violation count then becomes the metric, and effort flows toward the things that move the metric rather than toward the things users hit. Stating the claim precisely — "no enabled rule fired" — is what keeps the rest of the programme honest. ```js // The rule check passes; both of these are defects. <img src="chart.png" alt="image" /> <button aria-label="Submit form">Save</button> ```
- Give a concrete example of markup that passes every rule and is still broken for users.`<button aria-label="Submit form">Save</button>`. Every presence and validity rule is satisfied: the button has a name and the attribute is legal. But a screen-reader user hears a different word from the one on screen, and a voice-control user saying "click Save" gets no match. Only a human comparing the announced name to the visible label catches it.
- How do you decide which flows get a manual screen-reader pass, given the budget is finite?Rank by user impact and irreversibility: the flows where failure costs someone the ability to complete a real task — authentication, payment, search, forms that submit data. Cover those on a schedule, cover new interaction patterns once when they are introduced, and let the automated suite carry the long tail of repeated components.
- How do you keep manual findings from being rediscovered every quarter?Convert the machine-checkable residue of each finding into an assertion. If the audit found focus was not moved into a dialog, add a test asserting focus lands inside on open and returns to the trigger on close. The judgment part stays manual, but each pass permanently retires some of its own scope.
- Is it worth checking contrast in the component suite at all?Not in a simulated DOM — the rule cannot resolve rendered colours, so it returns an undecided result rather than a verdict, and a green run says nothing. Check contrast where pixels exist: in a real browser, or by reviewing the design tokens themselves so the palette is correct by construction rather than per component.
saying these in an interview costs you the question
- Zero violations means the product is accessible
- Assumes automation checks whether alt text is meaningful
- Thinks correct roles guarantee correct keyboard behaviour
- Believes contrast is covered by the component suite
- Treats violation count as the accessibility metric