skip to content

Testing & Audits

7 roadmaps5 questions

What automated rule engines can and cannot decide, manual keyboard and screen reader passes, and where checks sit from design review to audit. Interviewers want evidence, not opinion.

part ofDesign systems & UX foundationsoverview, primer and where to startread it →
on this pageshow

explore

questions

5

How do you run a keyboard-only pass and an assistive-technology pass over a task, and why in that order?

level: juniorimportance: must knowfreq 66%

answer

  1. Two passes, two different questions
  2. Test the whole task, not one screen
  3. One asks can you operate it at all
  4. The other asks is what you hear enough
  5. The second pass is keyboard-driven too

basics

~20 s

Two passes over the whole task: complete it using the keyboard alone, then repeat it with an assistive technology, judging what is announced. Run the keyboard pass first — an inoperable screen makes every later finding point at the wrong cause.

solid answer

~50 s

Test a **task**, not a screen. The keyboard pass asks one question: can the task be completed end to end without a pointing device? You are looking for steps that cannot be reached or operated, anything that takes the focus point and will not give it back, a focus point you cannot see, and an order of controls that contradicts the meaning of the layout. The assistive-technology pass then asks whether what reaches a non-visual user is enough to act on: does each control announce a name that matches its visible label, is a change of state announced when it happens, and can you orient yourself from the structure alone. Run the keyboard pass first, because the second pass is also driven from the keyboard — operability defects contaminate every finding in it and get logged as announcement bugs.

code

pseudocode · 9 lines
pseudocode
MANUAL PASS RECORD
  task:      apply for a service contract (6 steps)
  pass:      keyboard-only
  duration:  35 min
  finding 3: step 4 - confirm control is reached before the two
             fields it confirms; order contradicts the screen
  finding 4: step 4 - focus point not visible on the machine picker
  finding 7: step 5 - overlay keeps the focus point after closing
  blocked:   findings 3,4 sit on the legacy screen (no owner)

go deeper

for a junior

Be ready to describe both passes concretely: put the pointing device away and finish the task, then repeat it with an assistive technology. Naming the two and saying keyboard comes first is the expected answer here.

for a middle

Explain what each pass is actually checking and why the order is not arbitrary — that the second pass is driven from the keyboard too, so operability defects would be logged against the wrong layer.

for a senior

Show how these passes fit real delivery: scoped by task, time-boxed, findings written so a fix can be judged, and defects on screens nobody will fix recorded rather than dropped.

for a principal

Own the question of who runs these and how often, what the passes cost against what they catch, and how the team avoids the trap of treating a manual pass as a ceremony that always ends in a pass.

## Two passes asking two different questions Automated rules settle the measurable criteria. The two manual passes exist to reach everything else, and they are not interchangeable — they ask different questions and produce different defect reports. - **The keyboard-only pass** asks: *can this task be completed with no pointing device at all?* It is a test of **operability**. - **The assistive-technology pass** asks: *is the information a non-visual user receives sufficient to complete the task?* It is a test of **perceivability and understandability**. Both are scoped to a task — "apply for a service contract", not "screen 4". Meaning-level criteria are about whether a person can finish something, and a screen-by-screen sweep systematically misses the joins between screens, which is exactly where the defects live. ## What the keyboard-only pass looks for Start at the first step of the task, put the pointing device out of reach, and finish the task. Record, per step: 1. **Reachability and operability** — every control needed for the task can be reached and activated. A step that cannot be completed without a pointer is a straight failure of the criterion requiring all functionality to be keyboard operable (2.1.1 Keyboard, level A). 2. **Keyboard traps** — anything that takes the focus point and will not release it. Covered by its own criterion (2.1.2 No Keyboard Trap, level A) precisely because it strands a user completely. 3. **A visible focus point** — you can always see where you are. The criterion requiring a visible indication of keyboard focus sits at level AA (2.4.7 Focus Visible). 4. **Order versus meaning** — the sequence in which controls are reached does not contradict the meaning or operability of the layout (2.4.3 Focus Order, level A). This one is a judgement, which is why no rule finds it. 5. **Recoverability** — after an error, a confirmation, or an overlay closing, you can still get back to where the task continues. ## What the assistive-technology pass looks for Repeat the same task with an assistive technology active, and judge only what a user could act on: - Every operable control announces a **name**, and that announced name agrees with the visible label. When the two disagree, a person using voice control can read the label out loud and have nothing happen. - A change of **state** — expanded, selected, checked, busy — is announced when it changes, not only when the control is first reached. - The **structure** is usable for orientation: you can move between the sections of the screen and know where you are. - Content that appears in response to an action is announced, rather than appearing silently below where you are working. - The task is **completable** — the end of the pass is a submitted form, not a tour of the controls. One discipline: record what a **user could or could not do**, never a claim about how a particular product behaves. If two assistive technologies differ, that is a note in the finding, not a criterion failure of its own. ## Why the keyboard pass runs first Three reasons, and the first is the one interviewers want: | Reason | What goes wrong if you invert the order | |---|---| | The second pass is keyboard-driven too | Every operability defect resurfaces inside the assistive-technology pass and gets written up as an announcement bug, so the report blames the wrong layer | | Keyboard defects are cheap to describe | They reproduce with no extra software, so they get fixed while the assistive-technology findings are still being triaged | | It establishes a clean baseline | On a screen you can already operate, a confusing announcement is genuinely about naming or structure, and the finding stands up | ## Making the findings usable A maintenance console for farm machinery has a 6-step application form for a service contract. A 35-minute keyboard pass over the 6 steps produced 9 findings; 4 of them were on one legacy screen at step 4 that nobody is willing to touch. The pass was still worth running, and here is why the write-up mattered more than the count: - Each finding names **the step**, what was attempted, what happened, and what was expected. "Step 4: after choosing a machine class, the confirm control is reached before the two fields it confirms." - Each finding names **the criterion in words**, so a fix can be judged: "the order in which controls are reached contradicts the meaning of the screen". - The four findings on the untouchable screen were recorded with the step they block and a severity, not quietly dropped. A defect nobody has budget to fix is still a defect, and an undocumented one becomes a surprise in the next audit. ## Common mistakes - Running the assistive-technology pass on a screen that is not yet keyboard operable. - Testing controls in isolation and never completing the task end to end. - Treating "I heard something" as a pass without judging whether the wording was actionable. - Skipping the manual passes because an automated scan came back clean — the two passes exist for the criteria that scan never evaluated.

  • You complete the whole task by keyboard with no failures. What have you established, and what have you not?
    You have established operability: every step can be reached and activated without a pointer. You have not established that a non-visual user can do it. Names may be wrong or missing, state changes may go unannounced, and content that appears may be silent. Operability is a precondition for the second pass, never a substitute for it.
  • How do you keep an assistive-technology finding from becoming a claim about one product?
    Write the finding as what the user could not do and what information was missing — 'no name was announced for the control that submits step 5' — rather than as a behaviour of a particular technology. If two technologies differ on the same construct, note the difference inside the finding. The defect is the missing or wrong information, which is the same on every platform.
  • The task spans a screen nobody will touch. Do you still run the passes over it?
    Yes, and you record the findings with the step they block plus a severity, even when no fix is planned. An unrecorded defect looks like a clean result later, and it removes the option of routing around the screen or warning users. Known and unowned is a legitimate state; unknown is not.

saying these in an interview costs you the question

  • Runs the assistive-technology pass before the screen is keyboard operable
  • Tests individual controls and never completes the whole task
  • Assumes hearing an announcement means the wording is usable
  • Skips both manual passes because the automated scan was clean
  • Records findings without naming which step of the task failed
open as a page

Which WCAG success criteria can an automated rule engine decide, and which need a human judgement?

level: middleimportance: must knowfreq 78%

basics

~20 s

An automated rule engine decides only criteria whose pass condition is mechanically measurable — a computed value, or a required property present or absent. Criteria turning on meaning, equivalence or order need a human. That boundary belongs to the criteria.

open as a page

Where do accessibility checks belong across design review, component acceptance, a build gate and periodic audits?

level: seniorimportance: should knowfreq 44%

basics

~20 s

Design review catches meaning decisions before they are built; component acceptance catches per-control naming and operability; a build gate catches regressions on the machine-decidable criteria on every change; a periodic audit catches whole-task problems the other three cannot see.

open as a page

A year after launch, how do you answer 'is this accessible?' with evidence rather than opinion?

level: principalimportance: should knowfreq 37%

basics

~20 s

Answer with evidence: dated records of which task was tested, by which pass, by whom, and what was found; a gate blocking regressions on every change; known unfixed defects with owners; and findings from people with disabilities.

open as a page

What must an accessibility line in a Definition of Done say to be able to fail?

level: middleimportance: nice to knowfreq 22%

basics

~20 s

A line that can fail names a check, a scope and a failing result: the task completes with no pointing device, every new control has an accessible name, the build gate adds no violations. 'Accessible' names none of them.

open as a page