A year after launch, how do you answer 'is this accessible?' with evidence rather than opinion?

## The question is a trap, and the answer is a portfolio "Is this accessible?" invites a yes, and a yes a year after launch is almost always built on the wrong thing — a scan run this morning, or a memory of an effort that finished eleven months ago. The credible answer has three parts, and a lead's job is to make sure the organisation can produce all three on demand. 1. **Evidence of what was tested.** 2. **Evidence that it has not decayed.** 3. **An honest statement of what is still unknown.** ## Part one: what was actually tested A reading is what a tool says today. A **record** is what somebody did, when, and what they found. The record that holds up a year later has, per entry: the **task** (not the screen), the **pass** used, the **date**, **who** ran it, the **findings**, and the **decision** on each finding. Two properties make it credible: - It is scoped by task, so it maps onto what a user was trying to do rather than onto a route. - It includes the failures. A record with no findings anywhere is evidence that the passes were not really run. Alongside it sits the **defect log**, with each entry labelled by the requirement it breaks in words — "the announced name does not match the visible label" — so the log stays readable when personnel change. ## Part two: what stops it decaying Accessibility decays by default, because every change is an opportunity to remove a name or reorder a screen. Four mechanisms, and the mix is a real budget decision: | Mechanism | What it protects | Cost profile | |---|---|---| | Automated gate on every change | The measurable criteria, continuously | High setup, near-zero marginal | | Guarantees at the shared library boundary | Per-control naming, role and operability, across every consumer | Moderate, and it propagates | | Manual passes attached to changes | Meaning-level criteria on the surface that moved | Linear in change volume | | Periodic deeper pass over whole tasks | Composition and real paths | Lumpy, and blind between runs | The measurement that matters here is not the count of open findings. It is **how defects re-enter**. For every regression, ask which mechanism should have caught it and did not: a naming regression that a gate could have failed means the gate is mis-scoped; a broken order that only the quarterly pass found means the manual pass is not attached to changes. A year of that question, answered honestly, is worth more than a year of violation counts, which fall whenever rules are disabled. ## Part three: what you still do not know The part that separates a credible answer from a sales answer. Name, with an owner and a date each: - **Tasks nobody has walked.** Coverage by task, stated as a fraction, is far more honest than coverage by screen. - **Criteria never evaluated.** Most of the standard is not in anybody's rule set, so if no human ever judged them, say so. - **Known and unfixed defects.** A maintenance console for farm machinery has a 6-step application form whose step 4 is a legacy screen nobody will touch. After 14 months there are 4 open findings on it, and a lead's obligation is to make that an explicit, owned decision — with the risk stated and a route around the screen offered — rather than a silence that reads as a pass. ## What involving people with disabilities changes It changes the question being answered. Criteria-based testing asks whether requirements were met; a session with somebody who uses an assistive technology daily asks whether the **task was completed**, and the two answers diverge constantly. Three things it reliably surfaces that no pass does: - **Cost, not just possibility.** A task that takes 40 steps and 3 wrong turns satisfies the criteria and is abandoned in practice. - **Priority.** Practitioners rank findings differently from teams; the defect an engineer calls cosmetic is often the one that ends the session. - **Paths nobody designed.** Real users arrive mid-flow, with settings the team never tried. Two cautions a lead owns. First, participation is **paid, consented work with real practitioners**, not a favour extracted from a colleague. Second, it does not replace criteria-based testing: a handful of sessions cannot cover the standard, and criteria coverage cannot tell you a task is usable. Each answers what the other cannot. ## Saying it out loud "Of the 12 core tasks, 9 have a dated keyboard and assistive-technology pass in the last two quarters. The measurable criteria are gated on every change and have blocked 7 regressions this year. Two practitioner sessions changed our ordering of the backlog. We have 4 known open defects on one legacy screen, owned and risk-accepted, and 3 tasks nobody has walked yet." That is an answer with a shape somebody can audit — and it is the shape to give even when parts of it are uncomfortable.