Your accessibility scan came back with four issues and a score in the nineties, and the question underneath it is whether that means you are compliant. It does not, and the reason has nothing to do with the quality of the tool. A scanner can only settle the failures that have a machine-checkable answer, and those are a minority of what the standard actually asks for.
That is not an argument against scanning. We run automated checks on every project and we build automated scanners, so the point here is narrower and more useful: a clean report tells you something real about one class of defect and nothing at all about the rest. Knowing which half of the report is missing is the difference between a site that passes a tool and a site someone can use.
Why the coverage numbers disagree
If you have looked into this at all, you have seen the coverage figure quoted several different ways, each time as a single number and each time with the confidence of a settled fact. The spread between those numbers is not sloppiness. The studies are measuring different things.
The UK Government Digital Service ran the cleanest version of the experiment. Its team planted 142 known accessibility barriers in a single page and ran thirteen automated tools across it to see how many each one found. The best performing tool found 40% of them, and the worst found 13%.
Deque, which builds the axe engine most of those tools are based on, ran the other kind of study. It analysed close to 300,000 issues across more than 13,000 pages drawn from over 2,000 real audits and reported 57% coverage. The methodology note matters more than the headline: Deque got there by changing the unit of measurement from WCAG success criteria to total volume of issues found, on the reasoning that some issue types are far more common than others and automation happens to be good at the common ones.
Both numbers are honest and they answer different questions. GDS asked how many distinct kinds of barrier a tool can recognize; Deque asked what share of the issues in a typical audit a tool ends up flagging. Neither tells you whether the problems on your site are the detectable kind, which is the only version of the question you care about.
We use roughly 30% as a working figure for the share of WCAG criteria a scan can settle on its own. It is the conservative end of the range, and it is the number we would rather be judged against when a client asks what an automated pass bought them. Whatever figure you use, treat it as a description of a method rather than a measurement of your risk.
What a scanner can actually decide
Automated tools are good at a specific and genuinely valuable job: checking conditions where the pass criterion is a value rather than an opinion.
- Contrast ratios between two declared colors
- Missing
altattributes, missing form labels, missing document language - Duplicate IDs and ARIA attributes pointing at elements that do not exist
- Heading levels that skip, and landmark regions that are absent
- Images, buttons, and links with no accessible name at all
These are worth catching, they are cheap to catch, and a scan will find every one of them in seconds. If you have not run one, start there. The free tools and how to use them are covered in the compliance post, along with which rules apply to your organization and by when.
The list above is also the whole list. Everything else in WCAG requires somebody to judge meaning, sequence, or the page as it actually renders, and that is where a report stops being able to help you.
The four kinds of failure that get through
It is more useful to sort these by why automation fails than by which success criterion they violate. There are four reasons, and together they cover almost everything a scan hands back clean.
Judgment about meaning. alt="IMG_4471.jpg" fails a scanner. alt="chart" on a revenue chart passes it and tells a screen reader user nothing they could not have guessed. The attribute is present, correctly formed, and useless. The same gap runs through link text that all reads “learn more,” headings that are large text rather than actual headings, and form labels that say “field 2.” A machine can confirm that a description exists. Whether it describes anything is a question about content.
Behaviour across a sequence. A keyboard trap, a modal that opens without moving focus into it, a dropdown that only responds to hover, a carousel you cannot pause. These failures live in the transition between states, and a scan evaluates one state at a time. Every element on the page can be individually correct while the path through them is unusable.
The rendered page. The contrast ratio between two declared colors is a fact a machine can settle, and settling it early is worth doing. We built a contrast ratio matrix tool for one design system that checks every foreground and background pairing in the palette against AA and AAA, so a failing combination gets caught before anyone builds with it. What a palette cannot tell you is what the text ends up sitting on. The same two colors are a different ratio in practice once the text is laid over a photograph, a gradient, or a video still, and deciding whether it holds requires looking at the rendered page. Reading order has the same shape, because it can match the underlying markup exactly and still not match the order the reader sees.
Whether the accommodation works. This is the most expensive one, because the markup is right and the experience still fails. An ARIA live region that fires but announces the wrong thing. An error summary that exists but does not move focus, so a screen reader user submits an empty form and hears nothing change. A skip link that sits in the markup and never becomes visible on focus. Every automated check passes.
Some of that is abstract, so here is a concrete version. On an operations platform we took over rather than built, a compliance review turned up more than 200 WCAG violations. The contrast failures and the unlabeled inputs came off an automated pass in an afternoon, and they were the cheap half. What took the real work was that the application had no coherent keyboard path through it at all, which no scan reported, because every individual control was reachable in the markup and none of them were reachable in a sensible order. Getting to 2.1 AA was mostly that second problem.
What the industry publishes about itself
The strongest argument here is not ours. It comes from the organizations running the largest automated accessibility studies in the world, stated in their own reports.
The WebAIM Million analysis scans a million home pages every year, and its 2026 run found detected WCAG failures on 95.9% of them, averaging 56.1 errors per page. WebAIM states the caveat directly in the report: “not all conformance failures can be automatically detected,” and “absence of detected errors does not indicate that a page is accessible or conformant.” Because only automatically detectable failures were counted, they conclude the true rate of full A/AA conformance is “certainly lower than 4.1%.” The largest automated study in the field spends a paragraph explaining that its own method understates the problem.
The W3C’s Web Accessibility Initiative, which writes the standard, puts it in one sentence: “no tool alone can determine if a site meets accessibility standards. Knowledgeable human evaluation is required to determine if a site is accessible.”
This is also the ceiling that overlay widgets run into. A product that automates the fix inherits the same limit as the product that automates the detection, and it cannot repair what it was never able to see. That argument has its own post: why accessibility overlays don’t count as WCAG compliance.
The twenty-minute manual pass
You do not need assistive technology expertise to find out whether your clean scan was the whole story. Pick one page that matters commercially, a checkout or a signup or a contact form, and run these five steps against it.
- Use only the keyboard. Leave the mouse and trackpad alone and tab through the page. Can you reach every interactive element, in an order that matches the visual layout, and get back out of every menu, modal, and dropdown you enter?
- Watch the focus indicator at every stop. Can you see where you are, on every element, including custom components and anything inside a modal? Focus that disappears partway through is the most common failure this pass turns up.
- Break a form on purpose. Submit it empty. Is the error announced, and does focus move to it, or does the message appear somewhere above the part of the page you are looking at?
- Zoom the browser to 400%. Text should reflow into a single column with no horizontal scrolling, nothing clipped, and nothing overlapping.
- Turn on the screen reader already on your machine. VoiceOver on macOS, Narrator on Windows. Read one flow start to finish and listen for whether it makes sense, not for whether it speaks.
Be honest about what this produces. It is not an audit, it does not generate a conformance record, and it will not survive contact with a procurement questionnaire. What it does in twenty minutes is tell you whether the report you are holding described your site or described a subset of it, which is the decision in front of you right now.
What this changes about how you buy
Two practical consequences follow, and they point at different work.
Automated scanning is the right tool for catching regressions, and it should run on a schedule rather than once. A site drifts after launch as content, brand, and dependencies move underneath it, which is one of the things ongoing maintenance exists to catch, and a monthly check finds a slipped contrast ratio in the week it landed. That is the part of the Monitor work automation is genuinely good at.
A manual pass is what a first assessment needs, because the failures that keep users out are concentrated in the part a tool cannot reach. It is also the honest basis for any claim you make publicly. Our own accessibility statement says testing combines automated scans with manual assistive technology testing, because the automated half alone would not support the claim. If you want the manual half done properly, that is what the accessibility work covers.
An automated scan is a smoke detector: worth having, worth wiring into everything, and cheap enough that there is no argument for skipping it. Nobody has ever walked through a building, noted that the alarm was silent, and filed that as proof the place was safe.