What kinds of image questions do vision models still get wrong, and how do you design around it?

## The failure families **Counting.** This is the most reliably broken capability and the one most often assumed fixed. Ask for the number of product facings on a retail shelf photo and you get a fluent integer that is frequently off by a few — 11 where the truth is 14. The error is not random noise from a hard image; it is systematic under crowding, occlusion and repetition, exactly the conditions that make counting worth automating. Nothing in the output distinguishes a careful count from a guess. **Dense small text.** Covered by the resolution the model actually processed. Below that threshold, characters do not degrade gracefully — they are gone, and the model fills the gap with text that fits the surrounding context. A QA bot reading a toast notification at low resolution will report a message that sounds exactly like your product's copy and is not what was on screen. **Fine spatial relations.** "Is the submit button inside the modal or behind it?", "is this label misaligned with the field?", "which of these two icons is further left?" Models do far better at what is present than at precise relative geometry, and cluttered interfaces are the worst case. Anything downstream that consumes coordinates should treat them as hints, not measurements. **Unlabelled charts.** A quarterly revenue bar chart with gridlines but no data labels forces the model to read values off the axis. The right answer is an approximation with stated uncertainty ("roughly 4.2 to 4.5 million"); the common answer is "4.37 million", a precision the image cannot support. False precision is worse than a wide range because it survives review — a number with two decimal places reads as measured. ## Why there is no confidence signal All four share one root cause for the engineer: the model's linguistic fluency is decoupled from the strength of the visual evidence. It produces an equally well-formed sentence whether it enumerated carefully or pattern-matched. Asking "how confident are you?" produces another fluent sentence, not a calibrated probability. Any design that assumes the system will flag its own uncertainty is unsound. ## Designing around it **Crop before you ask.** If the answer lives in one region, send that region. The region then occupies most of the frame and survives resampling, and you pay for far fewer pixels. Where the layout is fixed — a known dialog position in a QA screenshot — the crop is deterministic. **Force enumeration.** Instead of "how many facings?", ask for a list: one line per item with a short description or an approximate position, then count the list length in your own code. This converts an unverifiable single number into a structure you can inspect, deduplicate and sanity-check. It is slower and costs more tokens, and it is the single most effective counting mitigation available at the prompt level. **Make abstention legal.** If your output schema has no way to say "cannot determine from this image", the model will always produce a value. Add the option explicitly and instruct that it be used. You will get some over-abstention; you will also stop shipping confabulated strings. **Demand ranges for chart reads.** Require a low and high bound plus the basis ("between the 4M and 4.5M gridlines"). A range you can act on beats a point estimate you cannot trust, and the stated basis makes the reasoning auditable. **Cross-check against non-visual truth.** If a number exists in a database, a log or an API, get it from there and use the image only to locate or contextualize it. Reading a value off a picture when the source of truth is queryable is a design smell. **Sample repeatedly where it matters.** For high-stakes counts, ask several times and compare. Agreement across independent samples is a weak but real signal; disagreement is a strong one and should route to human review. **Measure per task type.** Aggregate accuracy hides everything. Track counting, text extraction, spatial and chart-reading tasks separately against a labelled sample, because they fail at very different rates and only the per-type numbers tell you which parts of the product are safe to automate. ## Where the field is moving Frontier reasoning models increasingly mitigate small-text and counting failures themselves by inspecting regions of the image during reasoning rather than answering from one downscaled glance. That shifts the balance but does not eliminate the failure families, and it introduces its own cost and determinism trade-offs. The engineering posture stays the same: assume confident wrong answers are possible on these four task types, and build verification that does not depend on the model noticing its own uncertainty.