What is thinking with images, and when does it beat pre-cropping a screenshot yourself?

## What the technique is The naive way to answer a question about an image is one pass: encode the whole frame at whatever resolution the detail setting allows, then generate an answer. Everything below the effective resolution is unavailable, and everything the model failed to attend to on that single pass is gone. Thinking with images breaks that. During its reasoning, the model acts on the image — cropping to a region, zooming in, rotating — and then looks again at the result, folding what it finds into the ongoing reasoning trace. Functionally it is tool use where the tool operates on the pixels. A 4K QA screenshot no longer has to be understood all at once: the model localizes the error dialog, zooms into it, reads the 11px text at usable resolution, and answers from that. As of mid-2026 this is live in frontier reasoning models and in a growing set of open research systems, and it has become the expected answer to "how do you handle small text and counting failures?" — replacing the older answer of building a tiling pipeline yourself. ## Why it works The underlying constraint has not changed: an image gets a finite token budget, and detail below what that budget represents is lost. Zooming spends a fresh budget on a smaller area, so the same tokens now describe far fewer pixels each. A crop that covers 5% of the frame gives that region roughly twenty times the effective resolution of the full-frame pass. It also helps for reasons beyond resolution. Counting a crowded shelf becomes counting several sparse regions in sequence, which is a much easier task. Reading a cluttered dashboard becomes reading one chart at a time. Attention no longer has to be split across a whole busy frame. ## What it costs **Tokens.** Every re-inspection is another image in the context. A run that zooms four times costs roughly five images, not one. On a high-volume pipeline that difference dominates the bill. **Latency.** Each pass is a reasoning round trip. Response time grows with the number of inspections, and the number is not fixed. **Determinism.** Two runs on the same screenshot may inspect different regions and take different numbers of passes. Cost per request becomes a distribution rather than a number, which complicates budgeting and makes latency SLOs harder to hold. **Observability.** You are relying on the model's choice of where to look. When it answers wrongly, the useful diagnosis is which regions it inspected — so capture that from the reasoning trace where the provider exposes it, or you are debugging blind. ## When to crop yourself instead Deterministic pre-cropping wins whenever you already know where to look: - **Fixed layouts.** In a mobile-app QA pipeline the error dialog, the toast area and the nav bar are at known coordinates. Cropping to them is free, exact and reproducible, and it removes a whole class of "the model looked at the wrong thing" failures. - **Tight cost or latency budgets.** One crop at a known resolution is a fixed price. Agentic zooming is not. - **High volume, narrow task.** Reading one field from a million screenshots does not need open-ended visual search; it needs the same crop a million times. - **Auditability.** When a decision must be reproducible after the fact, a deterministic crop is far easier to defend than "the model chose to look here". Use the model's own inspection when the search is genuinely open-ended: an unfamiliar interface, a failure you cannot localize in advance, a document-like screenshot whose structure varies, or a question that requires comparing several distant regions whose positions are not known. ## The hybrid, which is usually the answer Most production designs combine them. Crop deterministically to a coarse region you are certain about — the app window, the content pane, the chart area — to remove irrelevant pixels and cut cost, then let the model zoom within it for whatever fine detail the question needs. You get the cheap, reliable reduction from your own knowledge of the layout and the adaptive search from the model's. A second useful pattern is capping the exploration. Where the platform exposes an effort or reasoning-budget control, a low setting keeps the pass count down for easy screenshots and reserves deep inspection for the ones that need it. Then measure: track accuracy and cost per task type with and without model-driven zooming, because on coarse questions the extra passes buy nothing and you are paying for exploration you did not need.