Image Understanding in Practice

## The knob and what it does Every mainstream vision API gives you some control over how much of an image reaches the model. OpenAI's version is the `detail` parameter, which as of mid-2026 accepts `low`, `high`, `original` and `auto`. The semantics generalize even where the parameter name does not: - **low** — the image is reduced to a small, fixed token budget. Cost is constant no matter how large the source image was. The model gets the gist: layout at a glance, dominant colours, obvious shapes, roughly what a squinting human sees from across the room. - **high** — the image is processed at meaningful resolution. Cost scales with the image's dimensions, so a large screenshot costs many multiples of a thumbnail. - **original** — added for dense, spatially sensitive input such as computer-use screenshots, where any resampling loses the answer. Most expensive. - **auto** — the provider picks based on the image. ## The cost model, as a budgeting decision On newer models, image billing is patch-based: the image is divided into small fixed-size squares (32×32 pixels on OpenAI's newer models), the patch count is multiplied by a per-model factor, and that is your image token count. Two consequences matter for budgeting. First, **cost is quadratic in linear resolution**. Doubling both width and height quadruples the patch count. Halving a screenshot's dimensions before sending cuts its cost to roughly a quarter — usually the single largest saving available on a vision request, and one you control client-side without touching any API parameter. Second, **cost is per image, per request**. In an agent loop that re-examines the same screenshot across five turns, you pay for it five times unless the conversation prefix is cached. A pipeline processing 400 screenshots at full detail has a very different bill from one processing them at `low`, and the difference is often the deciding factor in whether the feature ships. ## Choosing per request class, not per project The mistake is picking one setting for the whole system. Classify the questions instead: - **Coarse classification** — "which screen is this?", "did the app crash or render?", "is there a modal open?" — `low` is enough and is dramatically cheaper. - **Reading small text** — a toast message, a truncated button label, an 11px validation error. `low` will not merely be less accurate here; the pixels carrying the text no longer exist after downscaling, and the model will confabulate plausible text rather than report that it cannot read it. This is the failure that surprises people: there is no "blurry" signal in the output. - **Precise spatial questions** — element coordinates, alignment defects, whether a control overlaps another. Needs full detail, and `original` when the downstream consumer acts on positions. - **Charts and dense dashboards** — gridlines and axis labels are small text. Treat these as the fine-text case. ## Cheaper alternatives to raising detail Before paying for `original` on a 4K screenshot, consider whether the question needs the whole image. Cropping to the region of interest client-side — the error dialog, the header bar — often gives *better* accuracy than full detail on the whole frame, at a fraction of the cost, because the region occupies more of the model's effective resolution. If you know the layout is fixed, that crop is deterministic and free. If you do not know where to look, the model can crop and zoom for you during reasoning, at the cost of extra passes. Similarly, resizing to the smallest resolution at which the target text is legible to a human is a reliable rule of thumb: if you cannot read it in the resized image, neither can the model. ## Verifying rather than guessing Token counts for images are predictable from dimensions and the detail setting, and providers document the formula and offer counting endpoints. Measure the cost of a representative request before extrapolating a monthly bill from an assumption; per-model multipliers differ enough that an estimate carried over from another model can be off by several times. ## Version caveat The four-level `detail` vocabulary and patch-based billing reflect mid-2026 practice; earlier models billed images via 512-pixel tiles and offered fewer levels. The durable ideas — a resolution knob, cost scaling with processed pixels, and the silent loss of fine text at low settings — outlive any particular parameter spelling.