How do you count the tokens an LLM prompt will use before sending it?

## Why anyone counts tokens A token count is not trivia; it is the unit three separate budgets are denominated in. Providers bill per token, with input and output priced differently. Latency scales with tokens: the prefill phase processes the input, then each output token is generated in sequence, so a prompt twice as long costs roughly twice the prefill work. And every model has a hard context limit expressed in tokens — exceed it and the request is rejected or history is silently dropped. Estimating tokens before dispatch is therefore ordinary engineering work, the same way you would size a payload before a network call. ## A token is not a character, a word, or a byte A token is an entry in the vocabulary that a particular tokenizer learned. Common English words usually map to a single token, leading whitespace normally attaches to the following word, and rarer words break into pieces. The crucial consequence is that a token count is a property of a *(text, tokenizer)* pair, not of the text alone. The same paragraph can differ by tens of percent between two model families, and even between generations of one family, because vocabularies are retrained. ## Exact counting: two routes The first route is an offline tokenizer: the provider or the open-weight release publishes the vocabulary and the merge rules, and you tokenize locally at zero marginal cost. The second is a counting endpoint: you send the assembled request to the provider and get back the number of input tokens it would consume. Offline is cheaper and works on the hot path; an endpoint costs a round trip but is authoritative for closed models whose tokenizer is not published. Either way, count the *rendered* request. Chat models are not fed a list of strings; they are fed one sequence produced by applying a chat template that wraps each message in role and turn-delimiter tokens and prepends the system prompt and tool definitions. Tokenizing the raw concatenated message strings undercounts by a fixed amount per message — a small error on one long message, a large one on a hundred short ones. ## Heuristics and their honest range The ~4-characters-per-token rule (equivalently ~0.75 words per token) is a reasonable prior for English prose and nothing else. Observed ratios in practice: ordinary English prose lands near 3.5–5 characters per token; source code with punctuation and indentation lands near 2.5–3.5; minified code, base64 blobs, UUIDs and hex hashes fall toward 1–2 because random character sequences share no learned merges; and non-Latin scripts such as Japanese or Thai often approach one token per character. A per-conversation cost model built on the flat 4x rule will be wrong by a multiple on some traffic. When you genuinely cannot tokenize on the hot path, do not use the generic constant. Sample real traffic, measure a characters-per-token ratio *per locale and per content type*, take a pessimistic percentile rather than the mean, add a fixed per-message overhead, and validate the estimator against the usage numbers the API reports. ## What must be inside the count Everything the model reads is input: the system prompt, every retained turn of history, retrieved documents pasted into context, tool and function schemas (descriptions and JSON Schema are prompt text and are re-sent on every call), and the template markers. Tool schemas are the most commonly forgotten line item — a large toolset can occupy thousands of tokens on every request before the user has typed anything. ## Output shares the budget Input and output normally compete for the same window, so the number of output tokens you allow must be subtracted from what the input may use; many APIs reject a request whose input plus requested maximum output exceeds the limit. On models that expose a reasoning-depth dial, the hidden reasoning tokens are generated and billed as output too, so a high effort setting silently enlarges the output side of the budget. ## Operational practice Log the reported input and output usage for every call alongside your pre-flight estimate. The delta is a cheap regression signal: it jumps the moment someone adds a tool, expands the system prompt, or switches model. Re-run your estimator over a worst-case corpus in CI, and treat a model swap as a re-measurement event rather than a drop-in. ## Common mistakes Assuming one token equals one word; reusing another family's tokenizer because it was handy; counting only the user message; forgetting the output reservation; and treating the character heuristic as a guarantee rather than a guardrail.