How do you attribute token cost in a LlamaIndex app using TokenCountingHandler?

## Why this exists A RAG query is not one LLM call. It is one or more embedding calls to encode the query, possibly a rerank call, and then one or more generator calls — several, if the response synthesizer refines across chunks. The bill is the sum, and nothing in the query's return value tells you what that sum was. `TokenCountingHandler` is the callback that adds it up. ## Wiring it in It is a callback handler, so it goes into a `CallbackManager`, and the manager goes somewhere the framework consults. The global path is `Settings.callback_manager = CallbackManager([token_counter])`, after which every component constructed from those settings emits events into it. Components also accept a `callback_manager` directly, which is how you scope counting to one index or one query engine rather than the whole process. The constructor's important argument is `tokenizer`: a callable that turns a string into a list of tokens. Pass the encoder for the model you actually call — for OpenAI models that is `tiktoken.encoding_for_model(model_name).encode`. If you leave it to a default and then call a model with a different tokenizer, every count is an estimate with a systematic bias, and the error compounds across thousands of requests. `verbose=True` prints each event's counts as it happens, which is useful exactly once, while you are learning what your pipeline calls. ## What it exposes Four aggregate counters matter: - `prompt_llm_token_count` — input tokens sent to the generator. In RAG this is the big one, because it contains all the retrieved chunk text. - `completion_llm_token_count` — output tokens generated. - `total_llm_token_count` — the sum of the two. - `total_embedding_token_count` — tokens embedded, both at index build and at query time. Separating prompt from completion is not pedantry: providers price them differently, usually with output several times more expensive per token, and the two respond to completely different levers. Prompt tokens are driven by `similarity_top_k` and chunk size; completion tokens by your prompt's verbosity instructions and any max-token cap. `llm_token_counts` holds the per-event records rather than the totals, which is what you need when a single query fired several LLM calls and you want to know which one was expensive — a refine-style synthesizer that walks ten chunks makes ten calls, and the aggregate hides that entirely. ## The cumulative trap Counters accumulate for the lifetime of the handler. Nothing resets them between queries. Read them, then call `reset_counts()` — or read deltas. In a script this is a minor discipline. In a server it is a correctness bug waiting to happen: a handler installed on global `Settings` sees every request from every thread or task, so by the time you read the counter after handling request A, requests B and C have added their tokens to it. There is no request identity in the aggregate. The fixes are the obvious ones. Build a fresh `TokenCountingHandler` and `CallbackManager` per request and attach it to the query engine you construct for that request, so the counts are naturally isolated. Or accept the global handler as a process-level meter — useful for "what did this service spend this hour" — and get per-request attribution from a tracing backend, which records token counts per span with a trace id attached. ## What you do with the numbers Token counting turns qualitative pipeline decisions into arithmetic. Doubling `similarity_top_k` roughly doubles the retrieved text in the prompt; the counter tells you exactly what that costs before you argue about whether the quality gain justified it. A tree-summarize or refine synthesizer over many chunks makes multiple generator calls where a compact one makes fewer; the per-event list shows the multiplier. Reranking adds a call but often lets you cut top-k, and the net can go either way — measure it. The strongest use is pairing cost with quality on the same eval set: run the batch, record the pass rate, record the tokens, and report both. A configuration that improves faithfulness by two points while tripling prompt tokens is a decision, not an improvement, and putting both numbers in front of the decision-maker is the job. ## Limits It counts tokens, not money — you apply the price table yourself, and prices differ per model, so a mixed-model pipeline needs per-model attribution that the aggregate counters do not give you. It also counts what your process sent, which can diverge from what the provider bills if the provider counts differently or if retries occurred inside the client below the callback layer.