In ADK's adk eval, what do tool_trajectory_avg_score and response_match_score measure?

## What adk eval actually computes When `adk eval` replays a recorded case, it runs the agent against each recorded user turn and collects two things: the sequence of tool calls the agent emitted, and the final text it produced. Those are compared against what the eval case recorded, and turned into two named metrics. ## tool_trajectory_avg_score For each invocation, ADK lines up the actual tool calls against the `tool_uses` recorded under `intermediate_data`. The comparison is exact — tool name and argument values — and order-sensitive. An invocation scores 1.0 when the actual calls match the expected ones and 0.0 otherwise; the reported metric is the average over all invocations in the case, hence *avg*. The default pass threshold is **1.0**. That is a deliberate choice: for a deterministic tool path, anything less than a perfect match means the agent did something you did not sanction — called an extra tool, skipped one, or passed a different argument. It also makes the metric brittle by construction. An agent that calls `get_weather(city="London")` where you recorded `get_weather(city="london")` fails. An agent that calls a search tool twice because the first result was thin fails, even though the answer improved. When you knowingly accept variation, you lower the threshold, but a threshold below 1.0 on a two-call trajectory is a very coarse dial — 0.5 means "half the invocations may be wrong", not "a bit of wobble is fine". ## response_match_score The final response is scored with ROUGE-1 similarity against the recorded `final_response`: unigram overlap between the produced text and the expected text. The default threshold is **0.8**. Text scoring has to be fuzzy — no LLM emits the same sentence twice — but ROUGE-1 is a lexical measure, not a semantic one. Two consequences you should be able to name in an interview: - A correct answer phrased differently can score low. "London is cloudy, 15 degrees" against a recorded "It is 15C and cloudy in London" loses overlap on function words and formatting. - A wrong answer phrased similarly can score high. Swap the temperature and most unigrams still match. ROUGE-1 does not know which token carried the fact. So `response_match_score` is a **regression tripwire**, not a correctness oracle: it tells you the answer moved, and you look. Treating it as proof of correctness is the classic misuse. ## Setting criteria Neither threshold lives in the eval case. You put a `test_config.json` next to the eval files with a `criteria` object naming the metrics and their minimums; it applies to the eval files in that folder, and `adk eval --config_file_path` can point at one explicitly. Folder scoping is useful in practice: keep a folder of deterministic tool tests at trajectory 1.0, and a folder of open-ended conversational cases at a lower response bar, without touching a single case. When no config is supplied, the defaults above apply — trajectory 1.0, response match 0.8. ## Reading a failure The useful discipline is to read the two together: - **Trajectory fails, response passes** — the agent found the answer another way. Either the tools changed, or the model drifted to a different route. Often the correct response is to re-record, but first ask whether the new route costs more calls or touches something it should not. - **Trajectory passes, response fails** — the plumbing is intact and the language moved. Usually a prompt or model change; sometimes live tool data changed underneath a recorded case. - **Both fail** — treat it as a real behavioural regression, not a scoring artifact. `--print_detailed_results` gives the per-case breakdown you need to make that call rather than staring at a pass/fail summary. ## Where the metrics stop These two scorers are cheap, deterministic and offline, which is exactly why they belong in a fast loop. They cannot judge whether an answer was *good*, whether the agent was efficient, or whether a refusal was appropriate. Those judgments need a different instrument, and ADK's metric set is not the place to fake them. The productive framing in an interview: the ADK metrics are the regression net, and anything that needs judgment is a separate, more expensive tier you run less often.