In Google ADK, how does an .evalset.json file differ from a .test.json file?

## Two artifacts, one schema ADK's evaluation story rests on recorded expectations. Two file shapes carry them, and interviewers ask about the difference because it reveals whether you have actually run the tooling. ## The shared unit: the invocation Whatever the file, the atom is an **invocation**: one user turn and everything the agent did in response. An invocation records `user_content` (the user's message, in the standard content/parts shape), `final_response` (the text the agent should end up producing) and `intermediate_data`, whose `tool_uses` list names each tool call with its arguments. That triple is what the scorers consume: the tool calls feed the trajectory metric, the final response feeds the text-match metric. ## Test files — small, many, cheap A test file (`*.test.json`) represents a single, simple agent–model interaction: one session, often one or two turns. The idea is unit-test ergonomics. You keep a folder of them beside the agent, each pinning one behaviour — "asks for a city and calls get_weather with it", "refuses when the tool errors" — so a failure points at one thing. They are cheap enough to run on every change during development, and because each is small, updating one after an intentional behaviour change is a small diff rather than a re-record of a long conversation. ## Eval sets — few, multi-turn, recorded An eval set file (`*.evalset.json`) is a dataset. Its top level carries an `eval_set_id` and a list of `eval_cases`; each case has an `eval_id`, a `conversation` (the list of invocations, in order) and optionally a `session_input` block giving `app_name`, `user_id` and an initial `state` dictionary. That `session_input` matters: it lets a case start from a non-empty world — a user already identified, a preference already set — which is exactly what a multi-turn conversation needs and what a bare test file does not express. Eval sets are the integration tier. A case can be ten turns long and exercise a whole session's worth of state accumulation, sub-agent transfer and tool sequencing. They are also the artifact the dev UI produces: run a conversation by hand in `adk web`, open the Eval tab, create or pick an eval set, and save the session as a case. That recording path is the practical reason eval sets exist — capturing a real session beats hand-writing a ten-turn JSON blob. ## Running either one The CLI takes both: `adk eval <path_to_agent_module> <path_to_eval_file>`. You can restrict a run to specific cases by appending case ids to the eval set path, and `--config_file_path` points at the criteria file. From Python, `AgentEvaluator.evaluate(agent_module=..., eval_dataset_file_path_or_dir=...)` runs a file or a whole directory, which is how you fold ADK evals into a pytest suite and let your existing test runner report them. ## Thresholds live beside the files Neither file carries its own pass criteria. A `test_config.json` placed in the same folder supplies a `criteria` block that applies to the files in that folder. That folder-scoped design is deliberate: you can hold strict criteria over a folder of tool-trajectory tests and looser criteria over a folder of open-ended conversational cases without editing any case. ## Practical failure modes Recorded expectations rot. A recorded `final_response` is one phrasing the model happened to produce; a prompt tweak that improves answers can fail the case on wording alone. Recorded `tool_uses` assume the tool surface is stable — rename an argument and every case that touched it fails at once, which is informative but noisy. And a case recorded from a session whose tools hit live third-party services encodes that day's data into the expected response. The discipline is to record small cases for anything you want stable, keep multi-turn eval sets few and deliberately chosen, and re-record rather than patch when behaviour intentionally changes.