After a two-week team trial of an agentic coding tool, what can you legitimately conclude from it?

## A published comparison answers someone else's question Comparisons of these tools are run on somebody else's code, by somebody else's team, at a moment that has already passed. Your question is not which tool is better in general. It is which shape survives *our* codebase, *our* checks and *our* review capacity. Nothing published can answer that. The absence of a credible public benchmark is not a reason to skip evaluation; it is the reason the evaluation has to be yours, small, and honestly bounded. ## Trial it on the work you are afraid of The instinct is to try a new tool on something clean: a fresh project, a well-covered module, a ticket somebody already understands. That is usually the least informative choice available. The properties that decide the outcome on an old codebase are exactly the ones a clean project lacks — rules that were never written down, coverage that stops in the middle of a path, code whose behaviour *is* the specification. So pick real tickets, in the area of the system the team dreads, and run them the way you would actually run them. Four engineers and a claims-processing system old enough to have outlived its authors is the realistic version of this. If the trial never touches the corrections path nobody understands, it has tested the part of the codebase that was never the problem. ## Observations that discriminate Decide in advance what you are recording, because afterwards everybody remembers the good run: - **How often it had to be told something the repository already says.** That is a statement about what the shape could see, and it is often fixable — sometimes by putting the rule where the tool reads. - **How often a change passed your checks and was still wrong.** This is a finding about your checks, and it is usually the most valuable thing the trial produces. - **How much of a run you could reconstruct a week later**, without asking the tool to narrate itself. - **How large the unit was that arrived per review decision**, and whether the reviewers kept up with it. - **What each person had to correct**, recorded with the same care as what went well. ## What you must refuse to conclude | the conclusion you will be tempted by | why the trial does not support it | |---|---| | it works on our codebase | it worked on the parts you chose to try | | it will work for the other teams | you measured your review habits and your coverage, not theirs | | it is clearly ahead of the alternative | nothing here was controlled, so the two samples are not comparable | | this rate will hold | a fortnight measures a fortnight, novelty included | | this settles it | the tool and the model behind it change on a cadence you do not set | And one that is easy to miss: a trial run by the people who wanted it measures those people. Volunteers tend to route around rough edges without noticing and to narrate the parts that worked. Put somebody sceptical in the trial, or read the result as the optimistic bound. Whether the tool made the team faster overall is a further argument again, with traps of its own, and it is not settled by a fortnight of enthusiasm. ## The most durable finding is not about the tool The most useful thing a trial of this kind produces is usually an answer to a question about yourselves: **are our checks strong enough that a wrong change is likely to be caught?** If the answer is no, that is true whichever tool you adopt and whether you adopt one at all, and it changes what the team does next far more than the tool choice does. Findings of that kind survive the tool being replaced; findings about the tool expire with it. A second durable finding is where your codebase resists being read — the areas where every shape had to be told things, because those are the areas where a human joining the team also has to be told things. ## Write the decision with an expiry A decision record here should carry four things: 1. what you concluded; 2. what it rested on — which tickets, which part of the system, which people; 3. the boundary of the claim, stated as what it does not cover; 4. the observation that would make you reopen it. That last item is what separates an evaluation from a preference. State before you start what result would have made you reject it and you have an evaluation; state it afterwards and you have a rationalisation. ## Answering this in an interview Say what you would try it on and why that choice is the informative one, name two or three observations you would record, and then spend most of the answer on what you would refuse to conclude. Candidates usually pitch the upside. The move being screened for at this level is bounding the claim before anybody else has to.