The Instrument Problem
An agent evaluation is a repeatable test used to compare systems or decide whether a change should ship. This series examines four ways that an otherwise precise result can answer the wrong question: an incomplete benchmark, a weak behavioral signal, a ranking rule that disagrees with release policy, and a cost total that omits workers.
For teams evaluating coding agents, the evidence helps decide what each number can support, which failures require another measurement, and whether the release decision uses the same outcome the test actually counted.
Start here: AI Coding Agent Benchmark: What CodeTraceBench Measures. Begin with the boundary of a real coding-agent benchmark. Then test a behavioral signal against chance, align the winner with the release rule, and count every model-calling role before comparing cost.
AI Coding Agent Benchmark: What CodeTraceBench Measures
CodeTraceBench, an AI coding-agent benchmark, gave recursive and one-shot analysts nearly the same localization score while recursion cost 5.57 times more.
AI Agent Evaluation: Output Decay Is Weak Evidence
Three shrinking model responses are weak evidence of agent decay; five calls plus a 60% endpoint make this behavioral check harder to fool today.
AI Agent Evaluation: Match Selection to the Release Rule
An AI evaluation can pick the wrong winner when its average score disagrees with the fail-closed rule used to release an agent candidate safely.
AI Agent Cost Accounting: Count Every Child Worker
An AI agent run can look cheap when its ledger counts only the driver; reconcile child-worker tokens and mark missing usage as unknown before comparing systems.