An engineer asks an analyst to inspect a failed coding-agent session and point to the step where the agent went wrong. The analyst names several steps. The next question is harder: did the analyst help the engineer repair the failure, or did it only point at labels that a benchmark already knew?
That difference is the reason to read CodeTraceBench, a public benchmark, carefully. CodeTraceBench is a public benchmark for diagnosing agent trajectories, where a trajectory is the ordered record of actions and observations from one coding-agent run. Its step-level labels make one question measurable: did the analyst identify the steps marked incorrect or unuseful?
The benchmark is useful for that question. It does not by itself measure whether an analyst explained the cause, ran a check, proposed a safe repair, or reduced a reviewer’s work.
In a public comparison over 32 pinned trajectories and two repetitions, a recursive analyst scored 0.3644 and a one-shot analyst scored 0.3673 on the report’s completed-row F1, a measure combining precision and recall, which scores only run slots that produced a result. F1 combines precision, the share of reported steps that match labels, and recall, the share of labeled steps that the analyst found. The recursive run cost $6.73. The one-shot run cost $1.21. The ratio is about 5.57 to 1.
That result is not evidence that recursive investigation has no value. It is evidence that this benchmark score cannot show whether the extra investigation helped a reviewer.
A benchmark can score the intended field correctly and still omit the capability behind the product decision.
The terms behind the score
A trace is the record of an agent run. It can contain model messages, tool requests, observations, files, retries, errors, and final output.
A benchmark is a fixed collection of cases used to compare systems under the same conditions. An evaluation is the process that runs a system on cases and applies a defined scoring rule. A metric is the number produced by that rule.
An analyst is the second program that reads a trace and reports findings. It is different from the coding agent that produced the trace.
A baseline is the system used as the comparison point. A candidate is the system or configuration being considered against that baseline. A repetition is another run of the same case and configuration. A completion means that a run slot produced a result that the report could score.
The public CodeTraceBench dataset’s reported inventory contains 4,316 agent trajectories in two published splits. Its verified split contains 1,000 trajectories, and its full split contains 3,316. The dataset documentation says the trajectories have human-verified step-level annotations and cover coding and terminal tasks across several agent families and models.
The experiment discussed here used only 32 pinned trajectories from that larger dataset. It ran each trajectory twice, creating 64 run slots. The subset is the experiment’s denominator. It is not a new score for all reported 4,316 trajectories.
What F1 can and cannot see
The public scorer compares the analyst’s reported step IDs with the labeled step IDs. Two simpler quantities make up F1.
Precision asks how many of the steps the analyst named were labeled mistakes. Recall asks how many of the labeled mistakes the analyst found. F1 combines precision and recall into one number between zero and one.
Here is a small, runnable illustration:
const labeled = new Set(['step-2', 'step-5'])
const reported = new Set(['step-2', 'step-9'])
const hits = [...reported].filter((step) => labeled.has(step)).length
const precision = reported.size === 0 ? 0 : hits / reported.size
const recall = labeled.size === 0 ? 0 : hits / labeled.size
const f1 =
precision + recall === 0
? 0
: (2 * precision * recall) / (precision + recall)
console.log({ precision, recall, f1 })
// { precision: 0.5, recall: 0.5, f1: 0.5 }The example measures step overlap. It does not ask whether step 2 caused the failure. It does not inspect the evidence behind the accusation. It does not run a test to validate a repair.
Those omissions are not bugs in arithmetic. They are boundaries of the measurement.
Work one case all the way through
Imagine a trace whose labels mark step-2 and step-5 as unuseful.
Matching reported step sets make the scorer assign one F1 to analyst outputs with different evidence quality.
| Analyst output | F1 consequence | What a reviewer still needs to know |
|---|---|---|
| Reports step 2 and cites the command output that caused the mistake | Counts one correct overlap | Whether the cited output supports the explanation |
| Reports step 2 and writes a plausible but unsupported sentence | Counts one correct overlap | Whether the explanation came from evidence or guesswork |
| Reports steps 2 and 5 but reverses their causal order | Counts two correct overlaps | Whether the proposed order helps repair the run |
| Reports step 9 because it sounds suspicious | Adds a false positive | Whether the analyst can recognize uncertainty |
The scorer sees the same step sets in the first two rows. An engineer does not. The first finding may shorten an investigation because it opens the relevant command output. The second may lengthen it by giving the reviewer another claim to disprove.
A useful evaluation case should therefore retain the entire trace, the analyst’s evidence references, and the action taken after the finding. It should ask the analyst to explain what would change its mind. It should also allow abstention when the trace does not contain enough evidence.
Those additions do not make F1 obsolete. They preserve the step-localization number while adding a measure for the work the product asks an analyst to do.
The head-to-head experiment
The public recursive-run report compares three rows over the same 32 pinned trajectories, with two repetitions per trajectory and one GLM-5.2 model through the metered router. A router is a model-access service that forwards requests to a provider and can record the model, usage, and cost of those requests.
The recursive analyst could inspect the trace, write and run Python over it, and investigate in several steps. The one-shot analyst read the trace once and returned its reported steps. The CodeTracer row is the upstream tool from the public CodeTracer project.
| Analyst | Process | Scored F1 | Precision | Recall | Completed | Known cost |
|---|---|---|---|---|---|---|
| Recursive analyst | Inspect, execute Python, investigate repeatedly | 0.3644 | 0.3413 | 0.3909 | 60/64 | $6.73 |
| One-shot analyst | Read once and return reported steps | 0.3673 | 0.3333 | 0.4091 | 63/64 | $1.21 |
| CodeTracer | Upstream trajectory diagnosis tool | 0.3128 | 0.2857 | 0.3455 | 61/64 | Not reported in this summary |
The recursive and one-shot scores differ by 0.0029. That gap is smaller than the uncertainty created by 16 labeled cases and two repetitions. The report therefore treats them as tied for this experiment.
The completion counts matter. The recursive row produced 60 scored results out of 64 slots. The one-shot row produced 63. The recursive row also cost $5.52 more in absolute terms and about 5.57 times as much.
The report says that provider rate limits caused three recursive failures. One more failure came from a model-emitted block wider than the configured 12-step cap. When the provider omitted usage, the run charged the reserved maximum, so the reported recursive cost is an honest upper bound rather than a claim that every provider usage field was present.
Do not mix the F1 columns
The public repository reports more than one aggregate for this benchmark family. The head-to-head README above reports a completed-row scored F1. The trace-analysis leaderboard also reports an official all-row F1 that includes failed rows and a separate micro F1. Micro F1 pools step-level matches across rows before calculating precision and recall; it is diagnostic context here, not the ranking metric.
For the corresponding published rows, the leaderboard lists:
| Row | Official all-row F1 | Micro F1 | Failed slots |
|---|---|---|---|
| Direct one-shot with test artifacts | 0.1347 | 0.3273 | 1 |
| Recursive stock run | 0.1021 | 0.3282 | 3 |
| CodeTracer comparison | 0.1161 | 0.2754 | 2 |
These numbers are not interchangeable with the completed-row values in the first table. The all-row aggregate penalizes missing runs. The micro aggregate weights matched steps across rows. The completed-row summary answers a different question.
The follow-up report uses still another table for its failure-block experiment. It is honest to show the denominator and aggregation beside every number rather than choose the most flattering one.
A denominator changes the decision
The raw cost ratio is incomplete because completion changes how a buyer might read it. Using the report’s simple completed-result counts, the recursive run cost about $6.73 divided by 60 completed results, or $0.112 per completed result. The one-shot run cost about $1.21 divided by 63 completed results, or $0.019 per completed result. That rough comparison is about 5.84 times higher for the recursive run.
This is not an official benchmark metric. It is a transparent calculation from the report’s cost and completion columns. It also treats the reported recursive cost as an upper bound because missing usage was charged at the reserved maximum. The calculation is useful only as a decision aid, not as a replacement for the published aggregates.
The distinction between failure and missing evidence matters here. If a run fails before producing a finding, a product may prefer to retry it. If a run produces a finding but cannot cite its evidence, a product may prefer to route it to review. Both can appear as “not completed” in a coarse dashboard while requiring different operational responses.
Report at least three denominators together:
| Denominator | Question it answers |
|---|---|
| Scheduled slots | How often did the system complete the requested run? |
| Scored rows | How accurate was the output when a score was available? |
| Labeled steps or blocks | How much of the annotated material did the aggregate weight? |
Keeping those denominators visible prevents a small completed subset from becoming a claim about the whole workload.
The score rewards localization, not investigation
The recursive analyst did real additional work. It inspected the trace, executed Python, and returned findings tied to the investigation. The benchmark score awards credit only when reported steps overlap labeled steps.
That creates a clean interpretation:
| The score can answer | The score cannot answer |
|---|---|
| Did the analyst name labeled incorrect or unuseful steps? | Did its explanation identify the cause? |
| How many named steps overlapped the labels? | Did a human replay the cited evidence? |
| How did precision and recall change? | Did an executed check confirm the finding? |
| How did analysts compare on the fixed cases? | Did a proposed repair work? |
If the product decision is “Which analyst should help engineers review failures?”, the right next measurement needs to include those missing outcomes. The existing score can remain one column. It should not be promoted to the whole decision.
The same issue appears in the trace record. An analyst can point to a step without preserving the file diff, command output, or observation that makes the accusation inspectable. An evaluation that rewards only the step ID will not distinguish a supported finding from a lucky label match.
Widening the label did not fix the aim
The first follow-up hypothesis was that the one-shot analyst lost recall because it named one step from a failure block instead of the whole contiguous block. The failure-block report tested that contract over the same 32 trajectories and two repetitions.
The report’s full comparison is:
| Measure | One-shot baseline | Failure-block run |
|---|---|---|
| Scored F1 | 0.3273 | 0.3407 |
| Precision | 0.2727 | 0.2875 |
| Recall | 0.4091 | 0.4182 |
| Official all-row F1 | 0.1347 | 0.1296 |
| Matched steps | 45 | 46 |
| Predicted steps | 165 | 196 |
| Predicted steps landing on a labeled step | 19.7% | 16.5% |
| Completed | 63/64 | 55/64 |
| Cost | $1.208 | $1.219 |
This table belongs to the follow-up report. Its baseline summary is not the completed-row 0.3673 in the recursive-run README, so the two tables should not be merged into one ranking. The reports expose different run contexts and aggregates. The raw artifacts and labels are the authority for reproducing either comparison.
The mechanism did fire. The block analyst emitted 123 blocks with a mean width of 2.27 steps, a median width of 2, and a maximum width of 11. The change therefore let the analyst name more than one step.
It did not improve the product question. The block run found one additional matched step, but it added 31 predicted steps and completed eight fewer slots. Only 16.5% of its predicted steps landed on labeled steps.
For this small corpus, the evidence points toward localization rather than block width as the binding problem. That is an inference from 16 labeled cases, 55 labeled steps per repetition, and 28 labeled blocks. It should guide the next experiment, not become a universal claim about every trace analyst.
The report also records nine failed slots in the block run. Some failures came from contract violations such as a consequence step preceding the block’s first step or a block wider than the 12-step cap. Those failures belong in the result because a production analyst must handle malformed or impossible output.
Build the next measurement around the review decision
If engineers need a diagnosis they can act on, keep localization and add the evidence around it. An illustrative finding shape might be:
{
"stepIds": ["step-21", "step-22"],
"evidence": [
{
"traceEventId": "tool-17",
"artifact": "test-output-3"
}
],
"cause": {
"statement": "The edit targeted the generated file instead of its source."
},
"checks": [
{
"name": "reproduce failing test",
"exitCode": 1
}
],
"repair": {
"status": "proposed",
"artifact": "patch-4"
},
"abstained": false
}This is a proposal for a measurement, not the current CodeTraceBench schema. It separates five questions:
- Did the analyst locate the labeled step?
- Did it cite evidence that a reviewer can open?
- Did an executed check support the diagnosis?
- Did the proposed repair change the right artifact?
- Did the analyst admit when the trace was too thin to support a claim?
The benchmark can then report one column per question. Cost and completion should remain visible beside them. If a recursive analyst costs more but produces accepted repairs, that benefit can be measured directly. If it costs more and only produces the same step overlap, the cheaper baseline is the better choice for that task.
The public agent-eval package already describes the broader evaluation pattern as running cases, scoring results, comparing a candidate with a baseline, and analyzing traces or feedback. The benchmark-specific schema still has to decide which review outcomes matter for its users.
Keep the benchmark, change the claim
Keep CodeTraceBench when the question is step localization. Use its labels to compare analysts under a fixed corpus and fixed scorer. Report the case count, repetitions, completed slots, aggregation, cost, and model alongside the score.
Do not use its F1 alone to claim that a recursive analyst is better for engineering review. The head-to-head result is a tie at roughly one-fifth the cost for the one-shot run. The follow-up shows that allowing wider failure blocks barely changes localization and lowers per-step accuracy on this small sample.
Choose the recursive analyst when the product needs inspectable investigation and you have a separate measure for that value. Choose the one-shot analyst when step localization is the only measured requirement and the lower cost matters. Run a new comparison when the product requires evidence, checks, repairs, or abstention.
The trace systems article explains why those artifacts belong beside the score. This benchmark is a useful instrument. The agent observability guide shows how missing events should remain missing instead of becoming false success values. It becomes a misleading product decision only when the team asks it to measure a capability it never records.
What does CodeTraceBench measure?
It measures whether an analyst’s reported step IDs overlap human-verified incorrect or unuseful step labels in coding-agent trajectories.
Did recursion improve the benchmark?
No. The recursive analyst scored 0.3644 and the one-shot analyst scored 0.3673 on the same 32 trajectories with two repetitions. The public report treats that difference as a tie.
Was the recursive investigation real?
Yes. The recursive analyst inspected traces, executed Python, and returned findings from those checks. The benchmark does not score that work, so the result cannot establish its value to a reviewer.
Why publish a result where the new system did not win?
The tie prevents a false promotion based on an unmeasured capability. It also identifies the next measurement: evidence, checks, repairs, unsupported claims, completion, and cost.
Can I reproduce the numbers?
The public reports link the pinned corpus, labels, result files, scorer, and run configuration. Re-running model calls still requires the provider access described by the reports.
Is F1 enough for choosing an analyst?
Use F1 when step overlap is the decision. For engineering review, add evidence quality, executed checks, repair acceptance, abstention, completion, and cost.