An evaluation campaign can contain two different decisions that look like one. The winner picker chooses which candidate looks best among the candidates that were tested. The release rule decides whether that candidate is good enough to replace the current version.
If the picker averages every test cell but the release rule requires every repetition of an instance to pass, the two decisions can disagree. That happened in the public regression fixture behind agent-eval’s release-rule change. The CLAUDE candidate passed 5 of 12 cells for a mean of 0.417 and fully resolved 2 of 6 instances. The MERGE candidate passed 6 of 12 cells for a mean of 0.500 but fully resolved only 1 of 6.
The old mean-based picker chose MERGE. The fail-closed release rule preferred CLAUDE. The fix lets the campaign supply a ranking key built from the same reduction used by its release rule.
agent-eval is Tangle’s public TypeScript toolkit for running cases, scoring results, and comparing a candidate with an incumbent. The public test in this post isolates one comparison-path defect; it is not a broad benchmark of either candidate.
The point is narrower than “averages are bad.” An average is a reasonable summary for many continuous measurements. It is the wrong winner key when the product promise is “release only instances that pass every required repetition.”
Two decisions hiding in one campaign
An evaluation is a repeatable test of a system against a stated outcome. An evaluation campaign runs several candidates over a defined collection of cases and stores the results. The cases might be coding tasks, prompts, traces, or other fixed inputs.
A candidate is one version or configuration being compared. The incumbent is the version currently in use. An instance is one case in the campaign. An evaluation cell is one candidate–instance–repetition result. The repetition matters because a model can vary across runs even when the prompt and candidate are unchanged.
The winner picker answers:
Which candidate ranks highest among the candidates with complete results?
The release rule answers:
Does that candidate clear the bar that permits replacement of the incumbent?
Those questions can use the same score, but they do not have to. They must be deliberately related.
If the release rule is fail-closed, one failed required observation is enough to withhold release.
In the public fixture, an instance is counted as resolved only when both of its repetitions resolve.
That is an AND rule across repetitions: pass(rep0) && pass(rep1).
The NIST AI Risk Management Framework places measurement inside a larger process of defining validity, documenting limitations, and managing the consequences of a system decision. That is the right level for this bug. The arithmetic was correct. The decision boundary was inconsistent.
How the means invert the result
The public regression test uses six instances and two repetitions, so there are 12 cells per candidate. Here is the complete shape in a compact table.
T means the candidate resolved that repetition.
F means it did not.
| Instance | CLAUDE rep 0 | CLAUDE rep 1 | MERGE rep 0 | MERGE rep 1 |
|---|---|---|---|---|
| i1 | T | T | T | T |
| i2 | T | T | T | F |
| i3 | T | F | F | T |
| i4 | F | F | T | F |
| i5 | F | F | F | T |
| i6 | F | F | F | F |
| Passing cells | 5 | 6 | ||
| Fail-closed instances | 2 | 1 |
CLAUDE has two fully passing rows, i1 and i2.
Its third pass is a one-repetition success on i3.
That gives five passing cells out of 12, or 5 / 12 = 0.4167, shown as 0.417.
MERGE has six passing cells.
Only i1 has two passing repetitions, so it has one fail-closed instance out of six.
Its mean is 6 / 12 = 0.500.
The mean asks, “How many individual cells passed?” The fail-closed count asks, “How many instances passed every required repetition?”
Both are real measurements of the table. They answer different questions.
The historical selector sorted candidates by the mean composite.
It therefore placed MERGE above CLAUDE because 0.500 > 0.417.
The release rule cared about fully resolved instances and placed CLAUDE above MERGE because 2 > 1.
This is not a numerical rounding problem. Rounding 0.4167 to 0.417 and 0.500 to 0.500 makes the inversion easier to read, but it does not cause it. The inversion exists in the underlying reductions.
It is also not a claim that MERGE is universally worse. The fixture says only that this candidate pattern is a poor match for this fail-closed release contract.
Why the mean looked attractive
Mean scores are convenient. They use every observed cell, provide a smooth ordering, and make it easy to compare candidates with different strengths across cases. For a continuous outcome such as latency, calibration error, or a quality score whose release rule also uses an average, the mean may be exactly the right primary key.
The problem appears when the campaign’s visible composite and its release contract represent different units. The mean counts individual repetitions. The fail-closed rule counts complete instances. One noisy instance can contribute a partial win to the mean while contributing nothing to the release count.
A candidate with many isolated successes can therefore outrank a candidate with fewer successes that arrive in repeatable pairs. That is what the table shows.
The Demšar paper on statistical comparisons of classifiers makes a related point for experimental comparisons: the unit of comparison and the dependence structure matter when turning many observations into a conclusion. The paper is about classifiers rather than agent campaigns, so it does not prescribe this release rule. It supports the narrower discipline of naming the unit being counted before ranking systems.
The same concern appears in a software release pipeline. Suppose one test suite runs two repetitions per bug and the release promise is “a bug is fixed only if both checks pass.” Counting every green check as an equal vote lets a flaky one-off pass compete with a fix that survives both checks. The selector has silently changed the release promise from “repeatably fixed” to “frequently green.”
The fix is not to discard the mean. Keep it as a descriptive field and use it as a secondary tie-breaker when the release contract allows it. Change the primary winner key to the same fail-closed reduction that decides whether a candidate may ship.
The public fix: one injectable ranking key
The public change adds an optional selectionRankKey to the optimization path.
It returns an ordered list of numbers where each element is compared from left to right and higher is better.
The fail-closed consumer can return a key such as:
[fullyResolvedInstances, meanCellScore]The first number carries the release contract. The mean breaks ties among candidates that resolve the same number of complete instances.
The existing default remains the scalar mean for consumers whose outcome is continuous and whose gate does not require an AND across repetitions. That is an important boundary in the change. The fix changes the consumer with the mismatch; it does not pretend that every campaign should rank by fail-closed counts.
The public release-rule test file tests both behaviors. The default scalar selector promotes MERGE. The supplied fail-closed key promotes CLAUDE. Another test asserts that the winner has two fully resolved instances even though MERGE has the higher mean.
The linked source record also explains why the descriptive composite remains untouched. Reports can still show the mean. The ranking decision now uses the metric the release rule uses. That separation makes a report more informative, not less.
Selection is not the same as promotion
There is one more boundary worth preserving. A compatible winner key does not turn the selected candidate into an automatic release. It only makes the comparison among candidates speak the same language as the gate.
Imagine that CLAUDE wins the fail-closed count with two complete instances while the incumbent has two as well. The candidate has matched the incumbent on the primary key, not strictly beaten it. The campaign should keep the incumbent unless a secondary rule or a human decision says otherwise. If CLAUDE wins with three complete instances but has an incomplete result on a required case, coverage can still make it ineligible. If it clears coverage and the primary key, a safety test can still reject it.
This distinction prevents a second class of instrument bug. The first bug ranks the wrong candidate. The second treats ranking as permission to ship. Both can hide behind a green test if the test asserts only that some candidate was returned.
A useful campaign report therefore exposes three labels rather than one. It can say which candidate had the highest descriptive mean, which candidate won the release-compatible key, and whether the winning key cleared the incumbent and the minimum threshold. Those labels may agree, but a reader should not have to infer that they always do.
The public change keeps the historical scalar-mean default for consumers where a fail-closed reduction would be inappropriate. That is a design constraint, not an omission. The ranking key belongs to the consumer because the consumer owns the meaning of success.
If a new consumer introduces a different release contract, its tests should include at least one table where the old default and the new contract disagree. The disagreement is the regression fixture’s value. It proves that the new path uses the new unit of success rather than merely renaming the old average.
A runnable table and ranking example
This TypeScript example carries the public fixture’s decision shape without importing any project code. It calculates the cell mean, counts rows where every repetition passed, and ranks candidates by the fail-closed count before the mean.
Save it as winner-rule.ts and run it with npx --yes tsx winner-rule.ts.
import assert from 'node:assert/strict'
type Candidate = {
name: string
cells: boolean[][]
}
function mean(cells: boolean[][]): number {
const flat = cells.flat()
return flat.filter(Boolean).length / flat.length
}
function failClosed(cells: boolean[][]): number {
return cells.filter((repetitions) => repetitions.every(Boolean)).length
}
function rank(candidate: Candidate): [number, number] {
return [failClosed(candidate.cells), mean(candidate.cells)]
}
function compare(left: [number, number], right: [number, number]): number {
return left[0] - right[0] || left[1] - right[1]
}
const claude: Candidate = {
name: 'CLAUDE',
cells: [
[true, true],
[true, true],
[true, false],
[false, false],
[false, false],
[false, false],
],
}
const merge: Candidate = {
name: 'MERGE',
cells: [
[true, true],
[true, false],
[false, true],
[true, false],
[false, true],
[false, false],
],
}
assert.equal(mean(claude.cells).toFixed(3), '0.417')
assert.equal(mean(merge.cells).toFixed(3), '0.500')
assert.equal(failClosed(claude.cells), 2)
assert.equal(failClosed(merge.cells), 1)
assert.equal(compare(rank(claude), rank(merge)) > 0, true)
console.log({
claude: { mean: mean(claude.cells), failClosed: failClosed(claude.cells) },
merge: { mean: mean(merge.cells), failClosed: failClosed(merge.cells) },
winner: compare(rank(claude), rank(merge)) > 0 ? claude.name : merge.name,
})The expected output is:
{
claude: { mean: 0.4166666666666667, failClosed: 2 },
merge: { mean: 0.5, failClosed: 1 },
winner: 'CLAUDE'
}The key is not the tuple syntax. The key is that the first element is produced by the same rule that the release decision uses. If the release contract changes, the ranking key must change with it.
An even safer implementation makes the release reduction a named function and passes that function to both the selector and the gate. That avoids having one path count complete instances while another path counts individual cells. The public change uses an injectable key rather than duplicating one hard-coded ranking policy for every consumer.
What this result does not prove
The six-instance fixture is a regression test for a logic bug. It is not a benchmark of CLAUDE versus MERGE. It does not establish that one candidate is better across models, prompts, or future cases.
The fixture uses binary resolved/not-resolved outcomes. If your outcome is a continuous score, fail-closed counting may throw away useful information. If your outcome is a safety property where one failure must block release, fail-closed counting may be the appropriate primary rule.
The fixture also has only two repetitions per instance. More repetitions can expose more instability, but they cost more and can create their own sampling questions. The right number depends on the decision and the failure cost.
The result does not say that the mean should disappear from reports. The mean remains useful for showing partial progress, diagnosing where a candidate succeeds, and breaking ties after the release-compatible primary metric.
It does not say that a fail-closed winner must ship. Winner selection and promotion are still separate decisions. The selected candidate can fail a minimum threshold, a safety check, a cost limit, or a human review.
The NIST AI RMF is useful here because it treats measurement as part of risk management rather than as a single number that settles every question. The campaign should say what the score means, what it does not mean, and who is allowed to act on it.
A practical review before you trust a winner
Before a campaign promotes a candidate, write down its unit of success. Is success an individual cell, an instance across all repetitions, a whole benchmark, or a cost-adjusted outcome?
Then compare three items side by side.
| Item | Question to ask |
|---|---|
| Winner key | What number or ordered key chooses among complete candidates? |
| Release rule | What exact condition permits replacement of the incumbent? |
| Reported composite | Which descriptive summaries explain the result without controlling the choice? |
If the first two rows use different units, produce an explicit reason. If there is no reason, use one reduction for both.
For a fail-closed two-repetition contract, a defensible default is:
- Count fully resolved instances first.
- Use the mean cell score only as a tie-breaker.
- Require a strict improvement over the incumbent’s same key.
- Keep incomplete candidates visible in the report but ineligible for promotion.
The fourth rule matters because a candidate with missing cells can otherwise win by being judged on an easier subset. The public fix preserves coverage checks before ranking.
For more context, read AI Coding Agent Benchmark: What CodeTraceBench Measures and Evaluation Gates: The Rule That Decides Whether an Agent Improves. The first explains why a metric can omit the work a product cares about. The second explains why the release gate is part of the system being measured.
The honest decision is simple. Use averages when averages define success. Use a fail-closed key when repeatable completion defines success. Do not let a convenient selector silently rewrite the promise your release rule makes.
FAQ
What is the difference between a winner picker and a release rule?
The winner picker ranks the tested candidates. The release rule decides whether the selected candidate may replace the incumbent.
Why did MERGE win the average but lose the fail-closed rule?
MERGE passed 6 of 12 individual cells for a mean of 0.500, but only 1 of 6 instances passed both repetitions. CLAUDE passed 5 of 12 cells for 0.417 and fully resolved 2 of 6 instances.
Is a mean score always unsafe for agent evaluation?
No. It can be the correct primary metric when the outcome and release contract are both defined as averages. The defect appears when the picker and release rule count different units.
What does fail-closed mean in this example?
An instance counts as successful only when every required repetition succeeds. One failed repetition prevents that instance from counting as resolved.
Should a fail-closed winner always ship?
No. It is only the candidate that best matches the chosen release-compatible ranking key. Thresholds, safety checks, cost limits, coverage, and human review can still reject it.
The selector is part of the instrument. When it and the release rule disagree, the campaign can report a result that its own gate refuses to recognize.