A coding agent edits the right file, passes the visible test, and still breaks the release workflow. Its judge gives the patch a high score because the explanation is clear. The team promotes it, then discovers that the candidate never ran the required integration check.
An evaluation reported a good-looking output. An evaluation gate should have refused the release. Evaluation gates are the engineering rule that decides whether the candidate may replace the baseline.
An evaluation is a repeatable test of an agent on a defined case. A baseline is the version already trusted for comparison. A candidate is a proposed replacement. A holdout set is a protected set of cases that the improvement process cannot inspect while searching. A trace is the run record that contains the task, model turns, tool calls, artifacts, checks, costs, and outcome. A judge is a scorer for qualities that a deterministic test cannot fully capture. A validator is a deterministic or policy check that can block a release when an artifact is invalid. A harness is the software around the model that provides tools, state, checks, and trace capture. A promotion gate is the fixed rule that turns all of that evidence into promote, reject, hold, or human-review.
The short answer to “what are evaluation gates?” is this: they decide whether a candidate is better enough, safe enough, and evidenced enough to replace the baseline.
The gate is not the last administrative step. It is the boundary that defines what the optimizer is allowed to learn.
A leaderboard is not a release rule
A leaderboard asks which system received the highest score. A release gate asks whether this candidate should replace this baseline for this product and scope.
Those questions need different evidence. A leaderboard can hide cost, latency, missing traces, task mix, tool availability, and severe failures behind one rank. A gate must preserve the baseline when any of those facts make the candidate unsuitable.
| Evidence | What it protects |
|---|---|
| Paired cases | The two systems saw comparable work |
| Holdout cases | Search did not tune directly against promotion data |
| Deterministic checks | Tests, schemas, permissions, and required artifacts are real |
| Semantic judgment | Open-ended quality is scored where exact checks are insufficient |
| Cost and latency | Quality gains fit the user and infrastructure budget |
| Trace integrity | The result can be explained and replayed |
| Protected slices | High-value personas and failure classes are not averaged away |
The gate should be written before the candidate is scored. Changing the threshold, judge, task mix, or cost rule after seeing the result starts a new experiment.
How evaluation gates compare candidates
Suppose the baseline and candidate answer five identical support questions. Their paired scores are an illustrative example, not a production result:
| Case | Baseline | Candidate | Difference |
|---|---|---|---|
| Refund eligibility | 0.60 | 0.80 | +0.20 |
| Shipping exception | 0.80 | 0.80 | 0.00 |
| Account deletion | 0.80 | 0.60 | -0.20 |
| Duplicate charge | 0.40 | 0.80 | +0.40 |
| Escalation request | 0.60 | 0.80 | +0.20 |
The candidate’s average lift in this toy example is +0.12. That is useful evidence, but it is not enough to ship. The candidate regressed the account-deletion case, and five pairs cannot support much confidence.
The paired quantity is:
δᵢ = score(candidate, caseᵢ) - score(baseline, caseᵢ)The gate can inspect the median or mean of the deltas, the distribution by failure class, and the number of protected cases that moved in the wrong direction. Pairing removes one major confound: the candidate cannot win merely because it received easier questions.
Use the same agent profile for both sides except for the surface under test. An agent profile is the recorded bundle of model, prompt, tools, permissions, skills, runtime settings, and evaluator configuration that determines the run. If the candidate also changes its model or tool access, record that as a multi-surface experiment rather than calling it a prompt improvement.
Holdout is a firewall, not a second score
The search set helps generate a candidate. The holdout set decides whether the candidate generalizes.
search = cases visible to mutation, debugging, and iteration
holdout = cases reserved for final comparisonThe candidate should not receive holdout prompts, labels, reference answers, judge rationales, or traces that reveal the protected cases. The optimizer should also be unable to edit the gate, the holdout manifest, or the judge used for promotion.
A search score that rises while holdout score falls is a classic overfit signal. A candidate can also overfit more subtly by learning a particular judge’s phrasing, a benchmark’s file layout, or a fixed order of cases. Keep case identifiers, evaluator versions, and split membership in the run record so a later audit can reconstruct what the optimizer could see.
The HELM evaluation framework from Stanford evaluates language models across scenarios and metrics, showing why an agent evaluation should expose more than one headline number. Its structure encourages readers to ask which scenario, adaptation, and metric produced a result. Agent gates need the same discipline plus runtime context, tool calls, and artifact checks.
Deterministic checks outrank model judges
A model judge can assess whether a response is clear, complete, or aligned with a user’s intent. It should not override a failed test, an invalid schema, a missing artifact, a permission violation, or evidence that the backend never ran.
Use an order like this:
required artifact and schema checks
security and permission checks
real backend and trace checks
cost and latency policy
calibrated semantic judge
aggregate quality scoreThe exact order can vary by product, but the principle is stable. A soft opinion cannot rescue hard evidence that the task was not completed.
OpenAI’s public write-up about its in-house data agent shows the pattern clearly. The team evaluates generated SQL by executing it and comparing the resulting data with a golden query, then uses a grader for acceptable variation in cases where string equality is too strict. The article is useful because the grader supplements executable evidence rather than replacing it.
Judges are measurement instruments
Open-ended tasks need semantic evaluation. Exact checks cannot fully answer whether a research memo addressed the user’s intent or whether a coding explanation omitted a critical limitation.
A large language model (LLM) judge can help, but it is not a source of truth. The G-Eval paper describes model-based evaluation with structured criteria. G-Eval is a model-judge method, not a release policy. The MT-Bench and Chatbot Arena paper documents useful agreement with human preference while also identifying position, verbosity, and self-enhancement biases.
Treat the judge as software that needs calibration. Maintain a small human-reviewed set of examples, include clear pass and fail anchors, and test whether the judge orders pairs the way trusted reviewers do. Track the judge model, rubric, prompt, examples, and version in the agent profile.
If a judge changes, a score change may be evaluator drift rather than agent improvement. Run the baseline through the new judge too. If the baseline moves, the comparison is a new measurement regime.
Confidence needs a denominator
A gate should report more than “the candidate won.” At minimum, report:
- Number of paired cases and repeated runs.
- Quality lift with its uncertainty interval.
- Protected slices and their individual outcomes.
- Deterministic failures and missing evidence.
- Input and output tokens, model calls, tool calls, wall time, and dollars.
- Judge identity and calibration status.
- Backend identity and whether the run produced real provider activity.
- Trace completeness and replay status.
- The exact decision and the person or service allowed to override it.
An uncertainty interval is a range that describes how much the measured lift could vary across comparable samples. For a bootstrap interval, resample the paired deltas with replacement, calculate the statistic for each resample, and use the lower percentile as the conservative bound. The technique estimates sampling uncertainty. It does not repair a biased task set, a poisoned judge, or a broken backend.
The gate can express a conservative rule:
promote(candidate) only if
lower_bound(lift_on_holdout) > minimum_lift
and deterministic_failures = 0
and trace_integrity = pass
and cost <= cost_limit
and protected_slices = pass“Lower bound” means the pessimistic edge of the interval, not the average result. “Minimum lift” means the smallest improvement worth the operational change. Both values should be set before the candidate is inspected.
Cost is part of correctness
A candidate that gains quality by doubling calls may be the right product choice. It is not the same product as the baseline.
Record cost as a vector:
| Cost field | Why it matters |
|---|---|
| Dollars | Direct operating budget |
| Input and output tokens | Model consumption and context growth |
| Model calls | Retry and fanout intensity |
| Tool calls | External load and side-effect opportunity |
| Wall time | User experience and queue capacity |
| Sandbox time | Compute allocation |
| Human review time | Operational burden |
The runtime topology article explains why a request for more workers changes runtime budget and control. The test-time compute article shows how to compare those workflows against simple equal-budget baselines.
Rejecting a candidate for cost is not a failed experiment. It tells the team that the behavior is promising outside the current product envelope. The team can then choose whether to optimize efficiency, raise the price, narrow the task, or keep the baseline.
Backend integrity is a release condition
A test row can exist even when the model call failed, a provider returned a stub, authentication was rejected, or a local fixture produced canned output. If that row is scored as agent behavior, the gate learns the wrong lesson.
The run record should prove:
which backend answered
which model or route was used
whether provider usage was recorded
whether tools and sandboxes actually ran
whether the returned artifact came from that executionMixed real and stub rows need quarantine or explicit exclusion. An authentication failure is a harness failure until the system can prove that the agent received a valid task execution. Failing closed here protects both the candidate and the baseline from false conclusions.
A public TypeScript evaluation shape
The public Tangle agent-eval repository includes an offline example built around defineAgentEval. The following shortened shape keeps the same idea while using a tiny deterministic judge:
import { defineAgentEval } from '@tangle-network/agent-eval/contract'
const evalKit = defineAgentEval({
scenarios: [
{ id: 'refund', question: 'Can this ticket be refunded?' },
{ id: 'shipping', question: 'Where is this ticket?' },
],
agent: async (surface, scenario) =>
surface.includes('ticket-id') ? scenario.id + ': checked' : 'checked',
judge: {
name: 'ticket-id-present',
dimensions: [{ key: 'present', description: 'The answer names the case id.' }],
score: ({ artifact, scenario }) => {
const present = artifact.includes(scenario.id) ? 1 : 0
return { dimensions: { present }, composite: present, notes: '' }
},
},
baselineSurface: 'Answer politely.',
expectUsage: 'off',
})
await evalKit.evaluate()
await evalKit.evaluate({ surface: 'Answer politely and include the ticket-id.' })This is an illustrative adaptation of the repository’s public offline example. It proves the mechanics of comparing two surfaces on the same cases. It does not prove that a production judge is calibrated or that a support policy is correct.
Failure taxonomies make the next experiment possible
A scalar score says which version won. A failure taxonomy says what to change next.
Useful classes include:
| Failure class | Candidate surface it may implicate |
|---|---|
| Wrong tool choice | Prompt, skill, tool description, or runtime |
| Invalid tool arguments | Schema, validation, or model behavior |
| Missing source | Retrieval or memory |
| Permission denied | Profile or runtime policy |
| Deterministic test failure | Candidate artifact or harness |
| Unsupported claim | Evidence policy, retrieval, or judge |
| Budget exceeded | Topology, retry, model, or cost policy |
| Ambiguous request | Intake or clarification behavior |
| Missing trace | Harness instrumentation or backend |
The taxonomy should come from observed traces rather than a list invented after the result. The trace systems article explains how a complete trajectory gives the analyst enough detail to attribute a failure.
Tangle’s evaluation boundary
Tangle’s public agent-runtime repository describes execution loops that run workers, validators, and supervisors. The public agent-eval repository describes paired comparisons, score distributions, prompt or skill optimization, and candidate generators. The useful division is observable in the public interfaces: runtime code executes behavior, while evaluation code measures it against cases that the optimizer should not control.
If that agent is deployed as a Tangle service, define the deployment terms separately. A Blueprint is a template for software that can run as a service. A Service is a live instance created from that Blueprint. An operator runs a live Service from that template. A Router directs a job call to its handler. An x402 payment is an optional HTTP ingress path that authorizes and settles payment before a job is enqueued. An attestation is signed evidence about a trusted execution environment’s loaded code and hardware state.
None of those terms means “the answer was correct.” The trace and evaluation gate still need to verify the task result. The Tangle x402 documentation states that a paid request being accepted and enqueued is not the same as the job having completed. That distinction is exactly the one a release gate needs.
What a gate should do when evidence is missing
Missing evidence is not a neutral score. If the candidate has no holdout runs, no real backend proof, incomplete traces, or no rollback path, return hold or reject.
Use human review for a decision that truly requires judgment, such as a rare high-severity case or a new risk class. Do not use human review to make missing instrumentation disappear. The approval packet should contain the candidate identity, changed surface, paired results, protected failures, cost, trace links, and rollback action.
If the workflow is deterministic and changes are infrequent, a normal regression suite and code review may be simpler than an adaptive promotion process. Governance still belongs outside any optimized surface. The governance article explains why a candidate must not rewrite the judge or gate that evaluates it.
What is an evaluation gate?
It is a fixed release rule that compares a candidate with a baseline on protected cases and checks quality, cost, deterministic failures, trace integrity, and risk before allowing promotion.
Is an LLM judge enough?
No. Use a calibrated judge for open-ended qualities, but let executable checks, policy checks, backend evidence, and trace integrity block a release first.
How large should a holdout set be?
Large enough to cover the product’s important task and failure slices with tolerable uncertainty. There is no universal number. Report the denominator, repeated-run plan, effect-size threshold, and what the holdout does not cover.
Why not promote the highest average?
An average can hide a severe regression, unequal costs, task-mix differences, judge drift, or missing execution evidence. Promotion is a product decision, not a ranking exercise.
What should a first gate contain?
Start with one baseline, one candidate surface, a protected case set, a deterministic check, a semantic score only where necessary, explicit cost fields, and a fail-closed decision when the record is incomplete.