Suppose an agent fails a customer task with a score of 0.42. The team asks whether it used the wrong tool, received bad data, stopped too soon, misunderstood the task, or was judged incorrectly. The dashboard has one number and no way to tell.
That is the failure of score-only learning.
An agent trace is the structured record of one execution: the task, configuration, model calls, tool actions, observations, intermediate artifacts, timing, cost, errors, checks, and outcome. A span is one timed operation inside that record, such as a model call, tool call, retrieval step, or checker. An artifact is a durable output produced or consumed by the run, such as a patch, screenshot, test report, or retrieved document. An evaluation is the assessment that turns those outputs into quality, policy, or cost evidence. A checker is a repeatable test for an explicit condition on an action, artifact, or result. A judge is a scorer for qualities that a deterministic checker cannot fully measure. An agent runtime is the software that supplies tools, controls turns and budgets, and records execution. An agent profile is the versioned model, instructions, skills, tools, permissions, runtime, and limits used for a run. A baseline is the version already in use, and a candidate is a proposed replacement. A holdout set is a protected group of tasks kept out of candidate search until the release decision. A promotion rule is the explicit rule that decides whether a candidate may replace the baseline.
A score tells you how the run ended. A trace exposes tool arguments, observations, retries, and checks that an optimizer can change.
A score is a lossy projection
An agent trajectory can be written as:
tau = (task, state_0, action_1, observation_1, ..., action_T, observation_T, result)The evaluator compresses that trajectory into a score:
score = evaluate(tau)A single score supports scalar aggregation, ranking, and trend charts across runs. It is poor evidence for diagnosis when the cause of failure lives in the discarded steps.
Suppose two runs both receive 0.42.
One chose a correct tool and received an empty response.
The other received good data, called the wrong tool argument, and never recovered.
The score is identical while the fixes are unrelated.
The trace should preserve the variables that can identify responsibility:
which profile and model ran
which prompt and skill versions loaded
which tools were available
which tool arguments were sent
which observations returned
which branches and retries happened
which artifacts changed
which checks ran
which budget was spent
which evaluator produced the resultThe trace does not need every token to be useful. It needs enough structure to answer, “What would have changed the outcome if this step had been different?”
The smallest useful trace
A trace can start as a small structured document. The schema can grow later, but its joins should be decided early. The values in this illustrative shape are placeholders; a production trace must carry the identifiers and measurements from its own run.
{
"runId": "run_0187",
"scenarioId": "refund-delivered-order",
"profile": {
"model": "example-model",
"promptVersion": "refund-v3",
"skillVersion": "refund-review-v2",
"toolsetVersion": "orders-v5",
"runtimeVersion": "runtime-v4"
},
"spans": [
{
"id": "span_01",
"kind": "model",
"startedAt": "2026-08-03T16:47:02Z",
"durationMs": 812,
"inputTokens": 1430,
"outputTokens": 188
},
{
"id": "span_02",
"parentId": "span_01",
"kind": "tool",
"name": "orders.lookup",
"argumentsDigest": "sha256:...",
"status": "ok",
"observationDigest": "sha256:..."
}
],
"artifacts": [
{ "id": "artifact_01", "kind": "answer", "digest": "sha256:..." }
],
"outcome": {
"score": 0.42,
"failureClass": "missing-order-status-check",
"hardChecks": { "schema": true, "policy": false }
},
"budget": {
"modelCalls": 1,
"toolCalls": 1,
"wallClockMs": 1044,
"usd": 0.018
}
}| Record | Question it answers | Minimum evidence |
|---|---|---|
| Run identity | Which candidate and environment produced this result? | Run, scenario, profile, model, and split identifiers |
| Span tree | What sequence of decisions and actions occurred? | Parent-child links, operation kind, status, and timing |
| Artifact link | What did the run produce or consume? | Stable artifact id, type, and digest |
| Outcome | Why did the run pass or fail? | Score, hard checks, failure class, and evaluator version |
| Budget ledger | What did the run cost? | Calls, tokens, tools, wall time, and monetary estimate |
The example stores digests rather than sensitive bodies in the top-level record. The full bodies can live in access-controlled artifact storage when a reviewer needs them. The join keys let an analyst move from the score to the failed tool span, from the span to the observation, and from the observation to the returned artifact.
Run identity comes first
Every span is hard to use if nobody can tell which run, task, candidate, or environment produced it. Keep a stable identity for:
run
scenario
candidate
baseline
dataset and split
model and provider
agent profile
prompt and skill versions
tool set
runtime and code version
seed or replicate
parent runFor each run, the profile should include model choice, instructions, skills, tools, permissions, budgets, and relevant execution settings. Changing one of those fields should create a new profile identity or a recorded versioned override.
This prevents a common measurement error. The team thinks it is comparing two prompts, but the candidate also received a different model, a different tool schema, and twice the turn limit. The trace can make that asymmetry visible only if those fields are recorded.
Span trees preserve the mechanism
One flat log line cannot show the shape of agent work. A span tree can:
agent run
├── planner model call
│ ├── retrieval call
│ └── tool call
├── worker run
│ ├── model call
│ └── sandbox execution
└── checkerThe tree answers questions a final answer cannot. Did the checker run after the patch or before it? Did the worker see the planner’s observation? Did the retry repeat the same tool argument? Did the model call that produced the final answer happen after the test failed?
The public ReAct paper made the reasoning-and-action trajectory central to an agent’s interaction with an environment. The public Reflexion paper shows how feedback can be converted into language that influences later trials. Those systems rely on a record of actions and observations because a final answer alone cannot carry the same learning signal.
Raw provider evidence and structured spans serve different jobs
A structured model span says what the instrumentation believes happened. Raw provider capture preserves the request and response that crossed the provider boundary.
Both matter because the structured layer can be wrong or incomplete. A retry may replace the first response. A stream parser may drop usage fields. A proxy may report one model name while the backend uses another. A custom client may bypass the instrumented path.
For model calls that affect an evaluation, capture enough raw evidence to audit:
provider and endpoint
model name
attempt number
request and response metadata
status code
duration
token usage
retry reason
redacted fieldsDo not store secrets merely because raw capture is valuable. Hash, redact, encrypt, or omit sensitive bodies according to the data policy, and record that a field was redacted. An empty field and a redacted field imply different diagnoses.
Replay turns an old run into a new experiment
When provider responses and artifacts are captured safely, a replay can inspect the run again without making fresh model calls. That enables:
- judge comparison on identical outputs;
- rubric calibration without sampling noise;
- regression checks after changing the evaluator;
- failure analysis without spending another budget;
- determinism checks for parsers and selection rules.
Replay needs an explicit miss policy. For a promotion test, a missing captured response should fail closed rather than silently call the network. Otherwise the “replay” is a new experiment with an unknown sample.
The replay boundary also protects the evaluation split. Runtime code may use tool results, compiler errors, and production-visible checks. It must not receive private holdout labels, answer keys, or post-hoc judge rationales that the product would not have at deployment.
Trace integrity is an input to the release rule
A trace can be present and still be incomplete. Before using it for promotion, check the expected evidence:
run identity exists
expected model spans exist
expected tool spans exist when tools were required
artifact references resolve
budget totals reconcile
raw provider evidence covers scored model spans
outcome and failure class existIf a run expected a tool call and the trace contains none, do not treat the absence as a successful no-op. Treat it as a capture or execution failure. If the structured span exists but raw provider evidence is missing, quarantine the run for any claim that depends on provider behavior.
Backend integrity is another check. Zero tokens, zero cost, or a scripted response can mean the run used a stub. A stub can be useful for testing the evaluator’s wiring. It is not evidence that the agent succeeded or failed against a real model.
OpenTelemetry gives the transport shape
OpenTelemetry supplies common interfaces, conventions, and transports for traces, metrics, logs, and events. The OpenTelemetry GenAI semantic-conventions repository defines conventions for generative-AI clients, agents, tools, providers, and evaluation-related signals.
Those conventions help agent traces join ordinary service traces and use existing collectors, storage, dashboards, and retention controls. They do not define every field an improvement loop needs. An agent system still needs product-specific attributes such as candidate identity, task split, skill version, failure class, artifact digest, budget ledger, promotion decision, and analyst evidence.
The practical design is:
OpenTelemetry-compatible spans and events
+ agent-specific identity and evaluation fields
+ integrity checks before promotion
+ access-controlled artifacts and replayKeep the mapping to the current convention version in one place. The GenAI conventions are an evolving public project, so a trace reader should record its schema version instead of assuming field names never change.
Tangle’s public trace boundary
The public Tangle agent-runtime repository describes a runtime that records runs for chat turns, one-shot tasks, teams, bounded rounds, and improvement experiments. The public agent-eval repository describes paired case evaluation, analysis of existing runs and traces, and prompt or skill optimization without exposing final test cases to the optimizer.
The public TypeScript packages document this installation path:
pnpm add @tangle-network/agent-runtime @tangle-network/agent-eval @tangle-network/sandboxIn this setting, the runtime emits behavior and the evaluation layer preserves and analyzes it. An evaluation result is not a trace, and a trace is not a promotion decision. Each layer has a separate job. The runtime execution article covers the execution shapes whose branches and cancellations should appear in these records.
When the agent is deployed through Tangle’s service model, a Blueprint is the reusable template for the service and an operator is the party running a live instance from that template. An x402 payment can authorize a paid web job before it enters the runner. A router can dispatch that job to its registered handler. A signed attestation can bind an execution measurement to the recorded runtime and policy. Join those records to the trace with stable run, span, artifact, and payment identifiers; they still do not prove that the agent’s answer was correct. The product still needs a result-level check and an evaluation that tests user-visible behavior.
Findings must point back to evidence
An analyst can summarize a trace, but the summary should not replace it. A useful finding includes:
finding id
failure class
claim
confidence
span or artifact references
recommended change
validation planFor the refund example, a finding might say:
claim: the agent applied policy before checking order status
evidence: span_02 returned delivery_state = delivered after the policy span
recommended change: require order lookup before policy selection
validation: compare the candidate on delivered, pending, and unknown-status casesThe evidence references make the finding falsifiable. Another engineer can inspect the same span, disagree with the diagnosis, and propose a different candidate.
Privacy is part of trace design
Agent traces can contain credentials, customer records, private source code, browser state, retrieved documents, and model prompts. The trace needs enough detail to identify mechanism without becoming an unrestricted copy of every secret the agent could see.
Capture-time protections should cover obvious credentials such as authorization headers, cookies, passwords, service keys, and access tokens. Tenant, retention, and access policies should apply to raw bodies and artifacts as well as indexes and dashboards. Redaction should leave a typed marker or audit field so a reviewer can tell whether a value was missing, omitted, or intentionally hidden.
Over-redaction can erase the cause of a failure. Under-redaction can turn the evaluation store into a data-exposure system. The correct boundary depends on the product, but the decision must be explicit.
What traces do not prove
A complete trace does not prove that the agent was correct. It proves that the recorded path is inspectable.
An attestation does not prove output quality. It can support a claim about the execution environment and policy.
An x402 receipt does not prove the job result. It proves that the payment path accepted or settled according to its rules.
A trace does not prove that an evaluator is calibrated. It gives the evaluator something concrete to score.
OpenTelemetry does not prove that every important span was captured. It gives systems a common way to carry the spans that were emitted.
The promotion rule must combine capture integrity with deterministic checks, judge calibration, cost accounting, and product outcomes.
When a trace lies by omission
Score-only learning leaves the optimizer with no mechanism to repair.
Summary collapse replaces tool arguments, branches, artifacts, and errors with a fluent paragraph.
Orphan spans record a model call without the provider evidence needed to audit it.
Backend blindness treats a stub, auth failure, or partial provider response as a model result.
Trace amnesia keeps the final answer while dropping failed branches and retries.
Judge leakage feeds private evaluator information into runtime policy.
Artifact loss leaves a trace pointing at a patch, screenshot, or report that no longer exists.
Identity drift makes run records, metrics, traces, and findings impossible to join.
Redaction erasure hides the fact that a sensitive field was removed and makes missing data look like absent behavior.
The fix is not “capture everything forever.” The fix is to define the diagnosis questions, capture the fields that answer them, and preserve evidence with the right retention and access controls.
Build the trace before the next optimizer
Start with one failure the team can reproduce. Write down the decision the next engineer needs to make. Add run identity, profile, model spans, tool spans, observations, artifacts, hard checks, budget, outcome, and failure class. Then replay the run and confirm that the evaluator and analyst can reach the same evidence.
After that, add candidate comparison and promotion rules. Only then is it worth asking an optimizer to turn findings into prompt, skill, runtime, or model changes.
The trace is not the product and it is not a correctness certificate. It is the evidence boundary that lets a self-improving system learn from a real mechanism rather than from a number it cannot explain.
Sources
- Tangle agent-runtime
- Tangle agent-eval
- OpenTelemetry Semantic Conventions
- OpenTelemetry GenAI Semantic Conventions
- OpenTelemetry for Generative AI
- ReAct: Synergizing Reasoning and Acting in Language Models
- Reflexion: Language Agents with Verbal Reinforcement Learning
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
- GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
What is an agent trace?
An agent trace is a structured record of one run, including the task, profile, model calls, tool actions, observations, artifacts, timing, cost, checks, errors, and outcome. It preserves the path needed to diagnose why the result passed or failed.
Why are traces more useful than scores for self-improvement?
A score is a compressed outcome. A trace can show the wrong tool argument, stale observation, skipped checker, repeated retry, budget breach, or evaluator mismatch that caused that outcome.
Should traces include full prompts and model outputs?
Capture enough content to reproduce and diagnose the decision, but apply redaction, encryption, access control, tenant separation, and retention rules to sensitive data. Digests and controlled artifact storage can preserve joins without putting every secret in every dashboard.
How do traces connect to evaluation rules?
The release rule uses traces to check that the backend was real, expected actions happened, costs reconcile, and failures are diagnosable. The evaluation-rules article covers the separate decision about whether a candidate may replace the baseline. Once those traces feed a release decision, test-time compute covers how to compare extra work at equal cost.