BlogThe Self-Improving Stackpart 3 of 3

Harness Evolution for AI Agents: When Prompts Plateau

Harness evolution changes the software around an agent when prompts and skills cannot add tools, isolation, checks, traces, or rollback.

The agent keeps saying it inspected the repository before proposing a change. The run record shows no file reads. The prompt says “run the tests,” but the runtime has no shell or test adapter.

Adding another instruction will not create either capability. This is the moment to change the harness.

The harness is the software around the model that starts work, exposes tools, manages state and budgets, captures artifacts, runs checks, and decides what can continue. A trace is the record of the whole execution, including the final answer and the steps that produced it. A validator is a deterministic or semantic check that turns an artifact into evidence. A candidate is a proposed harness or profile change. A runtime is the software that starts model and tool work, manages limits, and records events. An agent profile is the versioned bundle of model, prompt, tools, permissions, skills, runtime settings, and evaluator identity for a run. An evaluation is a repeatable test that compares the candidate with a baseline on defined cases. A baseline is the version already trusted for comparison. A judge is a scorer for qualities that a deterministic validator cannot fully capture. A holdout set is a protected group of cases kept outside candidate search. A sandbox is an isolated execution boundary that limits files, networks, credentials, or processes. A mutable surface is the part of the system a candidate is allowed to change.

Harness evolution changes tools, workers, validators, traces, isolation, or release controls around the model. Because those changes affect execution, evidence capture, permissions, and rollback, they need broader tests than a prompt or skill edit.

On Tangle, a candidate harness runs as an agent profile through providerAsSandboxClient on a Tangle sandbox. Each run reports its cost on the done event (totalCostUsd), a check’s exit code from box.exec, and the trace the runtime records.

Find the missing action

Use the trace to identify what the current system could not do.

Trace evidence Missing capability Likely repair
The model describes a file read that never happened Repository access or tool dispatch Tool adapter and observable call
A requested second worker never appears Worker creation or topology Runtime driver
A failed test is mentioned but not run Execution or validator Test adapter and hard check
A branch changes shared files Isolation Sandbox or per-candidate workspace
A score cannot be traced to an artifact Capture or lineage Trace schema and artifact store
A candidate changes its own judge Outer control boundary Gate outside the mutable surface

The table is a diagnosis aid, not a reason to build a larger framework. If the agent already has the needed action and forgets to use it, a prompt or skill change may be enough. Evolve the harness when the missing move is not representable, not enforceable, or not observable.

Anthropic’s public agent architecture guidance recommends starting with the simplest composable system that fits the task. Cognition’s public account of using Devin to build Devin describes recurring playbooks, code review, automated fixes, and session insights as explicit workflow surfaces. The transferable lesson is not a particular vendor architecture. It is to turn repeated operational behavior into a named mechanism with a check.

The harness defines the reachable behavior

Let h be the current harness and M be the set of mutations it can express. The system can search only the behaviors reachable through M. If M edits prompt text, it cannot produce a new worker process, an isolated workspace, or a raw provider capture sink.

Surface Search can change Search cannot create by itself
Prompt Instructions, examples, tool descriptions A tool, process, permission, or durable record
Skill Reusable procedure and checks An execution engine that invokes the procedure
Runtime topology Worker order, fanout, retry, handoff, budget New evaluator or storage boundary unless exposed
Harness Tools, adapters, traces, validators, isolation, lifecycle A trustworthy release decision if it owns the gate too

Harness evolution begins when the runtime itself is the object that needs to change.

Structural change is different from knob tuning

Changing a concurrency limit or retry count can be useful. It is ordinary configuration search unless the change introduces a new mechanism.

A structural candidate might:

  • Replace a summary-only record with child spans and artifact lineage.
  • Replace a single retry with isolated branches and an explicit winner rule.
  • Add a deterministic validator before a semantic judge.
  • Add a replay adapter that can rerun a failed tool step.
  • Move candidate generation into an isolated workspace.
  • Attach a stable agent profile to every evaluation cell.
  • Add a cancellation path that reaches child workers.

A non-structural candidate might:

  • Add “be rigorous” to a prompt.
  • Rename a worker from reviewer to supervisor.
  • Raise the judge threshold after seeing a result.
  • Increase population size without changing the search mechanism.
  • Drop trace capture to make the candidate faster.

The difference is whether the agent can now perform, prove, or stop an action it could not handle before.

A candidate lifecycle that can be rolled back

Harness code should be treated as a release artifact. Each candidate needs an identity, a parent, a hypothesis, an isolated workspace, and a result.

freeze baseline
create isolated candidate workspace
apply one structural hypothesis
run a cheap smoke case
run paired development cases
inspect trace and artifact capture
run protected cases outside the search
promote, reject, or quarantine
retain rollback handle and lineage

The smoke case is a cost control. It should catch a missing dependency, broken adapter, malformed schema, or absent trace before the expensive campaign starts. The protected run is a control boundary. It should not be writable by the candidate. The rollback handle is an operational requirement.

Parallel candidate generation requires isolation. Two variants that share a mutable directory can overwrite each other, pass because one consumed the other’s artifacts, or leave a mixed result whose parent is impossible to identify. A candidate record should include:

Field Purpose
Parent identity Shows which mechanism the candidate inherited
Changed surfaces Prevents an unplanned profile change
Hypothesis Makes the result falsifiable
Smoke result Proves the candidate can run
Trace coverage Shows what the candidate did
Quality and cost Supports comparison
Promotion decision Records why it advanced or stopped
Rollback handle Makes activation reversible

Keep the evaluator outside the candidate change

Architecture search can attack its own measurement because the candidate can change the path being scored. A harness candidate can improve its score by:

  • Dropping hard cases from the adapter.
  • Retrying until a lucky answer appears.
  • Routing around a failing check.
  • Preferring judge-friendly prose over a correct artifact.
  • Skipping trace capture to reduce reported latency.
  • Reading protected examples through logs or error messages.
  • Editing the evaluator or release threshold.

The outer rule is:

candidate code can produce behavior
candidate code cannot rewrite the gate that promotes it

If the product needs to evolve the evaluator, that change must be a separate candidate with its own test set, owner, and review. Otherwise the same system can lower the bar and declare itself improved.

In CI, the regression gate applies this rule. The candidate prompt arrives in CANDIDATE_PROMPT; the scenarios, judge, and shipped baseline are fixed in the file, and the build fails when the candidate scores lower.

src/examples/evals/regression-gate.mtsRan on production 2026-11-02 from these bytes, exit 0, $0.1818 reported usage, @tangle-network/sandbox 1.0.168. View result.
import { Sandbox } from "@tangle-network/sandbox";
import { defineAgentEval, type Scenario } from "@tangle-network/agent-eval/contract";

const client = new Sandbox({
  apiKey: process.env.TANGLE_API_KEY!,
  baseUrl: "https://sandbox.tangle.tools",
});

// Run in CI: the build fails when the new prompt scores below the shipped one.
const shipped = "Fix the cause.";
const candidate =
  process.env.CANDIDATE_PROMPT ?? "Fix the cause. Change only what the test needs.";

const evalKit = defineAgentEval<Scenario & { task: string }, number>({
  scenarios: [
    { id: "cli-flag", kind: "fix", task: "The test tagged cli-flag fails." },
    { id: "null-guard", kind: "fix", task: "The test tagged null-guard fails." },
  ],
  baselineSurface: shipped,
  agent: async (prompt, scenario) => {
    const box = await client.create({
      git: { url: "https://github.com/tangle-network/sandbox-demo" },
      backend: { type: "claude-code", model: { model: "glm-5.3" } },
    });
    try {
      await box.prompt(`${scenario.task} ${String(prompt)}`);
      return (await box.exec(`npm test -- --test-name-pattern=${scenario.id}`)).exitCode;
    } finally {
      await box.delete();
    }
  },
  judge: {
    name: "suite",
    dimensions: [{ key: "pass", description: "the tagged test passes after the turn" }],
    score: ({ artifact }) => {
      const pass = artifact === 0 ? 1 : 0;
      return { dimensions: { pass }, composite: pass, notes: "" };
    },
  },
  expectUsage: "off",
});

const score = async (surface: string) =>
  (await evalKit.evaluate({ surface })).aggregates.byJudge.suite?.mean ?? 0;
const before = await score(shipped);
const after = await score(candidate);
console.log({ shipped: before, candidate: after });
if (after < before) {
  console.error("Regression: the candidate passes fewer cases.");
  process.exit(1);
}

The output is two means and an exit code. It proves that the candidate does not score lower on these two cases under one judge. It does not prove production gain; two cases are a smoke test, not a holdout set.

What the public Tangle runtime shows

The public Tangle agent-runtime repository describes one runtime used for chat turns, one-shot tasks, teams, and agent improvement. Its README separates the execution loop from the scoring and shipping decision provided by agent-eval and from sandboxed execution. providerAsSandboxClient sets the harness in the default profile. box.streamPrompt yields result and done events, and done carries totalCostUsd.

src/examples/runtime/any-cloud.mtsRan on production 2026-11-12 from these bytes, exit 0, $0.0740 reported usage, @tangle-network/sandbox 1.0.168. View result.
import { createTangleProvider } from "@tangle-network/agent-provider-tangle";
import { providerAsSandboxClient } from "@tangle-network/agent-runtime/kernel";
import { Sandbox } from "@tangle-network/sandbox";

// Any AgentEnvironmentProvider fits here; this one runs on Tangle.
const provider = createTangleProvider({
  client: new Sandbox({
    apiKey: process.env.TANGLE_API_KEY!,
    baseUrl: "https://sandbox.tangle.tools",
  }),
});

// The harness is a profile field: "claude-code", "codex", "opencode", "pi".
const client = providerAsSandboxClient(provider, {
  defaults: { profile: { name: "repo-reader", harness: "claude-code" } },
});

const box = await client.create({
  git: { url: "https://github.com/tangle-network/sandbox-demo" },
});
try {
  const task = "Read this repository. In two sentences, say what it does.";
  for await (const event of box.streamPrompt(task)) {
    if (event.type === "result") console.log(event.data.finalText);
    if (event.type === "done") console.log(`$${Number(event.data.totalCostUsd).toFixed(4)}`);
  }
} finally {
  await box.delete();
}

The output is the final answer and one cost line: the harness is a profile field, and the run is metered. It does not show a changed harness scores better; the regression gate above does.

The public README also exposes a simple improvement shape. The following is an illustration, not runnable code; baseProfile, executionRef, method, findings, the scenario sets, judge, and runProfile are application-provided.

import { improve } from '@tangle-network/agent-runtime'

const result = await improve(baseProfile, {
  surface: 'prompt',
  executionRef,
  method,
  findings,
  trainScenarios,
  selectionScenarios,
  testScenarios,
  judges: [judge],
  agent: (candidateProfile, scenario, ctx) =>
    runProfile(candidateProfile, scenario, ctx),
  runDir: 'support-prompt',
  costCeiling: 25,
})

if (result.decision === 'ship') {
  // A ship decision is evidence, not approval.
  await requestApproval(result.candidate.profile)
}

This snippet follows the repository’s public README shape. It is not a product integration: the application supplies the profile, method, scenarios, judge, agent adapter, and approval path. Improvement is a measured candidate operation, so the live agent does not rewrite itself in place. Search and activation stay detached: a ship decision says the candidate beat the baseline on held-out cases, and the application approves it before a separate step makes it live.

When the changed surface is harness code, run the code in an isolated workspace, record the exact change, execute the real backend, and evaluate the resulting behavior against the frozen baseline.

The research lineage

The Gödel machine paper describes a self-referential solver that rewrites its own code after proving a utility improvement. Practical systems usually replace proof with empirical evaluation.

FunSearch made the evaluator-driven pattern concrete for LLM-generated program search. AlphaEvolve extends the idea toward broader code and algorithm search with automated feedback. OpenEvolve provides a public implementation to inspect. The Darwin Gödel Machine paper studies an archive of self-improving coding agents that edit and validate descendants.

These systems benefit from a bounded search domain with an executable evaluator. Production agent harnesses are harder because the objective includes user intent, external tools, privacy, permissions, latency, and a release process that cannot be allowed to mutate freely.

A harness variant should make a visible prediction

Do not start with “make the harness smarter.” Start with a testable hypothesis.

Hypothesis Structural change Falsification
Retrieval failures come from missing source spans Capture raw retrieval inputs and selected spans No reduction in missing-source failures
Reviewers miss deterministic failures Run executable checks before semantic review Hard failures remain unchanged or cost exceeds limit
Parallel workers corrupt each other’s work Isolate workspace and state per branch Conflicts persist in isolated runs
The winner rule rewards style Replace it with a deterministic artifact check Quality does not fall when style cues are removed
Retries hide provider failures Record backend status and fail incomplete rows closed False agent failures disappear

The prediction makes the candidate useful even when it loses. A failed hypothesis tells the next search which mechanism did not cause the observed result.

Profiles and traces keep comparisons honest

The profile is the identity of the run conditions. It should capture the model, prompt, tools, skills, permissions, runtime topology, harness revision, and evaluator identity that can affect behavior.

If the candidate changes three of those at once, the score cannot attribute the gain to one mechanism. If the candidate changes a harness but the profile still says “baseline,” the comparison record is incomplete.

The trace should connect the profile to every child branch, tool call, artifact, validator, and final decision. An execution span is one timed operation in that trace. OpenTelemetry’s trace model gives the parent and child span vocabulary. Agent traces need domain fields on top: task identity, profile identity, tool arguments, artifact hashes, evaluator results, and cost.

Keep service deployment controls outside candidate promotion

If the evolved harness runs inside a Tangle Blueprint service, the Blueprint, its Service, the operator, and the Router do not prove that an internal harness variant is safe.

An x402 payment and an attestation (signed evidence about code and hardware in a trusted execution environment) prove payment and code identity, not a correct harness result. The evaluator still needs the trace, deterministic checks, and task evidence.

This is also why a harness candidate should not silently change payment, permissions, or isolation mode. Those are separate risk dimensions that belong in the profile and the promotion packet.

Failure modes after the harness evolves

Evaluator capture happens when the candidate learns the judge or test adapter.

Trace regression happens when the new path is faster because it records less.

Isolation leakage happens when branches share files, credentials, memory, or browser state.

Lineage loss happens when a merged candidate cannot identify which parent or hypothesis produced a change.

Backend confusion happens when provider failures are scored as model failures.

Rollback gap happens when the candidate can be measured but not safely activated or removed.

Complexity debt happens when the new mechanism adds more failure modes than it removes.

Measure the new branch count, latency, operator work, and failure surface. If the simpler baseline performs as well, keep the simpler baseline.

When to evolve the harness

Evolve the harness when the trace shows that the current system cannot:

  • Execute the requested tool or worker.
  • Isolate mutable state.
  • Enforce a validator or budget.
  • Capture enough evidence to diagnose failure.
  • Replay a critical step.
  • Compare candidates on the same profile.
  • Stop unsafe or wasteful child work.

Do not evolve it merely because a prompt feels inelegant.

Every optimizer searches a representation. Prompt search explores strings. Skill search explores reusable procedures. Runtime search explores execution shapes. Harness evolution explores the code that defines actions, observations, checks, records, and candidate lifecycle.

When the current representation cannot express the desired move, widen the representation first. Then freeze the gate, isolate the candidate, capture the trace, and let the mechanism compete against a hard baseline.

What is harness evolution?

Harness evolution changes the software around the model, such as tool adapters, drivers, validators, trace capture, replay, isolation, and candidate release flow.

How do I know a prompt is not enough?

Look for a missing or unenforced action in the trace. If the runtime has no tool, worker, workspace, check, or record needed for the behavior, prompt changes cannot create it.

Does harness evolution mean rewriting the whole system?

No. The best change is often one missing adapter, one hard validator, one trace field, or one isolation boundary. Change the smallest mechanism that can produce and prove the desired behavior.

What should be protected from the candidate?

Protect the holdout cases, evaluator implementation, release policy, credentials, shared state, and activation path. The candidate may propose a change, but an external control process should decide whether it persists.

Run one cheap smoke candidate through the real backend, confirm that traces and artifacts are captured, and verify that the release gate can reject it. Only then spend on a larger population or long campaign.

Primary sources