Optimization Theory for Agent Builders

An agent that answers support questions is missing the same refund rule three days in a row. The team keeps rewriting its system prompt. The wording gets longer, but the agent still reaches for an obsolete policy page.

That is not necessarily a prompt problem. The failure may live in retrieval, memory, the tool description, the runtime loop, or the evaluator that never noticed the regression. That is the kind of failure an optimization framework should diagnose. The framework asks which part of the system can change, which evidence can distinguish a better candidate from a lucky one, and which release rule is allowed to make the change real.

An agent is software that uses a model, tools, and state to pursue a task. The mutable surface is the part of that system the improvement loop is permitted to change. The runtime is the software that executes model turns and tool calls, manages limits, and records what happened. A trace is the record of one run, including inputs, actions, tool results, artifacts, costs, and outcome. An evaluator is the test or person that scores the run. The baseline is the version already in use. The candidate is the proposed replacement. A promotion gate is the rule that decides whether the candidate may replace the baseline. A deterministic check is a repeatable test that turns an artifact or action into evidence. A holdout set is a protected group of cases kept outside candidate search. A sandbox is an isolated execution boundary that limits files, networks, credentials, or processes.

The common loop is short:

candidate → run → measure → compare → keep, reject, or revise

The engineering difficulty sits inside the verbs. If the loop changes the wrong artifact, measures the wrong outcome, or compares unequal work, it can improve its number while making the product worse.

Start with the failure, not the optimizer

Suppose a research agent returns an answer without citing the source that supports its most important claim. There are at least five plausible repairs:

Observed failure	Likely surface to test first
The instruction never asks for a citation	Prompt or skill
The source exists but is not retrieved	Retrieval or memory
The agent cannot open the source	Tool permissions or runtime
The agent retrieves the source but ignores it	Procedure, deterministic check, or model behavior
The judge rewards fluent unsupported answers	Evaluator or promotion gate

The cheapest editable surface is not automatically the causal surface. Prompt text is easy to change, so teams often use it as a universal repair kit. That habit creates prompt maximalism: a growing instruction block that tries to simulate permissions, memory, concurrency, and verification.

Use the smallest surface that can produce the desired behavior. If the agent knows the policy and applies it inconsistently, a prompt or skill experiment is reasonable. If the policy is absent from its context, changing wording cannot supply it. If the agent needs a second worker or an isolated workspace, the runtime must expose that capability. If the candidate can change the evaluator, the outer release process must remain outside the search.

One loop, several search spaces

The same proposal-and-test loop can operate on different objects.

Search surface	Candidate can change	Evidence that matters most
Prompt	Instructions, examples, schemas, or rubrics	Held-out task behavior and instruction-following checks
Skill	A reusable procedure that survives across runs	Transfer to related tasks and quiet behavior on unrelated tasks
Memory	Stored facts, decisions, experiences, or warnings	Retrieval-conditioned task lift, provenance, freshness, and scope
Runtime topology	Order, parallelism, retries, handoffs, budgets, and cancellation	Execution traces, equal-cost comparisons, and failure propagation
Harness code	Tools, adapters, validators, trace capture, candidate isolation, and release flow	Real backend activity, executable checks, replay, and rollback
Model or adapter	Learned parameters or a parameter-efficient attachment	Deployment behavior, data governance, safety tests, and rollback

GEPA and MIPRO are text-search methods for language-shaped surfaces. The GEPA paper describes reflective prompt evolution, while DSPy’s optimizer documentation treats instructions and demonstrations as parts of an LM program that can be compiled against a metric. SkillOpt moves the same idea to a persistent skill document and accepts bounded edits only when validation improves. AlphaEvolve makes executable code the candidate and uses automated evaluators to search algorithm space.

These systems rhyme because they all search. They are not interchangeable because a prompt edit cannot create a sandbox, and a code mutation cannot be judged safely by a stylistic preference score alone.

A small equation that prevents vague claims

Let s be the mutable surface and x be a task. Let τ(s, x) be the trace produced when the system runs surface s on task x. Let Q be the quality score and C be a vector of costs.

The product objective is:

J(s) = E[Q(τ(s, x))] - λᵀ C(τ(s, x))

The notation is less important than the terms it forces into the conversation. Q might include a test result, citation correctness, task completion, or human review. C might include model calls, tokens, wall time, dollars, sandbox minutes, and human review time. λ expresses which costs the product cannot ignore.

If a new topology scores higher only because it uses four times as many model calls, the result may be worth buying, but it is a compute trade rather than a pure quality improvement. If a memory layer increases answer quality but exposes private notes to the wrong tenant, quality is not the only term that changed. If a judge rewards long answers, verbosity has entered Q whether the team intended it or not.

Writing the cost vector down is often the first useful optimization. It exposes the hidden budget that a candidate was spending to win.

Compare paired runs, not lucky averages

The baseline and candidate should see the same task, profile, tool surface, and replicate whenever possible. An agent profile is the versioned bundle that identifies those run conditions, such as model, prompt, tools, permissions, skills, and runtime settings.

For paired task i, calculate:

δᵢ = score(candidate, i) - score(baseline, i)

The list of deltas answers a better question than two unrelated averages. It shows whether the candidate tends to improve the same work the baseline attempted. It also exposes asymmetric failures, such as a candidate that wins easy tasks and loses every high-risk case.

A positive average is only a signal. The release decision should also state the number of pairs, the spread of deltas, the protected cases that regressed, and the uncertainty around the estimate. A bootstrap confidence interval is one practical way to resample the paired deltas and estimate how much the result could move under another draw.

Do not turn a confidence interval into a magic pass number. The minimum useful lift depends on the product. A one-point lift may matter for a high-severity failure class and not matter for a low-value formatting preference. The threshold should be set before the candidate is inspected.

The baseline must be hard enough to beat

An optimizer can look excellent when its comparison point is weak. An eight-branch agent should not be compared with one greedy sample if the product could have spent the same budget on eight simple samples.

For a test-time workflow, compare at least:

Candidate	Fair comparison
One supervisor plus four workers	One worker with the same total call and token budget
A new prompt searched over 40 trials	The old prompt evaluated with the same search budget or a documented human edit budget
A stronger model with fewer retries	The existing model with equivalent spend and latency
A new memory retriever	The same agent with retrieval disabled and the same task set

This is the equal-compute principle. It does not say that extra compute is bad. It says that the system should name whether the gain came from a better policy, a larger search, a stronger model, or a combination.

The test-time compute article develops this comparison in more detail. The evaluation gates article explains how to turn it into a release decision.

Where Tangle changes the boundary

Tangle’s public agent-runtime repository describes a TypeScript runtime for chat turns, one-shot tasks, and teams of agents. Its public README shows a driver reading worker output, composing the next step, and stopping after a validator passes. That is a runtime mechanism, not a prompt convention.

The companion agent-eval repository describes the measurement side. Its public examples compare a baseline surface with a candidate across the same cases and return score distributions. Those repositories are useful boundaries for an optimization design: runtime code executes the candidate, while evaluation code decides whether the measured change earned promotion.

Tangle’s network vocabulary describes a separate deployment boundary. A Blueprint is a reusable service template that defines runnable artifacts, jobs, metadata, and optional protocol logic. A Service is a live instance created from that template. An operator is the independent party that runs the Service and handles its jobs. A Router in the Blueprint Runner directs a job call to the handler registered for its job identifier.

If a customer calls an agent service through x402, x402 is an optional off-chain HTTP payment path that verifies and settles a payment before injecting a job call into the runner. Payment authorization says who paid and whether the request was accepted. It does not say that the agent’s answer was correct.

If a service uses a trusted execution environment, or TEE, an attestation is a signed report about the loaded code and hardware state that a policy check can compare with an expected measurement. Attestation can support an execution-integrity claim. It does not replace a task evaluator, a trace, or a rollback plan. Tangle’s public TEE and x402 explanation makes that separation explicit.

An agent profile still belongs in the optimization record. It tells the evaluator whether a candidate changed only a prompt or also changed the model, tools, permissions, router, runtime, or deployment boundary.

A worked choice

Imagine a paid document-review agent. The agent receives a document, retrieves policy sources, returns a finding, and charges per job. The product has two failures: unsupported claims and unpredictable cost.

The first experiment should not be a multi-agent rewrite. It should freeze the model and tools, add a source-citation check, and compare the current prompt with a candidate prompt on paired held-out documents.

If citations improve but the agent still retrieves irrelevant documents, the next experiment belongs to retrieval or memory. If the agent finds the right source but fails to include it in the output, add a deterministic output check or a bounded skill. If the agent needs several independent sources in parallel, test a real fanout topology with a budget cap. If the result is good but the trace omits retrieval spans, repair the harness before trusting the score.

The following decision record is illustrative; its case counts are placeholders, not a Tangle benchmark:

surface: prompt
baseline: profile-2026-05
candidate: profile-2026-05-citation
paired_cases: 120
quality: citation-supported findings
cost: dollars, tokens, wall time
hard_failures: unsupported high-severity finding
holdout: 40 documents never shown during search
decision: hold, promote, or reject

The point is not the field names. The point is that someone can tell what changed, what was measured, what remained fixed, and why the release decision follows.

Failure modes worth naming

Prompt maximalism hides missing tools, memory, or control flow inside prose.

Metric capture happens when the candidate learns the judge’s preferences instead of the user’s task.

Cost laundering hides extra retries, stronger models, or human review inside an apparently improved score.

Surface confusion blames the prompt because it is easy to edit even when the trace points to retrieval or permissions.

Holdout contamination makes a protected test set part of the search process.

Candidate drift changes the profile between generation and final scoring, so the measured object is not the object that ships.

Rollback blindness treats a successful experiment as a deployment plan. Every mutable surface needs a reversible activation path, especially memory, permissions, code, and model behavior.

Choose the layer that can move the outcome

Optimize the prompt when the failure is an instruction or example. Optimize a skill when a reusable procedure is missing or brittle. Optimize memory when the next run needs scoped, source-backed context. Optimize runtime topology when the system needs a different sequence, branch, worker, or stop rule. Optimize harness code when the current machine cannot express the needed action or cannot prove what happened. Touch model weights only when the behavior cannot be reliably expressed outside the model and the team is ready for the stronger data, safety, and rollback obligations.

The shortest trustworthy improvement loop is:

name the failure
freeze the profile
choose one mutable surface
run paired cases
measure quality and every relevant cost
inspect the traces
promote only on protected evidence

If the team cannot say which surface caused the improvement, it has a result but not yet an explanation. That is enough for a candidate. It is not enough for a system that is allowed to improve itself.

What is optimization theory for agent builders?

It is the study of proposing and comparing changes to an agent’s prompts, skills, memory, runtime, harness, or model under noisy feedback, finite budgets, and release constraints.

What should change first when an agent fails?

Start with the smallest surface that the trace identifies as causal. Do not assume the prompt is responsible because it is the easiest artifact to edit.

Is a higher score proof of improvement?

No. The candidate should beat a defined baseline on paired protected cases, survive deterministic checks, stay within its cost and risk limits, and remain explainable in the trace.

Where should a new team begin?

Begin with one task family, one baseline, one candidate surface, and one hard failure that a deterministic check can recognize. Add more roles, memory, or model adaptation only after that small comparison is trustworthy. When the evidence says the behavior must generalize across contexts, continue with post-training for agents.