Prompt Optimization for AI Agents: One Layer of a Bigger System

The support agent answers the customer’s question, uses the right tone, and still forgets to include the one field the downstream system needs. The team rewrites the system prompt, sees a higher score on twenty examples, and ships the change. Two weeks later, the agent is still missing the field on a different kind of ticket.

Prompt optimization is the right first experiment only when the missing behavior is already possible in the agent’s action space. It can improve behavior caused by language-shaped instructions, examples, tool descriptions, output schemas, and judging rules. It cannot create a missing tool, a parallel worker, a larger token budget, or an honest evaluator.

A prompt is the text that gives a model a task, context, constraints, or output format. An agent runtime is the software that calls the model, exposes tools, controls turns and budgets, and records the run. An agent profile is the complete configuration for one behavior cell, including the model, prompt, tools, skills, permissions, and limits. A trace is the run record that keeps model calls, tool actions, observations, artifacts, costs, and the final outcome. An evaluation is a repeatable assessment of those outcomes against a task set and a scoring rule. A selection rule is a deployable rule that chooses a candidate result from the results the runtime produced. A checker is a repeatable check that tests a result against an explicit condition. A holdout set is a protected group of tasks kept out of candidate search until the release decision. A judge is a scorer for qualities that a deterministic checker cannot fully measure. A baseline is the version already in use, and a candidate is a proposed replacement.

Prompt optimization is a search over text-bearing controls, but its causal reach stops at the capabilities the agent already has.

The question a prompt optimizer can answer

Let p be the mutable text, m the model, r the runtime, x a task, and S the evaluator. A prompt search estimates something like:

p* = argmax_p  E[S(run(m, r, p, x))] - cost_penalty

The fixed terms are as important as the variable. If m changes, the experiment no longer isolates the prompt. If r changes its turn limit or tool set, the candidate received a different action space. If S changes its rubric, the score is not comparable.

The causal path is clean when it looks like this:

prompt text -> model decision -> output or tool trajectory -> score

It becomes a different problem when the path is:

prompt text -> request to verify -> no checker exists -> unsupported output -> generous judge score

The second system needs a tool or a better judge before it needs a cleverer sentence.

“Prompt” names several different surfaces

Production agents rarely send one undifferentiated string. They assemble a program from multiple text-bearing components.

Surface	Example change	Typical blast radius
System instruction	Require a source check before answering	Every task in the profile
Planner instruction	Ask for a plan and a stopping condition	The whole trajectory
Tool description	Clarify an argument’s units or required fields	Calls to one tool
Output schema	Require `answer`, `evidence`, and `uncertainty`	Parsing and downstream actions
Few-shot examples	Show correct inputs, outputs, and edge cases	Tasks near the examples
Judge rubric	Define what “complete” means	The metric itself
Retrieval instruction	Choose sources and citation behavior	Knowledge-heavy tasks

The surface has to be named before it is optimized. A field description, a coordinator policy, and a judge rubric may all be text, but they have different owners, constraints, and failure modes.

The structured-program view is why DSPy’s MIPROv2 documentation describes joint optimization of instructions and few-shot examples inside a program. The optimizer is choosing components and combinations, not searching a single magic sentence.

What GEPA, MIPRO, and the older methods change

The family resemblance is simple:

propose a text candidate
run it on examples
score the result
keep useful candidates
propose the next edit

The proposal and feedback mechanisms differ.

Method	Candidate surface	Feedback used by the search	What to watch
Instruction-generation search (APE)	Instruction text	Scores of generated instructions	Instruction overfitting and weak examples
History-based prompt optimization (OPRO)	Natural-language solution or instruction	A history of candidates and scores	The optimizer’s own prompt and bias
Module and demonstration search (MIPROv2)	Instructions and demonstrations across modules	A task metric and search over candidate combinations	Demo leakage and module interactions
Textual feedback optimization (TextGrad)	Text variables in a compound graph	Natural-language feedback passed to variables	Feedback drift and unconstrained edits
Reflective prompt evolution (GEPA)	Text components in an agent system	Trajectories, scores, and reflective critique	Holdout overfitting and evaluator contamination

APE treats instruction generation as a search problem. OPRO uses a language model to propose new solutions from previously scored candidates. MIPRO searches instructions and demonstrations in multi-stage language programs. TextGrad passes textual feedback through a computation graph in a gradient-like role. GEPA adds trajectory-level reflection, prompt mutation, and Pareto-style preservation of complementary candidates.

Those descriptions identify the search operator. They do not establish that one method wins on your product. The task distribution, evaluator, model, runtime, and cost limit still decide the result.

A worked example: one missing ticket field

Suppose a support agent must return a structured object with the answer and the ticket identifier. The model already has a ticket lookup tool. The tool call is present in the trace, but the final response sometimes drops the identifier.

That is a good prompt-optimization candidate because the missing behavior is representational. The capability exists, and the failure appears between available evidence and the output format.

The smallest evaluation can be deterministic:

import { defineAgentEval } from '@tangle-network/agent-eval/contract'

type SupportCase = { id: string; kind: 'support' }

const evalKit = defineAgentEval<SupportCase, string>({
  scenarios: [
    { id: 'refund', kind: 'support' },
    { id: 'shipping', kind: 'support' },
    { id: 'cancel', kind: 'support' },
  ],
  agent: async (prompt, scenario) =>
    String(prompt).includes('ticket') ? `Ticket ${scenario.id}: on it.` : 'On it.',
  judge: {
    name: 'ticket-id',
    dimensions: [{ key: 'present', description: 'The answer includes the ticket id' }],
    score: ({ artifact, scenario }) => {
      const present = artifact.includes(scenario.id) ? 1 : 0
      return { dimensions: { present }, composite: present, notes: '' }
    },
  },
  baselineSurface: 'Answer politely.',
  expectUsage: 'off',
})

const baseline = await evalKit.evaluate()
const candidate = await evalKit.evaluate({
  surface: 'Answer politely and include the ticket id in the final object.',
})

This example uses the public defineAgentEval contract from the Tangle agent-eval repository. The production agent and judge can be replaced with a real model call and a schema validator, while the comparison remains the same.

The public package documents this installation path:

pnpm add @tangle-network/agent-eval

The candidate should then be tested on a protected set of tickets with the same model, tools, runtime, and output parser. The full trace should confirm that the ticket lookup still happened and that the candidate did not pass by omitting the tool call.

If the candidate improves only when the judge reads the new phrase, the prompt may be learning the judge rather than the task. If it improves on support tickets but breaks a separate escalation schema, the candidate needs a regression slice. If it needs twice the output budget because the instruction made every answer verbose, the quality result has a cost attached.

Tangle’s public placement

Tangle’s public agent stack separates execution from measurement. The agent-runtime repository shows an improve flow that can select a prompt surface while keeping the final test set out of the optimizer. The agent-eval repository exposes paired case evaluation, score distributions, and official GEPA and SkillOpt integrations.

The profile is the important unit. A prompt change is not a free-floating string when model, tools, skills, budgets, and policies affect behavior. It is a new profile cell that must be compared with the old cell.

A Blueprint is a reusable service template that defines the jobs and artifacts a service can expose. Tangle also uses the word router for a job traffic director inside a Blueprint Runner. A router validates a job call and sends it to the handler registered for that job identifier. That is different from a prompt optimizer and different from a model-routing policy. Keeping those meanings separate prevents a prompt experiment from quietly changing the service path.

The optimizer cannot repair a missing action

Consider an agent asked to verify a generated patch. The prompt says:

Run the tests, inspect the diff, and stop if the patch is unsafe.

If the runtime supplies a shell tool, a diff view, a test command, a budget, and a stop policy, the instruction may help coordinate them. If the runtime supplies only a text response, the instruction can produce a convincing claim about tests without any test execution.

The same distinction applies to coordination. “Ask three specialists to work in parallel” is operational only when the runtime can create workers, isolate their state, collect their results, and choose a winner. The runtime execution article covers that boundary.

Prompt optimization is also the wrong first move when the trace shows:

a missing tool or stale tool schema;
no retrieval result for the needed source;
a runtime that stops before the required check;
a selection rule that cannot distinguish good and bad candidates;
a judge that rewards unsupported fluency;
a model that cannot perform the task within the available budget.

The optimizer can be part of the repair after the missing control is supplied. It cannot supply the control by writing about it.

A promotion protocol that keeps the claim narrow

A prompt run should freeze the parts that are not being optimized. The minimum record includes:

baseline profile and candidate profile
model and provider endpoint
tool and schema versions
search, selection, and holdout task ids
paired scores and deterministic checks
input and output tokens
wall time and dollar cost
full traces for representative failures
judge version and calibration set
promotion decision and rollback reference

For paired tasks, define:

delta_i = score(candidate, task_i) - score(baseline, task_i)

A lower confidence bound is a conservative estimate of the lift after accounting for sampling uncertainty. Promote only when the lower confidence bound for the paired lift clears the product threshold, hard checks pass, and the candidate stays inside cost and latency limits. The exact confidence method can be bootstrap, permutation-based, or another justified method. The important part is that the rule is fixed before the holdout results are read.

Keep three roles distinct:

Set	What it is allowed to do
Search	Propose and debug candidates
Selection	Choose among candidates during the run
Holdout	Decide whether the selected candidate generalizes

Do not place private answer keys, final judge rationales, or promotion decisions inside the agent’s runtime context. The optimizer can learn from an evaluation after a run. The production agent cannot use information it would not have at deployment time.

When prompt search produces the wrong lesson

Prompt optimizers fail in repeatable ways.

Benchmark phrasing overfit occurs when the candidate copies patterns from the search examples and transfers poorly.

Judge coupling occurs when the candidate learns the evaluator’s favorite wording without improving the user’s outcome.

Schema drift occurs when a helpful instruction changes output fields that downstream code expects.

Verbosity inflation occurs when a judge rewards longer answers and the candidate spends more tokens to gain a point.

Demo leakage occurs when examples reveal facts, labels, or answer formats that should remain protected.

Surface misattribution occurs when a runtime, tool, or evaluator failure is blamed on language.

Safety regression occurs when a candidate becomes more compliant with the task by weakening refusal, permission, uncertainty, or privacy behavior.

The trace should make these failures inspectable. The release rule should reject them even when the aggregate score rises.

Choose prompt optimization when text is causal

Use prompt optimization when the failure is visible in the text-to-behavior path and the needed capability already exists. Start with the smallest surface that can plausibly cause the fix. Keep the model, runtime, tool graph, evaluator, and budget fixed while searching. Use traces to explain why a candidate changed behavior. Use a holdout and deterministic checks before promotion.

Move to skill optimization when the procedure must persist across future runs. Move to test-time compute when the system needs more attempts, search, or verification under a measured budget. Move to post-training when the behavior needs to generalize through model parameters and the team can accept the harder rollback and data obligations.

Prompt optimization is one coordinate in the self-improving stack. It earns its place when the prompt is the cause, the evaluator can see the difference, and the candidate remains better after the search set is gone.

Sources

What is prompt optimization?

Prompt optimization searches language-shaped control surfaces such as instructions, examples, tool descriptions, output schemas, retrieval guidance, and judging rubrics. It is useful when the changed text can causally affect the observed failure while the rest of the agent profile stays fixed.

Can prompt optimization create new agent capabilities?

No. A prompt can request a tool call or a parallel workflow, but the runtime must supply the tool, worker management, state isolation, budget, and stopping behavior.

How should I compare a prompt candidate with a baseline?

Run both on the same task cells with the same model, tools, runtime, evaluator, and budget. Keep full traces, deterministic checks, cost, and a protected holdout, then promote only when the paired lift survives the release rule.

Where does GEPA fit?

GEPA is a reflective text-evolution method that uses trajectory feedback to propose prompt changes and preserve complementary candidates. It is a candidate generator, not a substitute for runtime capability, trace integrity, or an independent promotion rule.