BlogThe Self-Improving Stackpart 1 of 3

Prompt Optimization for AI Agents: What It Can Change

Prompt optimization searches instructions, examples, schemas, tool descriptions, and rubrics for better agent behavior while the model and runtime stay fixed.

The support agent answers the customer’s question, uses the right tone, and still forgets to include the one field the downstream system needs. The team rewrites the system prompt, sees a higher score on twenty examples, and ships the change. Two weeks later, the agent is still missing the field on a different kind of ticket.

Prompt optimization is the right first experiment only when the missing behavior is already possible in the agent’s action space. It can improve behavior caused by language-shaped instructions, examples, tool descriptions, output schemas, and judging rules. It cannot create a missing tool, a parallel worker, a larger token budget, or an honest evaluator.

An agent runtime is the software that calls the model, exposes tools, controls turns and budgets, and records the run. An agent profile is the complete configuration for one behavior cell, including the model, prompt, tools, skills, permissions, and limits. An evaluation is a repeatable assessment of those outcomes against a task set and a scoring rule. A selection rule is a deployable rule that chooses a candidate result from the results the runtime produced. A holdout set is a protected group of tasks kept out of candidate search until the release decision. A judge is a scorer for qualities that a deterministic checker cannot fully measure. A baseline is the version already in use, and a candidate is a proposed replacement.

Prompt optimization is a search over text-bearing controls, but its causal reach stops at the capabilities the agent already has. Tangle’s surface for that question is defineAgentEval in @tangle-network/agent-eval: evaluate({ surface }) runs the shipped baselineSurface and a candidate surface over the same scenarios and returns aggregates.byJudge, a per-judge mean for each surface.

The question a prompt optimizer can answer

Let p be the mutable text, m the model, r the runtime, x a task, and S the evaluator. A prompt search estimates something like:

p* = argmax_p  E[S(run(m, r, p, x))] - cost_penalty

If m changes, the experiment no longer isolates the prompt. If r changes its turn limit or tool set, the candidate received a different action space. If S changes its rubric, the score is not comparable.

The causal path is clean when it looks like this:

prompt text -> model decision -> output or tool trajectory -> score

It becomes a different problem when the path is:

prompt text -> request to verify -> no checker exists -> unsupported output -> generous judge score

The second system needs a tool or a better judge, not a cleverer sentence.

“Prompt” names several different surfaces

Production agents assemble a program from several text-bearing components, not one string.

Surface Example change Typical blast radius
System instruction Require a source check before answering Every task in the profile
Planner instruction Ask for a plan and a stopping condition The whole trajectory
Tool description Clarify an argument’s units or required fields Calls to one tool
Output schema Require answer, evidence, and uncertainty Parsing and downstream actions
Few-shot examples Show correct inputs, outputs, and edge cases Tasks near the examples
Judge rubric Define what “complete” means The metric itself
Retrieval instruction Choose sources and citation behavior Knowledge-heavy tasks

The surface has to be named before it is optimized. A field description, a coordinator policy, and a judge rubric have different owners, constraints, and failure modes.

The structured-program view is why DSPy’s MIPROv2 documentation describes joint optimization of instructions and few-shot examples inside a program.

What GEPA, MIPRO, and the older methods change

The family shares one loop:

propose a text candidate
run it on examples
score the result
keep useful candidates
propose the next edit

The proposal and feedback mechanisms differ.

Method Candidate surface Feedback used by the search What to watch
Instruction-generation search (APE) Instruction text Scores of generated instructions Instruction overfitting and weak examples
History-based prompt optimization (OPRO) Natural-language solution or instruction A history of candidates and scores The optimizer’s own prompt and bias
Module and demonstration search (MIPROv2) Instructions and demonstrations across modules A task metric and search over candidate combinations Demo leakage and module interactions
Textual feedback optimization (TextGrad) Text variables in a compound graph Natural-language feedback passed to variables Feedback drift and unconstrained edits
Reflective prompt evolution (GEPA) Text components in an agent system Trajectories, scores, and reflective critique Holdout overfitting and evaluator contamination

APE treats instruction generation as a search problem. OPRO uses a language model to propose new solutions from previously scored candidates. MIPRO searches instructions and demonstrations in multi-stage language programs. TextGrad passes textual feedback through a computation graph in a gradient-like role. GEPA adds trajectory-level reflection, prompt mutation, and Pareto-style preservation of complementary candidates.

Those descriptions identify the search operator, not which method wins on your product. The task distribution, evaluator, model, runtime, and cost limit still decide the result.

A worked example: one missing ticket field

Suppose a support agent must return a structured object with the answer and the ticket identifier. The model already has a ticket lookup tool. The tool call is present in the trace, but the final response sometimes drops the identifier.

That is a good prompt-optimization candidate: the capability exists, and the failure is representational, between available evidence and the output format.

The smallest evaluation runs without a sandbox:

import { defineAgentEval } from '@tangle-network/agent-eval/contract'

type SupportCase = { id: string; kind: 'support' }

const evalKit = defineAgentEval<SupportCase, string>({
  scenarios: [
    { id: 'refund', kind: 'support' },
    { id: 'shipping', kind: 'support' },
    { id: 'cancel', kind: 'support' },
  ],
  agent: async (prompt, scenario) =>
    String(prompt).includes('ticket') ? `Ticket ${scenario.id}: on it.` : 'On it.',
  judge: {
    name: 'ticket-id',
    dimensions: [{ key: 'present', description: 'The answer includes the ticket id' }],
    score: ({ artifact, scenario }) => {
      const present = artifact.includes(scenario.id) ? 1 : 0
      return { dimensions: { present }, composite: present, notes: '' }
    },
  },
  baselineSurface: 'Answer politely.',
  expectUsage: 'off',
})

const baseline = await evalKit.evaluate()
const candidate = await evalKit.evaluate({
  surface: 'Answer politely and include the ticket id in the final object.',
})

The string agent proves only the wiring. Install with pnpm add @tangle-network/agent-eval, then replace the agent with a real model call and a schema validator.

Test the candidate on a protected set of tickets with the same model, tools, runtime, and output parser. The full trace should confirm that the ticket lookup still happened and that the candidate did not pass by omitting the tool call.

If the candidate improves only when the judge reads the new phrase, the prompt may be learning the judge rather than the task. If it improves on support tickets but breaks a separate escalation schema, the candidate needs a regression slice. If it needs twice the output budget because the instruction made every answer verbose, the quality result has a cost attached.

Measure the candidate on real sandboxes

evals/regression-gate runs the same comparison with a real agent. Each scenario creates a Claude Code sandbox from the sandbox-demo repository, sends the task with the surface appended, and passes only when npm test exits 0 there. CANDIDATE_PROMPT supplies the candidate.

Regression gate: shipped prompt against candidateRan on production 2026-11-02 from these bytes, exit 0, $0.1818 reported usage, @tangle-network/sandbox 1.0.168. View result.
import { Sandbox } from "@tangle-network/sandbox";
import { defineAgentEval, type Scenario } from "@tangle-network/agent-eval/contract";

const client = new Sandbox({
  apiKey: process.env.TANGLE_API_KEY!,
  baseUrl: "https://sandbox.tangle.tools",
});

// Run in CI: the build fails when the new prompt scores below the shipped one.
const shipped = "Fix the cause.";
const candidate =
  process.env.CANDIDATE_PROMPT ?? "Fix the cause. Change only what the test needs.";

const evalKit = defineAgentEval<Scenario & { task: string }, number>({
  scenarios: [
    { id: "cli-flag", kind: "fix", task: "The test tagged cli-flag fails." },
    { id: "null-guard", kind: "fix", task: "The test tagged null-guard fails." },
  ],
  baselineSurface: shipped,
  agent: async (prompt, scenario) => {
    const box = await client.create({
      git: { url: "https://github.com/tangle-network/sandbox-demo" },
      backend: { type: "claude-code", model: { model: "glm-5.3" } },
    });
    try {
      await box.prompt(`${scenario.task} ${String(prompt)}`);
      return (await box.exec(`npm test -- --test-name-pattern=${scenario.id}`)).exitCode;
    } finally {
      await box.delete();
    }
  },
  judge: {
    name: "suite",
    dimensions: [{ key: "pass", description: "the tagged test passes after the turn" }],
    score: ({ artifact }) => {
      const pass = artifact === 0 ? 1 : 0;
      return { dimensions: { pass }, composite: pass, notes: "" };
    },
  },
  expectUsage: "off",
});

const score = async (surface: string) =>
  (await evalKit.evaluate({ surface })).aggregates.byJudge.suite?.mean ?? 0;
const before = await score(shipped);
const after = await score(candidate);
console.log({ shipped: before, candidate: after });
if (after < before) {
  console.error("Regression: the candidate passes fewer cases.");
  process.exit(1);
}

The script prints the shipped and candidate means and exits 1 when the candidate is lower. That shows the candidate did not lose on two failing tests in this run. It is not a lift: two scenarios cannot support a lower confidence bound, so the promotion rule below still decides.

Tangle’s public placement

Tangle’s public agent stack separates execution from measurement. The agent-runtime repository shows an improve flow that can select a prompt surface while keeping the final test set out of the optimizer. The agent-eval repository exposes paired case evaluation, score distributions, and official GEPA and SkillOpt integrations.

The profile is the important unit. A prompt change is not a free-floating string when model, tools, skills, budgets, and policies affect behavior. It is a new profile cell that must be compared with the old cell.

A Blueprint is a reusable service template that defines the jobs and artifacts a service can expose. Tangle also uses the word router for a job traffic director inside a Blueprint Runner. A router validates a job call and sends it to the handler registered for that job identifier. That is different from a prompt optimizer and different from a model-routing policy.

The optimizer cannot repair a missing action

Consider an agent asked to verify a generated patch. The prompt says:

Run the tests, inspect the diff, and stop if the patch is unsafe.

If the runtime supplies a shell tool, a diff view, a test command, a budget, and a stop policy, the instruction may help coordinate them. If the runtime supplies only a text response, the instruction can produce a convincing claim about tests without any test execution.

The same distinction applies to coordination. “Ask three specialists to work in parallel” is operational only when the runtime can create workers, isolate their state, collect their results, and choose a winner.

Prompt optimization is also the wrong first move when the trace shows:

  • a missing tool or stale tool schema;
  • no retrieval result for the needed source;
  • a runtime that stops before the required check;
  • a selection rule that cannot distinguish good and bad candidates;
  • a judge that rewards unsupported fluency;
  • a model that cannot perform the task within the available budget.

The optimizer can be part of the repair after the missing control is supplied. It cannot supply the control by writing about it.

A promotion protocol that keeps the claim narrow

A prompt run should freeze the parts that are not being optimized. The minimum record includes:

baseline profile and candidate profile
model and provider endpoint
tool and schema versions
search, selection, and holdout task ids
paired scores and deterministic checks
input and output tokens
wall time and dollar cost
full traces for representative failures
judge version and calibration set
promotion decision and rollback reference

For paired tasks, define:

delta_i = score(candidate, task_i) - score(baseline, task_i)

A lower confidence bound is a conservative estimate of the lift after accounting for sampling uncertainty. Promote only when the lower confidence bound for the paired lift clears the product threshold, hard checks pass, and the candidate stays inside cost and latency limits. The exact confidence method can be bootstrap, permutation-based, or another justified method. The rule must be fixed before the holdout is read.

Keep three roles distinct:

Set What it is allowed to do
Search Propose and debug candidates
Selection Choose among candidates during the run
Holdout Decide whether the selected candidate generalizes

Do not place private answer keys, final judge rationales, or promotion decisions inside the agent’s runtime context. The optimizer can learn from an evaluation after a run. The production agent cannot use information it would not have at deployment time.

When prompt search produces the wrong lesson

Benchmark phrasing overfit occurs when the candidate copies patterns from the search examples and transfers poorly.

Judge coupling occurs when the candidate learns the evaluator’s favorite wording without improving the user’s outcome.

Schema drift occurs when a helpful instruction changes output fields that downstream code expects.

Verbosity inflation occurs when a judge rewards longer answers and the candidate spends more tokens to gain a point.

Demo leakage occurs when examples reveal facts, labels, or answer formats that should remain protected.

Surface misattribution occurs when a runtime, tool, or evaluator failure is blamed on language.

Safety regression occurs when a candidate becomes more compliant with the task by weakening refusal, permission, uncertainty, or privacy behavior.

The trace should make these failures inspectable, and the release rule should reject them even when the aggregate score rises.

Choose prompt optimization when text is causal

Use prompt optimization when the failure is visible in the text-to-behavior path and the needed capability already exists. Start with the smallest plausible surface. Keep the model, runtime, tool graph, evaluator, and budget fixed while searching. Use traces to explain why a candidate changed behavior.

Before a prompt change ships, run the regression gate on the candidate, then the paired holdout check. Keep the shipped prompt when the candidate mean is lower or the lower confidence bound misses the threshold, and read the trace for the missing control.

Move to skill optimization when the procedure must persist across future runs. Move to test-time compute when the system needs more attempts, search, or verification under a measured budget. Move to post-training when the behavior needs to generalize through model parameters and the team can accept the harder rollback and data obligations.

Sources

What is prompt optimization?

Prompt optimization searches language-shaped control surfaces such as instructions, examples, tool descriptions, output schemas, retrieval guidance, and judging rubrics. It is useful when the changed text can causally affect the observed failure while the rest of the agent profile stays fixed.

Can prompt optimization create new agent capabilities?

No. A prompt can request a tool call or a parallel workflow, but the runtime must supply the tool, worker management, state isolation, budget, and stopping behavior.

How should I compare a prompt candidate with a baseline?

Run both on the same task cells with the same model, tools, runtime, evaluator, and budget. Keep full traces, deterministic checks, cost, and a protected holdout, then promote only when the paired lift survives the release rule.

Where does GEPA fit?

GEPA is a reflective text-evolution method that uses trajectory feedback to propose prompt changes and preserve complementary candidates. It is a candidate generator, not a substitute for runtime capability, trace integrity, or an independent promotion rule.

Next in The Self-Improving Stack: Post-Training Agents: When to Change the Model