BlogThe Self-Improving Stackpart 2 of 3

Post-Training Agents: When to Change the Model

Post-training for agents moves improvement from prompts and skills into model weights or adapters, with harder data, release, and rollback decisions.

The coding agent follows a new tool-use procedure in one project after the team adds a skill. It forgets the procedure in another product, under another prompt, with another tool description. The team asks whether the behavior should move into the model.

Most product teams can improve an agent without changing weights. They can edit prompts, skills, retrieval, tools, runtime execution, traces, and release rules. Post-training is worth its cost when the behavior should generalize across many contexts and the team can accept a harder data, release, and rollback problem.

The Tangle products that carry this argument sit on the evidence side of the loop. evals/score-a-trace sends one sandbox run’s answer and every tool invocation to a judge on the Router, and returns a score per dimension. That score is what a training signal is built from. evals/regression-gate is the check a post-trained candidate has to clear: it runs the shipped and the candidate version on the same scenarios in real sandboxes, and exits 1 when the candidate passes fewer cases.

The terms used below:

  • Model weights are the learned parameters that map context to output probabilities.
  • Post-training updates those weights, or adds a learned adapter on top of a base model, after the initial broad training stage.
  • An agent profile is the versioned bundle of model, prompt, skills, tools, permissions, runtime, and limits used for a run.
  • A trace records one execution: actions, observations, artifacts, costs, and outcome.
  • An evaluation tests behavior against tasks, hard checks, policy rules, and cost limits.
  • A holdout set is a group of tasks kept out of candidate search until the release decision.
  • A baseline is the version in use. A candidate is its proposed replacement.

The model boundary changes the release unit

External changes are usually inspectable as files or configuration. A reviewer can read a prompt diff, compare a skill, inspect a tool schema, or revert a runtime change.

Weight changes ship as a model artifact. One update can change tool choice, refusal behavior, uncertainty, style, and task performance in contexts that were not in the trace that motivated it.

The loop keeps the same stages:

collect behavior
label or verify it
construct a training signal
update a model or adapter
evaluate the candidate
release or reject the artifact

Only the mutable object changes:

external loop: candidate prompt, skill, tool, memory, or runtime
weight loop: candidate model weights or an adapter attached to a base model

A local fix means something different in each loop. A prompt can be scoped to one profile. A model update can generalize across profiles, which is its value and its risk.

The post-training ladder

Each method uses a different signal and makes a different claim.

Method Plain-language signal Typical use Main failure
Supervised fine-tuning (SFT) Show the desired answer or action Teach format, vocabulary, and common patterns Imitates surface behavior without the decision rule
Reinforcement learning from human feedback (RLHF) Humans compare outputs and a reward model learns the preference Subjective helpfulness, style, and safety The model optimizes a preference proxy
Reinforcement learning from AI feedback (RLAIF) A model supplies preference labels under written principles Scale feedback over repeatable cases The evaluator and policy share blind spots
Direct Preference Optimization (DPO) Prefer one answer over another directly Simpler preference-based adaptation Pair quality and reference choice dominate
Process supervision Score intermediate steps Reasoning and tool-use behavior Step labels are expensive or incomplete
Check-backed reward Tests, schemas, proofs, or exact checks Code, math, and structured tool tasks The model finds a shortcut around the checker
Tool-use reinforcement learning (RL) Reward tool choice and argument behavior in an environment Multi-step actions and tool use Sparse rewards and unsafe exploration

Supervised fine-tuning teaches examples

Supervised fine-tuning, or SFT, trains on pairs of inputs and desired outputs, raising the probability the model gives the demonstrated response.

training row = (task, desired response)

For an agent, the response can be a tool call with structured arguments or a final artifact. SFT teaches the expected shape of a workflow.

It is weak when the dataset shows only the final answer and hides why that answer was chosen. The model can copy the style while missing the check that made the example safe, and it can learn a workaround after the underlying tool bug is fixed. Use traces to decide whether a training row should hold a final answer, an action sequence, a preference pair, or a result backed by a repeatable check.

RLHF and RLAIF turn preferences into a proxy

Reinforcement learning from human feedback, or RLHF, trains from human comparisons as well as demonstrations. The InstructGPT description from OpenAI has three broad stages:

human demonstrations -> supervised model
human comparisons -> reward model
reward model -> policy update with reinforcement learning

A reward model is a learned scorer that predicts which output a reviewer would prefer. It is a proxy for the user’s outcome, not the outcome.

RLAIF, or reinforcement learning from AI feedback, replaces some human comparisons with judgments from another model, often under written principles. Constitutional AI is a public example of a written constitution guiding critique, revision, and preference collection.

Scale comes with correlated-error risk: if the judge and the policy share a blind spot, the loop strengthens it. The team needs human anchor cases, judge versioning, and held-out tasks the optimizer cannot read.

DPO simplifies the preference path

Direct Preference Optimization, or DPO, trains from a preferred and a rejected response against a reference model. It does not fit a separate reward model and run online policy optimization as classic RLHF does. The DPO paper derives a classification-style objective from those pairs.

task -> preferred response
     -> rejected response
     -> update the candidate relative to a reference model

The pipeline is simpler, but the preference data matters as much as before. If the preferred response wins because it is longer, more flattering, or closer to a judge’s style, the model learns that bias.

For agents, a pair can compare a correct tool call with an invalid one, a verified patch with a plausible untested one, a source-backed answer with an unsupported one, or a safe refusal with an unsafe compliance. A reviewer should be able to name the test, policy, source, or outcome that made the winner better.

Process supervision scores the path

Outcome supervision scores the final artifact. Process supervision scores the intermediate decisions: the tool call, the observation the agent used, the check it ran.

An outcome-only label says the artifact failed. A process label can say the tool arguments were invalid, the observation was ignored, or the check ran on the wrong file. The public Let’s Verify Step by Step study showed why step-level feedback can matter on hard reasoning tasks. A step that happened is not a good step, and a judge’s praise does not make the judge correct.

The judge in this example reads the answer and every tool invocation from one claude-code run on a real sandbox. It scores two process dimensions: grounded, whether every file and line it names exists, and economical, whether no tool call repeats earlier work. A low economical score can flag a run that reached a correct answer by wasteful means.

src/examples/evals/score-a-trace.mtsRan on production 2026-10-31 from these bytes, exit 0, $0.0399 reported usage, @tangle-network/sandbox 1.0.134. View result.
import { Sandbox } from "@tangle-network/sandbox";
import { createChatClient, llmJudge } from "@tangle-network/agent-eval/contract";

const apiKey = process.env.TANGLE_API_KEY!;
const client = new Sandbox({ apiKey, baseUrl: "https://sandbox.tangle.tools" });

// A model judge on the Router reads the answer and every tool call.
const judge = llmJudge("trace-review", "Score this agent run against its task.", {
  chat: createChatClient({
    transport: "openai-compatible",
    baseUrl: "https://router.tangle.tools/v1",
    apiKey,
    defaultModel: "glm-5.3",
  }),
  dimensions: [
    { key: "grounded", description: "Every file and line it names exists" },
    { key: "economical", description: "No tool call repeats earlier work" },
  ],
});

const task = "List each place the API validates a request body, as file:line.";
const box = await client.create({
  git: { url: "https://github.com/tangle-network/sandbox-demo" },
  backend: { type: "claude-code" },
});
try {
  const run = await box.prompt(task);
  const score = await judge.score({
    artifact: { task, answer: run.response, tools: run.toolInvocations },
    scenario: { id: "body-validation", kind: "trace" },
    signal: AbortSignal.timeout(120_000),
  });
  console.log(run.traceId, score.dimensions, score.notes);
} finally {
  await box.delete();
}

The script prints the trace ID, the two dimension scores, and the judge’s notes. That proves this judge scored this run on these dimensions. It does not prove the judge agrees with a reviewer. Calibrate the judge on labeled examples before its scores become training labels.

Use deterministic process labels where a check exists:

valid tool schema
permission check passed
source returned
test executed
artifact changed as intended
rollback condition preserved

Use semantic judgments, like the one above, only where no deterministic check exists.

Check-backed reward is a strong narrow signal

A check-backed reward comes from a check that can be rerun and whose success condition is explicit. Examples include:

tests_passed
schema_valid
proof_checked
compile_succeeded
required_policy_assertions_passed
tool_result_matches_expected_state

The public DeepSeek-R1 paper describes reinforcement learning with rule-based rewards for reasoning-oriented tasks, alongside model-based rewards for more general data. That does not mean all reasoning reduces to exact tests. It shows why trainable environments with checkable outcomes are attractive.

A model can pass a weak test while producing a dangerous artifact. A code test can miss a payment authorization regression. A schema can be valid while the values are wrong. A tool call can return 200 while the requested state change did not occur.

A worked decision: teach the model or keep the procedure outside?

Suppose a coding agent repeatedly forgets to run the project’s migration check before opening a pull request. The team has four candidate fixes:

Candidate What changes Why choose it What can go wrong
Prompt Add a migration-check instruction Fast and easy to inspect It may be ignored in a crowded context
Skill Persist the procedure and trigger it for schema changes Reusable without changing weights It can activate too broadly or become stale
Runtime Make the check a required release step Enforceable and measurable It may slow unrelated tasks or need new tooling
Post-training Update model behavior across contexts Broad transfer and lower prompt dependence Harder attribution, rollback, and regression control

Start outside the model when the behavior is a policy or an enforceable workflow step. Move toward weights when many contexts show the same behavior, the data is clean, and the product needs the model to generalize it before a prompt or skill is loaded.

The model update should not be the first response to a missing capability. If the migration checker does not exist, post-training cannot create a trustworthy one.

Data provenance becomes the main control

For agent behavior, the training record must preserve:

source and ownership
privacy classification
consent or access boundary
model and tool versions
trace and artifact references
split assignment
checker version
judge or human label source
synthetic-data marker
deduplication identity
contamination status

Do not train on holdout cases. Do not treat model-generated answers as ground truth without labels or checks. Treat a judge rationale as a hypothesis until an independent check supports it. Keep user feedback, synthetic data, replayed data, and evaluation output distinct.

The model-collapse study in Nature documents how repeatedly training on model-generated data can remove diversity and distort the distribution. Agent teams should mark generated traces and control their share of the mixture, rather than feeding every successful-looking answer back into training.

Tangle’s public boundary

The public agent-runtime repository covers executing agents, teams, tools, bounded rounds, and improvement experiments. The public agent-eval repository covers comparing behavior, preserving evaluation cases, analyzing traces, and keeping final test cases away from the optimizer.

The public TypeScript packages install with:

pnpm add @tangle-network/agent-runtime @tangle-network/agent-eval @tangle-network/sandbox

These packages collect training evidence. They are not a public claim that any Tangle agent is fine-tuned in production. The model update remains a separate training and release system unless a product explicitly wires one in.

runtime executes the workflow
trace records the workflow
evaluation scores the workflow
analyst identifies a candidate signal
training system updates the model or adapter
release rule decides whether the artifact ships

An agent profile should identify the model artifact, prompt, skills, tools, permissions, runtime, and evaluator used to generate the training evidence. Without that record, a model update can absorb behavior from an untracked configuration and become impossible to reproduce.

Frontier Tuning is an environment-level example

Microsoft’s public Frontier Tuning announcement describes a private-preview approach that applies reinforcement learning inside a customer’s compliance boundary, with tuned models, skills, and runtime orchestration as one system.

For an agent team, the environment, tools, skills, and evaluation rules shape what a model can learn and what a customer can deploy. The announcement is a vendor description, not independent evidence of the lift a given organization will see.

The model release rule

A model artifact needs a stricter release rule than a prompt change. At minimum, compare:

held-out task quality
hard checker failures
profile and tool-use regressions
safety and refusal behavior
privacy and memorization probes
contamination checks
judge calibration
cost and latency
data and reward lineage
artifact reproducibility
rollback and deprecation plan

Use a release record that names:

base model id
adapter or candidate id
training-data manifest
reward and checker manifest
evaluation manifest
artifact digest
access policy
deployment scope
rollback target
approver

evals/regression-gate is the mechanism behind the pass-or-fail step. The example below runs the shipped prompt and a candidate on two fix scenarios, each in a fresh sandbox, and exits 1 when the candidate passes fewer cases.

src/examples/evals/regression-gate.mtsRan on production 2026-11-02 from these bytes, exit 0, $0.1818 reported usage, @tangle-network/sandbox 1.0.168. View result.
import { Sandbox } from "@tangle-network/sandbox";
import { defineAgentEval, type Scenario } from "@tangle-network/agent-eval/contract";

const client = new Sandbox({
  apiKey: process.env.TANGLE_API_KEY!,
  baseUrl: "https://sandbox.tangle.tools",
});

// Run in CI: the build fails when the new prompt scores below the shipped one.
const shipped = "Fix the cause.";
const candidate =
  process.env.CANDIDATE_PROMPT ?? "Fix the cause. Change only what the test needs.";

const evalKit = defineAgentEval<Scenario & { task: string }, number>({
  scenarios: [
    { id: "cli-flag", kind: "fix", task: "The test tagged cli-flag fails." },
    { id: "null-guard", kind: "fix", task: "The test tagged null-guard fails." },
  ],
  baselineSurface: shipped,
  agent: async (prompt, scenario) => {
    const box = await client.create({
      git: { url: "https://github.com/tangle-network/sandbox-demo" },
      backend: { type: "claude-code", model: { model: "glm-5.3" } },
    });
    try {
      await box.prompt(`${scenario.task} ${String(prompt)}`);
      return (await box.exec(`npm test -- --test-name-pattern=${scenario.id}`)).exitCode;
    } finally {
      await box.delete();
    }
  },
  judge: {
    name: "suite",
    dimensions: [{ key: "pass", description: "the tagged test passes after the turn" }],
    score: ({ artifact }) => {
      const pass = artifact === 0 ? 1 : 0;
      return { dimensions: { pass }, composite: pass, notes: "" };
    },
  },
  expectUsage: "off",
});

const score = async (surface: string) =>
  (await evalKit.evaluate({ surface })).aggregates.byJudge.suite?.mean ?? 0;
const before = await score(shipped);
const after = await score(candidate);
console.log({ shipped: before, candidate: after });
if (after < before) {
  console.error("Regression: the candidate passes fewer cases.");
  process.exit(1);
}

The surface here is a prompt. A post-trained candidate needs the agent function to run the candidate model instead; the comparison and exit code stay the same. The gate shows only that the candidate passes at least as many of these two tests as the shipped version. Holdout quality, safety probes, and cost need their own cases in the release record.

The release rule must be independent of the training process. A training loop must not be able to rewrite the evaluator that decides whether its own model ships.

Where post-training fails

Reward overoptimization produces outputs that satisfy the learned scorer while missing the real goal.

Preference shortcutting teaches the model to imitate reviewer style rather than task correctness.

Process-label noise rewards a step that correlates with success without causing it.

Checker gaming exploits a test, schema, or simulator that covers only part of the user’s requirement.

Catastrophic forgetting improves one domain while damaging previously useful behavior.

Data leakage turns private or holdout examples into model behavior.

Synthetic drift feeds generated outputs back into training until the distribution narrows.

Hidden coupling makes a model improvement depend on one prompt, tool schema, or runtime setting that was not recorded.

Rollback ambiguity leaves the team able to revert a pointer but unable to identify which data, reward, or policy produced the artifact.

These risks explain why post-training comes after traces, evaluation, data governance, and external controls.

Choose the model boundary

Choose prompt or skill changes for local, frequently changing, easy-to-inspect behavior, and runtime changes for behavior that needs enforcement, tools, or budgets. Choose post-training when the behavior should generalize across contexts, the training signal is defensible, and the release process can govern the artifact.

If the failure is still local to instructions or a recurring procedure, start with prompt optimization. If the team cannot explain the failure from a run record, read agent traces before changing model parameters.

What to run before a weight update

Score the traces behind the proposed training rows with evals/score-a-trace, and check a human-labeled sample against its dimensions. Then run evals/regression-gate with the candidate and require at least as many passes as the shipped version. If the judge disagrees with the reviewer or the candidate fails any case, keep the behavior in the prompt, skill, or runtime rule.

FAQ

What is post-training for agents?

Post-training updates model weights or a learned adapter after broad pretraining, so behavior changes across future contexts. For agents, the training signal can come from demonstrations, preferences, process labels, check-backed outcomes, or tool-use environments.

When should an agent team use post-training?

Use it when a behavior repeats across many contexts, should generalize before a prompt or skill is loaded, and has a clean training signal with a governed release path. Keep the behavior external when it is a fast-changing policy, an enforceable runtime rule, or a procedure that must stay easy to inspect.

Is DPO the same as RLHF?

No. DPO trains directly from preference pairs using a reference model. Classic RLHF commonly trains a reward model and then optimizes a policy against that reward. Both depend on the quality and coverage of their preference data.

Do traces become training data automatically?

No. A trace is evidence about a run, not a label that the run should be imitated. Training data needs provenance, privacy review, split assignment, checker or human support, and contamination checks before it becomes a model update.

Sources

Next in The Self-Improving Stack: Harness Evolution for AI Agents: When Prompts Plateau