Post-Training Agents: When to Change the Model

The coding agent follows a new tool-use procedure in one project after the team adds a skill. It forgets the procedure in another product, under another prompt, with another tool description. The team asks whether the behavior should be moved into the model.

That is the post-training decision.

Model weights are the learned parameters that determine how a model maps context to output probabilities. Post-training updates those weights or adds a learned adapter after the initial broad training stage. An adapter is a smaller learned parameter set applied on top of a base model. A prompt is the task, context, and constraints supplied to a model for a run. A skill is a reusable procedure that an agent can load for matching tasks. An agent runtime is the surrounding software that supplies tools, controls turns and budgets, and records execution. An agent profile is the versioned bundle of model, prompt, skills, tools, permissions, runtime, and limits used for a run. A trace is the record of one execution, including actions, observations, artifacts, costs, and outcome. An evaluation is a structured test of behavior against tasks, hard checks, policy rules, and cost limits. A checker is a repeatable check that tests whether an artifact or action meets an explicit condition. A holdout set is a protected group of tasks kept out of candidate search until the release decision. A baseline is the version already in use, and a candidate is a proposed replacement.

Most product teams can improve an agent without changing weights. They can edit prompts, skills, retrieval, tools, runtime execution, traces, and release rules. Post-training becomes appropriate when the behavior should generalize across many contexts and the team can accept a harder data, release, and rollback problem.

The model boundary changes the release unit

External changes are usually inspectable as files or configuration. A reviewer can read a prompt diff, compare a skill, inspect a tool schema, or revert a runtime change.

Weight changes are distributed through a model artifact. The same update can affect tool choice, refusal behavior, uncertainty, style, and task performance in contexts that were not present in the motivating trace.

The loop still has familiar stages:

collect behavior
label or verify it
construct a training signal
update a model or adapter
evaluate the candidate
release or reject the artifact

The mutable object is different:

external loop: candidate prompt, skill, tool, memory, or runtime
weight loop: candidate model weights or an adapter attached to a base model

That difference changes what “local fix” means. A prompt can be scoped to one profile. A model update can generalize across profiles, which is its value and its risk.

The post-training ladder

The methods below use different signals and make different claims.

Method	Plain-language signal	Typical use	Main failure
Supervised fine-tuning (SFT)	Show the desired answer or action	Teach format, vocabulary, and common patterns	Imitates surface behavior without the decision rule
Reinforcement learning from human feedback (RLHF)	Humans compare outputs and a reward model learns the preference	Subjective helpfulness, style, and safety	The model optimizes a preference proxy
Reinforcement learning from AI feedback (RLAIF)	A model supplies preference labels under written principles	Scale feedback over repeatable cases	The evaluator and policy share blind spots
Direct Preference Optimization (DPO)	Prefer one answer over another directly	Simpler preference-based adaptation	Pair quality and reference choice dominate
Process supervision	Score intermediate steps	Reasoning and tool-use behavior	Step labels are expensive or incomplete
Check-backed reward	Tests, schemas, proofs, or exact checks	Code, math, and structured tool tasks	The model finds a shortcut around the checker
Tool-use reinforcement learning (RL)	Reward tool choice and argument behavior in an environment	Multi-step actions and tool use	Sparse rewards and unsafe exploration

The ladder is not a ranking. It is a choice of what evidence the training loop can trust.

Supervised fine-tuning teaches examples

Supervised fine-tuning, or SFT, trains on pairs of inputs and desired outputs. The model is adjusted to put more probability on the demonstrated response.

training row = (task, desired response)

For an agent, the response can include a tool call, structured arguments, or a final artifact. SFT is useful for teaching the expected shape of a workflow:

how to format a tool request;
how to fill a required structured-data field;
how to use domain vocabulary;
how to return an approval request;
how to separate evidence from a conclusion.

SFT is weak when the dataset shows only the final answer and hides why that answer was chosen. The model can copy a style while missing the check that made the example safe. It can also learn a temporary workaround after the underlying tool bug is fixed.

Use traces to decide whether the training row should contain a final answer, an action sequence, a preference pair, or a result backed by a repeatable check.

RLHF and RLAIF turn preferences into a proxy

Reinforcement learning from human feedback, or RLHF, trains a model from human comparisons rather than only from demonstrations. The classic InstructGPT description from OpenAI has three broad stages:

human demonstrations -> supervised model
human comparisons -> reward model
reward model -> policy update with reinforcement learning

A reward model is a learned scorer that predicts which output a reviewer would prefer. The policy is the model being trained to produce responses. The reward model is not the user’s actual outcome; it is a proxy for that outcome.

RLAIF, or reinforcement learning from AI feedback, replaces some human comparisons with judgments generated by another model, often under written principles. Constitutional AI is a public example of using a written constitution to guide critique, revision, and preference collection.

The scale advantage comes from generating more comparisons under the same written principles. The risk is correlated error. If the judge and the policy share a blind spot, the loop can strengthen it. The training team needs human anchor cases, independent checks, judge versioning, and held-out tasks that the optimizer cannot read.

DPO simplifies the preference path

Direct Preference Optimization, or DPO, trains from a preferred and rejected response without separately fitting a reward model and running online policy optimization in the same way as classic RLHF. The DPO paper derives a classification-style objective using a reference model and preference pairs.

The practical shape is:

task -> preferred response
     -> rejected response
     -> update the candidate relative to a reference model

DPO is attractive because the training pipeline is simpler. It does not make preference data less important. If the preferred response wins because it is longer, more flattering, or closer to a judge’s style rather than more correct, the model learns that bias.

For agent training, a preference pair can compare:

correct tool call vs invalid tool call
verified patch vs plausible untested patch
source-backed answer vs unsupported answer
safe refusal vs unsafe compliance

The pair needs evidence for the preference. A reviewer should be able to point to the test, policy, source, or outcome that made one response better.

Process supervision uses the path

Outcome supervision scores the final artifact. Process supervision scores intermediate decisions.

Consider an agent trace:

read task
choose tool
send arguments
observe result
revise plan
run check
return artifact

An outcome-only label says the artifact failed. A process label can say the tool arguments were invalid, the observation was ignored, or the check ran on the wrong file.

The public Let’s Verify Step by Step study showed why step-level feedback can matter for hard reasoning tasks. Agent systems already produce candidate process signals in their traces, but those signals need careful labeling. The fact that a step happened does not make it a good step. The fact that a judge praised it does not make the judge correct.

Use deterministic process labels where possible:

valid tool schema
permission check passed
source returned
test executed
artifact changed as intended
rollback condition preserved

Use semantic judgments for what deterministic checks cannot cover, and calibrate those judgments against trusted examples.

Check-backed reward is a strong narrow signal

A check-backed reward comes from a check that can be rerun and whose success condition is explicit. Examples include:

tests_passed
schema_valid
proof_checked
compile_succeeded
required_policy_assertions_passed
tool_result_matches_expected_state

The public DeepSeek-R1 paper describes reinforcement learning with rule-based rewards for reasoning-oriented tasks alongside model-based rewards for more general data. The result does not mean all reasoning can be reduced to exact tests. It shows why trainable environments with checkable outcomes are attractive.

A model can pass a weak test while producing a dangerous artifact. A code test can miss a payment authorization regression. A schema can be valid while the values are wrong. A tool call can return 200 while the requested state change did not occur.

Check-backed reward is strongest when the checker covers the user’s real failure and its limits are known.

A worked decision: teach the model or keep the procedure outside?

Suppose a coding agent repeatedly forgets to run the project’s migration check before opening a pull request. The team has four candidate fixes:

Candidate	What changes	Why choose it	What can go wrong
Prompt	Add a migration-check instruction	Fast and easy to inspect	It may be ignored in a crowded context
Skill	Persist the procedure and trigger it for schema changes	Reusable without changing weights	It can activate too broadly or become stale
Runtime	Make the check a required release step	Enforceable and measurable	It may slow unrelated tasks or need new tooling
Post-training	Update model behavior across contexts	Broad transfer and lower prompt dependence	Harder attribution, rollback, and regression control

Start outside the model when the behavior is a policy or an enforceable workflow step. Move toward weights when many contexts show the same behavior, the data is clean, and the product needs the model to generalize it before a prompt or skill is loaded.

The model update should not be the first response to a missing capability. If the migration checker does not exist, post-training cannot create a trustworthy one.

Data provenance becomes the main control

The training record needs more than input and output text. For agent behavior, preserve:

source and ownership
privacy classification
consent or access boundary
model and tool versions
trace and artifact references
split assignment
checker version
judge or human label source
synthetic-data marker
deduplication identity
contamination status

Do not train on holdout cases. Do not treat model-generated answers as ground truth without labels or checks. Treat a judge rationale as a hypothesis until an independent check supports it. Do not lose the distinction between real user feedback, synthetic data, replayed data, and evaluation output.

The model-collapse study in Nature documents how repeatedly training on model-generated data can remove diversity and distort the distribution. Agent teams should mark generated traces and control their mixture rather than feeding every successful-looking answer back into training.

Tangle’s public boundary

The public Tangle agent-runtime repository focuses on executing agents, teams, tools, bounded rounds, and improvement experiments. The public agent-eval repository focuses on comparing behavior, preserving evaluation cases, analyzing traces, and keeping final test cases away from the optimizer.

The public TypeScript packages document this installation path:

pnpm add @tangle-network/agent-runtime @tangle-network/agent-eval @tangle-network/sandbox

Those are useful pieces for collecting training evidence. They are not a public claim that every Tangle agent is fine-tuned in production. The model update remains a separate training and release system unless a product explicitly wires one in.

The distinction keeps the stack inspectable:

runtime executes the workflow
trace records the workflow
evaluation scores the workflow
analyst identifies a candidate signal
training system updates the model or adapter
release rule decides whether the artifact ships

An agent profile should identify the model artifact, prompt, skills, tools, permissions, runtime, and evaluator used to generate the training evidence. Otherwise a model update can absorb behavior from an untracked configuration and become impossible to reproduce.

Frontier Tuning is an environment-level example

Microsoft’s public Frontier Tuning announcement describes a private-preview approach that applies reinforcement learning inside a customer’s compliance boundary. It describes an evolving environment, customer workflows and data, and tuned models, skills, and runtime orchestration as parts of one system.

That framing is useful because enterprise agent improvement rarely ends at weight updates. The environment, tools, skills, and evaluation rules influence what the model can learn and what the customer can safely deploy. The announcement is a vendor description of a product direction, not independent evidence that every organization will see the same lift.

The release question remains:

Which behavior belongs in model parameters,
and which behavior should remain visible in the runtime and policy files?

Keep behavior external when auditability, tenant separation, frequent policy changes, or fast rollback matters more than compression. Choose post-training when the signal is repeated, broad, privacy-safe, and valuable across the contexts where the model will be used.

The model release rule

A model artifact needs a stricter release rule than a prompt change. At minimum, compare:

held-out task quality
hard checker failures
profile and tool-use regressions
safety and refusal behavior
privacy and memorization probes
contamination checks
judge calibration
cost and latency
data and reward lineage
artifact reproducibility
rollback and deprecation plan

Use a release record that names:

base model id
adapter or candidate id
training-data manifest
reward and checker manifest
evaluation manifest
artifact digest
access policy
deployment scope
rollback target
approver

The release rule should be independent of the training process. A training loop must not be allowed to rewrite the evaluator that decides whether its own model ships.

Where post-training fails

Reward overoptimization produces outputs that satisfy the learned scorer while missing the real goal.

Preference shortcutting teaches the model to imitate reviewer style rather than task correctness.

Process-label noise rewards a step that correlates with success without causing it.

Checker gaming exploits a test, schema, or simulator that covers only part of the user’s requirement.

Catastrophic forgetting improves one domain while damaging previously useful behavior.

Data leakage turns private or holdout examples into model behavior.

Synthetic drift feeds generated outputs back into training until the data distribution narrows.

Hidden coupling makes a model improvement depend on one prompt, tool schema, or runtime setting that was not recorded.

Rollback ambiguity leaves the team able to revert a pointer but unable to identify which data, reward, or policy produced the artifact.

These risks do not make post-training a bad idea. They explain why it belongs after traces, evaluation, data governance, and external controls are working.

Choose the model boundary

Choose prompt or skill changes when the behavior is local, frequently changing, or easy to inspect. Choose runtime changes when the behavior requires enforcement, tools, budgets, or an execution shape. Choose post-training when the behavior should generalize across many contexts, the training signal is defensible, and the release process can govern the artifact.

The model boundary is worth crossing when the gain is broad enough to justify the loss of local visibility. That is the durable decision behind post-training agents.

If the failure is still local to instructions or a recurring procedure, start with prompt optimization or skill optimization. If the team cannot explain the failure from a run record, read agent traces before changing model parameters.

Sources

What is post-training for agents?

Post-training updates model weights or a learned adapter after broad pretraining so behavior changes across future contexts. For agents, the training signal can come from demonstrations, preferences, process labels, check-backed outcomes, or tool-use environments.

When should an agent team use post-training?

Use it when a behavior repeats across many contexts, should generalize before a prompt or skill is loaded, and has a clean training signal with a governed release path. Keep the behavior external when it is a fast-changing policy, an enforceable runtime rule, or a procedure that must remain easy to inspect.

Is DPO the same as RLHF?

No. DPO trains directly from preference pairs using a reference model, while classic RLHF commonly trains a reward model and then optimizes a policy against that reward. Both depend on the quality and coverage of their preference data.

Do traces become training data automatically?

No. A trace is evidence about a run, not a label that the run should be imitated. Training data needs provenance, privacy review, split assignment, checker or human support, and contamination checks before it becomes a model update. When the behavior should remain outside model weights and adapt through scoped writes, see agent memory flywheels.