Skill Optimization for AI Agents: Training Durable Procedures

The support agent knows the refund policy. It still forgets to check whether the order shipped before promising a refund. The same omission appears in tickets handled on Monday, Thursday, and the following week.

Adding another sentence to the current prompt may fix one run. The recurring omission calls for a durable procedure that can be loaded on future tasks and tested outside the original examples.

Skill optimization tests whether that procedure can persist across future tasks without activating on unrelated work.

A skill is a reusable procedure with a scope: when a task matches this situation, take these steps, use these tools, and check these conditions. A prompt steers one model call or run. Memory stores facts, observations, or previous outcomes. Model weights are the learned parameters that shape how the model responds. A tool performs an action or returns an observation. An agent runtime, the software controlling the agent loop, loads those materials, invokes tools, spends the budget, and records the run. An agent profile is the versioned bundle of model, instructions, skills, tools, permissions, runtime, and budgets used for a run. An evaluation is a repeatable assessment of those runs against tasks, checks, costs, and safety rules. A holdout set is a protected group of tasks kept out of candidate search until the release decision. A baseline is the version already in use, and a candidate is a proposed replacement.

Skill optimization changes the procedure that survives the current context. It does not change model weights, and it does not create a tool that the runtime does not expose.

Activation is the decision to load a skill for a particular task.

A skill is a procedure with a trigger

The word “skill” covers several artifacts that should not be evaluated the same way.

Artifact	Stores	Changes	Main risk
Prompt	Current instructions	Behavior in this run	Overfit wording
Memory	Facts and observations	What the agent recalls	Stale or poisoned context
Tool	Executable action	What the agent can do	Unsafe side effects
Skill	Reusable procedure	How the agent operates on matching tasks	Persistent bad habits
Runtime	Loop, budgets, and control flow	What runs	Instructions that sound operational but are not enforced

An agent skill can be a Markdown procedure packaged with optional scripts, references, and templates. The open Agent Skills specification uses a directory with a SKILL.md file containing metadata and instructions as its basic shape. The format makes the procedure portable enough to review and version, while the agent client decides how discovery and loading work.

A useful mental model is:

skill = procedure + activation condition + scope + version

If the artifact never persists, it is a prompt. If it persists only as a fact, it is memory. If it directly changes the world, it is a tool. If it controls the order, budget, or lifetime of work, it belongs to the runtime.

The boundaries can overlap in implementation. They still need separate names in an evaluation because each one has a different rollback and failure path.

The activation rule is part of the product

A skill body can be excellent and still make the agent worse. The skill may activate on unrelated tasks, conflict with another procedure, or arrive after the important decision has already happened.

That gives skill optimization two objectives:

body quality: does the procedure help when loaded?
activation quality: does it load when needed and stay quiet otherwise?

For a task set with known relevance labels:

activation precision = relevant activations / all activations
activation recall = relevant activations / all relevant tasks

The labels do not have to be perfect to be useful. They do have to be defined before a candidate is compared.

A broad description can make recall look good by loading the skill everywhere. A narrow description can make precision look good by avoiding the skill entirely. The product needs both behavior lift and controlled activation.

A worked `SKILL.md`

For the refund agent, a small initial skill might look like this:

---
name: refund-review
description: Use for refund requests when order status, delivery, or purchase date affects eligibility.
---

Before recommending a refund:

1. Identify the order and the policy version that applies.
2. Check the order status and delivery state with the order tool.
3. Apply the policy to the observed state, not to the user's summary alone.
4. State the decision, the evidence, and any missing information.
5. Do not submit a refund or change an order without the required approval.

The body is intentionally procedural. It names the trigger, the sequence, the evidence, and the side-effect boundary. It does not pretend that “check the order” is possible unless the agent profile includes an order tool.

The public Agent Skills guidance recommends keeping the discovery metadata small and loading detailed material only when a task matches. That progressive disclosure pattern reduces always-on context, but it does not remove the need to test whether matching works.

The first evaluation should include three slices:

Slice	Example	Expected behavior
Relevant	A delivered order inside the return window	Load the skill and check status before answering
Near miss	A billing question with no refund request	Stay quiet or use another procedure
Dangerous	A request to issue a refund immediately	Follow approval and side-effect rules

The near miss matters as much as the relevant case. A procedure that fires everywhere becomes a permanent context tax and can override more specific instructions.

What SkillOpt changes in the loop

SkillOpt treats the skill document as the trainable state of a frozen agent. The target model and execution setup stay fixed while a separate optimizer model reads scored trajectories and proposes bounded additions, deletions, or replacements. The candidate skill is kept only when validation improves.

The loop is:

current skill
  -> run tasks and capture trajectories
  -> separate recurring successes from recurring failures
  -> propose bounded edits
  -> evaluate on selection cases
  -> keep the best candidate
  -> test on protected cases
  -> export the deployable skill

The bounded edit matters. A broad rewrite can erase a procedure that was working while improving the small batch that motivated the edit. An edit budget plays a role similar to a learning rate: it limits how far the text can move at one update.

The public SkillOpt project page reports best or tied-best results in 52 of 52 model, benchmark, and execution-setting cells across six benchmarks and seven target models. That is a project-reported result, not a guarantee for a new agent. The same page also reports transfer across model scales and execution clients, which makes transfer a testable claim rather than an assumption.

The important architectural result is simpler than any particular score:

the deployed agent receives the final skill,
not the optimizer's private history or extra deployment-time calls

The optimizer pays during training. The target agent pays only for the skill and its normal execution.

Skill optimization versus memory

Suppose a trace contains the fact that an order was delivered on June 3. That fact belongs in memory or a source system. Suppose the agent repeatedly forgets to check delivery state before applying the policy. That recurrence points toward a skill.

Confusing the two creates bad persistence. The agent may save a specific answer as if it were a general rule, or retrieve a general rule when it only needed a current fact.

Question	Memory answer	Skill answer
What was observed?	Store the fact with source and time	Usually irrelevant
What should happen next time?	Retrieve a trusted policy or lesson	Store the procedure
What can go stale?	Facts, sources, and indexes	Procedures, APIs, and policies
What should be measured?	Retrieval-conditioned task lift	Procedure lift and activation behavior

The memory flywheels article covers source grounding and persistence. The distinction is worth preserving because a skill can be correct while the data it reads is stale, and current data cannot compensate for a missing procedure. If the failure is confined to wording or examples, prompt optimization is the narrower layer to test first.

Tangle’s profile boundary

The public Tangle agent-eval repository treats a changed skill as a changed behavior surface. Its evaluation examples compare a baseline surface with a candidate on the same scenarios, and the public agent-runtime repository documents improvement over named profile surfaces such as prompts and skills.

The public evaluation package documents this installation path:

pnpm add @tangle-network/agent-eval

Changing the active skill should therefore create a new profile cell, even when the model is unchanged. The trace should record which skill version activated, which instructions were loaded, and which tools ran.

If a skill becomes part of a Tangle service, the deployment vocabulary adds another boundary. A Blueprint is a reusable service template that defines runnable artifacts and jobs. An operator is the service host that runs a live instance from that template. The skill is behavior-bearing software inside the service, not passive documentation attached after the fact.

That distinction matters for review. An operator can identify the declared artifact for a service instance, while an evaluation still has to prove that the skill helps the intended tasks and does not expand unsafe behavior.

The security problem is semantic

A skill file is operational text. It can affect discovery, selection, context, tool use, and side effects.

The Agent Skills supply-chain study describes semantic attacks in which malicious instructions are hidden in skill packages and influence which capabilities an agent finds or trusts. The risk is not limited to executable scripts. The words that describe when a skill should activate can be part of the attack.

Every skill release therefore needs at least five checks:

admission: should this package enter the registry?
activation: when should the agent load it?
procedure: what actions and checks does it teach?
authority: which tools and data can those actions reach?
revocation: can the skill be disabled and the previous behavior restored?

Do not let an optimized skill edit its own evaluator, approval policy, or credential scope. Keep scripts and references reviewable. Treat tool outputs as data rather than as higher-priority instructions. Record provenance and version for every active skill.

A release test for a skill

The comparison must isolate the skill while testing its future blast radius.

1. Freeze the model, prompt, tools, runtime, and evaluator.
2. Register the baseline profile and current skill version.
3. Split tasks into search, selection, holdout, transfer, and near-miss sets.
4. Run with and without the skill on the same task cells.
5. Record activation decisions, full traces, costs, and hard failures.
6. Reject candidates that violate schema, permission, or side-effect rules.
7. Promote only when held-out lift and activation limits pass.
8. Keep the old skill and a tested rollback reference.

For paired task results:

delta_i = score(with skill, task_i) - score(without skill, task_i)

A lower confidence bound is a conservative estimate of the lift after accounting for sampling uncertainty. A practical release rule can be written as:

promote(skill) if:
  lower_confidence_bound(median(delta on holdout)) > minimum_lift
  and activation_precision >= minimum_precision
  and activation_recall >= minimum_recall
  and near_miss_regressions == 0
  and security_regressions == 0
  and cost <= cost_limit

The metric choices can change by product. The need to measure activation and interference does not.

Where durable procedures go wrong

Persistent overfit turns one lucky success into a rule that fails on future tasks.

Over-triggering loads the skill on unrelated work and pollutes every prompt.

Under-triggering leaves the procedure unused when it could help.

Stale procedure encodes an old interface, policy, or permission model.

Skill conflict creates contradictory instructions when several procedures activate together.

Authority hiding puts a risky tool call inside friendly prose and makes the side effect easy to miss in review.

Transfer failure makes a skill look good on one model or execution client while harming another.

Context inflation improves a narrow score while increasing latency and token cost on every matching run.

Registry poisoning lets an attacker make a malicious skill look relevant or trustworthy.

The right response is not to ban durable procedures. It is to treat them as behavior-bearing releases with scope, tests, provenance, and rollback.

Keep procedures at the skill layer when they recur

Use skill optimization when the trace shows the same procedure failure across tasks and the needed tools already exist. Use it when the procedure should survive a session boundary, transfer to nearby cases, or deploy without extra optimizer calls.

Use prompt optimization when the failure is local to wording or examples. Use memory when the missing input is a source-backed fact. Use runtime work when the procedure requires a new action, worker, budget, or stop condition. Use post-training when the behavior belongs in model parameters and the team can accept the stronger data and rollback obligations.

Skill optimization is the middle layer between a one-off instruction and a hard-coded application feature. Its test is not whether the file sounds wise. Its test is whether the right agent loads the right procedure, performs better on unseen work, stays quiet elsewhere, and can be rolled back when the environment changes.

Sources

What is skill optimization for AI agents?

Skill optimization trains a reusable procedure and its activation behavior while the target model stays fixed. The output is a versioned skill artifact that can guide future runs without requiring the optimizer to run at deployment time.

How is a skill different from a prompt?

A prompt usually controls the current context or run. A skill is a durable procedure that an agent can discover, load, version, transfer, disable, and evaluate across future tasks.

How is a skill different from memory?

Memory stores facts, observations, and prior evidence. A skill stores what procedure to follow when a task matches a defined situation. The agent may need both: current evidence from memory and a reliable sequence of actions from a skill.

What should a skill evaluation measure?

Measure task lift, activation precision, activation recall, near-miss interference, transfer, cost, security behavior, and rollback. A high score on relevant tasks is incomplete evidence when the skill triggers everywhere else. For the theory of comparing those changes under finite budgets, continue with optimization theory for agent builders.