Blog

AI Coding Agent API Integration: Measuring Agent Readiness

Measure AI coding agent API integration with current-contract tasks, hidden execution checks, calibration, and denominators.

A developer asks a coding agent to add a payment flow. The agent reads the provider docs, writes the client, runs a test, and reports success. The code compiles. The first real customer still gets the wrong subscription state because the API changed after the model learned the old example.

That is the problem with AI coding agent API integration. The agent must discover a live contract, turn it into a request, and produce the right side effect. Recalling an old tutorial is not enough.

The Agent Readiness Index measures this narrower ability: a benchmark of coding-agent configurations on current API-integration tasks. A result reports tested model configurations and their execution wrappers, not a guarantee about every Tangle product or every production integration.

In Tangle’s Sandbox SDK, client.create starts a machine with a claude-code backend, box.prompt returns the turn with success, usage and costUsd, and box.exec runs the check in that same machine. The reader sees the check’s exit code, not the agent’s summary.

The failure that looks like a provider problem

Stripe’s current documentation distinguishes pausing payment collection from pausing a subscription. The documented field named “pause_collection” leaves the subscription status unchanged, while the pause-subscription API endpoint can set the status to “paused”. The Stripe API reference and pause-subscription guide make that difference explicit.

A client written from memory can call the old-looking field, receive a successful HTTP response, and leave service delivery active when the product meant to pause it. The integration did not fail at compilation. It failed at the state the customer sees.

The pause endpoint is a separate POST /v1/subscriptions/{id}/pause operation. The current guide says it requires Stripe API version 2025-06-30.preview or later.

Stripe also removed subscription-level “current_period_start” and “current_period_end” in the “2025-03-31.basil” release and moved billing periods to subscription items. The dated Stripe changelog entry documents the replacement. Code that reads the old field can lose the value it expects without a compile-time failure.

Those are useful benchmark tasks because the stale answer can look reasonable. The checker must inspect the request, response, and resulting state rather than trusting a status code.

Define the complete agent profile

A leaderboard row is not a model result when several moving parts produced it.

An agent profile is the complete configuration presented to a task. It can include the model, prompt or instruction set, available tools, and other declared behavior.

The harness is the program and execution wrapper that runs the agent loop. The execution environment is the process, filesystem, network, and tool boundary in which that loop runs. The model, profile, harness, and environment can each change the result.

The benchmark treats a row as a configuration, not as a timeless property of a model. Two rows can show the same model and harness while representing different profile variants. If a published row does not expose the changed profile axis, the reader cannot attribute the difference to one cause.

An evaluation record should make the profile boundary explicit. An illustrative row:

{
  "model": "gpt-5.4-mini",
  "harness": "pi",
  "profile": "documented-api-search",
  "tools": ["public-web", "filesystem", "test-runner"],
  "environment": "isolated-node-runtime",
  "tasks": 12,
  "passes": 7
}

The object is a methodology example, not a published result. Its purpose is to show why the sample score of 58 percent is incomplete. The harness may change tool timing and retry behavior. The profile may change the instructions and available sources. The environment may change which files or network calls the agent can reach.

Four design choices keep the measurement honest

Choice Measurement decision Why it matters
Fresh contract Tasks come from dated API changes and current documentation Memory of old examples is not enough
Live discovery The agent can search public sources during the run The benchmark tests the source-search work required for integration
Hidden execution The submitted code runs against a server that checks behavior A prose answer or status code cannot pass alone
Calibration Empty, current-reference, and stale solutions are tested before admission A task must separate doing nothing, knowing the contract, and remembering the old contract

The benchmark task suite, mock server, and full task assertions remain private. That protects the hidden assertions, but it limits independent reproduction. The AI agent discovery guide covers the public surfaces an agent can inspect first.

A hidden execution test checks behavior

Each task gives the agent a goal and access to public documentation. The task does not hand over the exact request shape that the agent is expected to discover.

The fix-a-test example below runs the caller’s own test command after the agent reports a fix, and fails the run on a nonzero exit code.

Check the agent's claim with a command the caller runsRan on production 2026-11-12 from these bytes, exit 0, $0.0145 reported usage, @tangle-network/sandbox 1.0.168. View result.
import { writeFile } from "node:fs/promises";
import { Sandbox } from "@tangle-network/sandbox";

const client = new Sandbox({
  apiKey: process.env.TANGLE_API_KEY!,
  baseUrl: "https://sandbox.tangle.tools",
});

const box = await client.create({
  git: { url: "https://github.com/tangle-network/sandbox-demo" },
  backend: { type: "claude-code" }, // or "codex", "opencode", "pi"
});
try {
  const run = await box.prompt("`npm test` fails. Find the cause and fix it.");
  if (!run.success) throw new Error(run.error);

  // Check the result yourself.
  const tests = await box.exec("npm test");
  if (tests.exitCode !== 0) throw new Error("The tests still fail.");

  await writeFile("fix.patch", (await box.exec("git diff")).stdout);
  console.log(run.usage, run.costUsd); // metered per turn
} finally {
  await box.delete();
}

It proves the fix passed the caller’s test command, not that the fix is right beyond that test.

The submitted code runs against a mock server that implements the Stripe behavior recorded for the benchmark task. The mock returns the request and response behavior needed to check the integration without sending the benchmark run to a live payment account.

A useful grader checks fields and side effects:

{
  "request": {
    "method": "POST",
    "path": "/v1/subscriptions/sub_example/pause"
  },
  "expected": {
    "status": "paused",
    "service_delivery": "stopped",
    "invoice_generation": "paused"
  },
  "result": "pass only when the state matches the task"
}

The grader above is illustrative. Updating /v1/subscriptions/{id} with pause_collection pauses payment collection while the subscription remains active; the separate pause operation produces the paused status. The benchmark does not publish its task assertions. A client can receive status 200 and still fail if it changed the wrong resource state.

Calibration rejects easy or broken tasks

A task enters the suite only after three solutions are tested.

An empty solution must fail.

A current-contract reference must pass.

A stale-memory solution must fail on the intended behavior.

The calibration matrix is:

Candidate Expected result What it establishes
Empty implementation Fail The task requires work
Current documented implementation Pass The task is solvable
Deliberately stale implementation Fail on the trap The task separates current knowledge from memory

Calibration is an admission rule, not a score bonus. If a task fails calibration, remove or repair it before comparing agent profiles.

A percentage is a count with uncertainty

The denominator is part of the result. A 95% Wilson interval gives a range for the underlying pass rate under the benchmark’s sampling assumptions. On a small task set the interval is wide, so close configurations are hard to distinguish. If one additional task changes a row from 10/12 to 11/12, the displayed percentage moves by 8.3 points even though the underlying behavior may not have changed in a durable way.

A reviewer should keep three quantities together:

observed result = passes / completed attempts
uncertainty = interval around that result
comparison = same tasks, same execution procedure, same profile definition

If a row has fewer completed attempts, say so before comparing it. Do not fill a missing attempt with a guessed failure or silently compare a 10-task row with a 12-task row as if the denominators were identical.

The interval also does not correct for every bias. If the task set omits a provider feature, or if one profile gets a different tool boundary, the uncertainty around the pass rate can be narrow while the comparison is still unfair.

What a score means

A pass means the submitted integration produced the expected behavior for one task against the benchmark’s current contract. It does not mean the agent can build any integration.

The suite measures one provider, one task set, and one execution procedure. It does not measure:

  • General software engineering or repository-scale maintenance.
  • User-interface work or browser interaction.
  • Production credentials, rate limits, or live payment risk.
  • Every endpoint or version of the provider’s API.
  • Whether a company’s documentation is easy for a human to understand.
  • Whether an agent can recover after a task that was not included in the suite.

The score is also time-sensitive. Stripe’s API versioning documentation explains that major releases can contain backward-incompatible changes. A score from one release should not be treated as a permanent capability label.

Refresh the suite without changing the question

An API benchmark has two kinds of change: the provider’s contract changes, which the suite is meant to detect, and the benchmark’s own task or checker changes, which can make a new score incomparable with an old one.

When the provider changes, preserve the old task as a dated version if its failure mode is still useful. Add a new task for the new contract and calibrate it independently. Do not silently edit the assertion behind an existing task and call the resulting percentage a trend.

When the grader changes, publish the boundary of the change. It may be a bug fix that corrects a false pass, a new timeout that changes completed attempts, or an execution-environment change that gives agents a different tool path. Each can be legitimate, but the row needs a new run identity and an explanation before readers compare it with the previous snapshot.

The durable comparison is not “this month’s percentage minus last month’s percentage.” It is “the same calibrated task versions, same profile definition, same harness and environment, and same scoring rule produced this change.”

Keep the raw task outcomes behind the summary. A row with 8 passes out of 12 should retain which four tasks failed, whether the failure was a wrong request or a runtime error, and whether the task was completed after a retry. The average is the headline; the failure shape says what to fix.

Protect the grading boundary

A hidden execution test is only informative if the agent cannot read the answer key from its environment. If the mock server exposes its source, assertions, or reference implementation to the agent, the run can become a file-discovery task instead of an API-integration task.

The safe design keeps the task assertions outside the agent workspace, records the requests and tool actions needed for audit, and rejects runs that cross the boundary. The benchmark owner should publish the boundary and the contamination policy even when the task text remains private.

A model judge cannot decide a pass. A trace judge such as evals/score-a-trace can score whether the agent read the current docs and reran its checks, but the pass condition still comes from execution against the controlled contract. A model can write a persuasive explanation of an API call while the code sends the wrong field.

What an API company can do with a low result

A low score is useful when the per-task failure points to a documentation or API-design change.

Publish each breaking change in a dated, searchable page. Name the replacement field or endpoint in the error when doing so is safe. Prefer an error that tells an agent what to search for over a silent empty field. Keep the request and response examples aligned with the current API version. Then rerun the same calibrated task set after the documentation changes.

The company should also test whether a stale request fails loudly. A wrong field that returns 400 can trigger a search and retry. A removed field that returns an empty value can ship incorrect business behavior.

Decision rule

Use this method when you need to know whether coding agents can integrate a changing API from public sources. Read the task count, row identity, denominator, interval, and last-run date before comparing any two results.

Treat the score as a work list. Use per-task failures to improve docs and errors. Use fresh tasks to check whether the improvement survives the next API change.

API benchmark FAQ

How should a company use a low score?

Inspect the per-task failure pattern, make the current contract discoverable, make silent failures loud where possible, and rerun the same calibrated tasks after the change.