A developer asks a coding agent to add a payment flow. The agent reads the provider docs, writes the client, runs a test, and reports success. The code compiles. The first real customer still gets the wrong subscription state because the API changed after the model learned the old example.
That is the problem with AI coding agent API integration. The agent must discover a live contract, turn it into a request, and produce the right side effect. Recall of an old tutorial is not enough.
The Agent Readiness Index measures this narrower ability. It is a benchmark of coding-agent configurations on current API-integration tasks. A benchmark is a fixed task set and scoring procedure used to compare runs. An evaluation is the check that decides whether a particular run satisfied its task. Tangle publishes the index as a dated view of these measured runs. It reports tested model configurations and their execution wrappers, not a guarantee about every Tangle product or every production integration.
This article explains the method, the public Stripe snapshot, and the limits of the score. It does not claim to measure general coding ability or every provider’s API.
The failure that looks like a provider problem
Stripe’s current documentation distinguishes pausing payment collection from pausing a subscription. The documented field named “pause_collection” leaves the subscription status unchanged, while the pause-subscription API endpoint can set the status to “paused”. The Stripe API reference and pause-subscription guide make that difference explicit.
A client written from memory can call the old-looking field, receive a successful HTTP response, and leave service delivery active when the product meant to pause it. The integration did not fail at compilation. It failed at the state the customer sees.
The pause endpoint is a separate POST /v1/subscriptions/{id}/pause operation.
The current guide says it requires Stripe API version 2025-06-30.preview or later.
Stripe also removed subscription-level “current_period_start” and “current_period_end” in the “2025-03-31.basil” release and moved billing periods to subscription items. The dated Stripe changelog entry documents the replacement. Code that reads the old field can lose the value it expects without a compile-time failure.
Those are useful benchmark tasks because the stale answer can look reasonable. The checker must inspect the request, response, and resulting state rather than trusting a status code.
Define the complete agent profile
A leaderboard row should not be called a model result when several moving parts produced it.
An agent profile is the complete configuration presented to a task. It can include the model, prompt or instruction set, available tools, and other declared behavior.
The harness is the program and execution wrapper that runs the agent loop. The execution environment is the process, filesystem, network, and tool boundary in which that loop runs. The model, profile, harness, and environment can each change the result.
The benchmark treats a row as a configuration, not as a timeless property of a model. Two rows can show the same model and harness while representing different profile variants. If the public board does not expose the changed profile axis, the reader cannot attribute the difference to one cause.
That restraint is part of the methodology. A benchmark should not imply a causal comparison that its rows do not support.
An evaluation record should make the profile boundary explicit. An illustrative public row might carry:
{
"model": "gpt-5-mini",
"harness": "pi",
"profile": "documented-api-search",
"tools": ["public-web", "filesystem", "test-runner"],
"environment": "isolated-node-runtime",
"tasks": 12,
"passes": 7
}The object is a methodology example, not a claim about fields currently exposed by the board. Its purpose is to show why the sample score of 58 percent is incomplete. The harness may change tool timing and retry behavior. The profile may change the instructions and available sources. The environment may change which files or network calls the agent can reach. The score belongs to the whole declared configuration.
Four design choices keep the measurement honest
The index is built around four choices.
| Choice | Measurement decision | Why it matters |
|---|---|---|
| Fresh contract | Tasks come from dated API changes and current documentation | Memory of old examples is not enough |
| Live discovery | The agent can search public sources during the run | The benchmark tests the source-search work required for integration |
| Hidden execution | The submitted code runs against a server that checks behavior | A prose answer or status code cannot pass alone |
| Calibration | Empty, current-reference, and stale solutions are tested before admission | A task must separate doing nothing, knowing the contract, and remembering the old contract |
The public Stripe benchmark board shows the task count, public rows, denominator, per-task heatmap, and run date for this dated snapshot. The benchmark task suite remains private, so the board does not expose task contents. That protects the hidden assertions, but it also limits independent reproduction. The benchmark’s mock server and full task assertions are private. The public page documents the calibration conditions and the behavior it reports, but it does not publish enough implementation detail to independently reproduce the mock. The board publishes no separate run ID or Stripe API-version value, so treat this as dated benchmark evidence rather than a live Stripe guarantee. The AI agent discovery guide covers the public documentation and manifest surfaces an agent can inspect before writing an integration.
A hidden execution test checks behavior
Each task gives the agent a goal and access to public documentation. The task does not hand over the exact request shape that the agent is expected to discover.
The submitted code is run against a mock server that implements the Stripe behavior recorded for the benchmark task. The mock is a test double for the provider’s API. It returns the request and response behavior needed to check the integration without sending the benchmark run to a live payment account.
A useful grader checks fields and side effects:
{
"request": {
"method": "POST",
"path": "/v1/subscriptions/sub_example/pause"
},
"expected": {
"status": "paused",
"service_delivery": "stopped",
"invoice_generation": "paused"
},
"result": "pass only when the state matches the task"
}The object above is an illustrative grader shape for the pause-subscription operation.
Updating /v1/subscriptions/{id} with pause_collection pauses payment collection while the subscription remains active; the separate pause operation produces the paused status.
The public benchmark does not publish its task assertions.
The principle is the important part: a client can receive status 200 and still fail if it changed the wrong resource state.
Calibration rejects easy or broken tasks
A task enters the suite only after three solutions are tested.
An empty solution must fail. This proves that the checker does not accept no-op behavior.
A current-contract reference must pass. This proves that the task can be solved from the public documentation and that the expected result is implementable.
A stale-memory solution must fail on the intended behavior. This proves that the task tests contract freshness rather than an unrelated syntax mistake.
The calibration matrix is:
| Candidate | Expected result | What it establishes |
|---|---|---|
| Empty implementation | Fail | The task requires work |
| Current documented implementation | Pass | The task is solvable |
| Deliberately stale implementation | Fail on the trap | The task separates current knowledge from memory |
Calibration is an admission rule, not a score bonus. If a task fails calibration, remove or repair it before comparing agent profiles.
The public Stripe snapshot
The public board snapshot used here was generated on July 25, 2026. It contains 12 tasks, but some rows have fewer completed attempts because the board preserves the real denominator. The table below includes every public row and its 95 percent interval.
| Public row | Model | Harness | Pass rate | Attempts | 95% interval |
|---|---|---|---|---|---|
| 1 | glm-5.1 | opencode | 91.7% | 12 | 64.6%–98.5% |
| 2 | glm-5.2 | opencode | 83.3% | 12 | 55.2%–95.3% |
| 3 | glm-5.2 | opencode | 66.7% | 12 | 39.1%–86.2% |
| 4 | glm-5.1 | opencode | 66.7% | 12 | 39.1%–86.2% |
| 5 | gpt-5-mini | pi | 60.0% | 10 | 31.3%–83.2% |
| 6 | gpt-5-mini | pi | 36.4% | 11 | 15.2%–64.6% |
| 7 | sonnet | claude-code | 41.7% | 12 | 19.3%–68.0% |
| 8 | sonnet | claude-code | 30.0% | 10 | 10.8%–60.3% |
The duplicate model and harness labels are deliberate. They identify distinct public rows, but the board does not expose enough profile detail to explain the difference between them. Do not turn this table into a model ranking.
The intervals are wide because the task count is small. The top two rows have overlapping uncertainty with several lower rows. A one-task difference on a 10- or 12-task set should start an investigation, not a marketing claim.
The board also shows an opaque per-task view. That view changes how you should read the average. It can show whether a row passes consistently or reaches its total through a mix of easy and hard tasks. It can also reveal missing coverage that needs a new run. Because the task prompts remain private, the view supports diagnosis of consistency and coverage but not independent reproduction of each task.
A percentage is a count with uncertainty
The denominator is part of the result. The first public row is 11 passes out of 12 attempts, which is 91.7 percent. The fifth row is 6 passes out of 10 attempts, which is 60.0 percent. Those are not equally informative observations, and neither is a permanent property of the model name in the first column.
The board’s 95% Wilson interval gives a range for the underlying pass rate under the benchmark’s sampling assumptions. It is wide because twelve tasks provide little room to distinguish close configurations. If one additional task changes a row from 10/12 to 11/12, the displayed percentage moves by 8.3 points even though the underlying behavior may not have changed in a durable way.
A reviewer should keep three quantities together:
observed result = passes / completed attempts
uncertainty = interval around that result
comparison = same tasks, same execution procedure, same profile definitionIf a row has fewer completed attempts, say so before comparing it. Do not fill a missing attempt with a guessed failure or silently compare a 10-task row with a 12-task row as if the denominators were identical. The public board preserves those differences because the missing run is data about measurement coverage.
The interval also does not correct for every bias. If the task set omits a provider feature, or if one profile gets a different tool boundary, the uncertainty around the pass rate can be narrow while the comparison is still unfair.
What a score means
A pass means the submitted integration produced the expected behavior for one task against the benchmark’s current contract. It does not mean the agent can build any integration.
The current suite measures one provider, one task set, one execution procedure, and the public run denominators. It does not measure:
- General software engineering or repository-scale maintenance.
- User-interface work or browser interaction.
- Production credentials, rate limits, or live payment risk.
- Every endpoint or version of the provider’s API.
- Whether a company’s documentation is easy for a human to understand.
- Whether an agent can recover after a task that was not included in the suite.
The score is also time-sensitive. Stripe’s API versioning documentation explains that major releases can contain backward-incompatible changes. A score from one release should not be treated as a permanent capability label.
Refresh the suite without changing the question
An API benchmark has two kinds of change. The provider’s contract changes, which is the behavior the suite is meant to detect. The benchmark’s own task or checker changes, which can make a new score incomparable with an old one.
When the provider changes, preserve the old task as a dated version if its failure mode is still useful. Add a new task for the new contract and calibrate it independently. Do not silently edit the assertion behind an existing task and call the resulting percentage a trend.
When the grader changes, publish the boundary of the change. It may be a bug fix that corrects a false pass, a new timeout that changes completed attempts, or an execution-environment change that gives agents a different tool path. Each can be legitimate, but the row needs a new run identity and an explanation before readers compare it with the previous snapshot.
The durable comparison is not “this month’s percentage minus last month’s percentage.” It is “the same calibrated task versions, same profile definition, same harness and environment, and same scoring rule produced this change.”
Keep the raw task outcomes behind the summary. A row with 8 passes out of 12 should retain which four tasks failed, whether the failure was a wrong request or a runtime error, and whether the task was completed after a retry. The average is the headline, but the failure shape tells an API team what to fix and tells an agent builder what capability is missing.
Protect the grading boundary
A hidden execution test is only informative if the agent cannot read the answer key from its environment. If the mock server exposes its source, assertions, or reference implementation to the agent, the run can become a file-discovery task instead of an API-integration task.
The safe design keeps the task assertions outside the agent workspace, records the requests and tool actions needed for audit, and rejects runs that cross the boundary. The benchmark owner should publish the boundary and the contamination policy even when the task text remains private.
This is also why a model judge is insufficient. A model can produce a persuasive explanation of an API call while the code sends the wrong field. Execution against a controlled current contract gives the pass condition a concrete meaning.
What an API company can do with a low result
A low score is useful when the per-task failure points to a documentation or API-design change.
Publish each breaking change in a dated, searchable page. Name the replacement field or endpoint in the error when doing so is safe. Prefer an error that tells an agent what to search for over a silent empty field. Keep the request and response examples aligned with the current API version. Then rerun the same calibrated task set after the documentation changes.
The company should also test whether a stale request fails loudly. A wrong field that returns 400 can trigger a search and retry. A removed field that returns an empty value can ship incorrect business behavior.
Decision rule
Use the Agent Readiness Index when you need to know whether coding agents can integrate a changing API from public sources. Read the task count, row identity, denominator, interval, and last-run date before comparing any two results.
Treat the score as a work list. Use per-task failures to improve docs and errors. Use fresh tasks to check whether the improvement survives the next API change.
API benchmark FAQ
Why are there duplicate model and harness labels?
The public rows represent distinct agent-profile configurations. The board exposes the model and harness but does not expose every changed profile axis. Readers should not infer a cause the public data does not show.
How should a company use a low score?
Inspect the per-task failure pattern, make the current contract discoverable, make silent failures loud where possible, and rerun the same calibrated tasks after the change.