Plausible Analytics v2 Query
Two coding tasks that query the Plausible Stats API v2 through its single POST endpoint and strict JSON query grammar. Clients that send legacy v1 GET requests fail these tasks.
Use this board to compare tested agent configurations on this exact task suite. It measures task execution against a hidden mock contract, not general model quality or production reliability.
How to read this benchmark
Each task is graded by a hidden mock server implementing Plausible current v2 query contract (required body keys, prefixed dimensions, filter trees, and date ranges); malformed and v1-style requests return structured errors and fail. Pass means the agent client executes correctly against the mock, never a model judging its own work. Every task is calibrated three ways before it is admitted: an empty solution fails, a current-contract reference passes, and the stale-memory solution fails on the intended trap.
- Last run
- Held-out tasks
- 2
- Configurations
- 8 agent configurations
- Chart metric
- Pass rate on a 0–100% scale
An agent profile is the exact instruction and tool configuration used for a run. A harness is the coding tool that gives a model its agent loop and tools; “direct” is a model call without that coding harness. On coding boards, “default reviewer” means the run used VerticalBench’s standard between-attempt review, while “no reviewer” means the coding harness ran once without that review. Hover a chart label to see its immutable profile ID. 95% Wilson interval for observed pass/fail outcomes; each configuration shows its own sample size. Small suites are narrow capability checks, not evidence that one model is generally better.
Read the broader agent-evaluation methodology →
Claude Sonnetvia Claude CodeDefault reviewern=2Claude Sonnetvia Claude CodeNo reviewern=2
GLM-5.2via OpenCodeDefault reviewern=2GLM-5.2via OpenCodeNo reviewern=2GLM-5.1via OpenCodeNo reviewern=2GLM-5.1via OpenCodeDefault reviewern=2
GPT-5 minivia piDefault reviewern=2GPT-5 minivia piNo reviewern=1Per-task breakdown
2 held-out tasks · prompts private| Task | Claude Sonnet via Claude Code Default reviewer | Claude Sonnet via Claude Code No reviewer | GLM-5.2 via OpenCode Default reviewer | GLM-5.2 via OpenCode No reviewer | GLM-5.1 via OpenCode No reviewer | GLM-5.1 via OpenCode Default reviewer | GPT-5 mini via pi Default reviewer | GPT-5 mini via pi No reviewer |
|---|---|---|---|---|---|---|---|---|
| 01 | 100 n=1 9697 tok · 162.0s | 100 n=1 10592 tok · 160.1s | 100 n=1 457432 tok · 298.0s | 100 n=1 403075 tok · 263.6s | 100 n=1 249711 tok · 106.2s | 100 n=1 429016 tok · 207.7s | 0 n=1 11334 tok · 168.1s | 0 n=1 14412 tok · 167.8s |
| 02 | 100 n=1 12766 tok · 196.7s | 100 n=1 11782 tok · 194.1s | 100 n=1 583507 tok · 259.4s | 100 n=1 664055 tok · 249.3s | 100 n=1 741789 tok · 298.8s | 100 n=1 422724 tok · 196.7s | 0 n=1 35935 tok · 312.7s | not run |
Each cell shows pass rate, total input-plus-output tokens, and mean wall time for that task and agent profile. Cost was not captured for these subscription-harness runs; that does not mean the runs were free. “Not run” means the result artifact has no recorded attempt for that task and profile. Task prompts stay private so they remain held out; opaque task numbers and every measured result are published.