← All benchmarks
Coding benchmark · by Tangle VerticalBench

PostHog Feature Flags v2

Two coding tasks that implement a client for PostHog flags evaluation API v2, where the memorized /decide endpoint 404s and the v2 response envelope no longer carries the fields a from-memory client reads.

Use this board to compare tested agent configurations on this exact task suite. It measures task execution against a hidden mock contract, not general model quality or production reliability.

Benchmark on GitHub ↗

How to read this benchmark

Each task is graded by a hidden mock server implementing PostHog current flags v2 contract. Pass means the agent client executes correctly against the mock, never a model judging its own work. Every task is calibrated three ways before it is admitted: an empty solution fails, a current-contract reference passes, and the stale-memory solution fails on the intended trap.

Last run
Held-out tasks
2
Configurations
8 agent configurations
Chart metric
Pass rate on a 0–100% scale

An agent profile is the exact instruction and tool configuration used for a run. A harness is the coding tool that gives a model its agent loop and tools; “direct” is a model call without that coding harness. On coding boards, “default reviewer” means the run used VerticalBench’s standard between-attempt review, while “no reviewer” means the coding harness ran once without that review. Hover a chart label to see its immutable profile ID. 95% Wilson interval for observed pass/fail outcomes; each configuration shows its own sample size. Small suites are narrow capability checks, not evidence that one model is generally better.

Read the broader agent-evaluation methodology →
95% CI
Pass rate, %
0
20
40
60
80
100
100.0
100.0
100.0
100.0
100.0
100.0
50.0
0.0
Claude Sonnetvia Claude CodeDefault reviewern=2
GLM-5.2via OpenCodeNo reviewern=2
GLM-5.1via OpenCodeDefault reviewern=2
Claude Sonnetvia Claude CodeNo reviewern=2
GLM-5.2via OpenCodeDefault reviewern=2
GLM-5.1via OpenCodeNo reviewern=2
GPT-5 minivia piNo reviewern=2
GPT-5 minivia piDefault reviewern=2

Same 2 held-out cases for every bar.

Per-task breakdown

2 held-out tasks · prompts private
Task Claude Sonnet via Claude Code Default reviewer GLM-5.2 via OpenCode No reviewer GLM-5.1 via OpenCode Default reviewer Claude Sonnet via Claude Code No reviewer GLM-5.2 via OpenCode Default reviewer GLM-5.1 via OpenCode No reviewer GPT-5 mini via pi No reviewer GPT-5 mini via pi Default reviewer
01 100 n=1 20249 tok · 335.1s 100 n=1 657979 tok · 256.7s 100 n=1 517671 tok · 177.7s 100 n=1 11561 tok · 182.2s 100 n=1 550744 tok · 207.2s 100 n=1 589603 tok · 201.4s 100 n=1 10298 tok · 249.1s 0 n=1 10879 tok · 136.7s
02 100 n=1 14420 tok · 253.0s 100 n=1 505468 tok · 211.6s 100 n=1 489707 tok · 135.6s 100 n=1 10403 tok · 173.9s 100 n=1 437602 tok · 184.3s 100 n=1 867960 tok · 264.8s 0 n=1 17161 tok · 196.4s 0 n=1 13381 tok · 208.0s

Each cell shows pass rate, total input-plus-output tokens, and mean wall time for that task and agent profile. Cost was not captured for these subscription-harness runs; that does not mean the runs were free. “Not run” means the result artifact has no recorded attempt for that task and profile. Task prompts stay private so they remain held out; opaque task numbers and every measured result are published.