BlogBrowser AgentPart 1 of 2
AI E2E Tests That Pass When They Shouldn't: Catching False Passes
A false pass is an AI test that reports success the page does not show. We measured 6 in 15 browser-agent runs, fixed the cause, and reran: 0 in 15.
We asked an AI browser agent to buy a backpack and a bike light on a demo store. In 5 of 5 runs the cart ended up holding the backpack and a T-shirt. The agent reported that it had added the backpack and the bike light, and in 4 of the 5 runs its own verifier marked the test passed.
That is the failure that matters in AI end-to-end (E2E) testing. A test that fails costs a rerun. A test that passes when the page says otherwise ships the bug, and it teaches your team to trust the green. We call it a false pass: the agent reports success and the final page state does not show it.
The short answer: let the page decide the pass, not the agent. Check the final state (the URL, the cart contents, the list on screen) with something that does not read the agent’s summary, and count false passes separately from failures. That is how we measured the bug above in our open-source agent, Tangle Browser Agent (bad), and how we know the fix worked: 6 false passes in 15 runs on version 0.37.0, and 0 in 15 on 0.38.3 (issue #177).
How an agent passes a test it failed
The checkout goal was plain:
Log in with username standard_user and password secret_sauce. Add ‘Sauce Labs Backpack’ and ‘Sauce Labs Bike Light’ to the cart. Check out with first name Ada, last name Lovelace, postal code 10001. On the overview page read the Item total. Then press Finish to place the order.
Two separate faults turned a wrong cart into a green result.
The agent acted on a page that had already changed. It planned two clicks from one snapshot of the page: the first and second “Add to cart” buttons. After the first click, the backpack’s button turned into “Remove”, every remaining “Add to cart” button moved up one position, and the second click landed on the T-shirt. The agent did not read the page again between the two clicks.
The verifier graded the agent’s story instead of the page. The agent’s final answer named the items it meant to add and gave an item total of $45.98. The cart it asked for totals $39.98. The goal verifier, running on the same model, accepted the summary in 4 of 5 runs. In 2 of those 4 it also accepted “The order has been placed” from a run that never pressed Finish.
A to-do test showed the same pattern: in 2 of 5 runs the agent also completed an item it was told to leave open, described the result as correct, and the verifier agreed.
Neither fault is exotic. Any agent that acts on a cached view of the page can click a shifted element, and any check that reads the agent’s own account inherits the agent’s mistakes.
How we measured it
The 15 runs cover three public demo sites, 5 repetitions each, on gemini-2.5-flash:
- login: log in on the-internet.herokuapp.com/login and stop on the secure area.
- checkout: the Sauce Demo goal above.
- todo: on TodoMVC, add three items, complete the second, and show only the Active list.
Each run starts a fresh Chrome. A separate watcher attaches to the same Chrome through its DevTools port and records the final state the harness cares about: the URL, the cart’s item names, the to-do items on screen and in storage. The watcher never reads the agent’s output. The agent’s verdict comes from its own report. A run is a pass when the watcher’s check passes, and a false pass when the agent reported success and the watcher’s check failed.
| Flow | 0.37.0: passes / false passes | 0.38.3: passes / false passes |
|---|---|---|
| login | 5 of 5 / 0 | 5 of 5 / 0 |
| checkout | 0 of 5 / 4 | 5 of 5 / 0 |
| todo | 3 of 5 / 2 | 4 of 5 / 0 |
| total | 8 of 15 / 6 | 14 of 15 / 0 |
Same harness, same goals, same model, 15 runs per version. Source: the 0.38.2 and 0.38.3 results on issue #177. The runs used different sandbox machines, so the wall times in the records are not compared. Fifteen runs is a small sample: 0 false passes in 15 is still consistent with a true rate of up to about 18% (one-sided 95% bound), so treat it as “the bug we found is gone”, not as a guarantee.
The one 0.38.3 failure is worth reading. On a to-do run the verifier found every requirement met at turn 6, but two of its quotes did not match the page text verbatim, so the result was rejected. The runner rolled back to an earlier URL, TodoMVC’s local storage kept the items, the agent added them again, and the run hit its 30-turn limit. The agent reported success: false. That is a failure you see, not a false pass, and it points at the next fix.
What fixed it
The fixes are in the release history, each tied to #177:
- Clicks bind to the element the agent observed (#178). A batched click that would land on a different element stops the batch instead.
- The verifier judges from page evidence, not the agent’s claim (#178), and every quote it relies on must appear in the page evidence it cites (#182). A live login run had confirmed “Logged in as tomsmith” by quoting the agent’s own extraction, not the page.
- The verifier’s raw verdict is kept in the report (#184), so a rejected pass can be explained after the run.
The general lesson does not depend on our agent: a pass check that can read the agent’s summary will sometimes believe it.
Make the page decide in your own suite
Write the final condition as page state
“The invite worked” cannot be checked. “The members page shows [email protected] with status Pending” can: it gives a location, a string and a state. For checkout, check the item names in the cart, not the total the agent read out.
Know what your tool counts as a pass
With bad, a suite runs from a cases file:
bad run --cases ./cases.json --concurrency 4
The command exits with code 1 when any case fails, so a CI step fails with it. Shared defaults such as the provider, model, headless mode and turn limit go in a bad.config.ts file at the project root. But a case from a JSON file passes on the agent’s verified result: with no explicit success criteria, the test runner defaults to the agent’s outcome. Since the #177 fixes that verifier reads page evidence, but it is still a model.
To make deterministic checks part of the pass, use the SDK’s TestRunner with successCriteria. A case then passes only when the agent reports success and every criterion holds on the live page. The criterion types are url-contains, url-matches, element-visible, element-text, element-count and custom:
import { chromium } from 'playwright'
import { PlaywrightDriver, TestRunner } from '@tangle-network/browser-agent-driver'
const browser = await chromium.launch()
const page = await browser.newPage()
const runner = new TestRunner({
driver: new PlaywrightDriver(page),
config: { provider: 'google', model: 'gemini-2.5-flash' },
})
const result = await runner.runTest({
id: 'checkout',
name: 'Backpack and bike light reach the overview page',
startUrl: 'https://www.saucedemo.com/',
goal: 'Log in as standard_user / secret_sauce, add Sauce Labs Backpack and Sauce Labs Bike Light, check out as Ada Lovelace 10001, and stop on the overview page.',
successCriteria: [
{ type: 'url-contains', value: '/checkout-step-two.html' },
{ type: 'element-count', selector: '.cart_item', value: '2' },
{ type: 'element-text', selector: '.cart_list', value: 'Sauce Labs Bike Light' },
],
})
console.log(result.verified, result.verdict)
await browser.close()
This sketch is typed against the 0.38.3 source (TestCase); we did not run this file for this post. The 15-run measurement above used a separate watcher instead, which works with any agent: attach to the browser after the run and assert on its state.
Count false passes, and repeat each case
Report three numbers per case: passes, failures and false passes. A false pass needs your independent check to exist, which is one more reason to write it.
One run is not a sample. We ran each flow 5 times, and the bug showed in 4 of 5 checkout runs, not in all of them. If a model grades your runs, expect it to disagree with people: Browser Use reports 87% alignment with human labels for its gemini-2.5-flash judge, which is good and still wrong about one verdict in eight. A deterministic end-state check on an app you control does not have that error rate.
Keep the evidence a reviewer needs
Save the screenshots, the actions, the final page state and the verifier’s verdict with every run. When the agent says passed and your check says failed, the trace shows which one was looking at the real page.
The same goes for the setup. In the recorded run below, a coding agent was asked to install Playwright and screenshot a page inside an isolated sandbox. The egress allowlist named the Playwright download host, but the download redirected to storage.googleapis.com, which was not listed, and was refused with HTTP 403. The agent switched to the machine’s own Chromium with --no-sandbox and a custom library path, wrote both files, and returned success: true. Its summary mentioned the workaround; the flag did not. Read the tool trace, not only the success flag, and allowlist every host an install reaches, redirects included.
import { Sandbox } from "@tangle-network/sandbox";
const client = new Sandbox({
apiKey: process.env.TANGLE_API_KEY!,
baseUrl: "https://sandbox.tangle.tools",
});
// The agent installs and drives a real browser inside its own machine.
const box = await client.create({
backend: { type: "claude-code" },
resources: { cpuCores: 2, memoryMB: 4096 },
// The default allowlist, plus the browser download and the one site it reads.
egressPolicy: {
mode: "strict",
includeImplicitDomains: true,
allowDomains: ["cdn.playwright.dev", "playwright.download.prss.microsoft.com", "news.ycombinator.com"],
},
});
try {
const run = await box.prompt(
"Install Playwright with Chromium. Open https://news.ycombinator.com, " +
"save a full page screenshot to /home/agent/front.png, and write the five " +
"top story titles to /home/agent/top.json as a JSON array of strings.",
);
if (!run.success) throw new Error(run.error);
console.log(JSON.parse(await box.read("/home/agent/top.json")));
await box.fs.download("/home/agent/front.png", "front.png");
} finally {
await box.delete();
}Where agent tests fit next to Playwright
Keep deterministic Playwright tests for stable paths with known selectors. They are faster, cheaper and repeatable. Slack’s engineering team ran more than 200 agent-driven E2E runs and put it well: “Tests enforce journeys. Agents verify goals.” They also report agent runs at $15–30 each in their setup, which places agent tests in exploratory and goal-level checks rather than every commit.
An agent earns its place when the path changes often (copy, layout, conditional steps) and the outcome does not. It should still hand the verdict to a check like the ones above.
What a run costs depends on two meters: browser time, if the browser is hosted, and model tokens per step. Our 15 runs on 0.38.3 used 10,127 to 149,462 input tokens each (median 39,155); multiply by your model’s price per token. On the maintainers’ benchmark the model spend came to about $0.022 per run. With --replay and a persisted memory folder, bad replays a route that already passed, and its CI recipe puts those reruns at about 10–20% of a live run’s model cost; a replayed step that no longer fits the page falls back to the live agent. Hosted browsers bill by the hour: see Browserbase pricing and Browser Use pricing for the current rates. If you run the browser and agent on a Tangle Sandbox machine instead, compute is billed per core-hour and per GB-hour of memory with a minimum hourly charge; the rate card has the current numbers.
Choosing the browser and the agent
Two layers are easy to confuse:
- Browser infrastructure runs the browser. Browserbase hosts browsers billed by browser time; you bring the agent or use its Stagehand library. Amazon’s Bedrock AgentCore Browser hosts browser sessions and adds the
InvokeBrowserAPI for operating-system-level actions, such as native dialogs, that the page’s DevTools connection cannot reach. - The agent decides what to click and when the goal is met. Browser Use is an open-source agent library that also sells cloud browsers. Tangle Browser Agent is an agent that drives any browser through a driver interface: Playwright locally, or a Steel cloud browser.
The false-pass problem lives in the agent layer, so it follows you to whichever browser host you pick.
Our side-by-side comparison is in Tangle Browser Agent vs Browserbase and Browser Use. Vendor benchmark scores, ours included, come from small or public corpora. The bad README reports 15 of 15 against Browser Use’s 12 of 15 on the maintainers’ own 5-task, 3-repetition corpus (2026-10-11), at $0.0223 against $0.0126 in model spend per run, about 1.8 times. It is a small corpus run by the maintainers, so rerun it before you rely on it. Published WebVoyager scores cover 15 public sites. Neither predicts your app: run your own flows with an independent check, as above.
Public sites can also block automated browsers, especially from cloud IP ranges, so a benchmark run from a laptop can look better than the same agent in CI. For public sites, bad can launch system Chrome with fingerprint hardening, solves common CAPTCHAs by default, takes a proxy through BAD_PROXY_URL, or runs on a Steel cloud browser with residential proxies. For your own app, allow the test runner through your bot protection rather than relying on any of that.
Try it
bad is open source under MIT or Apache-2.0 and needs Node.js 22.12 or later:
npm install -g @tangle-network/browser-agent-driver
npx playwright install chromium
bad run --goal "Log in with username tomsmith and password SuperSecretPassword! and stop on the secure area" \
--url https://the-internet.herokuapp.com/login
Do not install the unscoped bad package from npm; it is unrelated. The agent reads OPENAI_API_KEY, ANTHROPIC_API_KEY or GOOGLE_GENERATIVE_AI_API_KEY. With no provider set it uses OpenAI when that key is present and otherwise the local claude-code provider; our measurements used gemini-2.5-flash, the model the maintainers benchmark on. The planner, executor and verifier roles can each use a different model. It reads the page’s accessibility tree by default, which is the fastest mode; --observation-mode vision works from labelled screenshots, and hybrid uses both, which the maintainers list as the most reliable. With the planner on, it plans the steps once and falls back to a step-by-step observe-and-act loop only when the page deviates.
Whatever agent you choose, run your three most important journeys five times each against a check the agent cannot read, and count the false passes before you let the result block a release.
Next in Browser Agent: AI Browser Automation: Verify Each Step Before the Next