BlogBrowser Agentpart 1 of 2
AI E2E Testing: Check the Final State of a Whole Journey
Follow a full browser journey across routes and backend state, using controlled test data and a final condition that proves the user's task finished.
A customer does not experience a unit test, an application programming interface (API) response, and a browser page as separate systems. They see a signup form, an invitation, a dashboard, and an email or status message that tells them whether the job finished.
AI E2E testing targets that complete journey. E2E means end-to-end: the test starts at a user-facing entry point, crosses the boundaries the user crosses, and checks the final state the user can see. An AI E2E test adds a natural-language goal and page interpretation to browser controls, while keeping enough evidence to fix a failure.
Tangle Browser Agent is Tangle’s browser driver for these goal-based sessions, with screenshots and DOM (the page’s element tree) evidence.
Its bad command starts a run from a terminal.
In the sandboxed example in this article, client.create starts a Tangle Sandbox machine, box.prompt returns a run whose success and error the test checks, and box.read and box.fs.download retrieve what the agent wrote.
A full journey has more than one assertion
Consider a workspace invitation. The product promise is that an owner can create a workspace, invite a teammate, and see the invitation in the workspace settings.
An incomplete browser case clicks “Invite” and stops when a toast appears. A useful E2E case follows the user-visible journey:
- Open the application’s sign-in page with a disposable account.
- Create a workspace named “Smoke”.
- Open the members screen.
- Invite a test address.
- Confirm the members screen shows the pending invitation.
- Stop before sending mail to a real person.
The browser has to see the route changes, form validation, loading states, and final list. The backend has to accept the request. The final page has to represent the result accurately.
The goal can stay in ordinary language:
bad run \
--url https://app.example.com \
--goal "Sign in with the disposable account, create a project named Smoke, invite the test address, and verify the invitation appears in the members list"
This command follows the public bad run shape documented in the Browser Agent driver repository. The web address, account, and test address are illustrative. Replace them with a disposable environment that your team controls.
The E2E bar
The test case has to make five things visible.
| Requirement | Weak case | Reviewable case |
|---|---|---|
| input | “Test the app” | One named journey and start URL (web address) |
| starting state | Hidden account and mutable data | Disposable account, known data, and documented permissions |
| browser | Mocked page or a single isolated component | Real browser session crossing the user path |
| final condition | Model says the flow looks complete | Visible list, URL, state, or response that satisfies the goal |
| evidence | One summary sentence | Screenshots, page observations, actions, and stop reason |
The final condition must be observable. “The invite worked” is weaker than “the members screen shows [email protected] with a Pending status,” which gives the test a string, a location, and a state to check.
An evaluation compares this expected condition with the observed result. It can use a selector, an API response, a screenshot, or several pieces of evidence. The agent’s explanation can help a reviewer, but it should not replace the condition being checked.
Put AI E2E testing in the right layer
| Layer | What it establishes | Typical owner |
|---|---|---|
| unit test | A function or component obeys a local invariant | Developer |
| API or contract test | A request, permission, and response shape agree | Service team |
| deterministic browser test | A stable UI path behaves under known selectors | QA or developer |
| AI E2E test | A changing user journey reaches a visible outcome | Product and QA |
| manual review | A new or high-risk experience deserves human judgment | Release owner |
Playwright is a good fit for deterministic browser cases. Its current documentation describes isolation, assertions, parallelization, and reports for modern web apps. The WebDriver standard defines the broader remote-control interface used by browser automation tools.
An AI case earns its place when page structure, copy, extensions, or conditional state make a fixed selector path expensive to maintain. It should still hand its result to deterministic checks where a known invariant matters. For example, an agent can navigate a billing page while a typed assertion checks that the displayed plan identifier matches the expected test plan.
Write a case a reviewer can rerun
A case needs enough context to explain whether a failure came from the application, the fixture, the browser, the network, or the agent.
{
"name": "workspace-invite-smoke",
"startUrl": "https://app.example.com",
"startingState": {
"account": "disposable owner account",
"workspace": "new workspace",
"invitedAddress": "[email protected]"
},
"goal": "Create Smoke, invite the test address, and verify the pending invitation appears",
"allowedActions": [
"navigate",
"fill forms",
"submit the workspace and invitation forms"
],
"stopBefore": [
"sending mail to a real address",
"changing a paid plan",
"deleting a workspace"
],
"evidence": [
"workspace form",
"members list",
"invitation status",
"first failure or stop reason"
]
}
This object is an illustrative case definition. The public driver accepts a cases file through bad run –cases, but the exact case schema belongs to the installed version and should be checked with its documentation.
The case should also record the browser version, viewport, observation mode, model or agent profile, and fixture reset method. An agent profile is a named bundle of model choice, page-observation mode, permissions, time limits, and recovery rules. Two runs with different profiles may take different paths even when the goal is identical.
Keep fixtures from becoming the bug
End-to-end tests cross enough services that setup failures can dominate the result.
| Fixture | Control to establish before the browser opens |
|---|---|
| account | disposable identity with the required role and no sensitive data |
| data | stable names, known empty states, and an explicit reset |
| browser | fixed browser family, viewport, locale, and extension list |
| network | reachable application and dependency health |
| time | deterministic clock or a tolerance for asynchronous states |
| credentials | least privilege, short-lived secrets, and no production access |
A health check can show that the application is reachable. It cannot prove the journey will succeed. A seeded database can make the run repeatable. It cannot prove the product handles a real empty state. Keep both the fixture description and the observed setup in the trace.
The runtime is where the browser, files, and model calls execute: a developer machine, a continuous-integration (CI) worker, or a sandbox.
The example below uses a sandbox. The agent is asked to install Playwright and Chromium itself, egress allows only the browser download hosts and one site, and the machine has 2 CPU cores and 4 GB of memory.
The test checks the files the run wrote, not only the run’s success flag or the agent’s description of them.
import { Sandbox } from "@tangle-network/sandbox";
const client = new Sandbox({
apiKey: process.env.TANGLE_API_KEY!,
baseUrl: "https://sandbox.tangle.tools",
});
// The agent installs and drives a real browser inside its own machine.
const box = await client.create({
backend: { type: "claude-code" },
resources: { cpuCores: 2, memoryMB: 4096 },
// The default allowlist, plus the browser download and the one site it reads.
egressPolicy: {
mode: "strict",
includeImplicitDomains: true,
allowDomains: ["cdn.playwright.dev", "playwright.download.prss.microsoft.com", "news.ycombinator.com"],
},
});
try {
const run = await box.prompt(
"Install Playwright with Chromium. Open https://news.ycombinator.com, " +
"save a full page screenshot to /home/agent/front.png, and write the five " +
"top story titles to /home/agent/top.json as a JSON array of strings.",
);
if (!run.success) throw new Error(run.error);
console.log(JSON.parse(await box.read("/home/agent/top.json")));
await box.fs.download("/home/agent/front.png", "front.png");
} finally {
await box.delete();
}In the recorded run the egress policy refused the Chromium download (HTTP 403), so the agent switched to the machine’s own Chromium with --no-sandbox and a custom LD_LIBRARY_PATH, loaded the live page, wrote both files, and still reported success: true.
The run proves that a sandboxed agent loaded a live page and wrote the requested files; the flag alone did not show that the setup had changed, so read the tool trace as well.
It does not prove that the same steps succeed behind a login, on another day, or with a different network policy.
The runtime sets network access, browser version, secret exposure, and reproducibility, so record it with the fixture.
Turn a run into a release decision
A release rule blocks or permits a change based on a test result. An AI E2E rule should be small enough to inspect and important enough to matter.
| Rule | Final condition | Safe failure response |
|---|---|---|
| signup | new account reaches the dashboard | keep the account disposable and attach the final screenshot |
| activation | first project or integration is visible | record the first missing state and the preceding click |
| billing | test plan and billing state agree | stop before real payment and record the review page |
| wallet | selected account and chain appear in the application | stop at signing unless test-wallet permission is explicit |
| support path | the reported customer journey reaches the expected page | preserve the input class and redact sensitive values |
Narrow cases are easier to own than many goals whose final conditions all mean “looks okay.” Choose the smallest set that covers the release risk, and review every case’s evidence.
A passing rule means the specified condition was observed under the specified fixture and runtime. It does not mean every path through the product is healthy, or that the model will follow the same route after a copy change.
Trace the first wrong step
A trace is the ordered record of goal, observation, action, result, recovery, final check, and stop reason. The first wrong step matters more than the final error because later actions may be consequences of the earlier mistake.
| Turn | Observation | Action | Result | Classification |
|---|---|---|---|---|
| 1 | Sign-in form is visible | Fill disposable credentials | Dashboard loads | expected |
| 2 | Empty workspace list is visible | Click “Create project” | Creation form loads | expected |
| 3 | Name field accepts “Smoke” | Submit form | Spinner remains for 30 seconds in this sample | environment or product |
| 4 | Error banner says service unavailable | Retry once within policy | Same error remains | failed |
| 5 | Final condition absent | Stop and save screenshot | fail with service error | actionable |
The last row is the evaluation result. The earlier rows show whether engineering should inspect the page, the backend, the fixture, or the environment. “Project creation failed” alone forces a rerun to recover that context.
A model judge can grade the same trace, but it grades against the condition the case states, not against the agent’s own summary. The example below scores a code-review run’s answer and every tool call on two dimensions.
import { Sandbox } from "@tangle-network/sandbox";
import { createChatClient, llmJudge } from "@tangle-network/agent-eval/contract";
const apiKey = process.env.TANGLE_API_KEY!;
const client = new Sandbox({ apiKey, baseUrl: "https://sandbox.tangle.tools" });
// A model judge on the Router reads the answer and every tool call.
const judge = llmJudge("trace-review", "Score this agent run against its task.", {
chat: createChatClient({
transport: "openai-compatible",
baseUrl: "https://router.tangle.tools/v1",
apiKey,
defaultModel: "glm-5.3",
}),
dimensions: [
{ key: "grounded", description: "Every file and line it names exists" },
{ key: "economical", description: "No tool call repeats earlier work" },
],
});
const task = "List each place the API validates a request body, as file:line.";
const box = await client.create({
git: { url: "https://github.com/tangle-network/sandbox-demo" },
backend: { type: "claude-code" },
});
try {
const run = await box.prompt(task);
const score = await judge.score({
artifact: { task, answer: run.response, tools: run.toolInvocations },
scenario: { id: "body-validation", kind: "trace" },
signal: AbortSignal.timeout(120_000),
});
console.log(run.traceId, score.dimensions, score.notes);
} finally {
await box.delete();
}The judge returns dimension scores and the trace identifier for storage beside the screenshots. A score is still a model’s grade, so it cannot replace the final condition.
Useful terminal states are:
| Status | Meaning |
|---|---|
| pass | The stated final condition was observed. |
| fail | The application or integration produced an observed result that violates the condition. |
| blocked | A required permission, login, human-verification challenge, dependency, or human approval prevented the case. |
| inconclusive | The evidence is insufficient to decide safely. |
Do not turn blocked or inconclusive into pass because the agent used all available steps. Do not turn a fixture outage into an application defect.
Parallel cases need separate worlds
Concurrency does not make shared state safe. Each case needs its own account, browser context, wallet fixture, and data namespace unless the application explicitly supports isolation.
The public driver README documents a cases-file entry point and a concurrency option:
bad run --cases ./browser-cases.json --concurrency 4
The command is safe when the four cases cannot affect one another, and dangerous when two cases share a wallet nonce, mutate the same workspace, or rely on one email inbox. Record the case and fixture identifiers with every artifact, so a screenshot from one account is not mistaken for another’s result. Use serial execution for stateful wallet flows, shared accounts, and cases that verify ordering. The state boundary decides, not the number of browser slots.
Keep the execution owner visible
If a team turns these cases into a service, a Blueprint defines the job’s inputs and returned artifacts, and an operator runs it. That packaging cannot validate a browser result on its own, so keep the final condition and its evidence in the test record.
Failure modes worth designing first
An invitation can be created twice because the page retried after the request succeeded. A wallet can show one account while the application keeps another account in memory. A dashboard can render cached data that looks complete while the API request failed. A login screen can send the agent into a loop that consumes steps without testing the product.
A test policy can still classify a challenge as blocked when the required human or account boundary was not safely verified. Do not treat a recovery attempt or a model summary as proof of identity.
The case should name the recovery policy. One bounded retry can be useful for a transient page load. Repeated clicks, changing user intent, bypassing a human-verification challenge, and submitting an unapproved payment are outside a safe recovery policy. When the boundary is reached, save the current evidence and stop.
What AI E2E testing cannot establish
One end-to-end run establishes one goal under one fixture, browser, model profile, and runtime. It does not prove every edge case, device, locale, account role, or network condition. It does not replace unit tests, API contracts, transaction simulations, or deterministic browser assertions. It does not make a production credential appropriate for an agent.
What is AI E2E testing?
AI E2E testing is end-to-end browser testing in which an agent follows a stated user goal, handles visible intermediate states, and checks the final user-visible condition.
Is AI E2E testing the same as Playwright?
No. Playwright is a browser automation framework. AI E2E testing adds goal interpretation and bounded recovery while still depending on browser controls and explicit assertions.
Is AI E2E testing reliable enough for CI?
It can be used in CI when fixtures are disposable, the final condition is explicit, the runtime is recorded, and every terminal result includes inspectable artifacts.
What should a first AI E2E case cover?
Choose one signup, activation, billing, wallet, or support journey whose failure would change a release decision. Keep the first case read-only or disposable until its stop policy is proven.
Make the release rule concrete
Use AI E2E testing when the user’s path crosses changing pages, conditional state, extensions, or visual decisions. Keep deterministic tests for stable invariants. Require a final condition, a saved trace, and a terminal status before the result can block or approve a release.
Next in Browser Agent: AI Browser Automation: Verify Each Step Before the Next