An agent is asked to verify a checkout flow. It opens the store, adds an item, fills a form, clicks Continue, and reports success. The total on screen may have changed, a validation message may have appeared, or the page may have stopped loading while the model continued from an old observation.
Browser automation for AI agents is useful when the run preserves enough evidence to distinguish those outcomes. The agent needs a natural-language goal, a real browser, structured page observations, bounded actions, recovery rules, and a final condition a person can inspect. Screenshots show what the user saw. The Document Object Model, or DOM, and accessibility tree show what the page exposed. A trace, the ordered record of one run, connects the observations, actions, errors, and stop reason.
Tangle Browser Agent provides a public CLI and TypeScript driver around this loop. The product boundary is deliberately narrow: drive Chromium, capture page evidence, and return a structured result. In this article, the runner is the process that owns the browser session and enforces the allowed actions. The service that owns a purchase, account change, or transaction still has to confirm that business effect.
The click that lied
A browser action is an attempt, not an outcome. The agent can click a button that is covered by a modal, use an outdated coordinate, submit a form with an invalid field, or reach a confirmation page that only looks successful.
Reliable browser automation therefore follows a short cycle:
- State the goal and the last safe checkpoint.
- Observe the current page.
- Choose one allowed action.
- Execute the action in the browser.
- Observe the new page and check the expected effect.
- Recover once when the failure has a safe, named alternative.
- Stop with evidence when the checkpoint is reached or recovery is exhausted.
This cycle makes the final answer a conclusion from recorded state rather than a guess about the last click.
Each observation has a limit
A browser exposes several views of the same page. Use each one for the claim it can support.
| Observation | Good evidence for | Blind spot |
|---|---|---|
| DOM | Text, attributes, form values, links, and element structure | Visual overlap, occlusion, and some browser-rendered state |
| Accessibility tree | Roles, accessible names, keyboard-visible controls, and semantic relationships | Styling, animation, and content that is not exposed to assistive technology |
| Screenshot | Visible layout, modal placement, totals, and visual regression | Hidden controls, exact element identity, and off-screen content |
| Browser console and network log | JavaScript errors, request failures, and HTTP statuses | Whether the server accepted the intended business action |
| URL and application state | Current route and visible workflow stage | Whether the agent interpreted the page correctly |
| Service confirmation | The system of record accepted the action | What the user saw before the confirmation |
The DOM specification defines the browser’s document tree. The WAI-ARIA guidance explains the roles and names that make the accessibility tree useful. Playwright’s documentation covers the browser control layer that launches pages, locates elements, and captures state.
For a read-only search, a URL and structured text may be enough. For a visual regression, add a screenshot and compare it with a known baseline. For a payment or wallet flow, stop at the final review screen and require confirmation from the system that owns the transaction.
Worked checkpoint: a review page is not an order
Consider a checkout test whose goal is to reach review without placing an order. The agent may observe the expected item and total, but those observations support only a page-level claim. They do not establish that an order was created, a payment was authorized, or inventory was reserved.
| Checkpoint | Evidence in the browser trace | Assertion that is safe to make |
|---|---|---|
| Cart | URL, item name, quantity, and page snapshot | The test cart contains the intended sample item |
| Address form | Field values after validation and any error text | The form accepted the test address or exposed a named validation failure |
| Review page | URL, item, total, currency, and screenshot | The browser reached the review state with the expected visible values |
| Payment boundary | The payment control is present and the runner blocks it | The test stopped before payment submission |
| Service record | Order ID or backend confirmation, when a test submits | The owning service accepted the order request |
If the browser trace ends at the review page, report “review reached” rather than “checkout passed.” If a click returns a success page, look for the order ID or confirmation from the service that owns the order. If that confirmation is absent, keep the result in an ambiguous state and preserve the screenshot, network response, and URL.
This distinction also changes how a retry works. A second browser run may create a second cart or send a second payment request. The runner should reuse a test fixture or caller-owned request ID only when the application documents replay behavior. Browser evidence explains what the page showed; the application’s own record explains what the business action did.
Read the public Browser Agent contract
Tangle publishes a Browser Agent manifest with the scoped package, CLI name, provider-key environment variables, SDK exports, and safe commands.
The package is @tangle-network/browser-agent-driver, and its command is bad.
The unscoped npm package named bad is unrelated to Tangle.
The discovery commands do not open a customer site or call a model:
npm install -g @tangle-network/browser-agent-driver
npx playwright install chromium
bad --help
bad run --help
bad snapshot --helpThe snapshot command is useful when an agent is unnecessary.
It produces a deterministic accessibility-tree dump without an LLM, which makes it a better fit for a stable page check.
Use an agent run when the task needs goal interpretation, flexible navigation, or bounded recovery.
A published SDK example
The public SDK uses Playwright to own the browser page and BrowserAgent to decide which allowed browser action to take.
This example reads a public page and closes the browser in every outcome.
import { chromium } from 'playwright'
import {
BrowserAgent,
PlaywrightDriver,
} from '@tangle-network/browser-agent-driver'
const browser = await chromium.launch()
const page = await browser.newPage()
const driver = new PlaywrightDriver(page)
const agent = new BrowserAgent({
driver,
config: {
model: process.env.BROWSER_AGENT_MODEL ?? 'gpt-5.4',
observationMode: 'hybrid',
plannerEnabled: true,
},
})
try {
const result = await agent.run({
goal: 'Read the page title and confirm that the page is public',
startUrl: 'https://example.com',
})
console.log({ success: result.success, reason: result.reason })
} finally {
await browser.close()
}The example is intentionally read-only, uses Playwright’s bundled Chromium, and uses the startUrl field documented in the public README.
The hybrid observation mode combines structured page information with a screenshot.
The planner can make a short plan before executing actions, but the runner still needs a maximum turn count, timeouts, and a stop condition for the actual task.
Model names and provider behavior change faster than browser APIs. Read the current README or manifest before pinning a model in CI. Keep provider keys in the process that owns the browser session rather than exposing them to the page.
A trace makes the run reviewable
A trace is the structured record of one browser run. It should preserve the goal, start URL, observations, actions, effects, recoveries, artifacts, and stop reason. The word “trace” describes the record, not a claim that every entry is true about the external system.
A useful trace can look like this:
{
"goal": "Reach the order review page and stop before payment",
"startUrl": "https://shop.example.test/checkout",
"profile": {
"observationMode": "hybrid",
"maxTurns": 12,
"allowedActions": ["navigate", "click", "fill", "screenshot"]
},
"turns": [
{
"turn": 1,
"observation": {"url": "/checkout", "screenshot": "turn-1.png"},
"action": {"type": "click", "target": "Add sample item"},
"effect": {"textChanged": true, "cartCount": "1"}
},
{
"turn": 5,
"observation": {"url": "/review", "screenshot": "turn-5.png"},
"assertion": {"totalVisible": true, "paymentSubmitted": false}
}
],
"stop": {"reason": "safe checkpoint reached", "actionTaken": "none"}
}The shape is illustrative application data. It shows why a final string such as “checkout passed” is too small for a reviewer to diagnose a failure. The Browser Agent evidence guide develops the same observation-to-result pattern in more detail.
A trace should redact passwords, access tokens, wallet seed phrases, and private page content before it leaves the test boundary. Evidence that exposes a secret creates a new incident even when the browser action was correct.
Recovery needs a budget
Real pages diverge from the plan. A consent dialog covers a control, an A/B test changes the label, a date picker opens a different month, or an API response returns an error page.
When the observation does not match the expected state, the agent should:
- Save the current screenshot, URL, structured page state, and error.
- Classify the deviation as recoverable, unsafe, or unknown.
- Try one bounded alternative, such as dismissing a consent dialog or reopening the form.
- Re-observe the page and test the same checkpoint.
- Stop with a failure reason if the checkpoint remains unreachable.
The Browser Agent README describes recovery strategies for modal blockers, form resets, date-picker stalls, and repeated-action loops. Those strategies can reduce operator work, but they do not justify unlimited retries. Every retry adds model calls, side effects, and another opportunity to cross a business boundary.
A good result distinguishes these outcomes:
| Result | Meaning |
|---|---|
| Success with checkpoint evidence | The observed session reached the stated condition |
| Safe recovery exhausted | The task failed under the allowed action budget |
| Policy rejection | The runner refused an action outside the profile |
| Navigation or network error | The browser could not observe the expected page |
| Ambiguous external result | The page changed, but the system of record did not confirm the business effect |
An agent profile is the versioned configuration that names the model, observation mode, action permissions, turn budget, credentials, and where the result is written for a run.
Store the profile identity beside the trace because changing hybrid to vision, or changing the allowed actions, changes what the result means.
Stop at the business boundary
The last safe checkpoint belongs in the goal, the policy, and the result assertion. Here is a checkout check that leaves payment to a person:
Goal: Verify that a sample order reaches the review page.
Allowed actions: open the public store, add the sample item, fill the test address, and capture screenshots.
Expected checkpoint: the review page shows the item and total.
Forbidden action: submit payment or place the order.
Stop reason: review page captured.The runner should reject a request to click the final payment control and record that rejection. The model should see the stop rule as context, but enforcement belongs outside the model’s text generation.
For a wallet flow, the last safe checkpoint may be a signature request. For an account workflow, it may be a confirmation screen before “Delete.” For a design audit, it may be a screenshot and accessibility report. The correct checkpoint follows the irreversible effect, not the number of browser turns.
DOM, vision, or both
Observation mode should match the claim being tested.
| Mode | Good fit | Cost or failure to plan for |
|---|---|---|
| DOM or accessibility tree | Stable labels, links, forms, and text assertions | Misses visual overlap and some rendered state |
| Vision | Layout, spacing, modal placement, and visible totals | Coordinates drift and text can be misread |
| Hybrid | Flows that need structure plus visual confirmation | More observation data and a rule for disagreements |
Pair an agent with deterministic assertions when the final condition has a stable locator or service response. The agent can navigate a changing page, while a script checks that the total, route, or server response has the expected value. This split reduces the amount of business logic hidden inside a model judgment.
Separate navigation from business assertions
The agent is often the flexible part of the test, not the authority for the final result. A useful case gives each important claim an owner.
| Claim | Owner | Example check |
|---|---|---|
| The agent found the checkout | Browser runner | The URL and page state match the review checkpoint |
| The total is the intended amount | Deterministic test code | The visible currency and amount match the fixture |
| The service created an order | Application or order API | A test order ID and status can be retrieved |
| Payment was accepted | Payment provider | The provider returns a confirmation for the request ID |
| The run was allowed to stop | Runner policy | The action record shows payment was blocked |
This split prevents a model from turning a visual impression into a business assertion. It also makes failures easier to route. An incorrect total belongs to the page or fixture, while a missing order ID belongs to the application or payment integration. The trace can contain all of those observations without pretending they have the same authority.
When the service has no test-facing confirmation endpoint, say so in the case definition. The result can still prove that the browser reached a safe page state, but it should remain “unconfirmed” for the business effect.
Browser automation is not an evaluation by itself
An evaluation, or eval, is a repeatable set of cases with a scoring rule and a comparison target. A browser run is one observation of one URL, account, browser build, model, profile, and network condition. An eval runs the same kind of goal across a named set of cases and records whether each final assertion passed.
For a checkout suite, the cases might include:
- the normal item flow,
- an invalid address,
- an empty cart,
- a slow payment provider,
- a consent dialog,
- a mobile viewport, and
- an explicit stop before payment.
The score should report the number of cases, the assertion results, the browser and model configuration, and the evidence path for each failure. One green case does not establish that the flow works for every user or locale.
Use Playwright’s testing guidance for deterministic paths with stable element locators. Use Browser Agent when the test needs natural-language intent, page interpretation, or bounded recovery. Use both when the agent’s navigation is flexible but the acceptance check should remain deterministic.
Browser and Sandbox have different jobs
The browser owns the page and the user-visible interaction. The Tangle Sandbox can hold test files, comparison scripts, screenshots, and reports around the browser session. The agent runtime guide explains how those pieces fit with model calls, state, and permissions.
Keep browser credentials scoped to the test account. Keep screenshots and DOM captures in a controlled artifact store. Delete temporary workspaces after copying the records that a reviewer needs.
What a browser run does not prove
A successful browser run proves what the runner observed for the tested inputs and session. It does not prove that another account, device, locale, browser version, or network condition behaves the same way. It does not prove that a business transaction was accepted unless the service that owns the transaction returns confirmation. It does not prove that a screenshot was interpreted correctly when the evidence and the model’s conclusion disagree.
The right response to ambiguity is a stopped run with evidence. Do not convert “the page looked right” into “the account changed.”
The decision for product teams
Use browser automation for AI agents when the problem is a user-visible flow that changes enough to defeat fixed locators, but still has a clear checkpoint and a bounded set of permitted actions. Start with a public, read-only task. Then add a disposable test account, one intentional failure, a deterministic final assertion, and a reviewer link to the trace.
Keep Playwright alone when locators and state transitions are stable. Add Browser Agent when interpretation and recovery are the missing capabilities. Require a person to approve the action whenever the next click can spend money, sign a transaction, delete data, or change production state. For model and provider selection around that browser task, see OpenAI-compatible routers for agents.
What is browser automation for AI agents?
It is browser control guided by a task goal, with structured observations, bounded actions, recovery, final assertions, and evidence preserved for review.
Is a browser agent the same as Playwright?
No. Playwright controls the browser and provides deterministic automation primitives. A browser agent adds goal interpretation, observation selection, action planning, recovery, and a structured result around those primitives.
What should a browser agent capture?
Capture the goal, start URL, browser and model configuration, page observations, actions, errors, screenshots, final assertion, and stop reason. Redact secrets before sharing the record.
Can browser automation confirm a payment?
It can reach a review page or observe a payment prompt. The payment service or application backend must confirm that the transaction was accepted.
How do I test a wallet or payment flow safely?
Use a disposable test account and environment, define the last safe checkpoint, block signing and payment by default, capture the wallet and page state, and require explicit approval for any irreversible action.