Natural language test automation lets a product team describe a workflow without writing a selector-heavy script. That is useful only if the English sentence becomes a real browser run. A model summary is not a test. A run with screenshots, actions, observations, and a final verifier is.
Tangle Browser Agent exposes that pattern through the bad CLI and SDK. For wallet-specific English tests, read Natural Language E2E Testing For Wallet Apps.
Write Goals Like Acceptance Criteria
Weak prompt:
Test onboarding.Useful prompt:
From the pricing page, start the free plan signup, create an account, finish onboarding,
and verify the dashboard shows the new workspace name.The second version gives the agent a start, path, final condition, and visible proof target.
Run Shape
bad run \
--url https://example.com/pricing \
--goal "Start the free plan signup, create an account, finish onboarding, and verify the dashboard shows the workspace"The browser driver should record the same evidence a careful QA engineer would save: screenshots, observed controls, actions taken, and the final reason for pass or fail.
When Natural Language Beats A Script
| Scenario | Why English helps |
|---|---|
| feature launches | flows change before selectors settle |
| product reviews | PMs can describe the outcome directly |
| design QA | visual state matters as much as DOM state |
| partner onboarding | each partner has slightly different setup |
| wallet apps | popups and extensions create branching paths |
Browser automation standards such as WebDriver and frameworks such as Playwright remain the base layer. Natural language should sit above them, not pretend the browser is magic.
Guardrails
| Guardrail | Why it matters |
|---|---|
| one goal per run | keeps the verdict meaningful |
| explicit final state | prevents “I clicked around” passes |
| saved artifacts | makes failures fixable |
| stable accounts | avoids setup failures hiding product bugs |
| known environment | separates app defects from test data drift |
If a run cannot say why it failed, it is not ready for CI.
Prompt Review
Natural-language tests still need review. The prompt is the test case.
| Prompt smell | Better version |
|---|---|
| ”make sure it works” | state the exact final screen or data condition |
| ”try checkout” | name product, plan, and confirmation target |
| ”test wallet” | name connect, network, sign, reject, or submit |
| ”look for bugs” | define visual, accessibility, or flow goal |
| ”do onboarding” | specify account state and completion condition |
This review step is cheap and prevents most false passes. A browser agent can adapt to UI changes, but it cannot infer the business outcome if the test author does not state it.
When To Convert To Code
Natural language is best while the flow changes. Convert to deterministic Playwright or lower-level tests when the invariant is stable and frequent.
| Keep natural language | Convert to code |
|---|---|
| launch review | stable checkout calculation |
| exploratory QA | permission matrix |
| partner-specific smoke | API contract |
| visual inspection | pure function behavior |
For CI gate design, read AI E2E Testing For Browser Flows. For the browser evidence loop, read Browser Automation AI Needs An Evidence Loop.
Keep a small prompt library and review failures weekly. When a prompt fails because the expected state was unclear, fix the prompt. When it fails because the app changed, decide whether the natural-language case or a deterministic test should own that behavior.
The library should keep examples of good and bad prompts. That helps product, QA, and engineering teams write consistent cases without turning the browser agent into an open-ended instruction follower.
What This Does Not Prove
Natural language test automation does not remove the need for engineering-owned tests. It gives non-test-authors a way to generate product-flow coverage while still leaving evidence engineers can inspect.
Decision Rule
Use natural language test automation when the workflow is user-facing and the desired outcome is easy to state but expensive to encode. Keep deterministic tests for stable invariants.
FAQ
What is natural language test automation?
It is test automation where a human describes the user goal in normal language and an agent drives the browser to verify it.
Can product managers write these tests?
Yes, if the system records artifacts and the goals are written as acceptance criteria.
Does this work with CI?
It can, but CI runs should save artifacts and return explicit pass, fail, blocked, or inconclusive outcomes.
How does this relate to Tangle Browser Agent?
Tangle Browser Agent is the browser automation product that turns natural-language goals into evidence-producing browser runs.