BlogCode Auditorpart 1 of 2

AI Code Audit: From Scanner Alert to Reproducible Evidence

Work one withdrawal alert from scanner output to a reproduced finding, testing reachability in a sandbox and logging the alerts it rejects.

A security scan reports that a withdrawal function sends funds to an untrusted address. The alert may be useful, but it does not say whether an attacker can reach the function, whether state changes happen before the external call, or whether a test can reproduce a loss.

An AI code audit turns that alert into a bounded engineering decision. The agent reads the project, runs its tools in an isolated workspace, tests the suspected path, and returns evidence that supports the claim or narrows or rejects it. At release time, an unreproduced warning extends review, while a reproduced finding gives an engineer a failure and a fix target.

In Tangle’s Sandbox, the audit agent clones the target into a machine with an egress policy, runs the project’s commands with box.exec, and gets back an exit code and captured output. A judge model on the Router then reads the answer and every tool call. The reader can check three things: the exit code, the costUsd of each agent turn, and the judge’s per-dimension scores. The primitives are public. The audit workflow built from them is a design direction, not a public audit endpoint.

This article uses a withdrawal example to show the workflow, then maps it to the Sandbox and its related service vocabulary.

Start with the path an attacker would need

A scanner examines source code, dependencies, or generated analysis data for patterns associated with bugs. Static analysis does this without running the complete application against live traffic.

CodeQL builds a queryable database, runs queries, and interprets the results as possible issues or data-flow paths. Semgrep’s rule glossary describes a finding as the result produced when a rule matches code.

Their rules and queries inspect every matching file, path, or data-flow edge on each run. The output is still a candidate until someone establishes the path from an external input to an affected asset.

Audit state Question answered Evidence worth keeping
Candidate What pattern or query produced the alert? Rule name, tool version, location, and raw output
Reachable Can the relevant caller reach the code under the stated setup? Call path, permissions, configuration, and input
Reproduced Does the suspected action create the claimed state change? Test, command output, request, or transaction trace
Rejected Which check, wrapper, or assumption blocks the claim? Negative test and the reason the candidate was closed

The state is a reporting aid. It shows which evidence changed the conclusion, so the next reviewer can check that evidence.

Work one withdrawal alert all the way through

Consider this small Solidity example. Solidity is the programming language, and the Ethereum Virtual Machine, or EVM, is the execution environment used by Ethereum-compatible contracts.

pragma solidity ^0.8.0;

contract Vault {
    mapping(address => uint256) public credit;

    function withdraw() external {
        uint256 amount = credit[msg.sender];
        require(amount > 0, "no credit");

        (bool sent, ) = payable(msg.sender).call{value: amount}("");
        require(sent, "send failed");

        credit[msg.sender] = 0;
    }
}

This is a teaching example with no deposit path or deployment configuration. The suspicious sequence is the external call followed by the balance update.

Reentrancy is the failure mode in which code called by a contract calls back into that contract before the first invocation has finished updating its state. The scanner can point at the call. The audit still has to show whether an attacker-controlled recipient can enter again and whether the second call changes the balance.

The agent should ask the questions a human reviewer would ask:

  1. Which function is public, and which caller controls the input?
  2. What storage value represents the asset or permission at risk?
  3. What must be true before the external call can run?
  4. What happens if the recipient calls withdraw again before credit is cleared?
  5. Does the test end with a balance, permission, or ownership state that should be impossible?

The minimum proof is one fixture, one command, and assertions for the attacker’s credit and the vault balance. For this example, fund the local vault with 20 ether, give the attacker 10 ether of recorded credit, and ask it to withdraw while its receiver calls withdraw once more. One ordinary withdrawal should remove 10 ether and leave zero credit. If both calls read the stale credit before either update completes, the observed result is a 20-ether decrease even though the attacker had only 10 ether of credit. The assertions check the initial credit, final credit, and vault balance. The test should also record the call depth or a transaction trace when the toolchain provides one.

The report should not say “reentrancy confirmed” because a model recognized a familiar pattern. It should state the caller, amount, initial state, sequence of calls, observed state change, and the code path that caused it.

The same process catches false positives. If an owner check rejects the attacker, or a non-reentrant lock blocks the callback, the agent keeps the negative test and the reason the path stopped.

Test outcome Safe report language
The attacker reaches the function and changes protected state “Reproduced under these inputs and setup”
The attacker reaches the function but the state change does not occur “Reachable, impact not reproduced in this test”
Authorization or initialization blocks the path “Candidate rejected by the tested guard”
The build or fixture fails before the path runs “Reproduction incomplete because setup failed”

Keep the experiment away from the host

An audit agent reads files, installs dependencies, runs compilers, and executes commands written by someone else. A sandbox is a separate computer or process boundary for that work, with its own filesystem, processes, resource limits, and network policy.

Tangle Sandbox documents this model as an isolated machine for an agent, implemented with a container or microVM and driven through an SDK, command-line interface, or dashboard. The public AI Agent Sandbox guide explains the same lifecycle in Tangle terms.

The fix-a-test example shows the calls an audit needs: client.create with a repository, box.prompt for the agent, and box.exec to run the tests itself. The check is the exit code of npm test, not the agent’s reply. The example fixes a failing test rather than auditing, so it shows the mechanism, not a withdrawal finding. It does not set the egress policy that an audit would pass to client.create.

The agent’s claim, checked by box.execRan on production 2026-11-12 from these bytes, exit 0, $0.0145 reported usage, @tangle-network/sandbox 1.0.168. View result.
import { writeFile } from "node:fs/promises";
import { Sandbox } from "@tangle-network/sandbox";

const client = new Sandbox({
  apiKey: process.env.TANGLE_API_KEY!,
  baseUrl: "https://sandbox.tangle.tools",
});

const box = await client.create({
  git: { url: "https://github.com/tangle-network/sandbox-demo" },
  backend: { type: "claude-code" }, // or "codex", "opencode", "pi"
});
try {
  const run = await box.prompt("`npm test` fails. Find the cause and fix it.");
  if (!run.success) throw new Error(run.error);

  // Check the result yourself.
  const tests = await box.exec("npm test");
  if (tests.exitCode !== 0) throw new Error("The tests still fail.");

  await writeFile("fix.patch", (await box.exec("git diff")).stdout);
  console.log(run.usage, run.costUsd); // metered per turn
} finally {
  await box.delete();
}

The exit code shows that the command ran and passed or failed inside that machine. It does not show that the withdrawal path changes state. The assertions in the reproduction section still carry that claim.

Keep secrets out of the workspace whenever a fixture or mocked service can replace them. If a live account, private registry, or chain fork is required, record that fact, grant the narrowest temporary permission, and state which production behavior remained untested.

Retain the artifacts before deleting the workspace:

Artifact Why the reviewer needs it
Source revision or archive digest Identifies the exact code under review
Scanner name, version, and configuration Explains how the candidate was created
Commands, inputs, exit codes, and output Lets another engineer repeat the attempt
Patch or suggested change Connects the finding to engineering work
Rejected candidates and failed attempts Prevents the same question returning without new evidence

Let the Sandbox run the test without becoming the verdict

The isolation boundary limits the damage a test can do to the machine that holds credentials or unrelated source. It does not make the project trustworthy, and it does not make the agent’s conclusion true.

A Blueprint is Tangle’s service package that defines jobs, metadata, contracts, and runtime rules. An operator is the person or service that runs the actual instance for other users. A service instance is the live registration and execution context for one service, not the reusable Blueprint package. The runtime is the environment that starts processes, exposes tools, and returns their results. The public AI Agent Sandbox Blueprint overview describes how those pieces split protocol lifecycle from live machine I/O.

An agent profile is the declared combination of model, instructions, tools, permissions, and resource limits used for one audit run. Pinning the profile explains why two runs may differ, but it is not a security proof.

A trace is the chronological record of a run, including actions, inputs, outputs, and results. Tangle’s Sandbox docs say a run can emit a trace, so a failed run stays inspectable after the client connection is gone.

An evaluation is a repeatable set of audit tasks with expected properties, used to compare runs. Useful measures include the fraction of candidates with reproducible evidence, the fraction of rejected candidates with a recorded reason, time to a reviewer decision, and findings the workflow missed.

Give every specialist the same evidence shape

Different codebases need different tools, but the report should mean the same thing in every language. Each specialist answers one narrow question and returns evidence in the same record shape as every other specialist.

Solana programs, Move packages, and zero-knowledge circuits each need different checks.

Code area Narrow question Useful evidence
Web application Can an unauthenticated request reach the protected action? Request, response, identity, route, and configuration
EVM contract Can a caller change funds or permissions outside the intended rules? Test, transaction trace, caller, and state diff
Solana program Do account ownership and instruction constraints hold for a hostile account set? Program test, account inputs, and failure output
Move package Do resource and ability rules reject the invalid transition? Unit test, expected failure, and package state
Zero-knowledge circuit Does an invalid witness satisfy the circuit constraints? Constraint test, witness, and proof result

The coordinator should route by detected language and build system, not by a fixed set of agent names. It also merges duplicates by root cause. Three alerts that point to one missing authorization check are one engineering problem, even if the tools reported different lines.

Severity follows the observed impact

Severity communicates impact, prerequisites, and scope. It is not a label added after a model writes an alarming paragraph.

Observed evidence Claim it supports
A test changes another user’s balance The tested input and caller can cause that state change
A request reaches an administrative action without authorization The tested environment permits that unauthorized route
A scanner matches a dangerous API but the path is blocked The pattern exists, but the tested attack path did not reach it
The build fails before the proof runs The security claim remains untested

The report names the affected asset, attacker prerequisites, tested configurations, and untested configurations. “High severity” without those details asks the reviewer to supply the missing reasoning.

Know where the workflow stops

An AI code audit can miss a dependency defect, a production-only configuration, a race between services, an economic attack, or a path absent from the fixtures. It can also write a plausible explanation for a test that never ran, if the runtime does not capture exit status and output apart from model text.

A clean sandbox does not cover a missing network integration. A passing unit test does not cover a different compiler version or deployment configuration. A protected execution claim does not establish contract correctness, and a paid request does not establish reviewer agreement.

Human review, monitoring, incident response, and independent testing remain necessary for systems that handle money, credentials, or personal data. The audit narrows the next review by preserving the source digest, command, output, state difference, and remaining uncertainty.

Make each command explainable after cleanup

The useful audit record is not a transcript of every token the model produced. It maps each assertion to the command, input, output, and state that support it.

For the withdrawal example, organize the record by phase:

Phase Input to preserve Observable result
Discovery Source revision, scanner rule, tool version, and configuration Candidate location and raw match
Reachability Caller identity, route or entry point, permissions, and relevant settings Path reached or guard rejected it
Reproduction Test name, arguments, initial state, and dependency versions Exit code, assertion, response, or transaction result
Impact Before-and-after balances, permissions, records, or files Specific state change and affected scope
Repair Patch revision and original proof Exploit proof fails while intended behavior remains

A reviewer can challenge one claim without rerunning the whole agent session. The shape also prevents a common error: the model’s summary survives while the command that should support it is discarded.

Keep model messages, tool output, and test assertions in separate fields. Model text can explain why the agent chose a test. The exit code and captured output show whether the test ran. An assertion shows that a balance changed. The source revision and initial state show which code and conditions produced it.

When the workspace is deleted, preserve the source digest, dependency lockfile digest, tool versions, command list, redacted inputs, and artifact checksums. Do not preserve secrets as proof. If a command can print a token, scrub the value and keep the fact that the command accessed a secret-dependent path.

Replay has a boundary too. A run against a local chain fork at one block is a new experiment if a later run uses a different block, even with identical command text. A database fixture needs its schema and seed version recorded, and it is not production behavior. A changed agent profile, source revision, or network policy gets a new run identifier.

An evaluation can then tell a stronger audit from a lucky answer, by whether the run kept the evidence needed for replay.

The practical decision

Run an AI code audit when a release needs more than scanner coverage and less than a full formal review. Require each escalated finding to name the code path, caller, inputs, reproduction result, affected state, proposed fix, and remaining uncertainty. Keep rejected candidates with their negative evidence.

For the execution boundary, start with the Tangle Sandbox quickstart. For the scanner-versus-audit choice, read AI Vulnerability Scanner vs. Agent Audit.

What is an AI code audit?

An AI code audit is an AI-assisted review that combines code analysis, repository context, executable tests, and an evidence-backed report.

How does an AI code audit differ from a scanner?

A scanner produces candidates from rules or queries. An audit tests reachability, runs a scoped reproduction, records negative evidence, and explains the next fix.

Does a sandbox make an audit safe?

No. A sandbox limits where commands run, while permissions, network policy, secret handling, and review scope determine what the test can still affect.

What should a finding contain?

It should contain the source revision, location, attacker path, inputs, command or test, observed result, impact, fix, and residual risk.

When should a team stop trusting the automated result?

Stop escalating when the build did not complete, the required environment was unavailable, the test did not reach the suspected path, or the report cannot separate model text from captured tool output.

Next in Code Auditor: AI Vulnerability Scanner vs. Agent Audit: Choose by the Risk