Should Your AI Automation Be Allowed to Run Its Own Code?

Should Your AI Automation Be Allowed to Run Its Own Code?

As of September 2026, yes, but only inside a sandbox you did not build yourself. Running model written code is now a managed product with published limits from several vendors. The interesting question is no longer whether it is possible. It is what you let the sandbox reach.

We have watched this shift change how we scope automation work. A year ago, letting a model write and run a script meant building your own isolation, and most teams sensibly refused. Now it is a tool call with a price per hour.

That makes the decision a design problem rather than an infrastructure project. This piece covers what the current products actually give you, the limits worth designing around, and where we still say no.

What Changed to Make This a Real Question in 2026?

Two things. Model providers shipped code execution as a first class tool with documented containers, and platform vendors shipped general purpose sandboxes aimed squarely at agent output. Both come with published specs, so you can now reason about them instead of guessing.

Anthropic's code execution tool is documented as running Python and Bash "in a secure, sandboxed environment," with tool versions named code_execution_20250825 and code_execution_20260120. That is a versioned API surface, not an experiment.

On the platform side, Vercel Sandbox describes itself as a way to "run untrusted or agent-generated code in isolated Linux microVMs," and its documentation names the isolation directly. Each sandbox runs in "a secure Firecracker microVM with its own filesystem and network."

The wording in both places is the tell. Vendors are no longer describing this as a code playground. They are describing it as a place to put output you do not trust, which is exactly the right frame.

What Does a Managed Code Execution Sandbox Actually Give You?

Isolation you did not have to build, a filesystem the model can write to, and a clean boundary for blast radius. If the generated code deletes everything it can see, it sees only the sandbox. That single property is most of the value.

The details differ by vendor and they matter. Anthropic's documentation says the container "has Python pre-installed," and that "each request runs in a new container unless you pass an earlier response's container ID back." So state is opt in, which is a safer default than persistence.

Vercel's default image, vercel/sandbox/universal, is documented as including "the current Node.js LTS, Python 3.14, coding agents, and common utilities," with full root access and the option to run "system-privileged processes" such as container runtimes and FUSE filesystem drivers. That is a much wider surface, aimed at longer running agent work.

Vercel also documents per agent isolation inside a sandbox, giving "each AI agent its own Linux user with a private home directory." If you are running several agents against the same workspace, that separation is worth understanding before you need it.

What Are the Real Limits You Have to Design Around?

Network access, memory, and lifetime. Get these three wrong and your automation fails in production on the first real input, usually because the code tried to install a package or download a file that the environment forbids.

Anthropic's documented runtime is specific. The container has 5 GiB RAM, 5 GiB of workspace disk, and 1 CPU. On networking the documentation is blunt: "No outbound network requests permitted," and "the container has no internet access, so Claude can't download or install additional packages at runtime: only the pre-installed libraries are available."

That constraint is a feature and a trap at once. It removes a whole class of security worry. It also means any workflow that assumed a pip install at runtime will fail, and the model will often try anyway because that is what the training data does.

Lifetime is the other one. Anthropic documents that "containers expire 30 days after creation," and that "after about 5 minutes of inactivity a container is checkpointed," with the container ID restoring it inside the 30 day window. So a long running job cannot assume a warm environment between steps.

What Does It Cost to Let an Agent Run Code?

Less than the engineering time it replaces, in the cases we have seen, but the pricing shape matters more than the rate. One vendor bills container hours, another bills the CPU your code actually burns, and agent workloads spend most of their wall clock waiting.

Anthropic's published terms give each organisation 1,550 free hours of code execution per month, with additional usage "billed at $0.05 USD per hour, per container." Its documentation also notes that code execution is free when used together with its web search or web fetch tools in the same request.

Vercel's pricing page takes the other approach and charges for active CPU rather than wall clock. Its documentation gives a worked example: a sandbox with "1 vCPU and 2 GB of memory" running for one hour at 10 percent CPU utilisation costs $0.0552 for CPU plus memory under active CPU billing, against $0.1704 under wall clock billing at the same rates.

Why the gap is that large is the interesting part. Vercel's documentation cites research finding that "waiting for model inference accounts for 71% to 98% of runtime" across five agent benchmarks. If that holds for your workload, most of your wall clock is idle, and a billing model that charges for it is charging you for waiting.

When Should You Use the Provider's Sandbox, and When Your Own?

Use the model provider's sandbox for self contained data work, and a platform sandbox when the code needs the network, your packages, or a long life. The deciding question is whether the task can finish with no outbound connection at all.

Situation	Model provider sandbox	Platform sandbox
Analyse an uploaded file and return a chart	Good fit	Overkill
Needs to call your internal API	Blocked by design	Good fit
Needs specific packages installed	Pre-installed libraries only	Custom image or install step
Runs for hours with checkpoints	Checkpointed, opt-in reuse	Persistent sandboxes and snapshots
Several agents on one workspace	One container per request by default	Per-agent Linux users
Lowest setup effort	One tool in the request	SDK or CLI plus an image decision

In practice most of our client automations sit in the first row and never need more. The temptation is to reach for the bigger tool because it feels more capable, and then own an image, a registry, and a network policy you did not need.

What Should Never Go Inside the Sandbox?

Production credentials, write access to a customer database, and anything that can send an email or a payment. A sandbox contains the code. It does not contain the consequences of a credential you handed it.

This is where teams get burned, and the failure is never the sandbox's fault. The isolation worked perfectly. The generated code ran a perfectly valid delete against the production Postgres connection string sitting in an environment variable.

Our rule is that a sandbox gets a scoped, read mostly identity with a short expiry, and nothing else. If the automation genuinely needs to write, it writes through a narrow endpoint you control that validates the payload. We laid out that pattern in our piece on least privilege access for AI agents.

The second rule is that the sandbox never holds the only copy of anything. Treat the workspace as disposable, because the documentation tells you it is. A 30 day expiry and a five minute checkpoint window are not a storage plan.

How Do You Tell Whether the Code Was Right?

You check the output against something you already know, not the code against your taste. Give the automation a case with a known answer, run it, and compare. If it cannot get the known case right, nothing about the code being elegant matters.

We keep a small set of these cases for every automation we ship. Three or four inputs with answers a human computed once. They run every time the prompt changes, which catches the slow drift you would otherwise notice a month late.

Reading the generated code still matters for anything that will run unattended, and the review looks different from reviewing a colleague's work. We covered what to actually look for in our piece on reviewing AI written code.

Logging is the third leg. If the sandbox output is not captured somewhere you can read later, every failure becomes a re-run rather than a diagnosis.

Does This Replace Writing the Script Yourself?

For one off analysis, often yes. For anything that runs on a schedule, usually no. A task you will run a thousand times should become a script you can read, version, and test, even if a model wrote the first draft inside a sandbox.

Our working split is by repetition. If it happens once, let the agent do it in the sandbox and keep the output. If it happens weekly forever, have the agent produce the code, then take that code into your repository where it can be reviewed and pinned. The sandbox becomes a drafting room rather than a runtime.

That distinction matters because a sandbox hides variance. The same prompt can produce different code on different days, and if the code is generated fresh each run, you have a new program in production every time. We went through the wider version of this trade in our piece on choosing an AI agent or a plain script.

Where Would We Draw the Line Today?

Sandboxed code execution is ready for data work, file processing, and drafting scripts a human will review. We would not yet put freshly generated code on a schedule against production systems, no matter how good the isolation is.

The reason is not the sandbox. The vendors have done that part well, and the published specs are clear enough to design against. The reason is that unreviewed code plus a real credential is the same risk it has always been, and a microVM does not change it.

If you are weighing where this fits in your own automation stack, we are happy to walk through the specific workflow and where the boundary should sit. You can find us at phoenix.studio.