Agent Sandboxes Compared on Cold Start, Pricing, Network Policy

A new comparison of agent sandbox platforms has been released, focusing on critical performance metrics and billing models that impact the execution of code-writing AI agents. The analysis evaluates five platforms that are frequently shortlisted by development teams: E2B, Daytona, Modal Sandboxes, Cloudflare Sandbox SDK, and Vercel Sandbox. It also considers Runloop, Fly.io Sprites, and Northflank where they influence the overall comparison. The evaluation prioritizes four key properties that significantly alter system architecture over other preferences: cold start under concurrency, filesystem persistence between turns, egress policy, and idle billing. These factors are deemed more impactful than feature matrices, which are often described as noise.

Cold start performance is a crucial metric because agent loops that create a new sandbox for every tool call can incur this delay thousands of times daily. Filesystem persistence determines whether an agent retains its environment, such as installed packages, from one turn to the next, or if it must rebuild its operational world each time. Egress policy dictates whether a sandbox can access the internet, if this access can be disabled, and if it can be modified mid-session. Idle billing is also a significant consideration, as agents spend a substantial portion of their execution time waiting for responses from language models, and someone must bear the cost of these idle seconds.

The report highlights that vendor claims regarding cold start times are often not directly comparable due to unstated conditions such as concurrency, region, image size, and the precise moment the timing clock stops (API acknowledgment versus first executed command). To provide a more objective measure, the analysis references the ComputeSDK’s sandbox leaderboard, an open-source dataset that runs on a schedule. This leaderboard measures Time to Interactive (TTI), defined as the elapsed time from initiating a sandbox creation request to the execution of the first successful command within that sandbox. The dataset involves 100 iterations per provider, launched concurrently in a single burst from a 4 vCPU host located in Northern Virginia.

Results from an August 21, 2026 run of the ComputeSDK leaderboard indicate that Vercel Sandbox achieved a median TTI of 0.67 seconds, with a 95th percentile (P95) of 1.04 seconds and a 99th percentile (P99) of 1.12 seconds, maintaining a 100% success rate. While specific figures for other platforms like Daytona (advertised sub-90ms sandbox creation) and E2B (commonly cited at roughly 150ms) are mentioned, the detailed leaderboard data provides a standardized benchmark for comparison. The comparison aims to clarify these metrics, offering developers a clearer understanding of platform performance beyond marketing claims.