If you ship agents to your customers or run a software factory, you run code that you didn’t write in your infrastructure. Every session clones a repository, installs packages, runs generated code and commands. This is why agent products like Grok Bot and OpenAI dots give each task a beefy sandbox (up to 8vCPU and 16 GiB memory) of its own. Unfortunately, not every company has the resources SpaceX and OpenAI have.
So the agent sandbox becomes part of the product, and its cost is part of your gross margin. It has to hold hostile code next to other tenants and still be cheap enough to start thousands of times a day. In the recent years a lighter type of VM called “microVM” has been trending to solve this exact issue. The caveat is that it requires hardware virtualization, which not all server products support. If the server is not bare metal, they need nested virtualization (passing hardware support to VMs within VMs) which has a performance penalty.
We think pagetable-based virtual machines (PVM) are a good alternative for agent fleets. Published in 2023, PVM runs a microVM with its own kernel inside an ordinary x86 cloud VM, keeps the provider’s hypervisor out of the hot path, and puts each guest behind two boundaries instead of one. Its even possible to use spot compute to gain significant savings. As with every nice thing, it comes with some downsides which we will discuss in this blog, along with some benchmarks.
Why are most agent sandboxes microVMs?
Containers work well for trusted code, but every container on a host shares one kernel, and a kernel bug reachable from one container is a path to every other tenant on the machine (Docker’s security documentation describes this model).
gVisor sits between the two: its userspace kernel, the Sentry, answers system calls itself, so only a filtered set reaches the host. Modal runs gVisor by default and added microVMs in October for workloads that need more of Linux than gVisor implements (i.e. some container workloads), which includes many of the developer tools agents run.
A microVM goes further and gives each sandbox a whole kernel of its own behind hardware virtualization, with a virtual machine monitor (VMM) stripped down to what a sandbox needs. E2B runs its sandboxes on Firecracker, which AWS built for Lambda; its specification targets guest userspace within 125 ms of the start call, at most 5 MiB of VMM memory per microVM, and more than 95% of bare-metal CPU performance.
Figure 1 shows where each approach draws the boundary, and in the bottom row, what each needs from the machine underneath. That row drives the cost of everything that follows.
- Agent code and userland
- Kernel owned by one workload
- Host kernel, shared
- Machine it runs on
- Isolation boundary
Docker
gVisor
KVM microVM
PVM microVM
Sources: Docker’s security model, gVisor’s architecture, E2B’s Firecracker isolation, and the PVM design.
The case for hardware virtualization
A Kernel-based Virtual Machine (KVM) microVM runs on Intel VT-x or AMD-V, which an ordinary cloud VM doesn’t expose. You can rent the hardware, or rent a VM whose provider passes the extensions through; both of which have pros and cons.
Bare metal: the most compute per dollar
Bare metal is the most direct option: no hypervisor between you and the CPU, and no hardware shared with other customers. At raw providers it is also inexpensive. Figure 2 compares a Hetzner AX42 with 16-vCPU, 64 GB instances from AWS, Google Cloud, and Azure; a hyperscaler vCPU is one hardware thread, so each has eight physical cores, like the AX42.
- Dedicated bare metal
- Cloud VM with nested virtualization
Sources: Hetzner price list, AWS m7i and m8i rates, GCP n2-standard-16, Azure D16s v5, ECB reference rate.
At on-demand rates the bare metal server costs about a fifth as much. Commitments narrow the gap but don’t close it: a three-year commitment brings the n2-standard-16 to about $255 a month, still more than twice the AX42. The CPUs differ per core, so benchmark before treating the price ratio as a performance ratio. EC2 On-Demand instances bill per second; a dedicated server bills by the month. Agent traffic arrives in bursts and neither the billing nor the provisioning latency holds up to it. You also give up spot pricing, which can be up to 80% cheaper.
The operations become yours as well: the host OS, the virtualization stack, patching, moving sandboxes off a failed machine. Customers who expect a SOC 2 Type II report will want evidence that those controls work over time, and a provider’s compliance reports cover only what it runs (AWS’s shared responsibility model).
Nested virtualization: available, but every exit costs more
All three hyperscalers now pass VT-x through to some of their VMs. AWS supports it on 7th- and 8th-generation Intel families such as M7i and C8i; Google Cloud on most Intel machine types but not E2, Arm, or most AMD; Azure per size series, including Dsv5. It’s the easiest way to run Firecracker on a hyperscaler, and a reasonable starting point for a prototype.
The cost is performance left on the table. A privileged operation in the microVM exits to the cloud provider’s hypervisor (L0), which reflects it to your KVM (L1), and resuming the guest traps to L0 again, so one round trip crosses the provider’s hypervisor twice. The PVM authors measured a nested world switch at 1.3 μs against 0.105 μs without nesting, and slowdowns of up to two orders of magnitude for memory-intensive workloads with many concurrent guests (Janus paper). Google warns of a 10% or greater drop for CPU-bound nested workloads; AWS recommends bare metal for performance-sensitive ones.
There is a security cost as well. To pass VT-x through, the provider’s hypervisor has to emulate it, which the PVM paper counts as extra attack surface in the most privileged software in the cloud. Nested virtualization also doesn’t work with AMD SEV or Intel TDX.
PVM: page tables instead of special hardware extensions
PVM is a paravirtualized execution backend for KVM, developed at Alibaba and Ant Group and published at SOSP 2023. Paravirtualized means the guest kernel knows it is a guest and cooperates with the host, through the same pvops interface Linux already uses for Xen. Applications in the guest see ordinary Linux, and the VMM above it still talks to the standard KVM interface.
That cooperation has a cost you take on before any benchmark: a PVM-capable host kernel and a PVM guest kernel. The patches remain out of tree, so you rebase them for every kernel update, or depend on a vendor that does.
PVM isolates the guest with page tables instead of VT-x. The guest application and the guest kernel both run at the CPU’s user privilege level. Each gets a shadow page table, built by the host, that maps only memory the guest may use. A small host component called the switcher handles every transition between them, and every exit to the hypervisor.
- Guest application
- Guest kernel
- Host code
- CPU privilege boundary
Source: redrawn from the PVM design slides
PVM also runs in production. Alibaba Cloud runs more than 100,000 PVM secure containers on over 400,000 vCPUs a day, and 36% of its secure-container users moved from bare metal to general-purpose instances once PVM made that possible.
Two boundaries instead of one
PVM is usually described as a way to run microVMs where KVM is unavailable. Its isolation properties are an equally strong reason to consider it.
The first property is privilege. Under hardware virtualization, a guest kernel runs in ring 0 of a guest mode the CPU maintains for it. Under PVM it never gets ring 0 anywhere: guest application and guest kernel both run in host user mode behind page tables the host builds, and the switcher keeps its own address space apart from the guest’s, in the style of the page-table isolation Linux adopted after Meltdown (PVM paper).
The second property is depth. A sandbox escape on bare-metal KVM lands on a physical machine with full access to its hardware. A PVM escape has to get through the guest kernel and then the PVM hypervisor, and what it reaches is one of your cloud VMs, still inside the provider’s hardware boundary. The interface is small, too: the paper counts tens of hypercalls between guest and host, against more than 250 system calls a container can reach under the default seccomp profile.
Finally, PVM requires nothing from the provider’s hypervisor, so there is no need to enable VMX emulation in the most privileged layer of the cloud.
These properties have limits. PVM’s own boundary is implemented in software, and no boundary protects a credential that leaks from inside the sandbox. Hovewer, because PVM’s boundry is in software it can be (already a planned feature) ported to other architectures such as ARM and RISC-V.
Faster than nested, close enough to bare metal
Under PVM, guest exits stay inside your cloud VM: a privileged operation goes to the switcher and the PVM hypervisor in L1 and never reaches the provider’s hypervisor.
- MicroVM (L2)
- Your cloud VM (L1)
- Cloud provider’s hypervisor (L0)
Nested KVM
- Privileged op
- Exit to L0
- KVM handles it
- Entry traps to L0
- Guest resumes
PVM
- Privileged op
- Switcher
- PVM handles it
- Switcher
- Guest resumes
Source: PVM paper
On Alibaba Cloud instances with identical hardware, the paper found:
- Running kernel builds, a file-server benchmark, and a Java benchmark in up to 16 concurrent guests, nested KVM collapsed at high concurrency while PVM stayed close to single-level virtualization.
- For memory-heavy workloads with many guests, PVM was up to two orders of magnitude faster than hardware-assisted nested virtualization.
Against bare-metal KVM, the paper’s LMbench runs put PVM close behind, and clearly behind only on fork, exec, and shell. That gap matters less than it first appears. Leased hardware stays on one CPU generation for years, while cloud instances move to new generations as providers release them: when Alibaba moved Spark workloads from bare metal to PVM on general-purpose instances, they ran 22.6% faster on average. The authors attribute the gain to newer processors rather than to PVM, which is the point: PVM makes the newer instances usable for sandboxes. You also get per-second billing back, and spot capacity on instance families that offer no nested virtualization at all.
An agent fleet is many small guests working at once, the case where nested KVM falls over. It also runs process-heavy tools, which is where PVM is weakest. Which brings us to the bill.
What PVM costs
PVM is slowest at the operations that hardware virtualization handles entirely inside the guest:
- System calls. Under hardware virtualization a guest system call never leaves the guest; under PVM it passes through the switcher. The paper measured
getpidat up to 7× slower than hardware-assisted KVM, about 1.3× with direct switching. - fork, exec, and many small allocations. These create page tables or faults the PVM hypervisor must handle, and the paper puts PVM behind hardware-assisted KVM on fork, exec, and shell, on bare metal and nested.
- x86 only, for now. CubeSandbox builds for Arm64, but its PVM deployment guide requires an x86-64 server, and the PVM authors list Arm64 as future work (LPC 2024 talk). Graviton, Axion, and Cobalt instances still require KVM.
On bare metal with VT-x and nothing else to worry about, plain KVM is faster, as the paper reports. Speed on a single host is only one part of the decision.
Stress testing PVM
To check these results on commodity infrastructure, we ran two tests of our own: how soon a new sandbox can run a command, and how many fit on one host. Both tests ran on an ordinary cloud VM with no /dev/kvm and no nested virtualization before setup; after boot the PVM host kernel’s kvm_pvm module provides /dev/kvm, and the sandbox platform uses it as it would hardware KVM. The lifecycle and capacity tests used two hosts of the same specification, running CubeSandbox, an open-source sandbox platform with PVM support.
- Provider
- UpCloud, Helsinki (
fi-hel1) - Instance
CLOUDNATIVE-4xCPU-32GB: 4 vCPUs, 31 GiB usable RAM, no swap- Storage
- 50 GiB ext4 root disk; 50 GiB XFS data disk with reflink for sandbox state
- Host OS
- Ubuntu 24.04.4 LTS
- Host kernel
6.6.117-opencloudos9.cubesandbox.pvm.host- Sandbox platform
- CubeSandbox v0.7.2, Python SDK
cubesandbox0.7.0 - Guest image
sandbox-codetemplate image pinned by digest
The scripts that set up the host and run each test, with the image digests and component hashes, are in our benchmark repository.
Time to a usable sandbox
Each lifecycle step ran 50 times after warm-up, with sandbox state on local XFS. A new sandbox started in 62 ms, ran its first command in 123 ms (median), 287 ms at p95.
- Median
- p95
Get a working sandbox
Park one for later
Source: our benchmark
A local snapshot completed in 40 ms and a pause in 88 ms. Coming back takes longer: a paused sandbox is running again in 167 ms, though its 401 ms p95 is the widest spread we measured, and a sandbox created from a snapshot runs its first command in 308 ms.
This matters because agents spend much of their time waiting, on a human review or a long model call. You can pause them in milliseconds and have them back well under half a second later. Restoring a snapshot takes about 185 ms longer than a fresh create, so it wins whenever preparing the workspace from scratch (installing dependencies, starting a browser) takes longer than that. Note that the snapshots are filesystem only, running processes do not survive it.
Remote snapshot storage is a different budget. With snapshots in S3-compatible object storage, a snapshot took about 1.5 s at the API and 5.8 s to finish uploading. The jump from milliseconds to seconds is clear and depends on network bandwidth of your provider.
How many microVMs fit on one host
We packed a host with microVMs of one size at a time, raising the count until a create failed, with the scheduler’s quota set high enough never to limit the test. Each size is ran twice: idle, and under load, with each guest keeping a quarter of each vCPU busy and touching a quarter of its RAM, closer to an agent installing packages and running tests.
- One host: 4 vCPUs, 31 GiB
- MicroVM that fit under load
- MicroVM that fit only while idle
- Wider tile = more vCPUs
Source: our benchmark
Idle, the host held 389 microVMs with 1 GiB each, 350 with 2 GiB, and 227 with 4 GiB: up to about 900 GiB of configured guest memory on a 31 GiB host, because an idle guest touches little of what it’s given. Every idle run ended on an API timeout, with all four CPUs busy and 9 to 12 GiB of host memory still free.
Under load the host held 67, 38, and 19, between a sixth and a twelfth of the idle counts. Every loaded run stopped at our memory reserve with 17 to 19 GiB touched inside the guests, so working capacity follows the memory agents actually use.
So one small cloud VM can keep a few hundred sandboxes parked but only a few dozen working. Because agents spend much of their time idle, both numbers matter. Size hosts for the working set, and use idle density to decide how many paused sandboxes each can hold. The guest image also matters, based on our test there is a 40 MiB difference in idle memory RSS between ubuntu and minimal alpine guest image.
Conclusion
PVM is worth a serious look if:
- you run on cloud VMs and want microVM isolation without leasing hardware;
- nested virtualization overhead shows up in your numbers;
- your customers’ security reviews care about blast radius, and you want an escape to be contained in a cloud VM rather than reach a physical host;
- you want a kernel per sandbox and gVisor’s compatibility isn’t enough;
- you can maintain a patched host and guest kernel, or work with a vendor who does (which won’t be a problem once PVM merges into mainline linux kernel).
It’s probably not worth it if you already operate bare metal successfully, if your fleet runs on Arm, or if gVisor covers the tools your agents run.
Before adopting it:
- Benchmark your own agent workload on PVM against KVM, including package installs and test runs, which exercise fork and exec.
- Plan who rebases and tests the out-of-tree kernel patches.
- You still need to scope credentials and network access to each task. A stronger boundary doesn’t make a leaked token safe.
To get started, CubeSandbox’s PVM deployment guide covers a self-hosted agent sandbox, and the Kata Containers PVM demo runs Kata on PVM inside an ordinary VM.
Honorable mention: check out Janus which is a new virtualization tech that came out in 2026 and promises to beat both PVM and KVM out of the park in performance.
Acknowledgements
Special thanks to UpCloud (and shout out to Samir from the team) for supporting our testing efforts and providing testing credits for their infrastructure. We are not affiliated with them in any way, they are just very nice people.
AI disclosure
Generative AI models were used in research, editing and image generation in this article.