[HN Gopher] Coding Agent VMs on NixOS with Microvm.nix
___________________________________________________________________
Coding Agent VMs on NixOS with Microvm.nix
Author : secure
Score : 75 points
Date : 2026-02-01 08:02 UTC (3 days ago)
(HTM) web link (michael.stapelberg.ch)
(TXT) w3m dump (michael.stapelberg.ch)
| clawsyndicate wrote:
| we run ~10k agent pods on k3s and went with gvisor over microvms
| purely for density. the memory overhead of a dedicated kernel per
| tenant just doesn't scale when you're trying to pack thousands of
| instances onto a few nodes. strict network policies and pid
| limits cover most of the isolation gaps anyway.
| secure wrote:
| Yeah, when you run [?]10k agents instead of [?]10, you need a
| different solution :)
|
| I'm curious what gVisor is getting you in your setup -- of
| course gVisor is good for running untrusted code, but would you
| say that gVisor prevents issues that would otherwise make the
| agent break out of the kubernetes pod? Like, do you have
| examples you've observed where gVisor has saved the day?
| clawsyndicate wrote:
| since we allow agents to execute arbitrary python, we treat
| every container as hostile. we've definitely seen logs of
| agents trying to crawl /proc or hit the k8s metadata api.
| gvisor intercepts those syscalls so they never actually reach
| the host kernel.
| rootnod3 wrote:
| And you see no problem in that at all? Just "throw a box
| around it and let the potentially malicious code run"?
|
| Wait until they find a hole. Then good luck.
| alexzenla wrote:
| This is why you can't build these microVM systems to just
| do isolation, it has to provide more value than that.
| Observability, policy, etc.
| alexzenla wrote:
| The reason why virtualization approaches with true Linux
| kernels is still important is what you do allow via
| syscalls ultimately does result in a syscall on the host
| system, even if through layers of indirection. Ultimately,
| if you fork() in gVisor, that calls fork() on the host (btw
| fork() execve() is expensive on gVisor still).
|
| The middle ground we've built is that a real Linux kernel
| interfaces with your application in the VM (we call it a
| zone), but that kernel then can make specialized and
| specific interface calls to the host system.
|
| For example with NVIDIA on gVisor, the ioctl()'s are passed
| through directly, with NVIDIA driver vulnerabilities that
| can cause memory corruption, it leads directly into
| corruption in the host kernel. With our platform at Edera
| (https://edera.dev), the NVIDIA driver runs in the VM
| itself, so a memory corruption bug doesn't percolate to
| other systems.
| zeroxfe wrote:
| I've used both gVisor and microvms for this (at very large
| scales), and there are various tradeoffs between the two.
|
| The huge gVisor drawback is that it __drastically_ slows down
| applications (despite startup time being faster.)
|
| For agents, the startup time latency is less of an issue than
| the runtime cost, so microvms perform a lot better. If you're
| doing this in kube, then there's a bunch of other challenges
| to deal with if you want standard k8s features, but if you're
| just looking for isolated sandboxes for agents, microvms work
| really well.
| dist-epoch wrote:
| LXC containers inside a VM scales. bonus point that LXC
| containers feel like a VM.
| souvik1997 wrote:
| Hey @clawsyndicate I'd love to learn more about your use case.
| We are working on a product that would potentially get you the
| best of both worlds (microVM security and containers/gVisor
| scalability). My email is in my profile.
| alexzenla wrote:
| This is the thesis of our research paper here, a good middle
| ground is necessary: https://arxiv.org/abs/2501.04580
| alexzenla wrote:
| This is a big reason for our strategy at Edera
| (https://edera.dev) of building hypervisor technology that
| eliminates the standard x86/ARM kernel overhead in favor of
| deep para-virtualization.
|
| The performance of gVisor is often a big limiting factor in
| deployment.
| souvik1997 wrote:
| Edera looks very cool! Awesome team too.
|
| I read the thesis on arxiv. Do you see any limitations from
| using Xen instead of KVM? I think that was the biggest
| surprise for me as I have very rarely seen teams build on
| Xen.
| rootnod3 wrote:
| That is quite an involved setup to get a costly autocomplete
| going.
|
| Is that really where we are at? Just outsource convenience to a
| few big players that can afford the hardware? Just to save on
| typing and god forbid...thinking?
|
| "Sorry boss, I can't write code because cloudflare is down."
| Cyph0n wrote:
| Keep in mind that this setup is a one-time cost. Also, a lot of
| the code is related to configuring it the way the author wants
| it (via Home Manager).
|
| Generally speaking, once you have a working NixOS config,
| incremental changes become extremely trivial, safe, and easy to
| rollback.
| aquariusDue wrote:
| To provide another data point: I too use NixOS and oh boy
| that one-time is really costly. And while we're sharing Nix
| stuff for LLMs there's this piece of kit too:
| https://github.com/YPares/rigup.nix
| Cyph0n wrote:
| Agreed, the learning curve is insane and docs are sparse.
| But it is truly worth it imo, even if you're just using Nix
| as a build tool, or using Home Manager on Linux or macOS.
| the_harpia_io wrote:
| The sandbox-or-not debate is important but it's only half the
| picture. Even a perfectly sandboxed agent can still generate code
| with vulnerabilities that get deployed to production - SQL
| injection, path traversal, hardcoded secrets, overly permissive
| package imports.
|
| The execution sandbox stops the agent from breaking out during
| development, but the real risk is what gets shipped downstream.
| Seeing more tools now that scan the generated code itself, not
| just contain the execution environment.
| ryanrasti wrote:
| Precisely! There's a fundamental tension: 1. Agents need to
| interact with the outside world to be useful 2. Interacting
| with the outside world is dangerous
|
| Sandboxes provide a "default-deny policy" which is the right
| starting point. But, current tools lack the right primitives to
| make fine grained data-access and data policy a reality.
|
| Object-capabilities provide the primitive for fine-grained
| access. IFC (information flow control) for dataflow.
| ATechGuy wrote:
| I agree. However, how to define these permissions when agent
| behavior is undefined?
| nh2 wrote:
| I find that a bit of a weird point.
|
| The goal of such sandboxing is that you can allow the agent to
| freely write/execute/test code during development, so that it
| can propose a solution/commit without the human having to
| approve every dangerous step ("write a Python file, then
| execute it" is already a dangerous step). As the post says: "To
| safely run a coding agent without review".
|
| You would then review the code, and use it if it's good.
| Turning many small reviews where you need to be around and
| babysit every step into a single review at the end.
|
| What you seem to be asking for (shipping the generated code to
| production without review) is a completely different goal and
| probably a bad idea.
|
| If there really were a tool that can "scan the generated code"
| so reliably that it is safe to ship without human review, then
| that could just be part of the tool that generates the code in
| the first place so that no code scanning would be necessary.
| Sandboxing wouldn't be necessary either then. So then
| sandboxing wouldn't be "half the picture"; it would be
| unnecessary entirely, and your statement simplifies to "if we
| could auto-generate perfect code, we wouldn't need any of
| this".
| giancarlostoro wrote:
| If that's the goal, why not just have Claude Code do it all
| from your phone at that point? Test it when its done locally
| you pull down the branch. Not 100% frictionless, but if it
| messes up an OS it would be anthropic's not yours.
| mystifyingpoi wrote:
| > not just contain the execution environment.
|
| See, my typical execution environment is a Linux vm or laptop,
| with a wide variety of SSH and AWS keys configured and ready to
| be stolen (even if they are temporary, it's enough to
| infiltrate prod, or do some sneaky lateral movement attack). On
| the other hand, typical application execution environment is an
| IAM user/role with strictly scoped permissions.
| heliumtera wrote:
| Couldn't you replicate all of your setup with qemu microvm?
|
| Without nix I mean
| rictic wrote:
| Yep. What nix adds is a declarative and reproducible way to
| build customized OS images to boot into.
| CuriouslyC wrote:
| Nix is the best answer to "works on my machine," which is a
| problem I've seen at pretty much every place I've ever
| worked.
| 0x457 wrote:
| It's also an answer to caching with /nix/store. I wish more
| cloud services supported "give me your nixosConfiguration
| or something similar" instead of providing api to build
| containers/vms imperatively. Dockerfile and everything that
| mimics it is my least favorite way to do this.
| Cyph0n wrote:
| It's fairly trivial to map your NixOS config into a VM
| image: https://nixos.org/manual/nixos/stable/#sec-image-
| nixos-rebui...
|
| An alternative is to "infect" a VM running in whatever
| cloud and convert it into a NixOS VM in-place:
| https://github.com/nix-community/nixos-anywhere
|
| In fact, it is a common practice to use the latter to
| install NixOS on new machines. You start off by booting
| into a live USB with SSH enabled, then use nixos-anywhere
| to install NixOS and partition disks via disko. Here is
| an example I used recently to provision a new gaming
| desktop: nix run github:nix-
| community/nixos-anywhere -- \ --flake .#myhost
| \ --target-host user@192.168.0.100 \
| --generate-hardware-config nixos-generate-config
| ./hosts/myhost/hardware-configuration.nix
|
| At the end of this invocation, you end up with a NixOS
| machine running your config partitioned based on your
| disk config. My disko config in this case (ZFS pool with
| 1 disk vdev): https://gist.github.com/aksiksi/7fed39f1703
| 7e9ae82c043457ed2...
| 0xcb0 wrote:
| I was looking for a way to isolate my agents in a more convenient
| way, and I really love your idea. I'm going to give this a try
| over the weekend and will report back.
|
| But the one-time setup seems like a really fair investment for
| having a more secure development. Of course, what concerns the
| problem of getting malicious code to production, this will not
| help. But this will, with a little overhead, I think, really make
| development locally much more secure.
|
| And you can automate it a lot. And it will be finally my chance
| to get more into NixOS :D
| NJL3000 wrote:
| A pair of containers felt a bit cheaper than a VM:
|
| https://github.com/5L-Labs/amp_in_a_box
|
| I was going to add Gemini / OpenCode Kilo next.
|
| There is some upfront cost to define what endpoints to map
| inside, but it definitely adds a veneer of preventing the
| crazy...
| phrotoma wrote:
| One problem with using containers as an isolation environment
| for a coding assistant is that it becomes challenging to have
| the agent work on a containerized project. You often need some
| janky "docker-in-docker" nonsense that hampers efforts.
| NJL3000 wrote:
| I was planning to have worktrees bind mounted systematically,
| but agree it's not super clean atm at scale (yet)
| mxs_ wrote:
| I there a way to make this work with macOS hosts, preferably
| without having to install a Linux toolchain inside the VM for the
| language the agent will be writing code in?
| mtlynch wrote:
| This is a similar macOS solution:
|
| https://github.com/lynaghk/vibe/
| messh wrote:
| I use shellbox.dev to create sandboxes through ssh, without ever
| leaving the terminal
| ghxst wrote:
| I'm working on a shared remote box for AI assisted development,
| will definitely look at this for some inspiration.
| giancarlostoro wrote:
| This brings me back to my college days. We had Windows, and Deep
| Freeze. Students could do anything on the computer, we restart it
| and its all wiped and new. How long before Deep Freeze realizes
| they could sell their tool to Vibe Coders, they have Deep Freeze
| for Mac but not for Linux, funnily enough.
___________________________________________________________________
(page generated 2026-02-04 23:00 UTC)