[HN Gopher] Firecracker: Start a VM in less than a second
___________________________________________________________________
Firecracker: Start a VM in less than a second
Author : ArmandGrillet
Score : 290 points
Date : 2021-01-23 16:09 UTC (6 hours ago)
(HTM) web link (jvns.ca)
(TXT) w3m dump (jvns.ca)
| daxfohl wrote:
| I just got this working in windows preview build too. If you
| enable KVM and rebuild the WSL2 kernel, then you can follow the
| Linux firecracker demo on github step by step and it works. I was
| able to launch 400 concurrent firecracker VMs on my laptop in 60
| sec.
| someonewithpc wrote:
| 60s total or each? I haven't tried WSL2 but I had to help
| people with _really_ bad docker performance running in WSL (php
| page load took around 60s)
| daxfohl wrote:
| Total. And it was like the first 200 were in 15 to 20 sec.
| freedomben wrote:
| Another incredible blog post from Julia Evans.
|
| Julia if you're reading, I'm a big fan. One request: could you
| add the date/time to the post, preferably near the title?
|
| I do see a `<time>` tag in the HTML, but it doesn't render any
| human readable text. The datetime is also part of the URL which
| is "good enough," but it always takes me a minute to remember
| that and Ctrl+F won't find it.
|
| I'm a little ADD about knowing when things were published, so far
| from an average person. If you like it the way it is, then don't
| worry about me. I just wanted to throw it out there.
| forix wrote:
| +1 on the ADD about timestamps :) and a +1 for Julia. Following
| her on Twitter and she's awesome!
| nix23 wrote:
| >Another incredible blog post from Julia Evans.
|
| Same same, everything grounded and then dug deep.
| eyelidlessness wrote:
| On reading this, I went to see if the time tag renders in
| reader mode, assuming that's part of why it's there, but at
| least on my mobile Safari it surprisingly does not.
|
| I'll second the request. And also second that I'm a big fan of
| Julia Evans.
| chrisweekly wrote:
| +1 you're not alone; seconding the request -- and esp the
| kudos/praise for Julia, one of my "secret heroes"
| williesleg wrote:
| Great! Now we can run all the shit code from the H1b's that need
| constant reboots! Yay innovation!
| ignoramous wrote:
| The blog post doesn't mention it but Firecracker was originally
| based on _crosvm_ [0] built at Google by the ChromeOS team for a
| WSL-like Linux sandbox on top of ChromeOS running debian-buster
| containers viz. _penguin_ (afaik) in a gentoo-based VM viz.
| _termina_ [1]. _crosvm_ inturn is part of a much bigger
| _crostini_ project [2], which I find to be super fascinating, as
| it supports UI workloads (over Wayland and X).
|
| If you're using ChromeOS to run any Linux app, you're using
| _crostini_ which launches those apps in _crosvm_ managed
| sandboxes (Container inside a VM) in seconds. In not so distant
| future, it looks like Android will sandbox platform workloads
| (running outside the Android framework?) managed by _crosvm_ , to
| considerably improve security [3].
|
| I looked for but couldn't find ChromeOS GCP instances. I mean,
| ChromeOS might be a great platform to run multi-tenant server
| workloads at this point.
|
| [0] https://archive.is/T1ZNJ
|
| [1] One can run custom Linux containers (other than the debian-
| based _penguin_ ) but not VMs (other than _termina_ ) at this
| point: https://chromeos.dev/en/linux/linux-on-chromeos-faq
|
| [2]
| https://chromium.googlesource.com/chromiumos/docs/+/master/c...
|
| [3] https://lwn.net/Articles/836693/
| pjmlp wrote:
| Android has been increasing the amount of sanboxing around
| security critical projects, so this might be the next step.
|
| https://source.android.com/security/enhancements
| ignoramous wrote:
| > _... so this might be the next step._
|
| Absolutely.
|
| When I first heard about KVM on Android, I thought Android
| was going to run every app in its own micro-VM managed by
| _crosvm_... But that 'd be too resource intensive for mobile
| devices, I think?
|
| What they are instead doing with this project led by Will
| Deacon is more towards isolating non-Android workloads that
| OEMs run (like Radios):
| https://news.ycombinator.com/item?id=10905643,
| https://news.ycombinator.com/item?id=24109856,
| https://news.ycombinator.com/item?id=14859602?
| antonvs wrote:
| > ... launches those apps in crosvm sandboxes in seconds.
|
| Technically true, but the initial VM launch is not all that
| quick. E.g., on a Pixelbook i7 launching a terminal session
| without the VM started, it takes about 20 seconds for the VM
| itself to initiate, and another 30 seconds for things like
| volume mounting and starting the container running Debian, for
| a total of 50 seconds.
|
| Subsequent launches once the VM is started are much quicker,
| just a few seconds.
|
| > I looked for but couldn't find ChromeOS GCP instances
|
| You're looking for Container-Optimize OS:
| https://cloud.google.com/container-optimized-os/ , which is
| Chromium OS based.
|
| It's the default OS for e.g. GKE nodes, so Google probably runs
| pretty large numbers of them.
| oauea wrote:
| And for some reason Crostini still isn't available on the
| majority of Chromebooks.
| koala_man wrote:
| You can use QEmu snapshots to start a VM in under a second (but
| more than 125ms).
|
| I used this in 2009 for an IRC bot that safely evaluated
| arbitrary shell commands for demonstration purposes, and a fork
| thereof is still chugging along to this day.
|
| https://github.com/geirha/shbot
| ahachete wrote:
| I wrote a similar post some weeks ago, going to similar deepths
| to script launching a VM with Firecracker.
|
| My main goal then was to provide the necessary automation to help
| using cloud images for the VMs, so you can easily leverage a wide
| array of existing images. Most of the credit is due to cloud-
| init, which helps automate instance configuration after boot.
|
| "Automation to run VMs based on vanilla Cloud Images on
| Firecracker": https://ongres.com/blog/automation-to-run-vms-
| based-on-vanil...
| jzer0cool wrote:
| Surprise I have not yet encountered firecracker until now with
| your post. Thanks for sharing.
| tptacek wrote:
| Regarding performance, I wrote a bit about this a few months ago
| in a post comparing different workload isolation techniques:
|
| https://fly.io/blog/sandboxing-and-workload-isolation/
|
| Two useful links from my Pinboard research for that post:
|
| A person at Red Hat optimizing QEMU boot time:
|
| http://oirase.annexia.org/tmp/paper.pdf
|
| An Intel deck talking about qemu-lite:
|
| http://events17.linuxfoundation.org/sites/events/files/slide...
|
| (The other thing to follow up on if you're interested in the
| background on this stuff is kvmtool).
|
| In both cases, a big part of the answer seems to be eliminating
| BIOS overhead; getting rid of oproms appears to have been the
| single biggest win for Intel. But the Red Hat article also finds
| lots of overhead in QEMU itself, and both pieces talk about
| kernel config issues (for instance, scrubbing the kernel you boot
| of subsystems that have expensive initcalls).
|
| By comparison: Firecracker is purpose-built in Rust for this one
| task, provides no BIOS, and offers only network, block, keyboard,
| and serial device support --- with tiny drivers (the serial
| support is less than 300 lines of code).
| bogomipz wrote:
| Regarding your first link I had questions about the following
| passages from it:
|
| >"People like to say "Docker isn't a security boundary", but
| that's not so true anymore, though it once was."
|
| Could you explain why that wasn't true and why it is now?
|
| >"Systems security people spent almost a decade dunking on
| Docker because of all the gaps in this simplified container
| model. But nobody really runs containers like this anymore."
|
| Could you elaborate? People don't run container like what? It
| isn't clear from the preceding paragraph.
| tptacek wrote:
| Modern Docker running in default configurations is reasonably
| well locked down. I wouldn't use it for multitenant
| scenarios, though.
| varun_chopra wrote:
| Can you elaborate why? Would love to know where it falls
| behind and how to fix these without going the Firecracker
| route.
| steventhedev wrote:
| With containers, both the kernel and the hypervisor are
| shared. With vms, only the hypervisor is shared.
|
| It's a matter of having a smaller attack surface. There
| are plenty of container images that run with root access
| by default, which is almost full access to the kernel.
| This means that if the application running in the
| container is compromised, you need to rely on the kernel
| enforcing the sandbox between containers. This is a
| relatively new threat (root not being fully trusted), so
| beyond there simply being more attack surface, there's
| likely to be more bugs/vulns out there to be discovered.
| With effort and care you can safely run this but reducing
| attack surface is a good idea for defense in depth.
| fpoling wrote:
| If one only allows container to run as a non-root user
| (no user namespace either) with all privileges dropped
| with strong mount isolation and some form of syscall
| filtering, then the attack surface is similar to that of
| hypervisors if not smaller while the performance is
| significantly better.
|
| But yes, quite a few services assumes they have root
| privileges and do not work as is in such containers, like
| recent OpenSSH. For those cases VM isolation makes for
| much smaller attack surface.
| tptacek wrote:
| The blog post linked upthread goes into some detail about
| why; answering that question is kind of the point of the
| post.
| iampims wrote:
| Not OP but most public cloud need stronger isolation than
| what Docker provides. GKE uses gVisor, AWS uses Firecracker
| now. When they started, lambda code was scheduled on a
| different EC2 instance per customer. VM isolation is just so
| much stronger. Firecracker hopes to make launching VMs as
| quick as launching containers while retaining the great
| isolation benefits of a VM.
| m00dy wrote:
| Has anyone know that Firecracker can access GPU ?
| ptomato wrote:
| not currently, though there's some people working on that.
|
| https://github.com/firecracker-microvm/firecracker/issues/11...
| syntaxing wrote:
| Are VM and containers different in this context? I'm confused
| where Firecracker would be better than Docker for this usage.
| StreamBright wrote:
| Firecracker would be better from the security isolation point
| of view. It can use everything that Docker has to offer in the
| security domain and some more.
|
| https://github.com/firecracker-microvm/firecracker/blob/mast...
| majkinetor wrote:
| > Firecracker can start a VM in less than a second!
|
| So this makes it comparable to containers when speed in question.
|
| Anybody using this instead typical VM's in production (and not
| being Amazon) ?
| tptacek wrote:
| Yep. At Fly.io, we run customer containers on our own hardware
| around the world --- the normal workflow just pushes Docker
| containers to our registry --- by converting them into root
| filesystems and running them as Firecracker instances.
| alexchamberlain wrote:
| What exactly is involved in converting them to root
| filesystems?
| iampims wrote:
| I'd love to know more about this as well. That's the part
| usually nobody talks about.
| CodesInChaos wrote:
| The linked article contains a script building the file
| system.
|
| It creates a block device using `qemu-img`, adds an empty
| filesystem using `mkfs.ext4` and then simply mounts it and
| copies in the files.
|
| The previous posts cover this topic in more detail:
|
| Day 43: Building VM images -
| https://jvns.ca/blog/2021/01/21/day-43--building-vm-images/
|
| Day 44: Building my VMs with Docker -
| https://jvns.ca/blog/2021/01/22/day-44--got-some-vms-to-
| star...
| nicwolff wrote:
| It may not be what fly.io does but there's a walkthrough of
| the basic idea here https://iximiuz.com/en/posts/from-
| docker-container-to-bootab...
| e12e wrote:
| Have a look at:
| https://stackoverflow.com/questions/23436613/how-can-i-
| conve...
|
| Or "slim": https://news.ycombinator.com/item?id=20182141
|
| Typically, a docker filesystem doesn't include a proper
| unit (docker will inject "tiny init") - so you would
| probably have to add a kernel, and init, somehow.
|
| I think I'd prefer to just create a vm, rather than re-use
| the pre-built docker.
| e12e wrote:
| Ed: a proper "init" not "unit"
| tptacek wrote:
| It's not a big secret or anything but it's changed recently
| and Jerome would do a better job of describing it than I
| would; apart from the filesystem stuff, we have an init we
| wrote in Rust that does a bunch of the lifting.
| freedomben wrote:
| May I ask why you use firecracker, especially when you
| already have Docker images in your registry? Do you need root
| and/or a kernel in your containers for your application?
| tptacek wrote:
| Because of multitenancy. It isn't safe to run jobs from
| different customers alongside each other in namespaced OS
| containers; instead, we give every customer instance its
| own VM and its own OS. This is the same model that Lambda
| and Fargate use (of course, that's what Amazon implemented
| Firecracker for).
|
| I linked to a blog post we wrote about the rationales here,
| upthread.
| freedomben wrote:
| Cool, thanks much. I'll find the link.
| FourSigma wrote:
| What is the difference between Firecracker vs LXC/LXD?
| dharmab wrote:
| When a VM context switch happens, the CPU uses extensions like
| Intel VMX to isolate the virtual machine code from the host
| code. Usually the hypervisor also forces a cache flush to
| mitigate CPU vulnerabilities as well.
| StreamBright wrote:
| VMs vs containers. One uses KVM under the hood the other uses
| cgroups. Btw. you can run Firecracker VMs with container
| isolation on the top.
|
| https://github.com/firecracker-microvm/firecracker/blob/mast...
| e12e wrote:
| Linux containers are containers, not VMs. They are more like
| docker (although, lxd/lxc typically are _used_ more like jails
| /VMs - a "full" user land, rather than just an application
| binary, like with a docker container wrapping a service
| implemented in go).
|
| Technically, docker/lxc uses kernel namespaces to isolate a
| process tree - firecracker starts up a virtual machine.
| jabl wrote:
| I read somewhere (?) that firecracker has poorer io performance
| than KVM/qemu. Is that still the case, or has it been fixed?
| StreamBright wrote:
| I am not so sure about that. Amazon uses Firecracker because of
| its performance and how lightweight it is. Qemu VMs used to be
| heavier and I think Qemu devs started a project to have a
| lightweight version like Firecracker recently.
| riyakhanna1983 wrote:
| You can read about its IO performance against QEMU in their
| NSDI paper (table 8,9):
| https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
|
| Even though the startup times are fast, the IO performance is
| poor compared to native. This is primarily because of IO
| emulation. We've been working on a new hypervisor that can
| directly run isolated containers (no VMs). Email me if you are
| interested in learning more.
| nhooyr wrote:
| Hi can you post more info on this new hypervisor publicly?
| Who's involved?
| riyakhanna1983 wrote:
| We will post more publicly soon.
___________________________________________________________________
(page generated 2021-01-23 23:01 UTC)