[HN Gopher] An EPYC escape: Case-study of a KVM breakout
___________________________________________________________________
An EPYC escape: Case-study of a KVM breakout
Author : headalgorithm
Score : 161 points
Date : 2021-06-29 16:53 UTC (6 hours ago)
(HTM) web link (googleprojectzero.blogspot.com)
(TXT) w3m dump (googleprojectzero.blogspot.com)
| mdaniel wrote:
| While reading over that kernel code, I kept wondering how it
| could _possibly_ be code reviewed before landing. The amount of
| context required to spot mistakes must be overwhelming
|
| If there was ever a case for Literate Programming, I'd vote for
| using it on kernel code before I would on rando CRUD app
| jeffbee wrote:
| Wonder no more: hardly anything in Linux has ever been
| meaningfully reviewed. Code looks like it couldn't possibly
| pass review because it can't, and didn't.
| yjftsjthsd-h wrote:
| > There is, instead, a somewhat involved (if somewhat
| informal) process designed to ensure that each patch is
| reviewed for quality and that each patch implements a change
| which is desirable to have in the mainline.
|
| - https://www.kernel.org/doc/html/latest/process/2.Process.ht
| m...
| jeffbee wrote:
| That's basically the process by which gatekeepers tell
| newcomers to go away. It doesn't really apply to the vast
| majority of changes from established contributors. You only
| have to hang out on l-k for a few minutes to get the flavor
| of things.
|
| The only patches getting thorough code reviews are getting
| them from organizations with their own internal code review
| culture.
| mhh__ wrote:
| People think that because what they are doing is arcane or
| complicated that it then has the right for the code to be
| arcane and complicated.
|
| This is not a local effect. Many compilers have more use of
| goto than you might expect - because goto _is_ acceptable in
| some parts of a compiler, it becomes less unacceptable in parts
| where it should remain so.
|
| It doesn't help that C is an awful programming language, which
| actively encourages repetition and the expansion of the surface
| area from which bugs can occur.
| saddlerustle wrote:
| Remember too that the majority of the linux kernel isn't
| covered by any unit or integration tests.
| tptacek wrote:
| A unit test isn't going to spot a bug like this, nor would a
| pre-written integration test. The kernel is _extensively_
| fuzzed.
| jcims wrote:
| It makes you wonder what else is lurking in there, yet to be
| discovered.
|
| Agree on your concern around code review. It's even worse than
| that, though, because it seems that the same internalized
| context that allows you to see mistakes also allows your brain
| to buy in to the same assumptions that allowed the bug to be
| written in the first place.
|
| I spent most of my career in infosec. When I write code I try
| to follow most of the practices I've been preaching, but I
| certainly don't make fewer mistakes.
| formerly_proven wrote:
| Huge difference between knowing better and doing better.
| tyingq wrote:
| That was a really good read. And nothing that something like
| Rust would solve. Just knowing that userspace with access to
| more than one core could outrace the kernel between the
| validation checks and the action.
| wyager wrote:
| I believe that Rust actually would allow you to protect
| against this sort of thing if you integrated the low-level
| API with rust's ownership/linearity system. Most rust APIs
| for embedded SOCs ("HALs") do this correctly.
|
| In the example in the OP, ownership rules were violated as
| there were two direct mutable handles to an object.
| monocasa wrote:
| I've done something really similar (albeit in modern C++)
| that created a type safe guard around DMAed structures to
| provide mutual exclusion between the device and the driver
| (and to handle the appropriate cache management as the
| buffer transitions on non coherent systems, but that part
| isn't really applicable to this article).
| nyanpasu64 wrote:
| How do kernels written in {C, Rust} handle that userspace
| code can create UB situations in kernel code, and prevent
| userspace memory unsafety from causing the kernel to
| malfunction?
| monocasa wrote:
| Typically by simply not having important data structures
| like that in untrusted memory. You're either dealing with
| raw, fairly untyped buffers, or structs that get marshalled
| into kernel space from user space before even being
| validated. I did hear about some similar TOCTOU bugs though
| with seccomp filtering, that then required a fix to move
| the marshalling to happen before the BPF filter was run.
| SubzeroCarnage wrote:
| > Assuming our guest can get full unrestricted access to any MSR
| (which is only a question of timing thanks to init_on_alloc=1
| being the default for most modern distributions)
|
| Can someone elaborate on how init_on_alloc would be helpful to an
| attacker?
| tptacek wrote:
| My guess is because the exploit requires zeroing out an ACL
| bitmap that the host uses to control MSR access, and
| init_on_alloc zeroes out memory as it's allocated, which is the
| state you want as an attacker here.
| ece wrote:
| I wonder if KASAN could've helped here.
| bluedino wrote:
| The old Theo quote... _You are absolutely deluded, if not stupid,
| if you think that a worldwide collection of software engineers
| who can 't write operating systems or applications without
| security holes, can then turn around and suddenly write
| virtualization layers without security holes._
| legulere wrote:
| A smaller attack surface should still lead to less security
| holes though.
| throw-8462682 wrote:
| Taking that quote at face value, it is shallow and needlessly
| binary. It's shallow because it is a truism (paraphrasing, you
| cannot write bug free software). It is needlessly binary
| because there is such a thing as the size of your trusted
| computing base.
| tptacek wrote:
| There's three ways of looking at this, one in which he's wrong,
| and two in which he's saying something much less interesting
| than it sounds like he's saying.
|
| As this P0 post says repeatedly: KVM has a relatively small
| attack surface. It's audited pretty carefully relative to the
| rest of the kernel. The idea behind KVM-based workload
| isolation is that it trades a very large attack surface (the
| entire kernel) for one that is by definition much smaller (a
| subsystem of the kernel that handles virtualization). The rest
| of the kernel runs _behind_ that subsystem, as far as isolated
| applications are concerned. This is a very good trade in
| practice. The Linux kernel (really, all Unix kernels) are much
| less trustworthy than kernel VMM drivers.
|
| So if de Raadt means to be saying that virtualization is by
| design less secure than shared-kernel isolation, he's wrong,
| just sort of plainly.
|
| On the other hand, there are at least two valid points he can
| be making with this line.
|
| First: virtualization systems consist of more than just the
| kernel virtualization driver. If you include QEMU in the mix,
| for instance, it's debatable whether you've gained much over a
| single exposed Linux kernel. Especially at the time de Raadt
| wrote this, it would be totally fair game to say that
| virtualization was a security shitshow compared to jailing
| processes or whatever. Of course, the future belongs to memory-
| safe VMMs that use an smaller and smaller subset of memory-
| unsafe kernel code.
|
| Second, it's just hard to write anything without security
| holes, so if "it's not bug free" is the dunk here, well, let
| him cast the first stone, &c.
| wahern wrote:
| Option #4, Theo was responding to virtualization advocates
| who claimed VMs offered as good or better security isolation
| than physically separate boxen.
|
| At the time and in some cases still today, plenty of
| advocates still make that claim. Many others just assume the
| truth of it, if only because to question it would cause
| cognitive dissonance with the prevalence of cloud hosting.
| (Notwithstanding that some savvier companies use EC2 much
| like they would a traditional server leasing provider, using
| instance types that take up the entire machine. Security and
| convenience is sometimes a trade-off, but some bargains are
| better than others if you don't succumb to simplistic,
| categorical claims. Which is what Theo was railing against.)
|
| EDIT: For context, here's the original post
| https://marc.info/?l=openbsd-misc&m=119318909016582. It's
| from 2007, when hardware virtualization extensions were new
| and all VMMs had to emulate network interface cards and
| similar hardware. These days virtio devices are common place,
| which helps to substantially reduce footprint. OpenBSD even
| has its own native VMM, which of course only supports virtio
| devices.
| temptemptemp111 wrote:
| EPYC is supposed to be the hardest to break out on considering
| SEV & SME... Unfortunately zero cloud vendors have those features
| enabled. (GCE supposedly does, but I haven't verified it.)
| Agingcoder wrote:
| This is a work of art.
|
| Thanks for posting.
___________________________________________________________________
(page generated 2021-06-29 23:00 UTC)