[HN Gopher] MacOS in QEMU - ARM edition [pdf]
___________________________________________________________________
MacOS in QEMU - ARM edition [pdf]
Author : watusername
Score : 207 points
Date : 2023-07-27 04:57 UTC (18 hours ago)
(HTM) web link (kvm-forum.qemu.org)
(TXT) w3m dump (kvm-forum.qemu.org)
| [deleted]
| fredgrott wrote:
| It would be interesting for this audience to compare QEMU against
| VirtualBox..
|
| My own bias is that I will be purchasing a Mac Studio around XMas
| as a flutter app build machine.
|
| I know that VirtualBox has guest OS additions to pass through
| graphics acceleration from the Host OS to the Guest OS. Does QEMU
| have this for MS Windows and Linux *6 guest OSes?
| roblabla wrote:
| As a personal VM machine, vbox is fine. But for anything more
| involved, it's got a lot of rough edges.
|
| We used to use virtualbox at work to run our CI workload
| (spawning a fresh VM for every job). Long story short: It does
| not scale well at all. Its scripting API is terrible, we ran
| into endless network problems (for instance, its builtin dhcp
| server would sometimes fail to flush expired IPs, leading to
| running out of IPs to give), and it would sometimes just freeze
| up entirely, requiring us to restart the virtualbox services.
|
| We switched over to kubevirt[0] (essentially qemu/kvm inside a
| kube cluster) to handle our CI, and it's been a much, much
| smoother sailing since then. And it even allowed us to run
| macos VMs on linux hosts, something that we never managed to
| get working in vbox.
|
| > Does QEMU have this for MS Windows and Linux *6 guest OSes?
|
| QEmu has virgl[1] for paravirtualized graphics, but its windows
| drivers are unsigned afaict, so you can't really use it there.
| Works fine on linux, so long as your mesa and kernel aren't
| super ancient.
| xxpor wrote:
| The quick summary is VirtualBox vs kvm/qemu is a classic
| commercial product vs open source comparison. Without going too
| deep into the actual architecture, VirtualBox is much easier to
| get started with (although check out something like Proxmox),
| but qemu is much more powerful. AFAIK, there are no features in
| VirtualBox that can't also be done with kvm/qemu, but there's a
| lot of things that can be done with kvm/qemu that can't be done
| in VirtualBox.
|
| I say kvm/qemu because the actual hypervisor is kvm, while qemu
| is used for control and device emulation. qemu can also emulate
| full machines, which is obviously much slower but can be
| extremely useful.
| sovietmudkipz wrote:
| I recently (yesterday) ran into emulation issues when I tried to
| run my dedicated game server build for Linux (that I produced
| from Unity) inside an amazon Linux docker container on my M2 Mac.
| I use docker for apple silicon that supposedly allows you to run
| amd64 images on arm (which M1/M2 macs are). Yes even specifying
| the correct ---platform and upgrading the emulation software.
|
| It was only when I opened up all logging in a dev build did I
| discover there was some syscall from the mono side of things that
| wasn't emulated properly enough. I thought it ironic recalling
| how many people had sold containers as lighter weight VMs that
| remove "works on my machine" hassles. Anyways, I just spun up an
| AWS EC2 instance to test this nothing part of the process.
|
| I know podman uses qemu and maybe it could run this Unity based
| workload and maybe I could uninstall docker but eesh the boring
| image tech is starting to become a bit of a hassle. I'll probably
| just keep using EC2 instances or write up a vagrant file to
| create a local VM. I'm sure eventually my workload will be fully
| supported eventually so I can optionally just wait.
|
| I love and hate knowing so much.
| magic_hamster wrote:
| Containers can and do remove the "it works on my machine"
| problem when you're on the same platform, and preferably the
| same OS exactly. The big deal here at least in my experience is
| the ability to completely replicate your environment, however
| you still need the underlying OS, because docker is not a VM.
|
| It sounds to me like you're running a container on a massively
| different architecture and I think it's still pretty cool that
| it runs at all. Obviously when trying to run a game, which
| needs access to hardware and system calls beyond the use case
| of a simple CLI tool, running the container on a different
| architecture becomes more of a long shot.
|
| I think given the circumstances it's actually kind of amazing
| you got this far, and if you're able to go all the way with
| your game with emulation, that'd be pretty cool.
| saagarjha wrote:
| Docker is a VM on macOS
| verst wrote:
| And on M1/M2 Macs that means an arm64 Linux VM. So whatever
| issue came up here probably would come up attempting to run
| amd64 containers on regular arm64 linux (assuming the right
| compatibility tooling is installed).
| Steltek wrote:
| Even when the arch matched, macOS was a bad platform for
| running containers. I'm not even sure you can call them "edge
| cases" because it was basic things like volumes not working
| right.
|
| macOS is engineered for users and desktops, not servers and
| development. Apple's long history of blocking virtualizing
| macOS is a testament to this.
| mannyv wrote:
| MacOS has been virtualizable in x86 since mac-on-x86 was
| released.
|
| It also is engineered for development. Maybe not your
| particular type of development. Apple silicon isn't
| technically development-unfriendly, it's just apple silicon
| makes it less easy to do x86 development. But so does mips,
| sparc, ppc, and other architectures.
| Steltek wrote:
| Do you understand the limits apple put on virtualization?
| Are you aware of how it's handled for cloud providers?
|
| I didn't say it was technically impossible. I said apple
| has a history of blocking it.
| naikrovek wrote:
| maximum of two virtual machines running MacOS on a single
| Apple device, no matter how it is configured.
|
| this is in the EULA and is enforced by
| Virtualization.Framework.
|
| cloud providers get no exception to this.
| smoldesu wrote:
| MacOS can be simultaneously "engineered for development"
| and "a worse container host than a chromebook". They aren't
| mutually exclusive.
| saagarjha wrote:
| This should work, since it spins up a full VM to run your
| container.
| ksala_ wrote:
| Isn't Docker for Mac just using VMs in the background?
| [deleted]
| fathyb wrote:
| It is, and Docker for Mac uses QEMU for `--platform`, just
| like Podman AFAIK. The issue here is that it's QEMU user-
| space emulation, which does not support some syscalls, like
| `ioctl`.
|
| The fix here would be to spin an amd64 VM and run the binary
| or install Docker there, instead of relying on the more
| convenient - but less compatible - user-space emulation of
| Docker. Not very related to this very interesting
| presentation on virtualizing ARM macOS guests on QEMU hosts..
| zbentley wrote:
| > Docker for Mac uses QEMU for `--platform`
|
| Only on older setups and on the default config. Updated
| MacOS (13+) and updated docker provides an option to use
| Rosetta and the new virtualization system. That, I believe,
| replaces QEMU for x86 emulation with the Apple emulator.
|
| I've had substantial success with that method; not only
| does it seem a bit faster, but some containerized apps that
| very frequently failed to start (e.g. Apache Pulsar) work
| consistently in that mode. I suspect this difference in
| experience speaks to bugs in QEMU that trigger in the
| presence of certain application behavior (in Pulsar's case,
| something weird happening with the JVM on startup).
|
| QEMU's still an incredible feat of engineering and
| something I frequently use on Linux, though. It just seems
| that the Apple emulation on Macs is better (which makes
| sense, given that it's built by the people who made the
| semi-proprietary silicon it's emulating from).
| xp84 wrote:
| Can I ask a question? It seems like your post is
| combining two separate things. As I understand it and
| based on earlier comments in the thread:
|
| 1. To run a Linux container you need to be running Linux
| 2. This is why Docker for Max has always run a VM under
| the covers. It uses qemu. 3. This was true even before
| Apple Silicon when everything was x86_64 and nothing was
| emulated
|
| Now you are bringing up Rosetta and x86 emulation. While
| I understand sometimes it's necessary to run a container
| that hasn't been compiled for aarch64 (or whatever they
| call Apple) so I can see why the actual cpu emulation
| capabilities of qemu or Rosetta would be necessary there,
| this seems to be beside the point.
|
| if you have no Linux system involved because you have no
| VM, how would that work? What would the base system
| "under" the container be and what would respond to
| Linux's OS-level calls?
| zbentley wrote:
| This is the core of my challenges:
|
| > sometimes it's necessary to run a container that hasn't
| been compiled for aarch64
|
| I have several containers in that boat, for which no ARM
| versions exist at all. Those containers were previously
| using QEMU to emulate x86 on ARM linux (in Docker's VM on
| an ARM Mac). That emulation encountered the failures I
| described, and was slow.
|
| Switching to use Rosetta 2 in Docker solved the problems;
| the process for enabling it is described here:
| https://levelup.gitconnected.com/docker-on-apple-silicon-
| mac...
|
| Something I don't know and am curious about is how
| Docker-for-Mac is actually _using_ Rosetta 2. Is it
| running an additional Linux VM containing an x86 Linux
| OS, and running that VM through a Rosetta-2-enabled
| hypervisor? Or is Rosetta 2 distributed as a Linux
| program that is being invoked inside Docker 's pre-
| existing aarch64 Linux VM instead of (or inside of?)
| QEMU?
|
| Edit: as for your question:
|
| > if you have no Linux system involved because you have
| no VM, how would that work?
|
| That's not my situation, so I'm not sure.
| sgerenser wrote:
| Yes, Rosetta 2 is being used by Docker's pre-existing
| aarch64 Linux VM. So it's only translating userspace x86
| code to Arm, then running it on the ARM Linux VM.
| Syscalls, etc. are still handled by the (aarch64) Linux
| VM.
| Thoeu388 wrote:
| Why not just use native Amd64 Linux machine for development?
| Using emulators and syscall debugger is not very productive
| experience!
| duped wrote:
| > I thought it ironic recalling how many people had sold
| containers as lighter weight VMs that remove "works on my
| machine" hassles.
|
| Containers aren't VMs, they're Linux namespaces. If you want to
| run a container you need Linux. If you want to run Linux on
| MacOS you need virtualization. There's no "lightweight" VM
| about it - it's still a VM.
| giantrobot wrote:
| On Windows and macOS Docker runs Linux in a VM and then all
| the containers attach to that VM. I think this has changed to
| WSL2 on Windows of late but it's still the case on macOS
| AFAIK.
| duped wrote:
| Windows actually has some kind of native containerization
| but I believe it's based on virtualization and not an
| equivalent of namespaces on Linux. I'm not even sure if
| such a feature would make sense on Windows in the same way
| it does on Linux.
|
| But that's my point. It seems from this thread people don't
| understand what docker _is_ except that they can kinda use
| it like a VM. But it 's fundamentally different.
| watermelon0 wrote:
| WSL2 is a Linux VM (albeit with some advanced features such
| as dynamic memory allocations, GPU & file sharing, etc.) on
| Windows hypervisor, and Docker for Windows just uses their
| own WSL2 VM.
|
| It also 'magically' exposes Docker socket to your own WSL2
| instances/VMs.
| 2OEH8eoCRo0 wrote:
| Containers are virtualization without the machine part.
| packetlost wrote:
| No, they are not. Containers are namespaced process groups
| (CGroups) with a sandboxed filesystem and (optionally)
| sandboxed network layer via bridge interfaces and virtual
| devices.
| sleepybrett wrote:
| While the machine part may not matter, the OS certainly
| does. Linux containers depend on the Linux kernel which is
| external to the container.
| monocasa wrote:
| When running Linux containers on Mac, the containers run
| within a full virtual machine.
| hiatus wrote:
| Docker Desktop on Linux works this way, too.
| worksonmine wrote:
| > I thought it ironic recalling how many people had sold
| containers as lighter weight VMs
|
| That's your problem there, containers are by design not VMs.
| Anyone who sold them as such aren't understanding the tech and
| you shouldn't have blindly trusted them.
| irusensei wrote:
| I'm using UTM (nice frontend to Qemu) and the performance is
| okay-ish on a base mbp m1. The desktop is fluid enough for basic
| usage. I use it mainly for remote access client apps that I don't
| want running on my OS, like Citrix.
| nortonham wrote:
| what operating systems are you running on utm? I've tried it,
| and for some it's great, for others, usually one of the bsd's,
| there's always some small problem. And the gui feels kind of
| incomplete.
|
| Weirdly, virtualbox on linux was usually my go to for trying
| out other operating systems. It also felt like the most
| straightforward to use.
| sandyarmstrong wrote:
| UTM is fantastic for macOS-on-macOS emulation on Apple Silicon
| devices. Historically, macOS VMs have been so slow on every
| other virtualization platform. UTM uses Apple's native
| virtualization framework and it's a godsend for testing older
| or newer OS's.
| larusso wrote:
| I used parallels in the past. Mainly to spin up macOS VMs for
| setup testing with ansible (I maintain a macOS build farm for
| our studio) And I found the performance decent enough. It all
| tanks the moment more than one instance is running though.
| mickgardner wrote:
| Worth noting that you can't use your appleid or icloud on a vm of
| macos
| coldtea wrote:
| As in technically can't, or mere license-wise "you're not
| supposed to"?
| amelius wrote:
| I wouldn't be surprised if there are fuses inside the
| processor that tie it to your Apple ID.
| 130e13a wrote:
| i could be completely wrong here, but i do seem to remember
| something about there being some physical chip in the machine
| that needed to be present for e.g. iMessage to work. this was
| a while ago but iirc they also somehow incorporated local
| Apple ID account management into that system.
| saagarjha wrote:
| Virtualization at least does not have a functional sign-in
| alex7o wrote:
| I don't use them on my real Mac why would I need them on my
| fake (VM) one
| ziftface wrote:
| I think that means no iOS development so if you need that you
| need a real Mac
| jeroenhd wrote:
| I can see why that would block publishing apps to the app
| store, but surely you can still locally develop
| applications in the iOS emulator?
| ziftface wrote:
| I'm sure it's possible, depending on the kind of
| application and the tools you use to build it. But xcode
| won't build your application if you don't link your
| account. I believe it needs that to "sign" the
| application with apple.
|
| There may be ways to get around this but I don't of any.
| saurik wrote:
| You can manage your signing identity entirely using
| Apple's website and just not use automatic provisioning.
| (I do that anyway even though I build on macOS as I just
| don't want our build system tied to Xcode just so we can
| use the automatic provisioning.)
| qalmakka wrote:
| Well, QEMU has no graphical acceleration for macOS (no
| virtio drivers for XNU and Quartz) so it's basically
| unusable - software rendering has been historically borked
| on macOS, and never well tested.
| EMIRELADERO wrote:
| Note that the EU's Digital Markets Act will probably stop
| all that bs, as requiring to use one core platform service
| to develop for, and distribute to, another (iOS) is against
| that law.
|
| Third-party app stores and outright website-based app
| distribution are coming next year anyway.
| tmpX7dMeXU wrote:
| If Apple was compelled to support a first-class iOS
| development experience on Windows, all this would do is
| immovably stunt the continued improvement of iOS
| development on macOS. There is a material degree of OS
| integration / cooperation in IOS development with Xcode.
| If Apple released a second-class development experience
| I've no doubt that you'd spare no time in making similar
| insinuations that it's some sort of artificial
| constraint.
| jwells89 wrote:
| Xcode and iOS Simulator on other platforms would involve
| porting AppKit as well as writing all-new Darwin
| virtualization for those platforms. Xcode is thoroughly a
| Mac native app, and the iOS/watchOS/tvOS/etc simulators
| currently just run the respective userlands on top of the
| underpinnings that macOS shares with other Apple
| platforms.
|
| It'd be rather similar to the situation with Docker,
| where Xcode and the simulators would run markedly better
| on macOS than on other platforms due to fewer layers
| being necessary. Developing for iOS on Windows or Linux
| would technically be possible but it wouldn't be very
| pleasant.
| incrudible wrote:
| XCode is a hog, if there was a supported way to develop
| applications without it, _that_ would be a first class
| experience.
| parski wrote:
| Been using Xcode since 2012 and I think it's great. It
| has it's flaws, sure, but overall I think it's a really
| good development experience. What do you dislike about
| it?
| jwells89 wrote:
| I've been using it since it was known as Project Builder
| and also don't have many complaints. It's not perfect,
| but it's not bad. I spend plenty of time in the IntelliJ-
| based Android Studio and haven't found it to be much of
| an improvement (and in fact, find that its "smarts" get
| in the way as often as they help).
| detourdog wrote:
| like the xcode cli tools. all th e basic tools are there.
| INGSOCIALITE wrote:
| This is more than accurate. I wanted to start iOS
| development, but even opening Xcode is painful.
| ziftface wrote:
| That's interesting, I did not know about this
| legislation. I'm not European, but it seems that even
| outside of Europe, people will benefit from this change.
| sofixa wrote:
| https://en.wikipedia.org/wiki/Brussels_effect
| qalmakka wrote:
| You can, but only if you follow an hackintosh guide and spoof
| it with a "virgin" ID, it's tricky but doable.
| wuming2 wrote:
| So what is for rent on AWS or MacStadium are not VMs on genuine
| Mac hardware?
| petree wrote:
| They're not really VMs. They are metal instances, which run
| without a hypervisor, but have virtualized network and
| storage attached instead.
| mschuster91 wrote:
| How are these secured against people messing around with
| flashable components?
| saagarjha wrote:
| There aren't many.
| wkat4242 wrote:
| On Intel macos it did work with iCloud accounts etc. At least
| last time I tried (which is a few years ago admittedly).
| defied wrote:
| You can't use AppleID or iCloud because Apple's
| virtualization.framework does not support this (yet)
| https://developer.apple.com/forums/thread/707459
|
| If you run macOS without the virtualization.framework, which is
| only possible on non-ARM mac's, then it will work.
| chewmieser wrote:
| ?
|
| I have Mac running on ESX signed into my Apple account. I
| haven't used it too much but it's on my list of devices on my
| account.
|
| Even says "VMWare" for device type.
| prynhart wrote:
| Same here. Mac VM which is all signed in to iCloud/iMsg etc
| hosted on ESXi. Haven't had any issues
| mrpippy wrote:
| Yes, it works on Intel but not ARM VMs (it's a conscious
| limitation by Apple)
| mrpippy wrote:
| Yes, it works on Intel but not ARM VMs (it's a conscious
| limitation by Apple)
| londons_explore wrote:
| VMware passes through the id of the physical hardware into
| the guest VM. So if you're running on apple hardware, this
| works. If you're running on a PC, there are hacks to 'borrow'
| the serial number of some ancient imac and have it work still
| (even ancient systems are still allowed to log into icloud,
| and those ancient systems didn't do any cryptography so all
| you need is a valid serial number that nobody else is using).
| jwells89 wrote:
| > If you're running on a PC, there are hacks to 'borrow'
| the serial number of some ancient imac and have it work
| still (even ancient systems are still allowed to log into
| icloud, and those ancient systems didn't do any
| cryptography so all you need is a valid serial number that
| nobody else is using).
|
| Hackintosh users have to do something similar to be able to
| sign into iCloud.
|
| The serial number doesn't have to be valid (as in existing
| on a physical machine) to work, though. It just needs to
| _look_ valid (be generated using the same methodology as
| real serials). In fact in order to prevent accidentally
| using a serial tied to a machine owned by somebody else,
| the recommended procedure is to generate a serial and check
| its AppleCare status to verify that it 's not tied to a
| real machine, and if it is to regenerate and check until
| you find one that isn't.
| GeekyBear wrote:
| > even ancient systems are still allowed to log into
| icloud, and those ancient systems didn't do any
| cryptography so all you need is a valid serial number that
| nobody else is using
|
| The ancient random serial numbers are also in a standard
| format.
|
| > Apple devices manufactured after 2010 generally have
| 12-character alphanumeric serial numbers, with the first
| three digits representing the manufacturing location, the
| following two indicating the year and week of manufacture,
| the next three digits providing a unique identifier, and
| the last four digits representing the model number.
| chii wrote:
| But if this serial number was shared between several
| people, all of whom might have logged in with a different
| apple cloud id, would that somehow cause apple to trigger
| something?
| londons_explore wrote:
| I think they log you out if someone else logs in with
| another appleid on the same hardware id.
|
| Mine got logged out when I used a publically shared id,
| and stopped being randomly logged out when I picked a
| random number.
| darkclouds wrote:
| The command line strings for Qemu can be enormous and complicated
| because they have changed over time, whats shown on page49 is
| very simplified, but should get some people going.
| deaddodo wrote:
| Well, not just that, but Qemu isn't a platform virtualization
| utility like Hyper-V or Virtualbox. Qemu is an emulator of
| general machine platforms that happens to offer virtualization
| APIs for performance boosts in specific configurations (x86 on
| x86, most specifically).
|
| This is what allows it to simultaneously pretend to be a
| reference PREP platform, G3 machine, G4 machine, ARM9TDMI
| mobile device, ARM-Cortex EFI platform, SPARC server or even
| some hodge-podge machine of mixed emulated components. People
| treat it as equivalent to VBox or VMware; but it's not, it's a
| different tool that happens to offer a similar overlap of use
| cases (like using the butt of a screwdriver as a hammer).
| darkclouds wrote:
| > Qemu isn't a platform virtualization utility like Hyper-V
| or Virtualbox
|
| I know, thats why I could run arm based OS on those twos on
| an x86 or x64 cpu.
|
| Theres a lot of work thats gone into Qemu to emulate the
| different processors, question is have they found
| undocumented features of cpu's when trying to emulate them?
| wyldfire wrote:
| Alex's presentation was among some of the cooler ones at this
| year's KVM Forum.
|
| HVF sounds like it has some unique features that make it nicer
| than KVM for some use cases.
| sandreas wrote:
| There is also a nice little guide on how to get macOS Ventura
| working on Proxmox (using QEMU), although x64 this is really
| working like expected:
|
| https://www.nicksherlock.com/2022/10/installing-macos-13-ven...
| ACAVJW4H wrote:
| KVM Forum 2023 Day 2 10:15 AM https://youtu.be/hyrw4j2D6I0?t=4684
| solarkraft wrote:
| Thank you. Slides don't really tell the whole story.
| p0w3n3d wrote:
| thank you, without the video, slides do not make sense
| mikkelam wrote:
| how well does this work? Is this the new hackintosh?
| kapildev wrote:
| It is MacOS only. So, no, this is not the new Hackintosh.
| john_alan wrote:
| I'm using QEMU via UTM as my primary virtualisation solution for
| Linux and macOS guests on macOS.
| sneak wrote:
| What guest OS do you use for GPU accelerated Linux guests? I
| have not been able to get GPU accelerated full res Linux guests
| working on M1/M2 via UTM.
| password4321 wrote:
| Semi-related (MacOS x86-64):
|
| _Quickemu: Quickly create and run optimised Win-10,11
| /macOS/Linux on Linux_
|
| https://news.ycombinator.com/item?id=28797129 (2021)
___________________________________________________________________
(page generated 2023-07-27 23:02 UTC)