[HN Gopher] How Fat Does a Fat Binary Need to Be?
___________________________________________________________________
How Fat Does a Fat Binary Need to Be?
Author : jart
Score : 98 points
Date : 2021-02-11 16:46 UTC (6 hours ago)
(HTM) web link (justine.lol)
(TXT) w3m dump (justine.lol)
| comex wrote:
| > That's how big a statically-linked executable needs to be to
| run natively without dependencies on Linux, Mac, Windows,
| FreeBSD, OpenBSD, NetBSD, and BIOS.
|
| * on x86.
|
| There was a time period when it was reasonable to just assume
| everything that mattered was x86. It started sometime in the
| early 2000s, as non-x86 servers died out and Macs switched to
| x86. And it ended last November, when Apple announced Macs would
| switch to ARM.
|
| The author has suggested sticking with x86 and handling other
| architectures using emulation. But x86 is the worst possible
| architecture to emulate, because its strong memory ordering model
| means that emulators on other architectures essentially must
| limit the emulated process to a single core. (Unless you have
| special hardware support like M1 Macs do... for now.) Also, any
| hardware CPU architecture is going to be slower to emulate than a
| purpose-built bytecode - and we already have a good enough
| bytecode, WebAssembly.
|
| Emulating x86 might make sense anyway if you assume that x86 is
| and will be the norm, and non-x86 an edge case that can afford to
| take a performance hit. But that sure doesn't seem to be the way
| the wind is blowing.
|
| [1] https://justine.lol/ape.html
| spijdar wrote:
| I'll admit to being biased up front, as I run a lot of ARM and
| POWER stuff, so I have a vested interest, but I have to agree
| that this feels short sighted. ARM at the very least has a big
| potential to take a much bigger slice of the market, with ARM
| servers and ARM Macs "finally" coming out in force, while ARM
| has been dominating smartphone and mobile devices for at least
| the last, say, 15 years, and has gotten a lot of mindshare
| through SBCs like the Raspberry Pi.
|
| It's a fun project, but I don't think the assumption that
| everything is x86 is as safe as the author proposes.
| heavyset_go wrote:
| > _And it ended last November, when Apple announced Macs would
| switch to ARM._
|
| This is revisionism. It ended with the smartphone and netbook
| era, both of which put more ARM devices in homes than Macs ever
| will.
| comex wrote:
| Well. I haven't heard of ARM netbooks being a big thing. But
| yes, smartphones and other non-desktop-operating-system
| devices have been ARM for a long time. And other
| architectures still show up on other embedded devices, like
| MIPS on routers.
|
| By "everything that mattered" I guess I should have said
| "everything that mattered that you have a shell on".
|
| For some of those non-desktop-operating-system devices,
| getting a shell is impossible without bypassing lockdown
| measures. For others, it's possible, but not particularly
| common as far as I know, for a variety of reasons. They tend
| to run stripped-down OSes; they don't have great keyboards;
| they were pretty slow compared to desktops until recently.
|
| Yes, this is a huge overgeneralization. Plenty of people do
| run shells on their phones or routers. But I do believe that
| a major desktop platform switching to ARM is qualitatively
| different from what we'd seen before.
| LordDragonfang wrote:
| Many chromebooks are on ARM, and chromebooks all have the
| native ability to run a true linux shell at this point, and
| have for several years now.
| greyandgreen wrote:
| I know a guy who long ago gave up MacBooks for
| Chromebooks for development. He programs the systems that
| keep HVAC/large refrigeration systems running. He buys a
| new machine every year and his company pays for it.
| Cheap. His last Chromebook had 8 GB RAM and his current
| one 16. For all intents and purposes, it's faster than
| any Mac or PC that you would use for programming and the
| horsepower needed is on a remote server. He can never
| lose any code or other project notes, photos, since they,
| too, live in the cloud. Not for everyone, obviously.
| minikites wrote:
| I thought most "netbooks" had Intel Atom processors?
| jart wrote:
| Author here. These binaries will re-exec themselves under Qemu
| if you run them on different architectures. So they truly are
| build-once run-anywhere.
|
| I think improving performance on other architectures is great.
| There's nothing that inherently limits the Actually Portable
| Executable format from baking in native support for other
| architectures. It's simply a question of focus.
|
| What Cosmopolitan does, it does really well. No other C library
| is able to target multiple operating systems and all the x86
| microarchitectures. You can now easily distribute software that
| can be easily run on the vast majority of PCs and servers. Want
| it to do more? Consider becoming a sponsor. The impact the
| project is already having should be all the proof we need that
| it could be doing much more. In order for that to happen it
| needs your support.
| lhorie wrote:
| Semi off-topic, but having just come across this project, I
| just started looking back at some of your stuff and saw a
| comment about an ape chibicc fork. What's the state of that?
| Is it usable yet?
| jart wrote:
| It's usable but not polished yet. Lots of GNU extensions
| and an integrated assembler were added. You can read about
| them here: https://github.com/jart/cosmopolitan/blob/master
| /third_party... I have automated tests which do a two-stage
| build and test on every platform each time I run make, so
| it's going to stay stable. The two other things I want are
| an integrated linker and VLA parameters. Rui is helping me
| implement those. So I anticipate we can start distributing
| a APE build of complete chibicc toolchain very soon that
| you can use to build APE binaries on any platform.
|
| Keep in mind this compiler doesn't optimize. So if you're
| OK taking a 2x performance hit, you're going to find that
| your code compiles significantly faster than GCC or Clang
| and you're going to have the convenience of the fact that
| the whole toolchain will be a single .com file that you can
| just scp on any server for a fully deterministic
| reproducible build experience. It's also the most readable
| and hackable compiler codebase I've ever seen, so if you've
| ever wanted to pull out the dragon book at try your hand at
| optimizing, you'll find that chibicc is fertile ground to
| do it.
| lhorie wrote:
| Very exciting stuff! I'll try to take it for a spin
| tonight :)
| jart wrote:
| Here's something you can paste into your Linux terminal
| to get started: git clone
| https://github.com/jart/cosmopolitan cd
| cosmopolitan make -j8
| o/third_party/chibicc/chibicc.com -include
| libc/integral/normalize.inc -o hello.com.dbg
| examples/hello.c o/cosmopolitan.a
| o/third_party/gcc/bin/x86_64-linux-musl-objcopy -SO
| binary hello.com.dbg hello.com ./hello.com
| eximius wrote:
| > Want it to do more? Consider becoming a sponsor.
|
| How?
| Jayschwa wrote:
| https://github.com/sponsors/jart
| jart wrote:
| https://github.com/sponsors/jart
|
| Thank you Hacker News for your support!
| comex wrote:
| Fair enough. I'm not against your project. I realize that
| open-source projects have time/funding constraints and that
| you don't owe me anything.
|
| By "baking in native support for other architectures", I
| guess you're thinking of an approach similar to Apple's "fat
| binaries", which are just multiple independent binaries built
| for different architectures bundled together? That's a good
| pragmatic option, and definitely the best choice size-wise
| for small programs. On the other hand, it lacks forwards-
| compatibility: you're limited to the set of architectures the
| original developer decided on. And for large programs, the
| cost of N copies of the program can add up.
|
| Personally, I'm a fan of bytecode VMs such as WebAssembly.
| That way the binary can be truly architecture-independent,
| capable of supporting many different architectures or special
| architecture variants without bloating the size of the
| executable. On the other hand, if you wanted the binary to
| run out of the box with no dependencies, you would have to
| bundle an entire VM implementation into the binary for at
| least some architectures, and then you'd have your own share
| of "hello world" bloat. (I'm assuming there would be an
| option for the user to supply their own VM for less common
| architectures.)
|
| I do wonder how small one can make a WebAssembly VM, if its
| design prioritizes minimal size over maximal speed, while
| still aiming to get reasonable speed (i.e. not an
| interpreter)...
| jart wrote:
| Check out Wasm3 which is the fastest WebAssembly
| interpreter, and since they recently adopted Cosmopolitan
| it's now the most universal one too.
| https://twitter.com/wasm3_engine/status/1354699030165676033
| Cosmopolitan is designed to be complementary to visions
| like yours. If you believe bytecode is the future then
| Cosmopolitan can help your vision be successful by giving
| it the best performance and portability across PCs and
| servers.
| zamadatix wrote:
| The time period still continues until something other than x86
| is consistently runnable (emulation or not) or most apps have
| done significant rewrites, not when different hardware shows
| up.
| zxspectrum1982 wrote:
| Is it possible to build real applications with Cosmopolitan?
|
| I'm thinking stuff that requires other dependencies, e .g. build
| a universal Python interpreter by using Cosmopolitan. Would that
| be possible?
| cjohnson318 wrote:
| I wanted to see if I could use this with Python, so I Googled
| "python cosmopolitan", and this was the first result: Passion
| Python Sex Position - Cosmopolitan [0]. I had _completely_
| forgotten about that magazine.
|
| [0](https://www.cosmopolitan.com/sex-
| love/positions/a26874/passi...)
| trizic wrote:
| This looks great! Combining this with Nim would be extremely
| powerful. Is it a matter of changing compilers like musl or would
| some tweaks need to be done for it to work with Nim?
| mshockwave wrote:
| Shout out to the cosmopolitan project. It looks super cool.
| Though I can imagine there might be more ABI issues when you move
| to other languages like C++
| als0 wrote:
| There will be other architectures after x86 has died. Mark my
| words.
| brazzy wrote:
| Probably developed and run by intelligent rabbits to whom
| humans are fairy tales.
| KMag wrote:
| It wasn't that long ago that programmers had to be reminded
| that there were non-VAX processors.
| marcodiego wrote:
| Now that there is a possibility that arm machines may become more
| common, it would be interesting to be able to build multi-arch
| AppImages. Maybe Ryan Gordon FatELF could be part of it.
| refactor_master wrote:
| Using Docker and Python I can only say: very, very fat. The
| Tensorflow image is something like 5 GB.
|
| On the other hand, Docker images is literally the only thing
| taking up space on my hard drive by now.
| [deleted]
| lostmsu wrote:
| Or you can ship Java or .NET binary (obviously)
| ape4 wrote:
| Seems impossible on the face of it. Linux binaries have to start
| with 0x7F ELF and Windows binaries have to start with MZ or ZM so
| I don't get how it could work with both.
| icedchai wrote:
| Based on the generated code, you may need to broaden the
| definition of "binary" to include anything that is executable
| (shell script.)
| fao_ wrote:
| /bin/sh execution is supported at the kernel level iirc?
| Like, the minimum requirement for a linux distribution is
| /bin/sh, the rest of the system doesn't have to be there
| necessarily.
| [deleted]
| [deleted]
| chungy wrote:
| It's not really a shell script though, it just looks like one
| enough to the point that a bourne shell can modify it to be
| directly executable.
|
| I'd probably be most worried about the bourne shell part. It
| seems like it'll be easy for this to break on any "non-
| standard" shell that Linux might execute.
| icedchai wrote:
| This is true, but the shell is executing it "as if" it were
| a script, right? Until the binary part takes over...
| lpapez wrote:
| As explained by the author https://justine.lol/ape.html
| Dylan16807 wrote:
| In short: The start of the file is MZ but also a shell
| script. If you run it on linux it replaces its header with
| the ELF header, then execs itself.
___________________________________________________________________
(page generated 2021-02-11 23:01 UTC)