[HN Gopher] Actually Portable Executables
___________________________________________________________________
Actually Portable Executables
Author : krab
Score : 611 points
Date : 2021-02-28 10:06 UTC (12 hours ago)
(HTM) web link (ahgamut.github.io)
(TXT) w3m dump (ahgamut.github.io)
| EamonnMR wrote:
| I wonder if you could somehow get this working with libraries
| that are widely ported across platforms already like SDL.
| phire wrote:
| It might take a bit of effort, but should be possible.
|
| One problem for libraries like SDL is they disable all their
| per platform code with #ifdefs. To build a static, cross
| platform version you would have to convert them all those
| #ifdefs to runtime branching, dynamically load libraries and
| provide headers files for all platforms.
| temp00345 wrote:
| Amazing accomplishment !
|
| Not to detract from the technical breakthrough, but won't this be
| a major gift to virus writers ?
| Nextgrid wrote:
| The hard part when making a virus is to exploit the target
| system to run said virus. And if you have such an exploit, then
| supplying a compatible binary depending on which OS the target
| system runs is easy.
|
| Exploit attempts for embedded systems (routers, cameras, etc)
| typically start with attempting to execute code in a portable
| language such as Shell scripts (which would be the only thing
| this new project would replace), those scripts try to detect
| the architecture of the system and then download the
| appropriate binary (the server hosting the malicious binaries
| provides many variants for different architectures).
|
| In the end I don't see this solving a real problem when it
| comes to malware - this problem has already been solved through
| other means.
| jxf wrote:
| Just because an executable is cross-platform doesn't mean it
| can exploit issues on other platforms. Those issues have to
| actually exist there, and most vulnerabilities don't look like
| that.
| al2o3cr wrote:
| The dream of "Write Once, Debug Everywhere" will never die
| jart wrote:
| Cosmopolitan Libc author here. When I saw this article earlier
| today, it put a big smile on my face, because I would have
| thought there'd be so many more issues than there turned out to
| be! For additional context, we've got a GitHub issue that's
| tracking progress on the changes that need to be made based on
| what we learned: https://github.com/jart/cosmopolitan/issues/61
| s_Hogg wrote:
| Congratulations on all you've done so far, it's amazing.
| ahgamut wrote:
| I'm the author of the blog post!
|
| As someone who has never messed around with Libc-level
| programming, it was surprisingly straightforward (and
| exciting!) to compile Lua all the way.
|
| Cosmopolitan is incredible, thank you so much.
| egeozcan wrote:
| Only your own creativity seems to match your intelligence;
| excellent, excellent work!
|
| I was wondering if we can accompany runtimes written in C (like
| lua) with some scripts? So we have cross-platform, double-
| click-to-run scripting?
|
| For us poor souls who can't write reliable C code, that would
| be a great thing!
| jart wrote:
| Yes we can do that. There's a real opportunity here to define
| a novel API / framework on top of Cosmopolitan that does just
| that. It's something I myself have personally refrained from,
| mostly because fiddling with the low level bits is what I'm
| best at, and novel APIs require a certain kind of artist. But
| I'm certain that someone if not myself at some point is going
| to come along and use Cosmo for that purpose and it'll be
| pretty cool.
| ducktective wrote:
| Any recommendations to somebody who wants to learn C "properly"
| in $CURRENT_YEAR?
| himujjal wrote:
| APE makes it viable for a cross platform C library with a
| cross platform C compiler to make cross platform binaries.
|
| Besides I was in the same boat as you. I come from the world
| of JS/Python/Go. I even wrote
| https://github.com/Himujjal/ekon in pure C recently. The
| reason I thought C would be good is performance and
| portability. But I found it to be better to invest time in
| Rust rather than in C. C is a fantastic language but cross
| platform dependency management is difficult. Unit Testing is
| also difficult. There are solutions but not as efficient as
| Rust's ecosystem.
|
| BTW I wish there could be a Cargo for every language.
| kamranjon wrote:
| I am just trying to learn myself so take this with a grain of
| salt but I have found the github of
| Antirez:(https://github.com/antirez) - the author of Redis -
| to be an amazing resource. The code is incredibly well
| documented and to my eyes pretty well written. A good place
| to start I think is
| Kilo:(https://github.com/antirez/kilo/blob/master/kilo.c) - a
| 1000 line text editor with no dependencies.
| ktr wrote:
| I really enjoyed Expert C Programming
| (https://www.amazon.com/Expert-Programming-Peter-van-
| Linden/d...).
| alisonatwork wrote:
| I can't remember where I learned the basics, but if you
| aren't new to programming in general then I think there are
| only three important things you need to know: what is a
| pointer, what is a macro, and always free what you malloced.
| Reading a book or tutorial will be helpful for this. In my
| day it was K&R, no doubt there are more up-to-date ones now.
|
| But the next step, which I think is far more important, is to
| look at the source code of tools that you actually use in
| real life. Things like cp, or wc, or head. You've probably
| used them for years without thinking about it, but they're
| all written in C. Don't look at the modern GNU versions just
| yet, since they can be packed full of complex functionality,
| start with micro implementations like Busybox or Toybox. Then
| look at some OSes that are known for super clean code like
| NetBSD or OpenBSD. In those OSes you have the benefit of
| being able to go back to the version that existed 25+ years
| ago, so you can read through the diffs to see how they
| adopted new features and found new ways to address C's
| biggest flaw/risk - memory exploits.
|
| If you're interested in kernel programming, check the Linux
| and BSD sources, and lurk on the mailing lists for a while to
| see how people talk about the code. The review process tends
| to be a bit more brusque than you might be used to on Github,
| but it's often detailed and results in code that is of a
| relatively high quality, or at least a consistent standard.
| It's a great place to learn.
| unkulunkulu wrote:
| $CURRENT_BEST_BOOK Sorry :)
| dj_mc_merlin wrote:
| K&R -> do some random programming problems -> small project
| -> learn ASM -> learn how to do a buffer overflow (read
| "Smashing the stack for fun and profit") -> from here find
| your own path.
|
| The first three are just useful for getting the basics. The
| next two are for starting to learn low level stuff. You don't
| really understand C until you understand how it relates to
| lower level code.
| jtvjan wrote:
| I really liked "The C Programming Language (second edition)"
| by Kernighan & Ritchie. It doesn't teach C99 and later
| features, but if you want to write portable code you're not
| using those anyway.
| e12e wrote:
| From various discussions on Hn, I think the consensus is
| most people really like k&r as a technical book (it's
| concise, clear and a fun read) - but everyone working with
| C says its a terrible book for learning modern C.
|
| I tried to refresh my memory from previous threads, and I
| think the general trend has been to recommend (as seen in
| sibling comments):
|
| https://modernc.gforge.inria.fr/
|
| https://nostarch.com/Effective_C
|
| I've also seen the older:
|
| https://www.oreilly.com/library/view/21st-
| century-c/97814493...
|
| Mentioned.
|
| There's also Architecture of Open Source Applications,
| which can help with starting to read some larger code
| bases, some of which are in C:
| http://aosabook.org/en/index.html
|
| And there's a general recommendation to go read the source
| code of the Redis cache/db.
|
| Finally i came across a mention of this short article on
| gdb (nb: mention of TUI text ui should probably have been
| in the top, not a foot note):
|
| https://www.recurse.com/blog/5-learning-c-with-gdb
|
| I feel I'm missing a book that has come up often, but can't
| think of which one.
|
| I did like zed Shaws learn c the hard way, but I'm afraid
| it's getting a bit long in the tooth:
| https://learncodethehardway.org/c/
| sillysaurusx wrote:
| K&R is a horrible way to learn C. It's the best way to make
| someone run in the other direction, though.
|
| It's like telling someone that SICP is the best way to
| learn Scheme. Maybe, but lots of people learn differently.
|
| Unfortunately I don't have good suggestions myself, but at
| least they won't feel bad if K&R isn't their cup of tea.
| CodeArtisan wrote:
| C Primer Plus by Stephen Prata is the most complete i have
| seen, it does cover everything up to C11 and goes in details
| about stuff like array vs pointer.
|
| I don't like 21st Century C, it may have changed since then
| but it has a chapter titled "Object-Oriented Programming in
| C" which is confusing object with Abstract data type.
| edgyquant wrote:
| I assume object oriented C means using structs and function
| pointers?
| CodeArtisan wrote:
| No. For 21st Century C, an "object" is an incomplete type
| and procedures. struct Object;
| void Object_init(struct Object *o);
|
| That's ADT.
| vkazanov wrote:
| Try new books: effective c and the free modern c one. They
| actually teach the language and all the concepts.
| asicsp wrote:
| https://news.ycombinator.com/item?id=24361469 - 'Modern C'
| is this one?
| vkazanov wrote:
| Yup
| CraigJPerry wrote:
| https://www.oreilly.com/library/view/21st-
| century-c/97814919...
|
| I thought 21st Century C was good, i've still kept my copy.
| I'd happily recommend it.
|
| I like the K&R book too - it feels reeeeeeally old but it's
| really short.
|
| There's a few others that have helped me in various ways but
| these are a little older -
|
| Love C by Tim Love (free online, my copy is something i just
| printed out, it's not that long).
|
| Programming from the Ground Up (x86 assembly) - this is
| available freely online but i bought the book and that helped
| me a lot with C even though it's a book with only assembly...
| (to be fair, it does go through calling conventions).
|
| Finally there's another book i love, Advanced Programming in
| the Unix Environment by Stevens, i have the 6th edition
| updated by Rago after Stevens' passing. Fascinating book -
| but huge.
| shawxe wrote:
| So first of all, I just want to echo the general sentiment here
| and say that all of this is beyond awesome. Cosmopolitan libc
| seems to have the potential to literally re-define what
| constitutes portable code.
|
| With that out of the way, am I understanding correctly that the
| way this works on Linux/Unix is that the modifies itself (by
| overwriting the EXE file header with an ELF header)? This seems
| to have the consequence of making that specific file no longer
| portable. If I'm understanding things correctly, it also looks
| like the QEMU hack for non-x86_64 architectures will only work
| once per file, since after the first time running the file it
| will no longer run as a shell script on Unix so the QEMU
| invocation will be unreachable.
|
| Have you considered adding workarounds for this?
| mikewarot wrote:
| I'm an old Delphi/Pascal application programmer here, trying to
| get the lay of the land in the C/C++ world.
|
| Why is there a forest of files in Cosmopolitan Libc? I tried
| looking at it on Github to see how things were done, and there
| were a lot of .h and .S files, I couldn't actually find the C
| source, though I'm sure somewhere in there is a .c file.
|
| Doesn't fragmenting the source into thousands of files make
| compilation far slower than it needs to be?
|
| Also, I wonder if/how functions that aren't called in a program
| get trimmed away by the linker, and thus don't make the
| executable larger.
| jart wrote:
| It takes about 18 seconds to build the entire Cosmopolitan
| repo and run all its unit tests on my PC. During that time
| the make command builds 14,376 .o files, 66 .a archives, and
| 421 .com executables. It's an exceedingly fast build config.
|
| Having lots of objects is a good thing because it helps
| static linking work better. When the Unix linker loads a
| symbol from a static archive, it pulls in the whole module
| that defines that symbol. For example, if you define memcpy()
| and memset() in the same .c file and then your app only needs
| one of them, they'll both go towards bloating your binary.
| Workarounds exist like -ffunction-sections and -Wl,--gc-
| sections but a C library should make assumptions about the
| fewest flags feasible.
|
| What's the end result? We're able to build executables that
| are 12kb in size which run on seven different operating
| systems. The big codebase is what made tiny binaries
| possible: https://justine.lol/cosmopolitan/howfat.html
| krab wrote:
| May I ask - what was the reason for creating
| APE/Cosmopolitan?
|
| I read your post about actually portable executable format
| but I wonder if it's something that you found immediately
| helpful for some project or if you work on it just out of
| curiosity.
| matheusmoreira wrote:
| Do you build on tmpfs? I started doing that and it makes
| builds ridiculously fast. $XDG_RUNTIME_DIR is perfect for
| this, I just make a directory there and symlink the build
| directory to it. Even added some makefile logic to ensure
| the link target exists:
| build_directory_link := $(shell readlink
| $(build_directory)) $(if
| $(build_directory_link),$(shell mkdir -p
| $(build_directory_link)))
| ed25519FUUU wrote:
| Good optimization, thanks for the pointer!
| wyldfire wrote:
| Also, if you are using BFD ld, you should consider gold
| or lld. Significantly faster if the link is the slow
| step.
|
| Also much lower peak memory usage for lld, which is great
| if you have multiple concurrent link steps.
| simias wrote:
| Compiling ASM and even C is very fast these days. No
| generics, small compilation units etc... Even with optims
| it's usually a non issue in my experience.
|
| For a datapoint it takes me ~2 minutes to build a reasonably
| fully featured ARM linux kernel with a cold cache on a 5 year
| old i5. wc counts 1399 CC, 65 AS and 411 LD calls. And of
| course incremental rebuilds are only a fraction of that.
| moonchild wrote:
| > Doesn't fragmenting the source into thousands of files make
| compilation far slower than it needs to be?
|
| This allows incremental compilation to work much better than
| it otherwise would. Your first clean build may be _slightly_
| slower, but after that, you only have to recompile the files
| you change.
| RodgerTheGreat wrote:
| Have you considered writing an index for Cosmopolitan's API by
| topic? The current reference documentation is rather daunting,
| and it is very difficult to determine at a glance what sorts of
| functionality are available, and thus what kinds of
| applications could be written or ported.
|
| What would it take to create an analog of SDL- some kind of
| lowest-common-denominator interface for mouse/keyboard io,
| audio output, and software-rendered graphics?
| HexDecOctBin wrote:
| One question: is it possible to load .so files (through
| dlopen/dlsym) on Linux when compiling to APE?
|
| I was working on something similar (though much smaller in
| scope), but had to stop when I realised that `ld-linux.so` has
| some internal APIs that Glibc uses to setup dlopen()/dlsym();
| essentially meaning that it is very hard -if not impossible -
| to load any shared libraries if one does not link to Glibc.
|
| What I wouldn't give for a liblinux, similar to NTDLL.
| surajrmal wrote:
| It doesn't appear to be supported in their libc
| implementation:
|
| https://github.com/jart/cosmopolitan/blob/d7ac16a9ed56ebdc70.
| ..
| surajrmal wrote:
| Looks like you're also giving up position independent code
| as that also relies on the dynamic linker.
| e12e wrote:
| Not sure if it's been mentioned, but the logical next step
| would seem to be extending the zip hack[1] to allow embedding
| lua source with the binary?
|
| [1] https://news.ycombinator.com/item?id=26271117
| lostmsu wrote:
| Copy-pasted from thread about libc:
|
| I missed an opportunity to ask in the previous thread: what
| would it take to link an app in a different language (say,
| Rust) with this library? Is it enough to just build an object
| file, that has libc functions assuming LP64 ABI as unresolved
| exports?
| stephc_int13 wrote:
| In my opinion, the most important drawback to your work and its
| adoption is the use of fancy unicode characters in the name.
|
| I think your work is outstanding but it looks a bit
| childish/bizarre with this name, and thus it might prevent some
| people to trust its reliability.
| metalliqaz wrote:
| you must mean ape. But the package itself doesn't seem to use
| any such characters.
| lordofgibbons wrote:
| The future is now, old man
| [deleted]
| chacha2 wrote:
| Naming Your Child | David Mitchell's Soapbox
|
| https://www.youtube.com/watch?v=Xblh12XgQ4o
|
| The first half is relevant.
| dennis2 wrote:
| Nice
| barneygale wrote:
| It would be fantastic to see a byte-by-byte breakdown of the
| universal header.
| csmpltn wrote:
| I'm not sure what problem Cosmopolitan is trying to solve.
|
| I think it's an interesting research project, but from a
| practical perspective - there are significantly better (and more
| mature) ways to achieve cross-platform compatibility that don't
| involve many of the compile-time and run-time hacks required to
| get Cosmopolitan to work.
|
| "Actually Portable Executables" are not any more "portable" than
| a statically compiled .exe running natively on Windows, or using
| Wine[0] on a Linux, as an example.
|
| [0] https://en.wikipedia.org/wiki/Wine_(software)
| jart wrote:
| I'm sorry but WINE is the kind of interpreter that gives me
| enough time to take a coffee break as it loads solitaire.exe. I
| needed to use it a few weeks ago because I'm supporting someone
| who needs it for their CI environment. I discovered that APIs
| for things as fundamental as things like memory that Microsoft
| introduced back in 2006 hadn't been implemented in the WINE
| release Debian installed on my machine that was made in 2019.
| As far as I can tell the project wasn't funded for the longest
| time, which is unfortunate, because a once great tool then
| succumbed to tragedy of the commons.
|
| That's why I hope Cosmopolitan can have a positive impact.
| Because interpreters are great but we shouldn't need to rely on
| them as much as we currently do. The only thing your Actually
| Portable Executables need is the canonical stock preferred
| kernel interfaces that have indefinite stability promises
| backed by the most powerful technology corporations on earth.
| No one got fired for depending on the IBM's of the world. Your
| software is going to have a long life with little maintenance
| and best of all it's going to be fast.
| csmpltn2 wrote:
| Thanks, but I still can't tell from your response what
| problem Cosmopolitan is trying to solve.
|
| I came across some of the following statements in your
| website (https://justine.lol/cosmopolitan/index.html):
|
| > "Cosmopolitan makes C a build-once run-anywhere language,
| similar to Java, except it doesn't require interpreters or
| virtual machines be installed beforehand"
|
| Glancing at the code so-far, Cosmopolitan is really just an
| alternative libc implementation. It's important we
| acknowledge that there's only so much you can do with a libc
| implementation: it's a paper-thin abstraction layer for a
| small subset of rudimentary APIs.
|
| It's common for software today to depend on operating system
| interfaces (which are platform-specific), either directly
| (via static/dynamic linking with operating system libraries)
| or indirectly (delegating to a runtime environment such as
| Java, Python, .NET Core, Mono, Wine, and others). I don't see
| how Cosmopolitan caters for these scenarios, and I don't see
| why you would make that comparison in the first place.
|
| > "Cosmo provides the same portability benefits as high-level
| languages like Go and Rust, but it doesn't invent a new
| language and you won't need to configure a CI system to build
| separate binaries for each operating system"
|
| Cosmopolitan doesn't change anything about the C language
| itself, right? How portable a certain C project is depends
| entirely on the business logic within the code. Any C code
| using nothing but libc is pretty much guaranteed to be
| "portable" (it can be built using almost any combination of
| hardware, operating systems and compilers) - I'm not sure
| what Cosmopolitan contributes to the portability story?
|
| > "What Cosmopolitan focuses on is fixing C by decoupling it
| from platforms"
|
| C is already "decoupled from platforms", so I'm confused by
| this statement. There are a lot of projects out there that
| don't use libc at-all, or use their own custom abstractions
| on-top of platform-specific code (#ifdefs, fancy
| buildsystems, etc). There's nothing about Cosmopolitan itself
| that makes it somehow "decoupled from platforms": the
| decoupling happens by virtue of writing code that relies on
| the public interfaces exposed by libc, and the ability to
| swap libc at both compile- and runtimes.
| dvt wrote:
| HN has been obsessed with this project for the past few
| days, and that might be why you're getting so aggressively
| downvoted. Cosmopolitan is interesting from a technical
| perspective, but doesn't really bring anything new to the
| table.
|
| Packaging multiple entry points in the same file is a
| clever hack but is extremely brittle (for example,
| archiving/re-archiving can mangle relevant sections which
| ZIP doesn't care about -- this _will_ break your binary),
| ELF uses a pre-processing step which renders the binary
| non-portable once it 's run, and the cross-platform libc
| stuff has been done since time eternal.
|
| Not to mention that, as you say, _just C_ is nigh useless
| these days, anyway. It 's kind of funny to see people
| comparing this to Electron.
| roca wrote:
| Some of these statements are very misleading too.
|
| > Cosmopolitan makes C a build-once run-anywhere language,
| similar to Java, except it doesn't require interpreters or
| virtual machines be installed beforehand. Cosmo provides
| the same portability benefits as high-level languages like
| Go and Rust
|
| That is simply not true. Java, Go and Rust can run programs
| at native speeds on non-x86-64 architectures, e.g. ARM, but
| Cosmopolitan cannot, not now and not ever because x86-64 is
| a terrible IR.
| t0astbread wrote:
| For the record, I've tried running a recent game (made in
| Unity, I believe) on WINE a while ago and it worked like a
| charm. So I guess it's hit or miss?
| VMG wrote:
| I'm surprised by this. I'm using WINE to play League of
| Legends on Linux and it works surprisingly well.
| jonathanstrange wrote:
| Once an application is running it's reasonably fast but
| Wine has always taken endlessly time to start up, at least
| on my machine. It feels similar to Java VMs in that
| respect.
| throwaway525142 wrote:
| Wine is actually quite similar to this: It's not an
| interpreter or an emulator, it natively runs the executable
| on the host CPU. What it does is provide the WinAPI dynamic
| libraries that the programs expect (so they can call
| functions like CreateFile) and a loader that can load the exe
| files (PE) into memory and knows where to jump to.
| [deleted]
| bkmeneguello wrote:
| I think the major problem it solves is to not require multiple
| binaries to be distributed, one for each OS since them all runs
| on the same architecture. By distributing a single binary that
| really runs anywhere without requiring another solution, like a
| VM, an emulator or a compatibility layer, it simplifies the
| installation process, and allows the projects to be available
| for more users without increase the build complexity.
| alisonatwork wrote:
| You're thinking in the wrong direction. WINE can let Linux
| users run Windows binaries, but then as a programmer you need
| to write Windows-y C. What makes this appealing to me is that
| as a Windows user I can write UNIX-y C and it will just work.
|
| Obviously Windows could also be supported by recompiling under
| Cygwin, or MSYS2, but those tools are challenging. Compiling
| code designed for a UNIX on Windows is a pain in the ass, and
| sometimes the end result has strange behaviors. Upstream
| projects tend not to care much about Windows support, so trying
| to fix it you're on your own.
|
| Nowadays we have WSL so we can just run Linux binaries on
| Windows as-is, but that's still a fairly heavyweight solution.
|
| Wouldn't it be better to just compile once on UNIX, link to a
| magical portability library, and now you have a binary that
| just works everywhere? I think so.
| jancsika wrote:
| > Obviously Windows could also be supported by recompiling
| under Cygwin, or MSYS2
|
| MSYS2 is a dream. You install it, specify your dependencies
| using Arch's pacman, and go to town. No fiddling with Windows
| configuration files or paths, it just works. In fact it has
| been easier to get new devs up and running in that
| environment than in Linux, where they tended to smear build-
| time dependencies across the bare metal install. Of course
| there are solutions for that, but those solutions are more of
| a pain to set up than MSYS2.
| 1996 wrote:
| I'd like to see an MSYS2 compiled with cosmopolitan
| alisonatwork wrote:
| The cool thing is that it wouldn't really be MSYS2. It
| would just be coreutils, or binutils.
|
| The dream is that a programmer can just write their tool,
| using standard ISO C, maybe some POSIX, then compile and
| link it to Cosmopolitan and they're done. No need to set
| up any special configs or ifdefs. No need for any
| platform specific distributions. You just compile once
| and now you have the canonical version.
| GordonS wrote:
| I tried to get Wine to run a dotnet framework console app last
| week. It was the first time I've tried to use Wine, but it was
| way more complicated than I thought it was going to be - many
| magical incantations are needed, which vary by OS, hardware
| platform, OS platform and binary platform. And it's very slow.
| I spent about 4 hours trying to get it working, and after all
| that time, all I'd achieved was bombing on a Crypto API error.
|
| I gave up.
| cfstras wrote:
| Yes, as they require a significantly less amount of stuff you
| need to deal with. WINE needs to implement vast amounts of
| Windows APIs to make something like .net work, while
| cosmopolitan limits you to low-level C (and things built on it,
| like lua) at compile time.
| csmpltn2 wrote:
| > while cosmopolitan limits you to low-level C (and things
| built on it, like lua) at compile time
|
| You can build very little with the C Standard Library alone,
| wouldn't you agree? Things like graphics, anything beyond the
| most rudimentary I/O, etc - are not a part of the C Standard
| Library.
| A12-B wrote:
| This is a theoretically more pure solution because you are
| trying to break down compatibility to its common factors, the
| processor, and then do as little work as possible to get it to
| the point of portability. Less abstraction than other
| solutions. The holy grail would be a generalised solution for
| any processor, using nothing but electrical engineering first
| principles, but I don't know if that's possible and maybe the
| better solution is to just not have the frankenstein's monster
| of hardware we have today.
| krab wrote:
| I just saw this tweet -
| https://twitter.com/JustineTunney/status/1365805503561957376
|
| Cosmopolitan is going to be a big thing.
| andai wrote:
| >We've just created a 336kb Lua interpreter binary that runs on
| seven operating systems, thanks to Cosmopolitan Libc.
|
| https://ahgamut.github.io/c/2021/02/27/ape-cosmo/
| svenfaw wrote:
| Awesome! But why no download link?
| ahgamut wrote:
| You can compile it yourself if you like. I created a fork
| of the Lua Github mirror and made the necessary changes to
| compile with Cosmopolitan:
|
| https://github.com/ahgamut/lua/tree/cosmopolitan
| pfortuny wrote:
| Because as OP says it is not totally complete and lacks
| some parts.
| poc43 wrote:
| Likely only if it's forked away from anything to do with its
| current maintainer, for reasons only a Google search away:
|
| http://valleywag.gawker.com/why-does-google-employ-a-pro-sla...
|
| That's the problem with trolling like that: once you do
| something cool that's unrelated and try to move on it's not
| clear if you're genuine about anything any more. The history
| there will absolutely make larger interests hesitant to
| associate, contribute or integrate in something like, say, LLVM
| (regardless of the merit or lack thereof), particularly in
| today's less forgiving culture.
|
| Note that I'm not sharing an opinion on the content. It's a
| metaobservation about where we are, for better or worse. That's
| on page 1 of my Google search for "wow, who's this amazing
| coder?", so my path is the same anyone else would take doing
| due diligence on getting involved.
| 1996 wrote:
| > Likely only if it's forked away from anything to do with
| its current maintainer,
|
| And exactly why should I care about the current maintainer?
| Except maybe to say "thank you"?
|
| Let's put the question upside down: what have you (or the
| gawker journalist) brought to me? I mean, besides your
| outraged opinions (which anyone is entitled to have!)
|
| On one hand, you have a wonderful technical creation by
| someone asking nothing in return. On the other, you are
| providing... hatred towards the author? On a 2 hours old
| account?
|
| I'm sorry if she breaks the kumbaya illusion, but people are
| not equal. Some can't escape the working class, or the
| welfare class - not due to any personal failure or limits,
| but due to societal indoctrination.
|
| Some other people break free, and then bring gifts of the
| gods to mankind. They are called innovators, entrepreneurs -
| the name vary. But you know one when you see one. There is
| nothing is cosmopolitan that was "impossible" to achieve by
| anyone dedicated, even years before. There's nothing magic-
| except in the mind of the author, that associated the right
| pieces together.
|
| > The history there will absolutely make larger interests
| hesitant to associate, contribute or integrate
|
| If anything, your comment make me less likely to associate
| with you!
|
| > particularly in today's less forgiving culture.
|
| I couldn't care less. The less forgiving culture is a problem
| for those who need to work ("will code for food" as they said
| during the dotcom crisis) or integrate with the rest of
| society. Unfortunately, that means most of the people here,
| especially those working for the FANGs.
|
| But there are some of us who just don't care - except to
| lament that, when shown the way to escape, most geeks double
| down on their own mistakes, and remain chained to their FANGs
| masters.
|
| So please go on attacking the author if that makes you happy,
| while the rest of us admire the creativity and new things
| made possible by cosmopolitan (and make money out of that
| too!)
| fwip wrote:
| If you don't care, that's okay. Other people are allowed to
| care and state their opinions.
|
| You may want to notice that the author, Justine Tunney, was
| also chained to her "FANGs master" Google during the period
| described.
| 1996 wrote:
| > Justine Tunney, was also chained to her "FANGs master"
| Google
|
| And now she's free, which creates an even more compelling
| narrative: true creativity can only shine in freedom, not
| in chains
| anthk wrote:
| >Rather than inclusionary leftism.
|
| Traditional left didn't give a shit on gender, race or
| whatever bullshit you think you should declare yourself
| "inferior" because of an excuse. Just work and equality.
|
| I hate both poshy techno-aristocrats (I would kick them out
| to Somalia or a Mexico narco shithole, to see what they can
| do without social support or a state backed police), and SWJs
| with far more points in common with fascism (Fuck that so-
| called cultural appropiation) than the common worker.
| poc43 wrote:
| I mean, you're engaging with the espoused opinion of
| someone who simultaneously advocated class warfare and drew
| Google payroll, so you may as well be arguing with a
| figment of our collective imagination. If that's not
| obvious by nature of how incongruous it is I fear for the
| Internet.
| fwip wrote:
| Thank you for sharing this, I personally am much less excited
| about the code now.
|
| I took a look through her twitter, and her current opinions
| seem to be broadly similar to the ones she expressed in 2014.
| ericb wrote:
| Would these binaries work on Apple Silicon? It seems like no,
| since it isn't x86? What are the limits to the portability, and
| is there a path past those limits?
| momothereal wrote:
| That's right, it only supports x86. However the binary will
| detect other architectures and try to run itself with QEMU in
| that case.
| samatman wrote:
| I would love for someone to do this experiment.
|
| In principle, Rosetta should detect that it's x86, perform the
| translation, and everything should "just work".
|
| In practice, it's possible that the weird things APE is doing
| to get cross-platform compatibility will undermine one or more
| of Rosetta's assumptions.
|
| I don't have an M1 Mac, so I'm not in a position to try it. But
| someone should!
| antman wrote:
| Portable Lus is a great idea. By using Cosmopilitan does this
| also run on baremetal machines?
| [deleted]
| momothereal wrote:
| I wonder if there's a compatibility/sanity test suite for Lua? It
| would be nice to run this interpreter with real scripts to see
| how it compares.
| ers35 wrote:
| Maybe Cosmopolitan could be used with luastatic[1] to compile a
| Lua program to an "Actually Portable Executable":
| CC="" luastatic main.lua # Compile main.luastatic.c with
| Cosmopolitan Libc
|
| [1] https://github.com/ers35/luastatic
| antman wrote:
| Portable Lua is a great idea. By using Cosmopilitan does this
| also run on baremetal machines? Portable micropython might also
| be a low hanging fruit target.
| benatkin wrote:
| I'm not any more or less excited about this from a utility point
| of view than I am about golang's support for embedding data in an
| executable.
|
| It is pretty damn impressive, though. I'll give it that.
|
| It's like that cat pushing a watermelon meme. This is a binary
| that runs on seven operating systems. Your argument is invalid.
| zbendefy wrote:
| Could webassembly be also used for this?
| AstralStorm wrote:
| That's not as portable as you need an interpreter.
| jart wrote:
| Yes it can! Wasm3 is the fastest WebAssembly interpreter. Since
| they recently adopted Cosmopolitan it became the most universal
| one too. Check out the 150kb APE binary they've got on the
| release page.
| https://github.com/wasm3/wasm3/releases/tag/v0.4.8
| iFreilicht wrote:
| Depends on what you mean by "using". It is conceivably possible
| to compile a webassembly interpreter like wac[1] with
| cosmopolitan, which would then run on all OSes and bare metal,
| yes.
|
| [1]: https://github.com/kanaka/wac
| [deleted]
| dazhbog wrote:
| Does anyone know if there is any way to run another exe/binary
| that its bundled in a cross platform app, while its all enclosed
| in 1 binary file?
|
| Because Cosmo could work as a wrapper of windows/mac/linux binary
| executables and then the cosmo app can decide which one to
| invoke.
|
| Based on my understanding if program A is to run program B,
| program B needs to be addressable in the filesystem. So based on
| my testing, the bundled executables need to be written to disk
| first and then the main program could run them. This also might
| work with macos .app folder formats but not sure how to do it on
| linux to avoid chmod +x issues and how to have 1 binary file on
| windows.
|
| Any ideas?
| pwdisswordfish0 wrote:
| Huh?
|
| > Cosmo could work as a wrapper of windows/mac/linux binary
| executables and then the cosmo app can decide which one to
| invoke.
|
| It sounds like you misunderstand the reason for the project and
| that you're trying to add another layer of indirection to the
| thing that is already able to achieve the effect that you you
| seek to accomplish with that layer of indirection. No need for
| the indirection, just use the thing directly.
|
| If you want to run $APP on platforms X, Y, and Z, then you'd
| use Cosmopolitan to build the $APP binary as an Actual Portable
| Executable, and you'd run that executable on any of those
| platforms.
| dazhbog wrote:
| When the source is available, then of course everything can
| be compiled in one nice executable. ;)
|
| I am referring to cases where the source is not available or
| its hard to set up some projects together. Some projects
| (usually hardware related) require different compilers,
| compile options, etc. If the project already has the
| binaries, then for distribution maybe a wrapper could be a
| good option.
| ngcc_hk wrote:
| This actually come as a surprise. Just to ask as a testing case -
|
| can we have an actually portable (common) lisp/python interpreter
| with at least terminal interface?
|
| Can the terminal have something like VT-100 or other basic
| interface? and
|
| Can we even run javascript to do screen related items ... like
| https://github.com/d3/d3
| matheusmoreira wrote:
| Yes. She built a javascript interpreter with cosmopolitan.
| Since the executables are also zip files, the interpreter can
| even load its own sources from the binary itself.
|
| https://github.com/jart/cosmopolitan/blob/master/examples/he...
| rahimiali wrote:
| The author mentions the portable binary is smaller than the
| original. It seems Cosmopolitan's output is actually an
| executable pkzip file: https://justine.lol/ape.html
| comex wrote:
| That's how I read it the first time through, but on a closer
| look, they're not comparing a normal native binary to an APE
| binary. They're comparing an intermediate step of the APE build
| process to the final output.
|
| `objcopy -S -O binary` does two things. One is to strip all the
| ELF headers and just dump the section contents into an output
| file. It looks like the input to this command is an executable
| within an executable, with the BIOS boot sector, ELF segments,
| Mach-O headers, etc. all embedded as section contents within an
| outer ELF file; objcopy then removes the outer shell. Clearly a
| necessary step, but it won't make much difference in binary
| size. However, the `-S` tells objcopy to strip debug
| information, which is included in the original since `-g` was
| passed to `gcc`. This is presumably responsible for almost all
| of the difference; debug info is notoriously huge. You can get
| a similar size reduction in a normal compilation by running
| `strip`.
|
| Edit: But objcopy does not do anything fancy like compress the
| input. It looks like the pkzip part only contains associated
| files the binary might need (e.g. "hellojs.com" contains a
| JavaScript file and time zone files), not the executable code
| itself.
| pbronez wrote:
| Something like this could be great for open source projects that
| target a lot of platforms. For example, Conda-Forge [0] has
| automated build pipelines for Linux, Mac and Windows. Perhaps
| something like this could let you reduce that to an x86 build and
| an Arm build... removing an entire dimension from the build
| matrix.
|
| Gonna need to solve that 1/2=g problem before Numpy is happy
| though!
|
| [0] https://conda-forge.org/
| abhinav22 wrote:
| Can somebody explain in layman's terms what this is all about? I
| tried my best reading the other thread, but it went well above
| me.
| Gaelan wrote:
| Windows, macOS, Linux, and BSD all (usually) run on Intel
| processors, so in theory it should be possible to write a
| program that runs on all three. However, despite ultimately
| using the same instruction sets, the four OSes use different
| formats to store metadata about the programs, and have
| different ways for programs to communicate with the operating
| system. Justine Tunney created two things:
|
| 1) Actually Portable Executables, a clever way of formatting a
| program so that all four OSes interpret it a valid program in
| their own format.
|
| 2) Cosmopolitan libc, a library for communicating with the OS
| that handles each OS's interface, allowing programs to work on
| all four.
| shrx wrote:
| They also run on AMD ;)
| munawwar wrote:
| So it's like a compatibly layer like wine except built into
| the binary. Does this bloat the binary as well?
| A12-B wrote:
| What happens when intel goes bust and we all have to switch
| to new processors
| OOPMan wrote:
| A little optimistic about Apples SoCs aren't you?
|
| Also...given that AMD exists and the x86 instruction set is
| not some magic potion ingredients...
| abhinav22 wrote:
| Thank you (and thanks to everybody else for their helpful
| answers). Your note on intel processors meaning in theory
| programs should run on all operating systems was most helpful
| and got me across the line - now I get it!
| jart wrote:
| That's pretty much it. Cosmopolitan Libc is the simplest
| project that could have happened decades ago. Anyone could
| have built it. There's never been a technical reason why an
| x86 binary can't run on all x86 operating systems. It's
| just that traditionally only operating systems publish C
| libraries, and they have no incentive to spend money
| supporting their competitors. So it's the kind of project
| that could only happen if it was done by an indie developer
| just trying to have fun.
| lambda_obrien wrote:
| Does cosmolibc allow you to run on bare metal? If so,
| I've got an old project in mind that you may have just
| made happen, and I'm very very excited to know!
| fctorial wrote:
| Another reason is that it requires fairly thorough
| knowledge of _7_ different programming environments. Most
| people don 't understand 1.
| jart wrote:
| The only programming environment I used to build this was
| Emacs on Linux. The rest was simply researching the magic
| numbers needed to use the binary interface of each
| system. I figured life's too short to be pulling my hair
| out with MSVC/Xcode warnings.
| fctorial wrote:
| s/programming environments/systems/
| thysultan wrote:
| How does one unzip the .com file? Is that even possible
| or have i misunderstood how it works?
| jart wrote:
| I recommend using InfoZIP on the Linux or Mac command
| line. For example, to list the files that are in your
| .com executables you can run a command like:
| $ wget https://justine.lol/redbean/redbean-2021-02-27.com
| $ unzip -vl redbean-2021-02-27.com Archive:
| redbean-2021-02-27.com Length Method Size
| Cmpr Date Time CRC-32 Name --------
| ------ ------- ---- ---------- ----- -------- ----
| 426 Defl:N 215 50% 02-09-2021 09:24 f9cc9464
| tool/net/redbean.css 896 Defl:N 509
| 43% 02-09-2021 09:24 eb74f8a0 tool/net/redbean.html
| 16958 Defl:N 6093 64% 02-09-2021 09:24 6a76bd39
| tool/net/redbean.ico 5073 Stored 5073
| 0% 02-09-2021 09:24 86c2afe6 tool/net/redbean.png
| 554 Defl:N 285 49% 02-09-2021 09:24 0aa6870a
| usr/share/zoneinfo/Beijing 2335 Defl:N
| 1093 53% 02-09-2021 09:24 6ccd4450
| usr/share/zoneinfo/Berlin 2453 Defl:N
| 1170 52% 02-09-2021 09:24 7fed6d80
| usr/share/zoneinfo/Boulder ...
|
| Another ZIP tool that works well with APE is Windows 10
| file explorer. But you have to change the extension from
| .com to .zip temporarily.
| galfarragem wrote:
| Will this rush an optimized version of Electron or even
| deprecate it?
| arendtio wrote:
| Well, the main problem with Electron is that every app
| ships its own browser using tremendous resources at
| runtime, having huge attack surfaces and lots of unused
| functionality. So just giving one optimized electron
| version to all users, certainly does not solve the problem
| as APE does nothing that changes these problems.
|
| Speaking of deprecating it, it might be one scenario, but
| developers probably don't just choose Electron because it
| is platform independent. I mean, there were other
| frameworks that supported that use-case even before
| Electron (e.g. QT). However, what is specific to Electron,
| is that you can use web technologies to build Apps. But
| that is also nothing you can do with APE.
|
| So as much as I like to see one binary format for many
| operating systems, I doubt that it will change anything
| regarding the Electron problem.
|
| There certainly multiple ways of solving it, but in my
| opinion, Apple and Microsoft must come to a common ground
| and support some kind of web-view that is built-in to the
| OS and extended with a platform independent API for the
| things that normal browsers do not support/allow, like
| filesystems access.
|
| At some point both supported the Progressive Web App (PWA)
| movement for a while, but afaik Apple is the one holding it
| back at the moment. My guess is that they saw their App-
| Store business at risk. And even if PWAs were a thing, they
| are still some use-cases they can not cover in the current
| state (e.g. missing filesystem access).
| mihcsab wrote:
| the main problem with electron is js
| simias wrote:
| Note that packaged electron apps are not portable across
| OS, so it's orthogonal. What this could do is make the same
| electron package work on different OS without rebuild by
| using a portable browser binary.
|
| Meanwhile Cosmopolitan doesn't provide a portable GUI
| toolkit so it can't replace electron on its own. Maybe if
| you could package Cosmopolitan + Qt or something like that
| you could end up with something very interesting though.
| cxr wrote:
| > What this could do is make the same electron package
| work on different OS without rebuild by using a portable
| browser binary.
|
| Likewise for non-GUI apps, like command-line utilities
| that are implemented with NodeJS. As prior art, Fabrice
| Bellard's QuickJS is able to compile JS programs to spit
| out a native binary that requires no external qjs
| runtime. If you modified it to output Actual Portable
| Executable binaries instead, it would be even more
| portable than the programs installed with `npm install`.
|
| I've done some experiments and progressed to writing
| tooling that approaches that _I-want-ridiculous-
| portability_ problem differently. It doesn 't use APE, so
| it still requires a separate interpreter (such as NodeJS)
| capable of executing the compiled output, but the object
| files it outputs are such that even if you don't have a
| copy of NodeJS installed on your system (or the _right_
| copy of NodeJS installed), then you can piggy back off
| the interpreter built in to your browser. You get a
| similar double-click-to-run experience, but when you do
| double-click to run, it launches in the browser, and you
| can go from object file to modifiable source trivially,
| since the object file and the Git repo from which it is
| built are automorphisms. The downside, if you consider it
| one, is that it doesn 't work with arbitrary NPM modules.
| You have to really buy in to the scripting language
| dialect and the development philosophy to get anything
| out of it.
|
| It would be interesting the use the QuickJS+APE strategy
| above so that there is _no_ reliance on an external
| interpreter, at the cost of marrying yourself to x86 and
| introducing some small hurdles in the path to
| modification and also giving up the safety of the browser
| sandbox, since you're back to running arbitrary
| executables.
| Nextgrid wrote:
| Technically, I guess you could use this to compile a
| universal browser binary that would run on all platforms.
| However, this wouldn't solve any of the problems caused by
| Electron.
| dj_mc_merlin wrote:
| I think his point was that with Cosmopolitan native
| binaries one would no longer need Electron to provide
| cross-platform binaries. There is a lack of good GUI
| frameworks to build this native Electron replacement with
| though, nor is it certain the effort would not be too
| high.
| darkfirefly wrote:
| Or really any of the problems Electron attempts to solve;
| the GUI toolkits are still not (generally) not cross-
| platform in the way Electron is.
| gostsamo wrote:
| In short, there is a "framework" that allows to compile a
| single binary that can run under both Linux, Windows, bare
| metal, and many others. Like the Apple universal binaries, but
| actually universal, not just covering the m1's arm64 and x86
| 64.
| Gys wrote:
| It is x86 only, arm is not supported. But would be a very
| nice addition :)
| TickleSteve wrote:
| non-x86 is emulated and bare-metal limits the portability.
| snarfy wrote:
| It makes binaries that can run anywhere by shimming(?) libc
| functions and values with a custom loader/linker.
| max_ wrote:
| You can compile a program on MacOS. The result, an executable
| binary, can run on Linux, Mac, Windows, FreeBSD, OpenBSD,
| NetBSD, BIOS .... the same EXACT binary!
| matheusmoreira wrote:
| The author somehow came up with an executable file format
| that's compatible with everything. It's simultaneously a valid
| shell script, Unix ELF, Windows PE file, boot sector code and
| zip file.
|
| Then she wrote a standard C library that detects which OS
| you're using at runtime and branches to the appropriate
| implementation, allowing the same x86_64 code to run on any
| supported system. if (IsWindows())
| DoWindowsThing(); else DoUnixThing();
|
| Traditionally, these system differences are resolved at build
| time: only the code for the target platform is included. Why
| would anyone want Windows-specific code in a Linux build? It's
| dead code... right? The new run-anywhere executable format
| makes that code useful.
| IshKebab wrote:
| Are you sure? Other answers are suggesting it isn't a valid
| ELF file and I don't get why it would need to be a valid boot
| sector or zip file.
|
| You might be right though. The author might be very good at
| executable hacks but they're a terrible writer. Their
| explanation explains nothing.
| reikonomusha wrote:
| The author seems to be an excellent writer. They just may
| not have taken the enormous amount of time to answer tons
| of questions or write tons of documentation people are
| seeking. One person did all this!
| IshKebab wrote:
| You really think so? Here's their description of how it
| works:
|
| > Here's how it works:
|
| >
|
| > MZqFpD='
|
| > BIOS BOOT SECTOR'
|
| > exec 7<> $(command -v $0)
|
| > printf '\177ELF...LINKER-ENCODED-FREEBSD-HEADER' >&7
|
| > exec "$0" "$@"
|
| > exec qemu-x86_64 "$0" "$@"
|
| > exit 1
|
| > REAL MODE...
|
| > ELF SEGMENTS...
|
| > OPENBSD NOTE...
|
| > NETBSD NOTE...
|
| > MACHO HEADERS...
|
| > CODE AND DATA...
|
| > ZIP DIRECTORY...
|
| Ok well that clears it up. MZqFpD='. Of course!
| arendtio wrote:
| > Why would anyone want Windows-specific code in a Linux
| build? It's dead code...
|
| You might want to ask the thousands of Electron App users how
| much they care about dead code ;-)
|
| I mean, you are right. But if you offer your users no
| alternative they might not care enough to choose another
| product. In addition, some might even value it not having to
| choose the right binary for their system.
| baq wrote:
| No worries. You're in good company. This actually portable
| executable thing is batshit insane (in the best possible way).
| TazeTSchnitzel wrote:
| Previous discussion of Cosmpolitan:
| https://news.ycombinator.com/item?id=25556286
| qyi wrote:
| Cool hack but extremely fragile. It's actually pretty sad that
| people will consider using something like this in real projects.
___________________________________________________________________
(page generated 2021-02-28 23:01 UTC)