[HN Gopher] Show HN: A 166 KB file for cross compiling glibc for...
___________________________________________________________________
Show HN: A 166 KB file for cross compiling glibc for any version,
any target
Author : AndyKelley
Score : 160 points
Date : 2021-12-13 10:05 UTC (12 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| richardwhiuk wrote:
| What's the point in this? Surely you'll have more dependencies
| than glibc?
| zokier wrote:
| glibc is notoriously difficult to link against; most other
| typical libraries have simpler ABI.
|
| This RH blog post is good intro on the topic:
| https://developers.redhat.com/blog/2019/08/01/how-the-gnu-c-...
| klodolph wrote:
| You will, but glibc is a common dependency on a dynamic lib.
| It's nice to be able to compile something and target an old
| glibc so your program can run on old systems (and by "old" I'm
| just talking about LTS).
| throwaway525142 wrote:
| What does "targeting any glibc version" mean? Is it the solution
| to the problem where you have to compile on an ancient Debian
| version to produce a binary that works on all Linux distributions
| because it otherwise links to too-new symbol versions in glibc?
|
| Can I use that outside of Zig, for compiling C++ code?
| saurik wrote:
| The correct solution for that is to just cross compile to
| whatever older CPU you want and target a sysroot that has a
| base image of whatever older glibc you wanted: you don't need
| to actually compile _on_ the older device...
| zX41ZdbW wrote:
| Example: https://github.com/ClickHouse-Extras/sysroot
| wyldfire wrote:
| zig also functions as a c and c++ compiler via 'zig cc'.
| jokoon wrote:
| Oh, zig is not in https://learnxinyminutes.com/, someone should
| add it :)
| laserbeam wrote:
| I'd wait for zig 1.0 or 1.1 before that. Syntax may still
| change a bit and it would be a shame to maintain that.
|
| The zig language doc is pretty good and short, and the lib/std
| folder is full of pretty easy to understand examples.
| ridiculous_fish wrote:
| How does one go about testing against arbitrary versions of
| glibc?
| klodolph wrote:
| I'd say that the probable scenario here is that you're
| targeting some specific LTS Linux distros, and you want to be
| able to run on the specific version of glibc on those distros.
| Normally, binary compatibility goes forwards but not backwards.
| What this tool lets you to is target an old glibc without
| needing to install that specific glibc.
| ridiculous_fish wrote:
| In my case, older versions of glibc (< 2.24) have a bug which
| prevents using posix_spawn. I'm not sure exactly which
| version of glibc has the fix. I would like to test against
| all glibc releases in turn, to identify the first in which
| posix_spawn is usable, but AFAICT there's no easy way to do
| that.
| modeless wrote:
| Something is very wrong with the state of C/C++ compiler
| development that it takes a different language to come in and
| make these obviously useful convenience features. Why are GCC and
| Clang not doing this?
| klodolph wrote:
| This is not an issue with GCC or Clang as far as I'm aware, but
| with glibc and the way it handles backwards compatibility.
|
| Versioned symbols get added when a function or object breaks
| binary compatibility. You keep around the old version, and
| compile against the new version. This might be something simple
| like a struct layout change, or a change in the size of an
| array. Both versions have the same name. When you compile,
| you'll get an unversioned symbol reference. When you link, that
| will get resolved to whatever the latest version is.
|
| Windows handles this by giving a new name to new symbols. For
| example, MoveFile(), MoveFileA(), MoveFileW(), MoveFileExA(),
| MoveFileExW(). You get the correct symbol when you compile,
| either by calling the correct function directly, or by
| referring to it by macro (MoveFileEx is a macro for MoveFileExA
| or MoveFileExW). Newer versions of Windows also make you embed
| the GUID of the versions of Windows you support in your
| application manifest, and Windows will run your application in
| a compatibility environment (at runtime) matching the latest
| available version you specify.
|
| On macOS, similarly, it uses the preprocessor, but it's a bit
| different and macOS also supports something called "weak
| linking" (not the same thing as weak linking in GNU Binutils /
| ELF). Weak linking allows you to link against an old version of
| a library, but use a symbol from a newer version... if the
| symbol is not available, it is NULL. This is done with the
| preprocessor and the linker in tandem... there's a preprocessor
| macro which specifies which minimum&maximum version of macOS
| (or iOS, etc) you target, and that affects which symbols are
| declared as being weakly linked.
|
| The long and the short of it... you can use the latest macOS
| SDK to make binaries compatible with old versions of macOS, and
| same for Windows, but glibc maintainers have not made this
| possible for glibc.
| AndyKelley wrote:
| Some context:
|
| In master branch of Zig right now (before
| https://github.com/ziglang/zig/pull/10330 is merged), status quo:
| $ cat ../lib/libc/glibc/*.txt | wc -c 205219 $
| cat ../lib/libc/glibc/*.txt | xz | wc -c 22976
|
| Zig supports targeting every version of glibc for any target
| architecture. The information required to do this takes up 200 KB
| installation size / 22 KB tarball size, and it has the following
| problems (which are causing several bugs):
|
| * fails to represent symbols that migrate from one library to
| another between glibc versions
|
| * fails to represent a distinction between functions and objects
|
| * fails to represent object sizes
|
| With this new glibc-abi-tool: $ cat abilists |
| wc -c 169421 $ cat abilists | xz | wc -c
| 24880
|
| 165 KB installation size / 24 KB tarball size, and all the above
| issues are solved. It also should be slightly faster to
| load/parse for the compiler since it is fewer KB to load from
| disk.
|
| The code that uses this file is here:
| https://github.com/ziglang/zig/blob/0d7331b6f6af85bd45829514...
| It creates c.s, pthread.s, dl.s, etc., which are assembled into
| libc.so, libpthread.so, libdl.so, etc., which are placed on the
| linker line when cross compiling. Thanks to this improved
| dataset, they can now accurately model the glibc that will be on
| any CPU architecture, any glibc version.
|
| If I just naively shipped every version of the .abilist files:
| $ cat (find glibc/ -name "*.abilist") | wc -c 37041906
| $ cat (find glibc/ -name "*.abilist") | xz | wc -c 205036
|
| ...it would be 35 MiB installation size, 200 KB tarball size. So
| I have effectively achieved a compression ratio of 219:1 by
| implementing a bespoke encoding of this information.
| gavinray wrote:
| Can I use this incredible feature with other toolchains, or
| existing artifacts that have a glibc dependency?
|
| Say if I am using a language/toolchain which pulls in a C/C++
| compiler, are you able to substitute "zig cc" there and
| explicitly cross-compile to a different GLIBC version?
| AndyKelley wrote:
| Yep! Example: https://dev.to/kristoff/zig-makes-go-cross-
| compilation-just-...
| gavinray wrote:
| Incredible, thanks a ton for doing this work!
| formerly_proven wrote:
| Totally off-topic: I installed Zig on Saturday afternoon and by
| the evening I had my first program do some useful stuff, on
| Sunday I added some C code, then added some tests and by Sunday
| evening I had a pure Zig project again (and afterwards learned
| that translate-to-zig is a thing). I gotta say I'm very
| impressed with Zig and partial evaluation is just such a nice,
| mentally simple technique for generics and lots of other
| things. The syntax is pretty nice, too, though the prefix-[]
| are kinda weird (just way too used to C/C++'s postfix array
| notation) and ranged loops are somewhat awkward. The @ naming
| for builtins confused me for a bit because I thought the @
| itself was syntax. And array literals are kinda weird sometimes
| though I've seen that apparently in 0.9 "&[_][]const u8{
| "-std=c99", "-O3" }" can just be written as ".{ "-std=c99",
| "-O3" }". And I was kinda surprised about array types in
| function signatures being implicitly const, I think they're
| like that to make array types behave mostly like value types
| while only passing a pointer?
| dnautics wrote:
| > the prefix-[] are kinda weird (just way too used to C/C++'s
| postfix array notation)
|
| I think over time the C/C++ way is going to go the way of the
| dodo; IIRC, rust and go both use the prefix-[]. It creates a
| clear, unambiguous way of defining the type (no "spiral
| typing" jokes for the modern languages).
| nyanpasu64 wrote:
| I _wish_ Rust used prefix &[]i32 for a slice, and
| [5][2]i32 for a 5x2 array. Instead it uses &[i32] for a
| slice, and [[i32; 2]; 5] for a 5x2 array. Reading types
| backwards is alive and well.
| mmastrac wrote:
| I prefer the unambiguous nesting of Rust's style. I can
| never remember the array shape when array bounds are
| listed in either prefix or postfix style. It's a tiny bit
| of extra typing that ensures that my understanding of the
| code is always 100% correct.
| dnautics wrote:
| eh, the type surrounded by the [] "operator" is equally
| unambiguous as prefix-[], so it's not the worst.
| Spex_guy wrote:
| Your instinct is right about array types, semantically in Zig
| they are passed by value. Parameter values are always const
| though, which allows the compiler to sometimes use pass-by-
| const-ref to avoid copying large pieces of data. Since arrays
| are proper value types, we can also make strongly typed
| pointers to them, like `*[3][4]u32`.
| lhorie wrote:
| How much work is it to then maintain this bespoke encoding?
| Without having much context on it, it sounds similar to having
| to maintain unicode identifier character tables for unicode-
| aware parsers (e.g. Javascript's). Updating unicode tables
| after unicode version releases can be done w/ somewhat hack-ish
| scripting of parsing txt spec files but it's admittedly not
| super clean, I'm curious what's involved when dealing w/ abi
| lists over time.
| Spex_guy wrote:
| The process of adding new versions of glibc to the data set
| is described in the readme. If something changes in a way the
| format doesn't handle, we can just change it and update the
| code in the compiler at the same time. The file ships as part
| of the compiler distribution so we don't have to worry about
| backwards compatibility.
___________________________________________________________________
(page generated 2021-12-13 23:01 UTC)