https://lwn.net/SubscriberLink/904892/dba951441b61cbdc/ LWN.net Logo LWN .net News from the source LWN * Content + Weekly Edition + Archives + Search + Kernel + Security + Distributions + Events calendar + Unread comments + ------------------------------------------------------------- + LWN FAQ + Write for us User: [ ] Password: [ ] [Log in] | [Subscribe] | [Register] Subscribe / Log in / New account The ABI status of ELF hash tables [LWN subscriber-only content] Welcome to LWN.net Free trial subscription The following subscription-only Try LWN for free for 1 content has been made available to month: no payment or you by an LWN subscriber. Thousands credit card required. of subscribers depend on LWN for Activate your trial the best news from the Linux and subscription now and see free software communities. If you why thousands of readers enjoy this article, please consider subscribe to LWN.net. accepting the trial offer on the right. Thank you for visiting LWN.net! By Jonathan Corbet August 19, 2022 It is fair to say that some projects are rather more concerned about preserving ABI compatibility than others; the GNU C Library (glibc) project stands out even among those that put a lot of effort into preserving interface stability, So it may be a bit surprising that a recent glibc change is being blamed for breaking a number of applications, most of which are proprietary games. There is, it seems, a class of glibc changes that can break applications, but which are not deemed to be ABI changes. When the dynamic linker starts a program, it must resolve all of the symbol references into shared libraries (including glibc). That can involve looking up thousands of symbols in long lists. Since this process must complete before an application can actually start running, it needs to happen quickly. Nobody likes a long delay between starting nethack and facing off against that first kobold, after all. So it is not surprising that some effort has gone into optimizing symbol lookup. When the ELF file for a shared object is created by the linker, one of the sections stored therein contains a hash table for the symbols in that file. This hash table can be used to speed the lookup process and get the application underway. For many years, the System V standard for the format of this table has been DT_HASH; that format is supported by the toolchains on Linux. In 2006, though, the DT_GNU_HASH format was added as well; it includes a number of improvements intended to get nethack players into their dungeons even more quickly, including a better hash algorithm and a Bloom filter to short-circuit the search for missing symbols. This format is not well documented, but this 2017 blog post gives an introduction. Since the hash table lives in its own ELF section, there is nothing preventing an ELF file from having more than one of them. Linkers on Linux systems can be told to create one format or the other -- or to create both, each in its own section. Until recently, glibc has been built (by default) with a linker option explicitly requesting that both formats be created. That changed, though, with the glibc 2.36 release at the beginning of August; it contained a simple patch from Florian Weimer causing only the DT_GNU_HASH format to be generated. At this point, glibc 2.36 has been installed onto a number of systems running the faster-moving Linux distributions, and few people have noticed the change; the DT_HASH format has not been used for anything on those systems in many years, and the only consequence of its removal is regaining a small amount of disk space. Game players, though, were not so lucky; naturally, the problem relates to a piece of proprietary software that cannot be easily changed. The Easy Anti-Cheat (EAC) system is a proprietary tool from EPIC intended to prevent game players from cheating by way of modifications to the game executable itself. As one might expect, the actual heuristics used are not documented anywhere and the code is secret, but one of the techniques involved appears to be looking at the symbols in the game executable and ensuring that they match the expected values. To do this, EAC goes rooting through the DT_HASH tables; if an expected table isn't present, EAC proudly proclaims that it has caught a cheater and does not allow the game to run. Gamers, it seems, find this behavior disappointing. Arkadiusz Hiler, for example, blogged: I think this whole situation shows why creating native games for Linux is challenging. It's hard to blame developers for targeting Windows and relying on Wine + friends. It's just much more stable and much less likely to break and stay broken. Hiler reported the problem in the glibc bug tracker; the discussion later moved to the project's mailing list as well. The report stated that the change "`breaks SysV ABI compatibility'", suggesting that it should be reverted. The responses from the project were not entirely sympathetic to that cause, though. Weimer answered that: "`Any tool that performs symbol lookups needs to support DT_GNU_HASH these days'". Adhemerval Zanella said: "`I am not sure this characterizes as an ABI break since the symbol lookup information would be indeed provided (albeit in a different format)'" Carlos O'Donell agreed that this change was not an ABI break: Software that is an ELF consumer on Linux has had 16 years to be updated to handle the switch from DT_HASH to DT_GNU_HASH (OS-specific). While I'm sympathetic to application developers and their backwards compatibility requirements, this specific case is about an ELF consumer and such a consumer needs to track upstream Linux ELF developments. He went on to say that characteristics of the generated ELF file are not part of the glibc ABI, even if changes there break applications, and suggested that the bug should be closed as "won't fix". He also requested that the EAC developers provide reasons for why DT_HASH should be retained by default. As of this writing, the bug remains open. Blaming the EAC developers for not keeping up with Linux ELF hash-table formats might not be entirely fair. The DT_HASH format is mandated by the System V ABI specification, the DT_GNU_HASH format is undocumented, and there has been no deprecation campaign to get users to move on. Chances are those developers are as surprised as anybody and haven't just been ignoring the "switch to DT_GNU_HASH" entry languishing in their issue tracker for the last decade or so. Regardless of blame, though, something needs to be done to solve this problem and save gamers from the prospect of having to get some actual work done. If EAC were free software, of course, chances are there would already be a patch circulating to deal with the problem. As it is, only its owner can deal with this problem directly. Meanwhile, though, there is another workaround available: distributors can easily patch the glibc build to restore the DT_HASH section and make the problem go away for now. Doing that and giving EAC (along with a few other programs) some time to move to DT_GNU_HASH seems like the best solution from just about any point of view. [Send a free link] Did you like this article? Please accept our trial subscription offer to be able to see more content like it and to participate in the discussion. ----------------------------------------- (Log in to post comments) The ABI status of ELF hash tables Posted Aug 19, 2022 14:34 UTC (Fri) by josh (subscriber, #17465) [ Link] If EAC were Free Software (and still for some mystifying reason existed at all), someone could have observed before now that it used DT_HASH. Since it isn't, there was neither an issue tracker to file the bug in nor a straightforward means for someone to notice in order to file one. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 14:40 UTC (Fri) by immibis (subscriber, #105511) [Link] These kinds of anti-cheat systems fundamentally cannot be free software; rather, a free-software version of the game could not come with a useful anti-cheat system at all. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 17:13 UTC (Fri) by flussence (subscriber, #85566) [Link] Doing anticheat with free software isn't impossible, it just needs to be thought of differently; you could use the flight computer quorum model where each client does sanity checking and reporting on the input received from others and the server offlines anything that consistently runs out of spec. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 17:35 UTC (Fri) by tux3 (subscriber, #101245) [ Link] The problem is not just manipulated output from a client. If the server is not sufficiently fine-grained with the info it sends, some cheats can reveal people hiding behind walls, disperse fog of war, or reveal other such byte patterns in memory that are not normally rendered. Anticheats are in the business of preventing you from controlling the code that runs on your machine, so that you can't decide to run code that would give you an unfair advantage. Which, short of imposing remote attestation hardware DRMs, tends to be strongly at odds with a free software system. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 17:45 UTC (Fri) by tshow (subscriber, #6411) [ Link] The flight computer quorum model doesn't have to deal with the possibility that any or all of the systems may have malicious intent. Consider an esports context where an entire team's computers (and thus, half the computers in the game) could be colluding. It's a far thornier problem. That said, the companies that make anti-cheat software haven't exactly covered themselves in glory. A lot of what's in the field look suspiciously like what would happen if you got the enthusiastic intern to write a rootkit. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 20:39 UTC (Fri) by camhusmj38 (subscriber, # 99234) [Link] I'm suspicious of anything that requires me to run a driver in kernel mode just to play a game. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 17:50 UTC (Fri) by excors (subscriber, #95769) [ Link] That won't help with even the most basic cheats like aimbots (where the player's inputs follow all the normal rules and are processed perfectly legitimately, but they don't come directly from the player's hand) or wall hacks (where players can see through walls, detecting enemies who ought to be hard to see). You need to either lock down the client hardware so thoroughly that you can be certain it's only running the software provided by the game developer (as with consoles), or engage in an endless arms race of updating detection for cheats which are then updated to circumvent that detection. Game developers will never completely win, but in practice they usually do a good enough job to stop the game being made unplayable by a flood of cheaters. They'd find it much harder if they freely shared their detection algorithms with the cheat developers. (You can also limit yourself to developing games where players don't gain an unfair advantage by cheating, but a lot of people enjoy competitive PvP action games and it'd be a shame to rule them out.) [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 15:21 UTC (Fri) by fratti (subscriber, #105722) [ Link] I'm surprised EAC runs on Linux at all, last I heard that was a blocker for many games on the Steam Deck. Though I'm glad game developers are still engaging in the Sisyphean task of trying to establish trust on an untrusted client. The hours spent playing cat-and-mouse with zit-adorned teenagers who are telling a game server that they can in fact fly through the air seem well invested. On a more serious note, is there any reason why glibc shouldn't keep DT_HASH around? kilobytes in disk space are hardly a reason to break an application that users are running, even if said application is not one the glibc maintainers like. > [...] something needs to be done to solve this problem and save gamers from the prospect of having to get some actual work done. While it may seem silly, there is a large industry of online content creators for whom playing video games is an integral part of work. Besides, we must be wary of glibc maintainers. First they came for Flash Player's use of memcpy, but I did not speak for I was not trying to watch the youtubes. Next, they came for gamers, and I did not speak for I was not a gamer. Once they come for people who spend too much time reading e-mails, there will be no one left to speak for us. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 20:55 UTC (Fri) by etra0 (guest, #160378) [Link] > On a more serious note, is there any reason why glibc shouldn't keep DT_HASH around? kilobytes in disk space are hardly a reason to break an application that users are running, even if said application is not one the glibc maintainers like. So far the only two reasons I've read are * People should have been aware of an *new* existing technology, albeit not being documented. * You save 1% of disk space, "(...) which is considerable for an unused feature." [1] I'm surprised how anyone would consider 1% of disk space considerable in modern day and age. [1] https://sourceware.org/bugzilla/show_bug.cgi?id=29456#c9 [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 15:31 UTC (Fri) by mb (subscriber, #50428) [Link] > Carlos O'Donell agreed that this change was not an ABI break: > Software that is an ELF consumer on Linux has had 16 years to be updated to handle the switch from DT_HASH to DT_GNU_HASH (OS-specific). What if the software is not being maintained anymore? Nobody is going to fix it. Of course this is an ABI break. Lots of software is not going to be changed, if some random OS developer drops support for some feature. If glibc doesn't fix this, the only solution for users will be to grab old outdated versions of glibc and throw it into the game bin. The game should have shipped with all required .so libraries in the first place, though. Including libc. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 16:31 UTC (Fri) by NYKevin (subscriber, #129325) [Link] On platforms which are not Linux, it's generally considered incorrect to ship your own libc, because on such platforms, libc is usually the official syscall API (so if your libc is outdated, then your app may break with no warning, and it will be considered entirely your own fault). Even on Linux, I'd be skeptical of shipping my own libc. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 17:40 UTC (Fri) by pbonzini (subscriber, #60935) [Link] Any statically linked program, which includes any Go program, is essentially shipping its own libc. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 19:26 UTC (Fri) by int19h (guest, #159020) [Link] Yes, which is why Go binaries were broken on macOS and FreeBSD for a long time, because it directly used a private API (syscalls) instead of libc. Here's an example: https://github.com/golang/go/issues/16606 The only OS on which you can statically link libc and expect your binaries to work in future releases is Linux. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 20:14 UTC (Fri) by fw (subscriber, #26023) [Link] Even the last part about Linux isn't totally true. The x86-64 kernel interface for getcpu and gettimeofday has already changed once for many users, breaking old glibc versions before 2.15 (and presumably statically linked binaries). [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 21:47 UTC (Fri) by linuxrocks123 (subscriber, # 34648) [Link] Do you have a source for this? The kernel ABI is considered sacrosanct to the point that obsolete system calls from the early days of Linux are still around. I'd be surprised if they allowed a system call interface change that broke something as important as glibc. [Reply to this comment] statically-linked Go programs and libc Posted Aug 19, 2022 20:14 UTC (Fri) by dkg (subscriber, #55359) [Link ] @pbonzini wrote: > Any statically linked program, which includes any Go program, is essentially shipping its own libc. Are you sure about this? gosop is a Go program, and all the Go bits are statically-linked, but it still dynamically loads libc 0 dkg@alice:~$ ldd $(which gosop) linux-vdso.so.1 (0x00007ffe414fd000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdedda00000) /lib64/ld-linux-x86-64.so.2 (0x00007fdeddcc5000) 0 dkg@alice:~$ [Reply to this comment] statically-linked Go programs and libc Posted Aug 19, 2022 20:36 UTC (Fri) by dtlin (subscriber, #36537) [ Link] Depends on how it is built. cgo will use libc, but if a binary is built with CGO_ENABLED=0, then $ ldd /usr/bin/tailscale not a dynamic executable [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 16:49 UTC (Fri) by mokki (subscriber, #33200) [ Link] There is no reason to go back to older glibc versions. The correct solution is that the game or steam has a wrapper that creates a copy of glibc (and other libraries the game scans) library and adds the missing DT_HASH section from the modern version. If that does not already exist in elf tools, it could be added. And of course the wrapper should detect if glibc changes and do the above again when needed to keep in sync with security and OS updates. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 17:00 UTC (Fri) by mb (subscriber, #50428) [Link] >The correct solution is that the game or steam has a wrapper that creates a copy of glibc (and other libraries the game scans) library and adds the missing DT_HASH section from the modern version. That might work in this case. But it won't make the breakage go away until such a solution exists. And it would require support from the sw vendor. That breaks as soon as the system goes out of support. I would not count on the vendor to still be around, if glibc decides to introduce the next breakage 16 years from now. Real software has been broken. Therefore, the only real solution is to revert the breaking change. It's that simple. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 18:25 UTC (Fri) by cgutman (subscriber, #110037) [Link] > Adhemerval Zanella said: "I am not sure this characterizes as an ABI break since the symbol lookup information would be indeed provided (albeit in a different format)" This line of thinking is completely crazy. Having a replacement available has absolutely nothing to do with whether something is an ABI break or not. By this reasoning, you could argue that removing memcpy() is not an ABI break because memmove() can do the same thing already and it's been available for ages too. > Software that is an ELF consumer on Linux has had 16 years to be updated to handle the switch from DT_HASH to DT_GNU_HASH (OS-specific). and software has had decades to switch from using strcat()/strcpy()/ gets() to something more resistant to buffer overflows like strncat() /strncpy()/fgets(), but nobody is talking about removing those APIs from glibc. Why? Because it breaks the ABI! [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 18:32 UTC (Fri) by ndesaulniers (subscriber, # 110768) [Link] > When the dynamic linker starts a program, it must resolve all of the symbol references into shared libraries (including glibc). > Since this process must complete before an application can actually start running, it needs to happen quickly. I don't think that's generally true; it relies on the executable being linked with `-z now`, otherwise the symbols are resolved lazily (ie. the whole dance with the GOT and PLT). I suspect the implicit default for this flag is configure-able in GNU BFD though, and at this point I suspect most distros would set that as the default. This adds a layer of protection to prevent tampering with the GOT and PLT at runtime. When symbols are resolved lazily, not "all" symbols references need to be resolved, and the process doesn't need to "complete" before the process starts running. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 19:31 UTC (Fri) by ndesaulniers (subscriber, # 110768) [Link] I suspect the EAC developers just implemented scanning the simpler and documented symbol table section. That said, if glibc can break abi like this, I wonder if we can drop --hash-style=both from VDSO's in the kernel and just support DT_GNU_HASH? (Fun fact: I once had a bug where one of these sections (DT_GNU_HASH) wasn't being produced; I could only repro on builds from the server and not locally. I had to use `dd` to slice the vdso out of a running process on device using a build from the server. Ultimately, kbuild wasn't hermetic, and the server's linker was missing support for --hash-style=both. https://stackoverflow.com/a/54797221). [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 22:56 UTC (Fri) by jreiser (subscriber, #11027) [ Link] > -z now ... adds a layer of protection to prevent tampering with the GOT and PLT at runtime. There is no protection against tampering unless (in addition) various parts of the PLT (ProgramLinkageTable) are put into a .relro section and then into a PT_LOAD segment that lacks PROT_WRITE. Part of the original impetus for -z now was to prevent spending hours and hours during the first phase of a computation, only to have the process abort because some symbol relating only to the second phase could not be resolved due to mismatch of shared libraries or other configuration problems. -z now can interfere with explicit dlclose(); ...; dlopen() for managing a multi-phase program in a constrained address space, or user interaction that explores optional (but known-in-advance) "heavy weight" subsystems. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 18:38 UTC (Fri) by shentino (subscriber, #76459) [Link] Not GNU's fault if they didn't comply with the ABI in the first place. Using undocumented dependencies that aren't part of the ABI or API is by definition invoking undefined behavior and voiding the warranty. And microsoft using undocumented APIs to give itself an unfair competitive edge is something they've already been shamed for, and rightly so. If proprietary games are the ones breaking because they cheated and tried to bypass the warranty provided by the ABI then it serves them right for breaking the rules. Unfortunately commercial interests will probably spin this to denigrate open source software and unlevel the playing field back in their own favor with proprietary crap that hides dishonest coding practices [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 20:37 UTC (Fri) by camhusmj38 (subscriber, # 99234) [Link] But they did comply with the ABI - the new hash algorithm is not documented and is not the one mandated by the System V ABI. This is definitely an ABI break because a program that worked with an earlier version stopped working because the file format changed. The reason we have an ABI is so that existing programs continue to work without needing recompilation. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 19:24 UTC (Fri) by rwmj (subscriber, #5474) [Link ] I think the thing which confuses me most is why glibc is generating this section and not the linker (ie. binutils)? [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 19:24 UTC (Fri) by rwmj (subscriber, #5474) [Link ] .. and to continue that thought, why the linker wouldn't have a -Wl,--generate-old-style-hash flag or similar. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 19:29 UTC (Fri) by corbet (editor, #1) [Link] The linker is generating the hash tables; the glibc change was, at its core, a change to a linker option. That's why this problem would be a relatively easy thing for distributors to address in the short term; they can just put the --hash-style=both option back. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 20:34 UTC (Fri) by marcH (subscriber, #57642) [ Link] > Blaming the EAC developers for not keeping up with Linux ELF hash-table formats might not be entirely fair. The DT_HASH format is mandated by the System V ABI specification, the DT_GNU_HASH format is undocumented, and there has been no deprecation campaign to get users to move on No surprise so many Linux products and so much software running on Linux try to avoid glibc. GPLv3 is the legal icing on the cake. [Reply to this comment] The ABI status of ELF hash tables Posted Aug 19, 2022 21:13 UTC (Fri) by fratti (subscriber, #105722) [ Link] glibc is LGPLv2.1 though? [Reply to this comment] Copyright (c) 2022, Eklektix, Inc. Comments and public postings are copyrighted by their creators. Linux is a registered trademark of Linus Torvalds