[HN Gopher] The ABI status of ELF hash tables
___________________________________________________________________
The ABI status of ELF hash tables
Author : chmaynard
Score : 55 points
Date : 2022-08-19 16:16 UTC (6 hours ago)
(HTM) web link (lwn.net)
(TXT) w3m dump (lwn.net)
| miohtama wrote:
| Here is the crux of the issue:
|
| > Blaming the EAC developers for not keeping up with Linux ELF
| hash-table formats might not be entirely fair. The DT_HASH format
| is mandated by the System V ABI specification, the DT_GNU_HASH
| format is undocumented, and there has been no deprecation
| campaign to get users to move on.
|
| Most problems like this are communication and co-ordination
| problems.
| rwmj wrote:
| > _communication and co-ordination problems_
|
| Which only exist in this case because the software isn't open
| source. This would have been spotted years ago in open source.
| Spivak wrote:
| Unless there's bots out there scanning public code repos for
| esoteric stuff like this I kinda doubt it. If you mean it
| would have gotten caught by packagers then yeah but that
| presumes a world where `apt install eac` makes any sense.
| rwmj wrote:
| We do actually have a way to rebuild Fedora with particular
| changes (eg. when we change the baseline x86-64 version). I
| guess other distros can do that too.
|
| Of course it goes almost without saying that EAC would make
| about as much sense as an open source project as it does as
| a closed source project, which is to say none whatsoever
| because it's conceptually nonsense :-)
| Narishma wrote:
| An open source anti-cheat system doesn't make much sense.
| akira2501 wrote:
| Neither does a closed source one, it is running on _my_
| computer after all. Cynically.. keeping it closed just
| allows you to call it "anti-cheat" with a straight face,
| rather than what it actually is.. "mildly successful cheat
| mitigation and frustration techniques."
|
| Which is obvious once you notice there is an active "anti-
| anti-cheat" marketplace.
| jcranmer wrote:
| That's a poor argument because, even going strictly by the
| specification, DT_HASH _isn 't_ authoritative--the symbol table
| is provided by DT_SYMTAB.
|
| Personally, I feel that by the time you're messing with the
| _DYNAMIC entries of an ELF file, you need to be prepared to
| actually follow and keep up with ELF handling rules. Deleting
| DT_HASH feels no different than, say, changing the format of
| the structure pointed to by DT_DEBUG, which happened just last
| year (https://sourceware.org/bugzilla/show_bug.cgi?id=15971, ht
| tps://sourceware.org/git/gitweb.cgi?p=glibc.git;h=a93d9e03...).
| Which, you know, breaks debuggers--but no one's running around
| saying "oh no glibc's ABI story is complete and utter trash!"
| because of that change (I daresay most of the people commenting
| on this issue aren't even aware that it happened).
| mhh__ wrote:
| Finding the most up to date ELF spec for your system is
| remarkably annoying.
| Arnavion wrote:
| If glibc updates its version number and EAC starts failing
| because it used too small a data type for that, does that mean
| glibc broke ABI compatibility and should be stuck on a lower
| version forever?
|
| Closed source software that wants to hold back open source
| software can go pound sand. Gamers can run their games with older
| glibc in a chroot / container. I say this as someone who does
| exactly this for games.
| mrguyorama wrote:
| In the Windows world the answer to that question was
| emphatically "Yes, you should contort yourself to make sure
| users can still actually USE their computer", to the point that
| the official version number of windows was "3.95" because some
| apps looked at only the minor version number and would have
| broken otherwise.
| Arnavion wrote:
| Yes. And? What part of my comment applies to Windows?
| Certainly not the "open source software" part.
|
| BTW my comment lists two ways for people to continue "still
| actually USING their computers".
| dmitrygr wrote:
| There is the strict interpretation of the words "what is an abi?"
| and then there is the real world "what breaks things users would
| rather not be broken?"
|
| Maintainers of a package used by literally almost every
| executable really ought to use the latter thought process, IMHO.
| Hiding behind the "well, _technically_... " is kind of
| irresponsible. Hiding behind "It's been 16 years since we
| introduced $NEW_THING" is doubly so. Who the hell reads glibc's
| changelists for fun?
|
| You know the quote: "on display in the bottom of a locked filing
| cabinet stuck in a disused lavatory with a sign on the door
| saying 'Beware of the Leopard.`"
| jwilk wrote:
| Related:
|
| https://news.ycombinator.com/item?id=32471624 ("Win32 Is the Only
| Stable ABI on Linux?", over 500 comments)
| ajross wrote:
| FWIW: that whole frame is just wrong anyway. Microsoft breaks
| stuff _constantly_. Everyone out there (including anti-cheat
| vendors) is relying on hackery and trickery to make their stuff
| work on windows. And every windows release a ton of that stuff
| breaks and these companies drop everything to re-reverse-
| engineer whatever it is they need.
|
| What works compatibly[1] is the win32 C API implemented along
| with the specific set of DLLs that you built your application
| against[2]. MS curates the kernel/driver interface below that
| fairly well[3]. But the kind of trickery EAC is doing here is
| right in the middle of things _that absolutely do not work on
| windows either_.
|
| [1] In the sense of "do this and your app won't break".
|
| [2] Which is why windows binaries ship their own C libraries.
| FWIW this model exists in Linux too. It's basically the way we
| package Docker and Snap containers. Had EAC been written
| against a particular Snap environment, we wouldn't be having
| this problem either because it would be frozen on a particular
| glibc version.
|
| [3] And above that level, there are often other runtimes like
| .NET to which you can portably target.
| riskable wrote:
| Going through great pains to support EAC is a bit ridiculous. EAC
| itself is ridiculous from the get-go since it doesn't _actually_
| prevent cheaters it just pretends. It 's not even _meant_ to stop
| cheaters! Its entire purpose is to lock developers into Epic 's
| proprietary system and prevent them from porting to non-approved
| platforms.
|
| The entire premise of client side anti-cheat tools is, "trust the
| client." I think pretty much everyone at Hacker News knows that's
| the exact opposite of how you make a secure system.
|
| _Never trust the client._
| jsheard wrote:
| Not trusting the client can only get you so far, in some types
| of game it's unavoidable that the client must have enough
| information and control that a cheat can enhance human play or
| replace the human altogether. It will always be possible to
| enhance aiming in a shooter, or automate repetitive tasks in an
| MMO, or script perfect blocks and parries in a fighting game,
| or feed the state of an online chess game into Stockfish and
| have it play for you, even if you trust the client to the
| minimum possible extent.
|
| The only things that can be done in those cases are loosely
| "detecting" cheaters through heuristics and human review of
| suspicious behaviour, which won't catch more careful cheaters
| and may lead to false positives, and client side solutions like
| EAC which frustrate less skilled cheat developers and sometimes
| catch the more skilled ones off guard. Alternatively, give up
| on PC and focus on consoles which are sufficiently locked down
| that cheating is currently impossible regardless of how much
| you trust the client.
| kllrnohj wrote:
| Or server-side automated review using machine learning
| models, which is what Valve is trying with "vacnet" (with
| then manual review to confirm the model's results are
| accurate - something most tech companies should learn from)
|
| Great talk on it here:
| https://www.youtube.com/watch?v=kTiP0zKF9bc
| jsheard wrote:
| It's an interesting idea, but it's hard to say if it's
| really effective against anything besides blatant
| ragehacking. AFAICT the general consensus in the community
| is that Valves own CS:GO servers are still infested with
| cheaters, and to avoid them you need to go to ESEA or
| FACEIT (third party CS:GO matchmaking services) or Valorant
| (Riots CS:GO clone) which all primarily rely on very
| aggressive clientside scanning solutions.
| kllrnohj wrote:
| I suspect for ESEA & FACEIT the most effective aspect of
| their anticheats is just that they are paid services. If
| I'm going to cheat at a game, I'm generally going to bias
| towards minimal consequences if I do get caught.
|
| I can say that Valorant's anti cheat is the singular
| reason I've never tried it, even though I love counter-
| strike and similar games. Giving game devs (especially
| those owned by Tencent) ring 0 access is, uh, a step too
| far I think.
| jsheard wrote:
| Valve took away the methods of getting into Prime
| Matchmaking without paying (you must pay $15 now) and it
| doesn't seem to have improved the situation much if at
| all.
|
| The ship has sailed on ring0 really, all of the major
| anticheat players plug into ring0 on Windows. If you play
| online games on Windows there's almost no avoiding it.
|
| Interestingly Proton acts as a loophole, EAC and BattlEye
| don't touch ring0 on Linux even though they do on
| Windows. Gamedevs have to opt-in to allowing Linux
| clients though.
| kllrnohj wrote:
| Funnily enough I also stopped playing Destiny 2 when they
| added ring 0 requirements and primarily focus on single
| player games now :)
|
| If steam deck is successful enough I wonder how long
| until kernel modules become a requirement...
| jsheard wrote:
| > If steam deck is successful enough I wonder how long
| until kernel modules become a requirement...
|
| Yeah, there's already Linux-specific cheats in the wild
| which abuse the fact that the Linux anticheats are
| oblivious to anything happening outside of the current
| user, so they can manipulate the games memory with
| complete impunity simply by doing it from root or a
| kernel module. Developers may become more hesitant to
| allow Linux clients as abuse becomes more common, unless
| the anticheats start extending into ring0 on Linux the
| same way they do on Windows.
| brigade wrote:
| I don't think anyone is proposing to eliminate support for
| --hash-style=both. Correct me if I'm wrong?
|
| So at this point, the maintenance burden on glibc is however
| much effort they want to spend justifying breaking backwards
| compatibility.
| ajross wrote:
| They didn't break backwards compatibility, only forwards. The
| core issue here is that there is a new format that won't work
| with old runtime linkers. But the only way to present that
| format to those old linkers[1] is to take a binary compiled
| with a glibc version from the last few weeks (!) and run it
| against on a system where the ld-linux.so is from pre-2006
| glibcs. That direction of compatibility is not impossible,
| but it's not what one normally considers when discussing
| "breaking backwards compatibility".
|
| Fundamentally the only way to provide the kind of
| compatibility you're demanding, where any level of reverse
| engineering or parallel implementation must be supported in
| perpetuity, is to just Never Change Anything At All.
|
| [1] With the sole known exception of the "runtime linker"
| embedded in this EAC product under discussion.
| fweimer wrote:
| If the required symbol versions are provided by the old
| glibc version (which is a bit unlikely), it will still
| work. The internal glibc hash tables are ignored by the
| link editor, and most distributions override the link
| editor hash table default anyway. (What changed is that
| glibc no longer overrides the distribution defaults.)
| brigade wrote:
| Keeping a existing binaries working is _the_ definition of
| backwards compatibility in an OS. glibc broke that.
|
| Never Change Anything is _far_ from the ask here. The ask
| is that when _real actual breakage_ is found, the benefits
| of keeping said breakage justify not spending the
| engineering costs to fix backwards compatibility. Near as I
| can tell, the benefit is 16kB disk space savings, and glibc
| 's engineering cost to maintain compatibility is nothing.
|
| Like, another backwards compatibility breakage I saw
| recently was Graphene switching from 39-bit to 48-bit
| virtual address space, which also broke games. That has a
| real measurable security benefit, _and_ actual engineering
| costs to develop a backwards compatibility story. But they
| still spent effort to fix breakages.
| ajross wrote:
| > Keeping a existing binaries working is the definition
| of backwards compatibility in an OS. glibc broke that.
|
| No, that's simply wrong. No "glibc binary" that could
| have been even theoretically compiled within the past few
| decades will fail to run because of this issue.
|
| What happened is that this EAC product had reverse-
| engineered the internal formats glibc uses to provide the
| APIs it implements. And that format changed.
|
| I repeat what I said elsewhere: it is simply not possible
| for any software product to anticipate all possible
| external users of its internal implementation details.
| What you want, you cannot have. (And I repeat what I said
| elsewhere too: _you don 't have this, even on windows_.
| Microsoft breaks things like anti-cheat hackery with
| every release too, it's just that you don't notice
| because the proprietary vendors care enough to fix it,
| where they don't on Linux.)
| dark-star wrote:
| > What happened is that this EAC product had reverse-
| engineered the internal formats glibc uses
|
| No, you got it the wrong way around. EAC has used the
| well-documented and SysV-standard DT_HASH mechanism.
|
| The DT_GNU_HASH is the (almost) undocumented one that
| they would have to reverse-engineer from GNU ld's source
| code
| brigade wrote:
| > No "glibc binary" that could have been even
| theoretically compiled within the past few decades will
| fail to run because of this issue.
|
| That's obviously not true. The entire story is how a new
| glibc binary broke existing binaries.
|
| > What happened is that this EAC product had reverse-
| engineered the internal formats glibc
|
| Okay I missed this; I thought EAC was parsing the libc
| binary as per ELF. What was reverse engineered?
|
| > it is simply not possible for any software product to
| anticipate all possible external users of its internal
| implementation details
|
| Correct. Which is why I never asked for stuff that never
| breaks.
|
| What is important is that when stuff is _found_ to break,
| the breakage and refusal to provide backwards
| compatibility has a compelling reason. glibc 's
| justifications here are laughable; they're spending more
| effort to not revert the breakage than it would take to
| fix.
|
| Which is kinda why the story blew up.
| ajross wrote:
| I don't know what people expect glibc to do here. EAC is a tool
| that reverse engineered library data that is only supposed to be
| used by the system runtime linker. People reverse engineer stuff
| all the time, and it's universally agreed that the onus is on the
| Weird Hack to bother with maintenance. EAC isn't using a
| supported API at all (the API for this is dlopen/dlsym et. al.,
| not the hash format!).
|
| It's unfortunate, sure, but it's just not possible to prevent.
| And all the people screaming about "This Doesn't Happen On
| Windows" are just fooling themselves (or, to be blunt, being
| deliberately disingenous). This happens on windows _every single
| release_. Windows N+1 comes out and suddently a bunch of hacky
| drivers or tools or gadgets stop working until their authors
| manage to figure out what part of their trickery Microsoft broke.
| dark-star wrote:
| nothing was "reverse-engineered". Read the article. DT_HASH has
| been documented in SysV since ages. It's the DT_GNU_HASH that's
| (almost) undocumented and needs to be reverse-engineered from
| GNU ld's source code.
| ronsor wrote:
| Anticheat software like EAC (as well as commercial antiviruses
| and malicious rootkits) often reverse-engineer internal system
| data structures and call undocumented functions. You really
| can't blame system library developers for breaking
| compatibility with software that is doing things "the wrong
| way."
|
| Yes, I know Linus Torvalds says "don't break userspace," but
| that's for officially exposed kernel interfaces.
| cosmiccatnap wrote:
| Seems like there is some handwaving here that "DT_HASH" should be
| supported forever because "The DT_HASH format is mandated by the
| System V ABI specification"
|
| It was replaced by DT_GNU_HASH in the gnu implementation of libc
|
| ....I think these people have forgotten what GNU stands for.
| YouWhy wrote:
| I strongly favor the Linux\Linus "don't break userspace" attitude
| than what has just happened for glibc.
|
| This is especially salient given that the DT_GNU_HASH format is
| not even documented. Leaving it as the only-option-by-default
| going forward is unconscionable.
| formerly_proven wrote:
| > On typical distributions, none of the system libraries use
| DT_HASH, not even core libraries such as libgcc_s.so.1. Any
| tool that performs symbol lookups needs to support DT_GNU_HASH
| these days.
|
| It seems to me that EAC includes its own copy of a dynamic
| linker for the express purpose of validating that glibc
| functions are not obviously detoured in the current process.
| fweimer wrote:
| It's equally possible that they added DT_HASH lookup to
| achieve glibc 2.34 compatibility. The old way for getting the
| original dlsym function after interposing dlsym stopped
| working because the (internal, usually uninterposed, now
| unused) __libc_dlsym@@GLIBC_PRIVATE symbol went away in glibc
| 2.34. But a program can still obtain the dlsym function
| address by looking at the ELF data structures directly.
|
| This would explain why people are so annoyed: there's
| unplanned extra low-level work to enable glibc upgrades two
| years in a row.
| titzer wrote:
| This is part of the reason Virgil doesn't rely on _any_
| userspace libraries. It uses the kernel directly, which is very
| stable on Linux.
| cratermoon wrote:
| Sympathy for the problems for EAC seems to be in short supply,
| and I can sympathize. It doesn't really excuse the abrupt removal
| of the compatible hash table format without at least a
| deprecation effort ahead of time. What I would expect to see
| would be something like
|
| step 1: provide complete and accurate documentation for the new
| format and communicate it widely and prominently to the developer
| community
|
| step 2: set a target date & version for removal, and get feedback
| from the community to ensure it gives sufficient time and
| information to complete a migration. Provide a way for developers
| to ask for and get support to make the necessary changes. Release
| sample code, if possible.
|
| step 3: one release _before_ removal, loudly note the deprecation
| and the upcoming change.
|
| step 4: as the release that removes backward compatibility is
| being prepared, again, communicate the upcoming change _and
| provide guidance for those who can 't update_ on any ways to
| mitigate problems.
|
| step 5: finally, release the version with the breaking changes,
| and put the break boldly at the top of the release
| notes/changelog. Reiterate the previous guidance on mitigating
| problems and help.
|
| step 6: Go have a tasty beverage.
| cesarb wrote:
| > It doesn't really excuse the abrupt removal of the compatible
| hash table format without at least a deprecation effort ahead
| of time.
|
| I strongly suspect that the reason for the "abrupt" removal was
| that it wasn't expected that anything would break: that table
| is used for dynamic linking, and the dynamic linker comes with
| glibc itself, so it can be assumed to understand the new table
| (and the dynamic linker which comes with other C libraries
| would be used to load these C libraries, and not glibc since
| you cannot load more than one C library in a single process).
| AFAIK, the code in the dynamic linker which understands files
| with only the old table wasn't removed, so from that point of
| view, compatibility was being kept.
|
| That is, they didn't expect anything to be using that table
| without going through dlopen() and dlsym().
| dzaima wrote:
| If that were the case, the plan should've been to revert if
| things do break, i.e. revert an improper action when you find
| it's based on false premises. But that's not what we see
| here.
| fweimer wrote:
| Exactly. In addition to that, it seemed very likely that the
| glibc 2.34 reorganization already broke software which
| depended on such hidden implementation details.
| cratermoon wrote:
| Public vs. Published.
| https://martinfowler.com/ieeeSoftware/published.pdf
___________________________________________________________________
(page generated 2022-08-19 23:01 UTC)