[HN Gopher] Ubuntu 24.04 LTS will enable frame pointers by default
___________________________________________________________________
Ubuntu 24.04 LTS will enable frame pointers by default
Author : jonseager
Score : 155 points
Date : 2023-12-13 16:23 UTC (6 hours ago)
(HTM) web link (ubuntu.com)
(TXT) w3m dump (ubuntu.com)
| brancz wrote:
| Frame pointers are such a destructive micro-optimization to omit
| by default, I am beyond excited about this collaboration with the
| folks at Canonical to make Ubuntu debuggable by default!
| duskwuff wrote:
| This option was almost certainly a holdover from the bad old
| 32-bit x86 days, when disabling frame pointers gave you a
| seventh valuable general-purpose register. It's no longer
| beneficial on x86_64 -- even with rbp locked down, you still
| have fourteen registers there.
| brancz wrote:
| 100% the way I see it! On 32-bit the performance benefits is
| very major on just about everything, but not so on 64-bit.
| PaulDavisThe1st wrote:
| and 14 registers should be enough for everyone!
| mratsim wrote:
| It's not. 14 is still low compared to other ISAs.
| peterfirefly wrote:
| Spilling to an XMM register instead of memory is an
| option.
| jandrese wrote:
| Yeah, back in the 32 bit days -fomit-frame-pointer was the
| only optimization you could count on to really make a
| difference. It wasn't small either, often 10-15% speedup. No
| other flag on gcc would make even a full percentage
| difference in my testing.
|
| The Amd64 architecture fixed the underlying problem, so this
| is pretty much just a holdover. I'm surprised they even
| enabled it by default.
| cesarb wrote:
| To emphasize this point: on 64-bit x86 with frame pointers,
| you have twice as many registers as on 32-bit x86 without
| frame pointers, _and_ these registers are twice as wide. A
| 64-bit value (more common than you 'd expect even when
| pointers are 32 bits) takes two registers on 32-bit x86, but
| only a single register on 64-bit x86.
| bodyfour wrote:
| That much is true -- the difference between 6 and 7 registers
| is much larger than the benefit of going from 14 to 15.
|
| However, even under zero register pressure having a frame
| pointer is still an extra register that needs to be touched
| on every function invocation, extra instructions taking space
| in the I-cache, etc. It's a small thing, but it's still a
| cost that has to be paid by all compiled code.
|
| I'm not going to claim that re-enabling frame pointers was
| the _wrong_ choice -- the people involved in the debate know
| the tradeoffs and I have to start with the assumption that I
| would have made the same decision if I were in their
| position. It does make me slightly sad, though. The idea
| behind removing frame pointers isn 't that backtraces aren't
| important, it's that computing the frame pointer after-the-
| fact is possible -- i.e. for normal functions without
| alloca() or dynamically-sized stack arrays map %rip -> frame
| size.
|
| The problem seems to be that despite years of experience with
| "no-frame-pointer" being the default I guess the profiling
| tools never got as reliable or good as the with-frame-pointer
| variants. My personal hope was that the problem would fade
| over time as tools improved, but it seems that's unlikely to
| ever happen. After all, once no-frame-pointer stops being the
| default there won't be any pressure for tools to improve. The
| towel has been thrown in.
| brancz wrote:
| Profiling tools have already solved the ability to reliably
| unwind in the absence of frame pointers[1], but there are
| plenty of tools that this kind of investment is simply too
| much that it won't ever happen, like bpftrace or bcc-tools.
|
| [1]
| https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-
| bas...
| rightbyte wrote:
| Ye. Bytecode interpreters very much benefit from the extra
| register. And any function that looks similar to one.
|
| I mean e.g. getter and setter functions get alot of extra
| code to run.
| kmeisthax wrote:
| Also, apparently Intel is planning to extend x86_64 to 32
| GPRs, with an extension called... _sigh_ [0]... Intel APX[1].
| So the overhead of frame pointers will be even lower in the
| future.
|
| [0] Intel APX is _extremely_ confusable with the iAPX 432, a
| failed non-x86 architecture Intel made that 's completely
| unrelated to doubling the size of the x64 register file.
|
| [1] https://www.intel.com/content/www/us/en/developer/article
| s/t...
| p1mrx wrote:
| iAPX has been dead for almost 40 years; there probably
| won't be much confusion.
| karmakaze wrote:
| > Ubuntu leaps forward with frame pointers by default
|
| Yeah it's more like "...will stop omitting frame pointers by
| default".
| amelius wrote:
| > to make Ubuntu debuggable by default!
|
| How's that, you'd still need the debug symbols
|
| Also has anyone else noticed that running stuff through
| Valgrind is really only possible if the program was made with
| Valgrind in mind? For example, Python and its many extensions
| generate numerous errors and warnings, so many that any real
| problem becomes hidden.
|
| I'd say that modern Linux systems are very far from being
| debuggable.
| rightbyte wrote:
| You offent need to make excemption file.
|
| My tip is to run some thing one program execution, and then
| run it twice during the same program execution, and diff the
| reports.
| brancz wrote:
| Ubuntu hosts a debuginfod server that you can automatically
| discover debuginfo from any binary from using the binary's
| build id.
|
| https://sourceware.org/elfutils/Debuginfod.html
| pryz wrote:
| The post from Polar Signal blog
| https://www.polarsignals.com/blog/posts/2023/12/13/embracing...
|
| Exciting!
| westurner wrote:
| Call stack > Structure > Stack and _Frame pointers_ :
| https://en.wikipedia.org/wiki/Call_stack#Stack_and_frame_poi...
|
| What do the Coding Guidelines listed in e.g. awesome-safety-
| critical say about Frame pointers? https://awesome-safety-
| critical.readthedocs.io/en/latest/#co...
|
| (Edit)
|
| /? "cert" "frame pointer"
| https://www.google.com/search?q=%22cert%22+%22frame+pointer%... :
|
| - Stack buffer overflow > Exploiting stack buffer overflows:
| https://en.m.wikipedia.org/wiki/Stack_buffer_overflow :
|
| > _In figure C above, when an argument larger than 11 bytes is
| supplied on the command line foo() overwrites local stack data,
| the saved_ frame pointer, _and most importantly, the return
| address_
|
| What about the Top 25?
|
| /? site:cwe.mitre.org "frame pointer"
| https://www.google.com/search?q=site%3Acwe.mitre.org+%22fram... :
|
| - CWE-121: Stack-based Buffer Overflow
| https://cwe.mitre.org/data/definitions/121.html
|
| This is closer to a better approach for security, debuggability,
| and performance IMHO:
|
| https://news.ycombinator.com/item?id=38138010 :
|
| > _gdb on Fedora auto-installs signed debuginfo packages with
| debug symbols; Fedora hosts a debuginfod server for their
| packages (which are built by Koji) and sets `DEBUGINFOD_URLS=`_
|
| > _Without debug symbols, a debugger has to read unlabeled ASM
| instructions (or VM opcodes (or an LL IR))._
| daoistmonk wrote:
| hasn't this already been in fedora for almost a year?
| brancz wrote:
| You're not wrong, but isn't it exciting that it's gaining more
| traction? Ubuntu is also very widely deployed, so I'll
| celebrate every distro that makes this the default, be it
| Ubuntu in this case, or Debian, or Arch, or Suse, or or or...
| bravetraveler wrote:
| I believe so, I remember a fair amount of hubbub over it.
|
| Can't say it's been useful here, any development I do is miles
| away from this. It hasn't hurt either so... cool, I guess
|
| Call me pessimistic, but I'm not convinced this being the
| default will lead to _more_ profiling. There 's plenty that
| could be done without this, that isn't, so I'm not buying it.
| brancz wrote:
| While profiling benefits immensely, imagine all the other
| ramifications. Cheap tracing of MySQL, postgres, or anything
| else using bpftrace or any other bcc tools. Getting stack
| traces of processes that are already core dumping...
|
| Profiling is one part, but the debuggability this enables is
| going to be huge in the long term I predict.
| pavon wrote:
| Yeah. To be fair there is a stronger argument for it being
| enabled in fedora than an LTS release (RHEL or Ubuntu) since it
| has more cutting edge software that needs more frequent
| debugging, is less likely to be used in production where the
| (minor, but uneven) performance hits may matter, and has so
| many upstream developers using it as their daily driver.
| brancz wrote:
| I would argue that LTS releases are going to be deployed
| millions of times and stay around effectively forever with
| lots of very critical software being deployed on it. Having
| all processes/binaries be debuggable cheaply and easily in
| stressful situations is a major improvement that's now here
| to stay.
| kelnos wrote:
| Agreed that cutting-edge software likely needs more frequent
| debugging, but I don't think that means LTS releases
| shouldn't be easier to debug.
|
| Consider that you're a big company deploying software to
| hundreds or thousands of machines, and you hit a difficult-
| to-diagnose performance issue, crash, etc. You'll _very_ much
| appreciate if the OS has made it easier for you to debug
| things.
|
| Put another way, Fedora users/developers might appreciate
| having frame pointers because they have to debug more
| frequently, but RHEL/LTS release users might appreciate frame
| pointers because on the less-frequent occasion when they need
| to debug, the stakes are much higher.
| rwmj wrote:
| Yes: https://fedoraproject.org/wiki/Changes/fno-omit-frame-
| pointe...
| herodoturtle wrote:
| For the layman, what does this mean?
| supportengineer wrote:
| Exciting new failure modes
| saagarjha wrote:
| Better stack traces for profiling tools when you don't have
| debugging symbols, or can't use them for whatever reason
| im3w1l wrote:
| I was going to write a long essay about the relation between C
| and assembly but after thinking for a while, I think there is
| an easier way to explain it.
|
| Stack frames are basically a (single) linked list of
| information about the call stack. Every frame corresponds to
| one function call, and says where the local variables are
| stored, and where the function should return after it has
| finished.
|
| The head of this list is stored in a register (a scarce
| resource, superfast memory). So to use frame pointers, you have
| to spend one register, and also every function has to do some
| work to maintain the linked list. Two instructions worth of
| work when the function is enterred, and one when it exits.
|
| The alternative to doing this explicitly is to keep track of it
| all implicitly which is faster but a bit more complex.
| brancz wrote:
| We wrote pretty extensively about what was needed to be able to
| profile things without frame pointers [1]. It's still possible
| at less than 1% overhead with the right set of technologies,
| but frame pointers unwinding is virtually free.
|
| [1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-
| bas...
| mratsim wrote:
| Intel VTune, Apple Instruments and perf have been able to
| profile without frame pointers.
| brancz wrote:
| A lot of profilers have, it's literally what unwind
| information is for, but we did it in kernel therefore not
| the entire stack needs to be copied to user space so it's
| way less overhead.
| jcranmer wrote:
| There are two key effects of this decision.
|
| The first effect is that it makes one additional general-
| purpose integer register unavailable for use for code. x86-64
| has 16 general-purpose registers, but one of these is the stack
| pointer and basically can't be used for any other purpose; this
| would add a second reserved register for the frame pointer.
| This effect _may_ cause slowdowns if the 15th register was
| critical for performance.
|
| The second effect is on the ability to identify (and
| potentially unwind) the stack trace. With frame pointers, the
| pseudocode for computing a stack trace is essentially:
| do load return address, previous frame pointer from
| current frame pointer print return address move
| previous frame pointer into current frame pointer until
| current frame pointer is invalid
|
| Without frame pointers, the way you have to do this procedure
| is: while current address has corresponding
| entry in unwind table: parse unwind table entry to find
| a program to run run this program on the current frame
| to generate return address print return address
| move return address to current address
|
| It turns out that there is a full Turing-complete program
| described in the unwind tables to be able to generate a return
| address. This makes unwinding quite expensive, and also can
| create lots of security headaches if you want do something like
| unwind in the kernel (since the unwind table is arbitrary user
| code!). It can also be pretty unreliable at times, especially
| in cases where your program crashed due to stack smashing so
| that you have to expect that the data being randomly
| overwritten with garbage and thus horrifically inaccurate.
| nerpderp82 wrote:
| Since it hasn't been mentioned in this entire thread, frame
| pointers are required to get good high resolution flame
| graphs.
|
| https://www.brendangregg.com/flamegraphs.html
|
| With systems like Phlare/Pyroscope SRE can monitor
| application performance in a very granular way in realtime.
|
| https://grafana.com/blog/2023/03/15/pyroscope-grafana-
| phlare...
|
| https://github.com/grafana/pyroscope
| mFixman wrote:
| Could programs compiled in architectures with 16 general
| purpose registers fail in one with 15?
| globular-toast wrote:
| Well, for a start you probably mean compiled _for_ an arch
| with 16 registers. It doesn 't actually matter what arch
| the compiler ran on (assuming modern cross-compiler like
| GCC).
|
| If a program uses 16 registers then it needs 16 registers.
| But note it's the program itself that decides to use a
| frame pointer, it's not being reserved by the operating
| system or something. Programs don't even have to use the
| stack pointer as a stack pointer, they could use all 16 as
| general purpose, but in practice almost all programs use a
| call stack (I guess all C programs must do, but you might
| be able to disable it if you make no function calls?)
| mFixman wrote:
| Ohh, I misunderstood what Ubuntu was doing.
|
| So the only change is that GCC and its toolkit will
| compile programs using a register as frame pointer by
| default? That seems like a very reasonable change. If
| having an extra general purpose register is critical for
| performance of then this can be disabled in that
| program's the makefile.
| brancz wrote:
| Correct and there are already known exceptions such as
| the python interpreter in which the "interpret function"
| function actually falls into this case and so for the
| foreseeable future python is going to continue to be
| compiled omitting frame pointers. But by default this
| destructive micro optimization is off by default until
| proven a performance bottleneck, just like any
| performance issue should be!
| kelnos wrote:
| Not really? I mean, there are architectures with fewer than
| 16 GP registers (IA32 is one of them); if you can compile
| some C code on that architecture, then surely you can also
| compile it on x86_64 with 15 (well, 14, really, due to the
| stack pointer also being reserved) rather than 16 (15).
|
| If someone is writing in assembly, then they've already
| decided if they're going to allocate a register for the
| frame pointer, and Ubuntu's change isn't going to affect
| that, as this is about compiled code, not assembled
| assembly.
|
| The only real issue is performance: if a program has a
| particular hot-path function (or just many functions
| overall) that really benefit from having that extra
| register available, and would otherwise have to spill data
| into memory, then this change could have a big negative
| impact. But that's not really a big deal; the packager can
| decide to omit frame pointers just for that particular app
| or library.
| mathiasgredal wrote:
| I thought modern speculative cpu's had way more registers
| than you can normally access. Why does it reserve these
| registers for speculative execution instead of exposing them
| to the program if it needs them?
| andreyv wrote:
| The number of registers available to the program is fixed
| in the instruction set. The program cannot address more
| registers without recompiling it to an extended instruction
| set.
| mdpye wrote:
| Because then it would need _more_ registers for the other
| purpose?
|
| But actually the decision about which general purpose
| registers to use for what is made at compile time (hence
| we're discussion a compiler flag here, the frame pointer is
| not a hardware dictated feature), so the question is
| actually kind of moot. If the compiler is out of registers
| to allocate and instead uses the stack, the CPU isn't
| reasonably going to be able to undo that.
| mathiasgredal wrote:
| Sure, but wouldn't it make sense to extend the
| instruction set to allow the compiler to use these
| registers instead of reserving them for speculative /
| out-of-order execution? It was just a thought i had after
| watching a talk by a compiler guy:
| https://youtu.be/2EWejmkKlxs?feature=shared&t=2409
| peterfirefly wrote:
| jcranmer got it right. Read that reply (and mine). And
| then maybe rewatch watch Chandler Carruth says.
|
| The current practice allows for CPUs to transparently
| increase their physical register count (to gain
| performance) and still run old code -- and older CPUs can
| still run new code. That's usually quite practical...
|
| Adding more register names takes more bits for the
| register numbers -- which leads to larger instructions.
| It also leads to more complicated encodings if we want
| backwards compatibility. AMD64 does that by adding an
| optional prefix byte that carries a payload of 4 more
| instruction bits. That's one bit each for the three
| possible register names encoded in a traditional IA32
| instruction + a bit to indicate whether to operate on
| 32-bit or 64-bit data (the actual rules are a bit more
| complex). Intel published a whitepaper recently
| suggesting a future encoding with a different (optional)
| prefix that encodes 8 more bits -- so each of the three
| register names can be extended to 5 bits (32 register
| names). It all ends up being quite complicated + new code
| won't run on older CPUs, which is not great.
|
| I think you are suggesting not just bigger register names
| but also doing away with register renaming -- that would
| be... less than entirely useful because you would lose
| almost all your out-of-order capability and thereby
| almost all your ability to hide cache misses. Cache
| misses are very, very hard to predict statically (before
| actually running the code on a real CPU with real data)
| so good luck trying to do magic ahead-of-time allocation
| of those registers...
| jcranmer wrote:
| That's not really how registers or speculative execution
| works. Intuitively, you can think of assembly as trying to
| describe a graph of instruction dependencies. Having 16
| registers in the ISA allows you to have 16 live outputs at
| any given "time". Speculative execution allows instructions
| to execute out-of-order, and to enable this, it has ~140
| registers that allow it to have 140 live outputs at once,
| so that it can run some code while a really long load is
| waiting for its data.
|
| From the ISA perspective, however, adding more registers
| means you have to spend more bits naming a register. With
| 16 registers, you need 12 bits of your instruction just to
| name the operands of a typical 3-address instruction (rA =
| rB op rC). With 128 registers, that is now a whopping 21
| bits, which means code density is a more pressing issue.
| mathiasgredal wrote:
| I get that there is a tradeoff with code density,
| although if you could have an encoding scheme or extended
| register mode to alleviate this. I was just thinking that
| if you have e.g a loop where you run out of registers,
| since you only have 16, then the compiler will swap the
| values to memory and reuse that register which creates an
| instruction dependency that doesnt really have to exist.
|
| If the compiler could use the hidden registers, then the
| cpu would know that it could run this instruction ahead
| of time.
|
| It is probably not worth it, since it adds a lot of
| complexity to an already complex system, which is why it
| isn't done.
| peterfirefly wrote:
| There is a difference between registers and register names.
|
| he AMD64 architecture only has 15 general-purpose registers
| (because the stack pointer is mostly treated as if it were
| a GPR as well). It is customary to use one of those
| (bp/ebp/rbp depending on mode) as a base pointer register.
| That leaves 14 GPR register names.
|
| The physical CPU the code runs on might have 200 physical
| registers -- those are the ones that matter for
| speculative, out-of-order execution -- but the code itself
| can only refer to 14 (or 15) GPRs at a time and has to
| include instructions to transfer values to/from memory or
| to/from XMM registers if that's not enough. Those extra
| instructions take up space + might slow the code down.
| PeterisP wrote:
| Does this provide there any benefit at all to the majority of
| machines which aren't intended to be used for development of
| binaries (i.e. user workstations, developers working with
| interpreted languages, and servers), what would be the use
| cases where frame pointers would help on those machines?
|
| Like, even for developers, I assume a random web development
| shop using Ubuntu and hosting stuff on Ubuntu would likely
| not ever attempt debugging a binary executable, and likely
| don't have any employees who could do it if they wanted. Of
| course there are companies who can and do debugging and
| profiling of binaries running on their servers, but IMHO
| those who are capable and willing to do that a relatively
| small minority of users of Ubuntu systems.
| rwmj wrote:
| I wrote this about the change when Fedora did it about a year
| ago: https://rwmj.wordpress.com/2023/02/14/frame-pointers-vs-
| dwar...
| dmpk2k wrote:
| Thank god. AMD making the base pointer optional in the x86-64 ABI
| was foolish.
| stephendause wrote:
| For anyone else who didn't know what a frame pointer was:
| https://softwareengineering.stackexchange.com/questions/1943...
| vaylian wrote:
| For x86 that value is typically stored in the stack base
| pointer register (bp):
| https://en.wikibooks.org/wiki/X86_Assembly/X86_Architecture
| amne wrote:
| Even in context it is hard to understand this: "The performance
| wins that these can provide far outweigh the comparatively tiny
| loss in performance."
|
| My guess is that on average the potential performance discovered
| with the techniques this enables is higher than the guaranteed
| negligible performance loss.
| saagarjha wrote:
| Yes, that's correct.
| o11c wrote:
| That is highly optimistic though. Profiling is hard even when
| you know what you're doing, and doing it wrong can easily lead
| to _pessimization_ (thinking particularly about if your
| profiling workload exercises a different set of branches).
| brancz wrote:
| Hyperscalers have long been doing infrastructure-wide
| profiling (or "Google-Wide Profiling" as the first whitepaper
| on the topic calls it [1]). This tech allows Google to reduce
| infra-resource usage by multiple percentage points per
| quarter.
|
| [1] https://research.google/pubs/google-wide-profiling-a-
| continu...
| kapilvt wrote:
| hmm.. I wonder if the 10% performance regression in python has
| been resolved.
| https://discuss.python.org/t/python-3-11-performance-with-fr...
| jonseager wrote:
| We're waiting on the benchmarks once the archive has been
| rebuilt to double check if we'll be affected in that way, but
| if indeed we do find that sort of regression, we'll exclude
| Python from this change (and any other package where there is a
| substantial hit).
| therealmarv wrote:
| Pro tip for performance optimisation in Ubuntu (you also gain
| more RAM):
|
| Remove snap (or choose a Ubuntu distro variant like Pop OS
| without snap)
| solarkraft wrote:
| Is that even practical nowadays? My understanding is that snap
| is deeply integrated. Don't some apt packages point to their
| snap variants?
| woodruffw wrote:
| It's still possible, although increasingly annoying.
|
| I believe Firefox now points to its snap variant, which I
| discovered when it broke a bunch of my browser extensions.
| Switching to the official Mozilla PPA was easily enough, but
| left a bad taste; if Canonical continues down the route of
| silently nudging users onto snap, I'll probably switch to
| Debian.
|
| (I have no particular opinions about snap itself, other than
| that it seems poorly documented and doesn't adhere to the "do
| what I say" philosophy when it's secretly injected into apt.)
| uxp8u61q wrote:
| > if Canonical continues down the route of silently nudging
| users onto snap, I'll probably switch to Debian.
|
| Serious question: why don't you switch now? I guess I just
| don't see the point of ubuntu anymore.
| woodruffw wrote:
| Laziness.
| ghotli wrote:
| I somewhat painstakingly figured this out the hard way pulling
| core dumps off of embedded linux devices that gdb had a hard time
| working with. At the time I was like whew why is omitting frame
| pointers the default at all in so many places when it didn't seem
| to make a measurable difference in the performance of the
| software. I guess it's just vestigial these days and yes please
| use a compiler flag like this when it measurably makes sense.
| Making debugging simpler for the rest of us is the way to go.
| o11c wrote:
| Something has to be seriously wrong with the way it was
| compiled for gdb to have trouble. Debugging (or in-process
| exception dumps), which only does a reasonable number of
| backtraces, should always be able to use separate unwind _data_
| sections.
|
| Including frame pointers should only have a performance effect
| for sample-based profiling, which does a very large number of
| backtraces. And the general fact is - people don't profile, and
| if they do, they don't do it correctly.
|
| Omitting frame pointers has significant performance wins on
| platforms with about 6 registers, like 32-bit intel x86. It's
| much less of a win on platforms with about 14 registers, like
| 64-bit x86 or 32-bit ARM, let alone platforms with about 30
| registers, like 64-bit ARM.
|
| Since modern architectures are strongly trending toward designs
| that support more registers, a frame pointer isn't unreasonable
| to choose. But that's still no excuse for all the shitty
| software that refuses to work _correctly_ without them, rather
| than merely more slowly.
|
| (Note that theoretically it is possible to design an ISA/ABI
| combo that supports easy and fast unwinding even without frame
| pointers, but there's always going to be _some_ overhead and to
| my knowledge this choice hasn 't been done.)
| brancz wrote:
| Founder of Polar Signals here, the profiling product that's
| mentioned in the blog post. We've already made it possible to
| unwind without frame pointers relatively cheaply (<1%
| overhead), and have written about this extensively [1].
|
| That said, no matter how we spin it, frame pointer unwinding
| is always going to be cheaper, and while profiling is getting
| better, I think I'm almost more excited about the other
| aspects of debuggability this is gaining: out-of-the-box
| working bpftrace, bcc-tool and anything else that needs to
| deal with unwinding with just about anything that's running
| on the box. I think we'll see a huge gain in capabilities
| over the next few years with frame pointers more prevalent in
| fedora and ubuntu and I'm sure more will now follow.
|
| [1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-
| bas...
| stefan_ wrote:
| Embedded devices, certainly in the days of 16 MiB NOR flash,
| do not contain unwind or debug information. Even today
| OpenWrt and similar will routinely strip all binaries
| installed to the final firmware image.
|
| There are some structural issues in the Linux world, too; the
| default of debug data contained within the binary is often
| undesirable, symbol servers (they finally learned about those
| in Ubuntu 22) require extra setup & tooling support that
| isn't often invested in, widely used libraries like libunwind
| are both arcane and terrible (yes, an instruction pointer of
| 0 will not have associated unwind information; use your brain
| and realize someone called a NULL function pointer).
| ghotli wrote:
| > Something has to be seriously wrong with the way it was
| compiled for gdb to have trouble.
|
| Cargo culting culture in embedded devices of just using some
| old toolchain copy pasted from some vendor seems to be
| pervasive. I cut fresh compilers and align their output with
| the old crusty toolchains. Made entire classes of issues go
| away. Regardless, agreed the omission of the frame pointer
| was merely one issue at play with those particular core dumps
| on those particular devices, years back at this point. :)
| zX41ZdbW wrote:
| ClickHouse has always-on profiling without frame pointers. But
| the implementation is very hard - it required patching of LLVM's
| libunwind to make it 100% async-signal safe. Using frame pointers
| should be easier and faster.
| ur-whale wrote:
| What about binary size increase ?
| brenns10 wrote:
| This is a good idea for the short-term. As of now, frame pointers
| are the most reliable way to ensure that software can be profiled
| by tools like perf*. The core issue is that the kernel must be
| the one to unwind the userspace stack, and it only knows how to
| unwind stacks with frame pointers**. The .eh_frame data will
| never be supported by the kernel, because it involves a turing-
| complete program that must be executed to compute the necessary
| unwind info***.
|
| For the long term, the more exciting option that's emerging is
| SFrame[1]. This is a new data section which would be generated by
| the compiler and contains unwind tables which the kernel will be
| able to understand. Unlike DWARF/.eh_frame, these tables would
| remain in the final binary (i.e. not be stripped away), and on
| exec(), the kernel would store them for use during profiling.
| Since the format is quite similar to ORC(*), and Steven Rostedt
| is quite invested in the format, it seems a safe bet that support
| will land in the kernel.
|
| My hope isn't necessarily that a distribution completely disables
| frame pointers once this format becomes available... though it
| could be an interesting thing to try. Rather, there can be a
| conscious choice about whether frame pointers are used, or
| SFrame, which would be useful for cases like Python, where it's
| mentioned that frame pointers may still have a significant
| performance impact. The kernel should be able to fall back to
| frame pointers when SFrame is unavailable, which means that
| either will be acceptable. Ideally, in a few years time we'll be
| able to go back to forgetting about frame pointers for most cases
| :)
|
| ---
|
| * Ironically, the kernel itself tends not to use frame pointers!
| It has its own unwind format called ORC, which gets generated by
| an in-kernel program called "objtool" which essentially reverse
| engineers the assembly generated by the compiler. It's
| x86_64-specific and frequently needs adjustment when the compiler
| changes code generation. It can't be used for userspace programs.
|
| ** it also knows how to unwind kernel stacks with ORC (see above)
|
| *** There is an option to allow perf to unwind with DWARF, but
| it's a total hack (though a very effective one). By passing
| --call-graph=dwarf, you can instruct the kernel to copy the
| userspace stack (by default, 8k bytes!) into the perf event
| buffer with each sample (this can be as many as 100 or 1000
| samples per second, per CPU...). Later, the perf userspace
| program will use that info, along with information about each
| process's address space, and the debuginfo for each program, to
| unwind the stacks. This has huge performance overhead, and it
| requires that you have easy access to debuginfo, which may not be
| the case, especially for container workloads.
|
| [1] https://lwn.net/Articles/940686/
| brancz wrote:
| We've also figured out an alternative format to use from within
| eBPF to unwind stacks (we happen to only support dwarf at the
| moment but theoretically any source information could work):
| https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...
| brenns10 wrote:
| Yeah I saw Vaishali & Javier's presentation [1] at LPC last
| year! Great stuff, & certainly available to use now rather
| than when SFrame becomes available and supported.
|
| In the same spirit, it seems that the .eh_frame -> BPF unwind
| table process could be (relatively) easily modified to
| produce SFrame, which you could attach to the binaries if you
| have a trustworthy way of doing that (which is... a big if).
| So that once SFrame support becomes available in the kernel,
| you could apply it to applications without rebuilding them.
|
| [1]: https://lpc.events/event/16/contributions/1361/
| brancz wrote:
| I would need to double check with the team on this detail,
| but if I recall correctly the architecture as it is is
| specifically designed to make the bpf verfier happy and we
| didn't think it was going to be possible with existing
| formats. But happy to reconsider, we'd of course much
| rather use a standardized format if possible!
| londons_explore wrote:
| So the whole world should take a 1-2% performance penalty on
| everything so some users can maybe run a profiler?
|
| Wouldn't it make more sense to just have an 'apt reinstall all
| --with-frame-pointers' command that power users could run before
| they wanted to profile something?
| alfalfasprout wrote:
| This is very hyperbolic and inaccurate.
|
| In 2023 it's not a 1-2% performance penalty anymore and
| certainly not for most use cases. Only if the 15th register is
| critical for performance on an x86_64 CPU.
|
| Certain workloads might suffer more, but most will certainly
| suffer less than a 1-2% hit.
| akira2501 wrote:
| Using any of the higher 8 registers on an x86_64 requires an
| opcode prefix and makes your instruction 1 byte longer. There
| is still a small reward for avoiding r8-r15.
| issafram wrote:
| very allegorical
| m463 wrote:
| You are prematurely optimizing.
|
| "can make use of this improved debugging information to
| diagnose and target performance issues that are orders of
| magnitude more impactful than the 1-2% upfront cost."
|
| Also, can't you get reliable stack dumps when something goes
| wrong too?
| dymk wrote:
| FTA
|
| > I've enabled frame pointers at huge scale for Java and glibc
| and studied the CPU overhead for this change, which is
| typically less than 1% and usually so close to zero that it is
| hard to measure.
| gloryjulio wrote:
| At the age I would think the observability and the debugability
| are the qualities that I'd like the systems to have. The
| productivity gains r immeasurable.
| aseipp wrote:
| Many systems take various kinds of performance hits in return
| for things all the time; reliability, observability, safety,
| etc. Many systems can be run at higher peak throughput in
| return for various instabilities, even. Performance is not
| actually a uniform number across the system. You're looking at
| an aggregate, but changes like this can make it much, much more
| practical to diagnose _specific_ performance issues for users
| in specific scenarios, which may have extremely large impacts
| far beyond 1-2%. That 's very important in practice especially
| when users can often feel those outliers, e.g. why does this
| application enter a spinning state and suddenly burn CPU for 1
| minute before returning to normal.
|
| > Wouldn't it make more sense to just have an 'apt reinstall
| all --with-frame-pointers' command that power users could run
| before they wanted to profile something?
|
| I don't see why it makes any more sense than just changing the
| default that the distribution uses. For one it's way more work,
| maintaining another copy of everything for a ~1% performance
| difference is not an obviously good tradeoff for the distro
| teams to make. Not to mention it often isn't possible to do
| this in the cases people want it i.e. they want to continuously
| profile an existing production system that they can't just run
| apt on willy nilly.
| alexey-salmin wrote:
| I've seen how instantly-available profiles affect the
| engineering culture on practice and it's transformative. The
| difference between "yeah strange, I'll deploy an fp build some
| time later and check... maybe" and "see this thing right here
| on the flamegraph" is huge and often repays 5-15x of the
| initial 1% slowdown.
| m463 wrote:
| interesting article on frame pointer optimization:
|
| https://community.ibm.com/community/user/wasdevops/blogs/kev...
| AndyKelley wrote:
| Why not let the upstream application developer decide, rather
| than choosing for them?
|
| This is one of the downsides of using C/C++ rather than a modern
| programming language like Rust or Zig. In the former case, the
| system maintainers reach across the table and change the
| settings, despite what the actual application developer has
| chosen. In the latter, the upstream developers' choices are
| respected more, mainly because the tooling is less standardized.
|
| Shoutouts to this NixOS bug which is still ongoing after causing
| much pain for many years:
| https://github.com/NixOS/nixpkgs/issues/18995
___________________________________________________________________
(page generated 2023-12-13 23:01 UTC)