[HN Gopher] KUtrace: Low-overhead Linux kernel tracing facility
___________________________________________________________________
KUtrace: Low-overhead Linux kernel tracing facility
Author : luu
Score : 150 points
Date : 2024-07-15 22:41 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| CalChris wrote:
| Site's article _Benchmarking "Hello, World!"_ is basically a
| KUTrace tutorial.
|
| https://queue.acm.org/detail.cfm?id=3291278
|
| Also, https://www.youtube.com/watch?v=D_qRuKO9qzM
| ndesaulniers wrote:
| Early in my career, I reached out to the author and was able to
| grab lunch with him; he was about to retire! Was insightful to
| hear his thoughts on system performance, particularly systems
| involving more than one machine, which is something they studied
| deeply.
|
| It gave me appreciation for the amount of knowledge one
| accumulates over a career, and what a loss it is to an
| organization when one so knowledgeable retires.
| c0deR3D wrote:
| Out of curious, does BPF now capable of capturing all the context
| switch events such as CPU trap?
|
| Also, if the overhead is negligible, maybe the author can try to
| merge this into mainline with the use of static key to make the
| incurred overhead switchable. In spite of the static key, the
| degree of the accompanied inteferences on cache and branch
| predictor might be an intriguing topic though.
| tanelpoder wrote:
| Lots of the low level exception/trap/fault handling functions
| are blacklisted, probably to avoid lockups and unwanted
| recursion mayhem: $ sudo wc -l
| /sys/kernel/debug/kprobes/blacklist 783
| /sys/kernel/debug/kprobes/blacklist
|
| Edit: Perhaps an alternative approach would be to attach probes
| to relevant (precise) PMU events. There's also this prototype
| of adding breakpoint/watchpoint support to eBPF [1]. But
| actually doing stuff within this context may get complicated
| very fast, so would need to be severely limited, if feasible at
| all.
|
| [1]
| https://ebpf.io/summit-2020-slides/eBPF_Summit_2020-Lightnin...
| usr1106 wrote:
| Sounds very interesting.
|
| But it works by patching the kernel, not just using eBPF like
| many performance tools recently. So it needs active maintenance
| all the time considering the current velocity of internal kernel
| changes. And I would not be surprised if it didn't build or work
| correctly if you have a heavily patched and customized kernel.
|
| On the positive side at a first glimpse the maintenance to adapt
| to new kernels looks very active.
| elric wrote:
| Is the naming intentional? Or just a weird coincidence? Kut being
| Dutch for cunt, by an author called dick..?
| crissaegrim wrote:
| KUtrace stands for "Kernel-User" trace as (most of) the
| tracepoints are at the kernel-userspace boundary. So it is a
| coincidence.
| xoranth wrote:
| How would this interact with `io_uring`, especially the polling
| modes (IO_SETUP_SQPOLL, IO_SETUP_IOPOLL)?
| rwmj wrote:
| You still have to wait for your cache to reload from main
| memory, or for disk or network I/O, or for processes to be
| scheduled to run, so while it's likely more efficient than
| epoll approaches, I doubt there's any really fundamental
| difference in the performance problems you would find.
| li4ick wrote:
| His book, "Understanding Software Dynamics", is one of the best
| technical books I've ever read. Top 3 for me.
| Quizzical4230 wrote:
| If you don't mind me asking, what are the other two?
| li4ick wrote:
| - Managing Gigabytes
|
| - Hacker's Delight
| jeroenvlek wrote:
| Much appreciated. I bought all 3.
| Quizzical4230 wrote:
| Thanks a bunch! Will check them out.
| crissaegrim wrote:
| I had the pleasure to work with Dick on getting KUtrace to work
| on Android devices last year. It was a great experience to work
| with one of the greats in systems performance. He was a wealth of
| information regarding performance bottlenecks and optimizations.
|
| KUtrace is absolutely one of the most powerful tools I've used
| for deeply understanding performance bottlenecks (after isolating
| issues) such as poor scheduling behavior. I would highly
| recommend reading his book "Understanding Software Dynamics" [1]
| if you are interested in learning more about KUtrace or
| performance bottlenecks/optimizations in general. The book is
| quite dense and dives deep into the performance characteristics
| of many examples of the five fundamental resources (according to
| Dick): CPU, Memory, Disk/SSD, Network, and Software critical
| sections.
|
| [1]: https://www.oreilly.com/library/view/understanding-
| software-...
| fschutze wrote:
| Interesting! Does it also work on non-rooted Android devices?
| crissaegrim wrote:
| No it unfortunately does not. You require root to remount the
| read-only system partition, to insert the kernel modules, and
| to turn SELinux off. We used a "userdebug" build to get root,
| but I imagine most of this could also be done with a phone
| rooted through other means (I haven't tried it, however).
| anonymousDan wrote:
| Thanks, looks interesting. Does it cover measuring memory
| bandwidth consumption? This is something I feel there is a lack
| of good tooling for.
| mgaunard wrote:
| Why not just use the eBPF system?
| jeffbee wrote:
| Most people would. The author makes a comparison in this
| interview:
| https://www.usenix.org/system/files/login/articles/login_fal...
|
| Note also that this work emerged within Google a decade before
| eBPF was really useful.
| wyldfire wrote:
| Wasn't there similar useful stuff from systemtap before eBPF?
| That's been around for quite a while.
| jeffbee wrote:
| True again, but SystemTap overhead is 20-100x higher.
| zokier wrote:
| Systemtap, for whatever reason, remained largely RedHat
| solution. It didn't really have much traction outside
| that diaspora.
| tanelpoder wrote:
| The other alternative is LTTng, released in 2005 (requires
| loading a kernel module):
|
| https://lttng.org/
| mgaunard wrote:
| Argument is that KUTrace is faster, which honestly I'm not
| sure of.
| mydriasis wrote:
| I thought this read "training facility", and I was excited to
| sign up!
___________________________________________________________________
(page generated 2024-07-16 23:01 UTC)