[HN Gopher] Looking for More Debugger
___________________________________________________________________
Looking for More Debugger
Author : luu
Score : 52 points
Date : 2021-01-28 09:55 UTC (1 days ago)
(HTM) web link (scattered-thoughts.net)
(TXT) w3m dump (scattered-thoughts.net)
| tomgs wrote:
| Disclaimer (and a bit of context): I work for a company that
| builds a special type of debugger, aimed at debugging live
| applications in production.
|
| As one would expect, we've spent an enormous amount of time
| thinking about debuggers, how they work from the inside and what
| they contribute to the general experience of developing software.
|
| Debuggers are, for developers well-versed in working with them,
| priceless tools (see [1] for an example from this very thread).
| There's so, so much value in walking through a program not only
| in one's head (as well-put by pg at one time[2]), but by using a
| tool that enforces a more structured, "formal" process for
| program exploration. I use it when trying to understand how code
| I write actually looks like IRL, but also - and maybe more so -
| for understanding code OTHER PEOPLE wrote with better context
| than any piece of documentation or simply reading the code would
| ever give me.
|
| But - as the article and some[3] of[4] the[5] comments[6] here
| point out - it has warts (quick plug - I recently wrote a blog
| post[7] that details my perspective on remote debuggers and how
| they can be better).
|
| One thing that I didn't see mentioned in this thread or in the
| article, and is a real PITA, is performance. Debuggers are not
| built to deal with performance problems - or collection of
| temporal data, for that matter - at all.
|
| Consider how one might go about debugging a real-life performance
| issue, especially in the context of modern applications - many
| threads, many hosts, many replicas of the same service, etc.
|
| You probably have a tracer - something like Zipkin[8] or
| Jaeger[9] - that helps you to break apart latency issues in your
| endpoints. You also probably have a set of instrumented metrics
| (graphed inside your APM, perhaps) for quick observation when
| something is not reacting as fast as you expect it to. But
| sometimes they don't tell the full story - there's just too many
| possible edge cases.
|
| Performance issues can be caused by data structures exploding in
| size, causing degraded performance and OOMs. They can be caused
| by one method, in a chain of dozens of different, inter-connected
| ones, that is especially slow. It can be related to non-optimized
| SQL queries (maybe hidden behind an ORM). It can be a weird code
| path - one the developer did not expect, and did not test for -
| that gets triggered by users with very specific characteristics.
|
| Hard.
|
| [1] https://news.ycombinator.com/item?id=25960923#unv_25960923:~.
| .....
|
| [2] http://www.paulgraham.com/head.html
|
| [3]
| https://news.ycombinator.com/item?id=25940064#unv_25957905:~...
|
| [4]
| https://news.ycombinator.com/item?id=25940064#unv_25957481:~....
|
| [5]
| https://news.ycombinator.com/item?id=25940064#unv_25957428:~....
|
| [6]
| https://news.ycombinator.com/item?id=25940064#unv_25957481:~...
|
| [7] https://lightrun.com/debugging/remote-debugging-re-
| examined-...
|
| [8] https://zipkin.io/
|
| [9] https://www.jaegertracing.io/
| souprock wrote:
| There are debuggers that nicely handle real-time with "many
| threads, many hosts, many replicas of the same service".
|
| If you don't mind paying, Concurrent has stuff for it:
| NightView, NightSim, NightTrace
| https://www.youtube.com/user/ConcurrentRealTime/videos
|
| Other approaches include whole-system emulation, digital logic
| analyzers, and SystemTap.
| bjg wrote:
| https://www.gdbgui.com/ is another good frontend for gdb.
| slezyr wrote:
| Qt Creator's debugger is almost perfect for my goals. It's,
| probably, the most user-friendly. However, I wish I could feed
| gdb commands directly to it.
| jcelerier wrote:
| > However, I wish I could feed gdb commands directly to it.
|
| you can: while debugging, enable "Debug Log" (in the View
| menu), and this adds a pane where you can input gdb command.
| Output is not very pretty though.
| flohofwoe wrote:
| Strange that VSCode + debugger frontend extensions (mentioned in
| the first part) caused such trouble, it's working flawlessly so
| far for me for mixed Zig/C projects, across Windows, macOS and
| Linux (Ubuntu 20)
|
| The inspection capabilities in the VSCode debugger are very bare
| bones though, that's true. At least the variable view panel could
| need some work.
| IAmAtWork wrote:
| I was so goo at debugging the hardware I underestimated just
| how difficult will it be to switch to software even when you
| really good at that language. But set up is important esp for
| web technologies as they are buggy by default. Interestingly I
| never had issue with C
| dvirsky wrote:
| It's worth mentioning that vim has added debugger integration
| support as a plugin, but that plugin is available with vim by
| default, you just need to enable it. It's not perfect but it's
| usable - you can set breakpoints visually and save/load them, you
| can step debug inside your code window (gdb is actually running
| in another vim window and can be controlled directly) and you can
| see the value of expressions in your code window.
|
| This is a nice guide to it:
| https://www.dannyadam.com/blog/2019/05/debugging-in-vim/
| kuter wrote:
| I don't think GDB is too bad. Most people probably don't use
| watch expressions, conditional breakpoints etc.
| madhadron wrote:
| > Most people probably don't use watch expressions, conditional
| breakpoints etc.
|
| That's an indictment of the available debugger interfaces that
| they wouldn't.
| Cloudef wrote:
| They become pretty much essential when reversing or debugging a
| program you dont have source for.
| modeless wrote:
| I use them all the time in Chrome dev tools where there is a
| nice GUI for them and they actually work properly.
| dan-robertson wrote:
| I think there are a few sides to this problem:
|
| 1. Debuggers are in some sense anti-unix. They only work well
| when they are thoroughly integrated with the rest of your
| environment. Trying to have some debugger program like gdb and
| then talking to it from emacs or a terminal or something will be
| a big pain. Emacs is also anti-unix but seems to have enough mass
| that it doesn't matter so much (or maybe the debugger and emacs
| fighting for importance is problematics)
|
| 2. Getting a debugger to work well is hard. Apart from needing
| information to work out what bit of code is executing (and
| passing all that information through weird compilation steps or
| linking), the debugger needs to understand the memory layout of
| objects and interpret some subset of the language so that people
| can enter eg watch expressions. This is hard when the compiler
| and debugger are quite separate and just agree on the format for
| possible debug information. Lack of integration hurts again
|
| 3. Some people are so used to not having a debugger that they
| don't know what they're missing. Debuggers work well for what,
| windows and websites? I guess Java too? Lots of people end up
| debugging with prints and that just isn't so effective. I think
| I've barely used a debugger for more than trivial websites in the
| last 10 years.
|
| 4. The problem of debugging has become harder: there are now more
| threads in programs and it is more common for languages to have
| some kind of async mechanism. If you naively debug that, you'll
| see crap stacktraces (a few frames sat atop a scheduler) and
| execution may move randomly between threads in a pool. I feel
| like most debuggers can't cope. (I guess website debuggers do
| understand async better and there are also new inventions like
| reversible debugging with rr, but the problem with integrating a
| debugger with a language or an editor remains.)
| alexhutcheson wrote:
| > Trying to have some debugger program like gdb and then
| talking to it from emacs or a terminal or something will be a
| big pain.
|
| If you're already familiar with Emacs commands, then the M-x
| gdb interface is as good as any debugger interface in an IDE.
|
| If you're not already an Emacs user, then the TUI[1] (gdb -tui)
| is surprisingly good - much better than the "old school"
| approach I was taught in college of just interacting via the
| gdb prompt.
|
| > Debuggers work well for what, windows and websites?
|
| I spend most of my time on a large C++ project that does
| neither of these things, and I still use gdb a lot. In my
| experience, it's most useful for unit tests - you identify a
| bug, write a unit test that triggers the bug, and run the unit
| test under gdb to poke around and understand what's going on.
|
| gdb and lldb don't work great when run on the whole program,
| mainly because it's hard to specify the actual behavior you
| want when debugging a system where multiple threads are
| executing concurrently. Almost all of our unit tests are
| single-threaded, so this isn't a problem when debugging unit
| tests.
|
| If you're working on a C or C++ program and using a supported
| processor, I also highly recommend trying out rr[2], the record
| and replay debugger, which you mentioned in your comment.
| Unlike gdb, it's reasonably easy to use and understand even if
| your program has lots of concurrency, and it pretty much
| eliminates the need to do "printf" debugging, since you can
| print variables from any line in your program without
| recompiling and re-running. You can still use the Emacs or TUI
| gdb interfaces when using "rr replay".
|
| [1]
| https://sourceware.org/gdb/current/onlinedocs/gdb/TUI.html#T...
|
| [2] https://rr-project.org/
| yakubin wrote:
| _> > Debuggers work well for what, windows and websites?_
|
| _> I spend most of my time on a large C++ project that does
| neither of these things, and I still use gdb a lot._
|
| I think that, when GP wrote "windows", they meant the
| operating system. And when they wrote "debuggers work well",
| they meant the quality of debuggers, not whether using a
| debugger is a good idea. Quality of Windows debuggers is
| generally on a whole other level than on Linux. Chromium
| DevTools also has a nice debugger. I don't know about Java.
| dan-robertson wrote:
| By windows, I really meant visual studio at its peak.
| dan-robertson wrote:
| I spend my life in emacs (comments passim) but I put it to
| you that M-x gdb is mostly unpleasant, as is gdb --tui. Both
| pale in comparison to visual studio 10-15 years ago. The OP
| and the post which the OP is following up to both mention
| those debugger interfaces.
| mijoharas wrote:
| I need to throw a million +1's at the `rr` suggestion. it's
| really great. It's a shame that it doesn't work on ARM
| processors, but there are good fundamental reasons why it
| doesn't (it's basically impossible iirc[0])
|
| EDIT - just to mention, rr is mostly compatible with gdb, so
| frontends such as gdb-many-windows in emacs can work with it.
|
| In terms of U.I's I found `gdb-many-windows` to be the nicest
| i found. While I hate almost everything about the platform, I
| have to concede that visual studio's debugging integration is
| actually really good. The only thing I found annoying was
| that I needed to use Visual Studio for it! (emacs on windows
| could be annoying enough that I got used to it eventually,
| but still preferred my linux environment. This is back when
| working on a cross platform C++ project if that much weren't
| already clear).
|
| [0] https://github.com/rr-
| debugger/rr/issues/1373#issuecomment-1...
| pflanze wrote:
| The last part of that page mentions that it has recently
| been made to work on aarch64 given the right CPU and all
| code including libraries built so they avoid using the
| offending instructions.
| touisteur wrote:
| About composability since I've taken to fuzzing I've been
| improving my gdb chops and now I mostly write gdb scripts if
| the program is not too long to run, or I just use a simple
| bisect script with with core dump checkpointing and detecting
| changes of whatever I'm interested in. With a large tmpfs it's
| quite fast and modular.
|
| If I'm doing real-time, Intel PT is great, especially if you
| can afford the signal-shit happening when dumping core (you get
| Intel PT trace in core dumps now and gdb supports them if
| compiled with...)
| greggman3 wrote:
| > Getting a debugger to work well is hard
|
| And yet somehow Microsoft made theirs work well since the early
| 90s
| jcelerier wrote:
| > 1. Debuggers are in some sense anti-unix.
|
| it's not a problem when you accept that the unix philosophy is
| extremely far from the holy grail of software design. "things
| that do one thing well" certainly has good sides for specific
| problems, mainly quick'n'dirty linear text processing
| dataflows, but is atrocious for other problems (anything that
| need integration and discoverability: art software, web
| browsers, or revolves more around user interaction than
| computing something)
|
| > 3. Some people are so used to not having a debugger that they
| don't know what they're missing. Debuggers work well for what,
| windows and websites?
|
| if debuggers took less time to start than a full "add print ->
| recompile -> relaunch app" cycle as is my experience with C++,
| they would certainly get more use.
|
| Hell, in my main project I've been unable to use any version of
| gdb & lldb for something like two months as both segfault a
| couple seconds after reaching a breakpoint for some reason.
| It's a shitshow.
| DoofusOfDeath wrote:
| > Hell, in my main project I've been unable to use any
| version of gdb & lldb for something like two months as both
| segfault a couple seconds after reaching a breakpoint for
| some reason. It's a shitshow.
|
| I'm really surprised. In my experience it's really rare for
| either of those debuggers to crash, let alone for _both_ to
| crash on the same target.
|
| A bit of a tangent, but would you mind sharing some details
| about your compiler / OS / debugger / target-hardware
| versions?
| jcelerier wrote:
| amd64 arch linux, so latest version of clang++ and
| gdb/lldb. Project built against c++20. Here's what I'm
| getting from gdb:
|
| https://twitter.com/jcelerie/status/1348259387127824387/pho
| t...
|
| (lldb does not give me an error message, just segfault)
| DoofusOfDeath wrote:
| Interesting. I just took a look at the line of gdb code
| that's crashing on you. I'm only slightly familiar with
| gdb's internals, but I can imagine a bunch of different
| sources for this bug:
|
| - gdb has a misconception about which c++ ABI was used
| when compiling the target program, or
|
| - your program is stomping on an object's vtable pointer,
| leading gdb to follow invalid pointers when trying to
| parse the object's RTTI, or
|
| - there's a bug in clang++, or
|
| - there's a mix of abi versions in your program's code,
| perhaps because of unexpected linking issues (static or
| dynamic), or perhaps because of inconsistent compiler
| flags when building your code
|
| I'm curious what happens if you build your program with
| g++ instead of clang++.
| tkinom wrote:
| Same here.
|
| CLI version of gdb has worked extremely reliable for me
| across multiple projects and architectures (x86, arm,
| arm64, etc).
|
| I have yet to encounter a bug can't be debug with gdb with
| some test automation + gdb script with conditional
| breakpoints / hw breakpoints. As long as I can reproduce
| the issue with test automation, I can debug it with gdb.
|
| Most complex bugs are I have to debug was multiple threads
| (20+) memory leak shows up after days of testing.
|
| I don't use GUI gdb. cli + gdb script is the best and most
| reliable debugging environment. I don't add printf to my
| code anymore, a new gdb breakpoint script + with proper
| debug build can replace the printf anywhere and anytime.
| TwoBit wrote:
| No offense but I'm guessing your debugging needs aren't
| as complicated as some others' needs.
| dan-robertson wrote:
| I agree that debuggers being anti-unix isn't a problem if
| they embrace that and integrate deeply with their environment
| but such a phrase does not describe gdb or lldb.
| fouric wrote:
| > if debuggers took less time to start than a full "add print
| -> recompile -> relaunch app" cycle as is my experience with
| C++, they would certainly get more use.
|
| This is a really good point - programmers seem to prefer
| making "dead"/batch-processing programs over "living",
| interactive, persistent systems (according to Steve Yegge's
| definition of those[1]).
|
| This style, in addition to being very Unix-y, also lends
| itself really well to printf debugging - you just add
| printfs, recompile, and re-run.
|
| I build "live" programs, and make heavy use of the debugger -
| although that might be because of tight integration of my
| debugger with the rest of my tools - although _that_ might be
| because other users of this language tend to prefer the live
| style and build their tools to adapt.
|
| [1] http://steve-yegge.blogspot.com/2007/01/pinocchio-
| problem.ht...
| jcelerier wrote:
| I also build live programs, I just have a simple system
| that reloads its state from before a crash upon statup -
| that ends up working fine 99% of the time and I can get
| back to just before the faulty interaction without loosing
| time.
| majormajor wrote:
| > 3. Some people are so used to not having a debugger that they
| don't know what they're missing. Debuggers work well for what,
| windows and websites? I guess Java too? Lots of people end up
| debugging with prints and that just isn't so effective. I think
| I've barely used a debugger for more than trivial websites in
| the last 10 years.
|
| They're extremely useful for dynamic languages (Python, Ruby,
| etc) since you can step through call chains and see the full
| objects being passed around. Wonderful way to learn a new
| codebase where you have a bunch of functions passing stuff
| around with no type information in the code.
|
| The debugger is the main reason you're not gonna pull me away
| from IntelliJ for JVM or dynamic language stuff... it could be
| even better - like how CLion is discussed - but I'm not seeing
| a better debugger-less alternative.
|
| Hooking into async callback stuff and losing useful stacktraces
| isn't great, but it's not like print statements are gonna do
| any better there either...
| dummy_physicist wrote:
| Some years ago I learned about ProDBG[0] and was pretty excited.
| It is going through a rewrite, I hope it gets usable soon.
|
| [0] https://github.com/emoon/ProDBG
| saagarjha wrote:
| > It doesn't actually seem to be trying to write "/tmp". It
| actually doesn't issue any syscalls at all when I press "Debug
| Main Project" and trigger the message.
|
| Perhaps a subprocess is doing this?
| jcalabro wrote:
| If you use Windows at all, you should check out RemedyBG[0], it's
| excellent. But Windows-only.
|
| [0] https://remedybg.itch.io/remedybg
| corysama wrote:
| They say "early access". Can anyone speak to its current
| stability/issues?
| samrat wrote:
| I recently discovered https://github.com/weirdNox/emacs-gdb/ .
| The installation is a bit unconventional compared to the usual
| Emacs package installation process, but works pretty well.
|
| This recreates the Visual Studio style workflow described in
| https://twitter.com/pervognsen/status/1098453808651202560 and
| https://youtu.be/0woxSWjWsb8?t=254
| wallstprog wrote:
| The one thing I really miss about Windows is the Visual Studio
| debugger -- it was frickin' awesome even back w/VS 2005 and I
| have to believe it's gotten better.
|
| On Linux I use ddd, and while it's definitely quirky (in that
| weird X way), it works just fine. I would guess that many of the
| poster's problems have to do with Linux's mediocre GUI support in
| general. I've recently switched to doing development on Ubuntu
| (vs. RH/CentOS) and all the GUI stuff works _much_ better in that
| environment, incl. ddd.
|
| I haven't tried Eclipse's C/C++ debugger, but its Perl debugger
| (part of EPIC) is quite nice and works well on both Mac and
| Linux.
| GartzenDeHaes wrote:
| I continue to be shocked at the lack of productive development
| tools on linux. My current c/c++ toolset is: a makefile generator
| that I wrote after years of wasting time on autotools problems,
| Anjuta for debugging and rudimentary editing, and Visual Studio
| 2008 on Windows XP running in VirtualBox for most productive work
| (it's a cross platform project).
| Cloudef wrote:
| vim, gcc, gdb, radare2, valgrind, callgrind, helgrind,
| coreutils, git, make
___________________________________________________________________
(page generated 2021-01-29 23:02 UTC)