[HN Gopher] Writing a debugger from scratch: Breakpoints
___________________________________________________________________
Writing a debugger from scratch: Breakpoints
Author : ingve
Score : 282 points
Date : 2023-09-27 06:31 UTC (16 hours ago)
(HTM) web link (www.timdbg.com)
(TXT) w3m dump (www.timdbg.com)
| parttimenerd wrote:
| I've done something similar in Python: A Python debugger from
| scratch in Python
|
| - https://github.com/parttimenerd/python-dbg/ - Part 1:
| https://mostlynerdless.de/blog/2023/09/20/lets-create-a-pyth... -
| Part 2:
| https://mostlynerdless.de/?p=1102&preview=1&_ppp=a17cda3e36
| ekidd wrote:
| This is a great series!
|
| I noticed that the author was using https://github.com/hydro-
| project/rust-sitter as a parser. Which is based on https://tree-
| sitter.github.io/tree-sitter/. I've been hearing about Tree-
| sitter a lot recently, so I dug into it.
|
| Tree-sitter is a tool for generating fast, incremental parsers.
| In particular, the algorithm is suited towards writing "language
| servers" for IDEs, which re-parse code incrementally as the user
| works. These kinds of incremental parsers have historically been
| a huge problem. It looks like Tree-sitter is an enormous
| practical advance in this area.
|
| And discovering that there's a way to use Tree-sitter from Rust
| is _fantastic_. From the post:
| #[rust_sitter::language] pub enum EvalExpr {
| Number( #[rust_sitter::leaf(
| pattern = r"(\d+|0x[0-9a-fA-F]+)", transform
| = parse_int )] u64 ),
| Symbol( #[rust_sitter::leaf(
| pattern = r"(([a-zA-Z0-9_@#.]+!)?[a-zA-Z0-9_@#.]+)",
| transform = parse_sym )] String
| ), // ...
|
| Getting easy access to fast, incremental parsing is a huge win.
| And Tree-Sitter has support for being used from a huge list of
| languages, not just Rust.
| timmisiak wrote:
| Absolutely, rust sitter is fantastic. I haven't used any other
| parsers in Rust so I don't have much of a comparison point, but
| it's probably hard to get much more clear and concise, which I
| think really helps.
| jamra wrote:
| The part that makes tree-sitter useful for this kind of thing
| is the error recovery. It's hard to do error recovery
| correctly. Tree-sitter gives you the ability to continue
| parsing your code which makes it useful for authoring tools.
| junon wrote:
| Tree sitter also has a bunch of deficiencies that don't make it
| ideal for a number of usecases, or sort of act bizarrely in
| some edge cases. Just evaluate tools like this cautiously, of
| course. But I like what it's done for the ecosystem as a whole!
| a1o wrote:
| I still don't know how to deal with forward declarations when
| using tree-sitter. :/
| jamra wrote:
| Because tree-sitter lexes as it parses, you may have to use
| an external scanner in order to deal with this kind of
| stuff. Where are you stuck trying to deal with forward
| declarations?
| a1o wrote:
| It's a simple parser that was originally made to be used
| through Atom that I would like to repurpose elsewhere
|
| https://github.com/edmundito/tree-sitter-ags-
| script/issues/1
|
| If this could be solved, we could port this AGS Script
| parser to the AGS Editor. Today, the parser Adventure
| Game Studio uses for the needs like auto-complete and
| it's very simple refactor like things uses a custom
| handmade parser built in C#. I think if we could leverage
| tree-sitter we could speed things up and repurpose it to
| build things like a LSP for AGS Script.
| jamra wrote:
| It looks like a bug in the grammar. I'll bookmark this
| and see if I can make time for it later. Probably won't
| be able to. I recently build a grammar from scratch so
| I'm okay at tree sitter
| xvilka wrote:
| Writing cross platform (different OS, POSIX and not, different
| architectures, different endianess, etc) debugger, this is where
| all pain lies.
| matt3210 wrote:
| Does anyone know a similar article using c/c++? Interesting
| concept.
| SebastienWae wrote:
| Yes, the "Writing a Linux Debugger" series in C++.
| https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/
|
| And more generally there is "The Debugging Book" in python.
| https://www.debuggingbook.org/
| [deleted]
| dgb23 wrote:
| This is amazing! I've been thinking about writing a debugger (for
| learning how they work etc.). This series is going to be a
| massive help!
| timmisiak wrote:
| Honestly it's a great exercise for learning how low level stuff
| works in general. Happy to answer any questions you have!
| Andrew018 wrote:
| [dead]
| heelwood wrote:
| [flagged]
| [deleted]
| _hl_ wrote:
| @dang
| danparsonson wrote:
| Great article, thanks - one question I couldn't see answered
| there is, what do you do when you want to set more than four
| breakpoints at once?
| vinge wrote:
| Others have already mentioned software breakpoints where the
| instruction is replaced, another option is to run the code in
| an emulator that supports a virtually unlimited set of
| breakpoints. For example, using QEMU with its GDB stub.
| saagarjha wrote:
| Use software breakpoints (which are mentioned but not
| described, the short story for those is you overwrite the
| address you care about with an illegal instruction and
| execution traps when it encounters that code, and then you undo
| it to continue).
| ithkuil wrote:
| You use software breakpoints.
|
| Basically you overwrite the instruction you want to break at
| with a breakpoint instruction (e.g. int 3 on x86). This will
| cause the process to trap and the OS will then let the debugger
| process know about out somehow, e.g. via the SIGTRAP signal on
| Unix.
|
| The debugger then replaces the int 3 opcode (which is a single
| byte conveniently) with the first byte of the original
| instruction so that the execution can continue.
| hinoki wrote:
| If you revert the int 3 to the original instruction's byte,
| when do you put it back? The breakpoint could still be
| active.
|
| In a trivial example, the breaking instruction could be a
| jump to itself, which you'd expect to immediately break into
| the debugger again.
|
| I thought the debugger had to emulate the instruction
| instead, but it's not like I've ever implemented one...
| i_don_t_know wrote:
| I believe when you resume the debugger, you can tell the
| process/thread to single-step over one instruction. So it's
| something like this:
|
| 1. Overwrite instruction with int 3.
|
| 2. When you hit the breakpoint, restore the original
| instruction.
|
| 3. Single-step over the original instruction by changing
| the thread's EFlags (Intel).
|
| 4. Restore the breakpoint with int 3.
|
| 5. Resume normally.
| hinoki wrote:
| Wouldn't that race against any other thread in the
| process? I guess you could stop all threads when you hit
| the breakpoint and start them again after you restore the
| breakpoint, but the synchronisation of that would be
| really tricky too.
| timmisiak wrote:
| Yes, software breakpoints are difficult to get correct (the
| main reason why I started with hardware breakpoints). It
| gets more complicated with kernel debugging, where a single
| step (trap flag) could get pre-empted by an interrupt
| handler. And you can't always single-step a CPU and leave
| all other CPUs frozen.
| spc476 wrote:
| As i_don_t_know stated, if the CPU has the ability to
| single step an instruction, you use that. Otherwise:
|
| * Restore the original instruction byte.
|
| * Find the next instruction, and set a temporary software
| breakpoint there.
|
| * Resume the one instruction
|
| * Restore the original instruction byte at the temporary
| software breakpoint.
|
| * Set the software breakpoint in the original instruction
|
| * Resume running
|
| The other thing to keep in mind is dealing with JMP, CALL
| and conditional branch instructions. It can get pretty
| messy pretty quick, which is why I find low level debuggers
| on old 8-bit CPUs a marvel as they had to deal with only
| software breakpoints.
| ithkuil wrote:
| Emulation is an option, rotating hardware debug registers
| is another option, detecting self-jumps is another option.
|
| I really only implemented a debugger for the esp8266 and it
| was just good enough for me and my team to get our job done
| so it didn't handle many edge cases like that
| whartung wrote:
| I'll share this anecdote told by a friend of mine.
|
| He was on a team building a Modula-2 compiler for OS/2, and his
| group was working on the debugger.
|
| At some point a debugger becomes feature complete enough that you
| can use the debugger to ... debug the debugger.
|
| But this was OS/2 which has true multiple processes (unlike it's
| contemporary Windows 3.1). So you could, naturally, run the
| debugger in one process and attached it to another process which,
| just so happens to be another instance of the debugger.
|
| As with all things, while doing this they encountered bugs in the
| debugger that, well, needed to be debugged.
|
| He said there was a certain epiphany when they realized, because
| of the multi process nature of OS/2, that they could debug the
| debugger debugging the debugger.
|
| I would imagine this took a bit of focus. Turn away for a moment
| and probably really messes with your head.
| gpderetta wrote:
| Surely you would then attach the debugger being debugged to the
| original debugger to debug it and stop the recursion (and of
| course instantly deadlock).
| meepmorp wrote:
| It's bugs all the way down.
| mywittyname wrote:
| I want to hear stories about debugger-on-debugger heisenbugs.
| timmisiak wrote:
| I think my record when I was on the WinDbg team was 5 debuggers
| deep.
|
| I honestly think one of the best parts of writing a debugger is
| being your own recursive customer. I think that's something you
| only get to do for a few things. Debuggers,
| languages/compilers, and operating systems. And probably a few
| others.
| lzybkr wrote:
| Not 5 levels, but I once wrote debug visualizers for a
| compiler using funceval (the visualizer uses the debugger to
| run code in the target process).
|
| I think I once had to debug the debugger debugging the
| compiler compiling itself which felt like another really
| weird kind of recursion.
| robertlagrant wrote:
| Font authors?
| cbsks wrote:
| I worked at a company that made JTAG probes. When you wanted to
| debug the firmware on the probe you'd attach another probe to
| it. And if you encountered a bug while debugging that probe,
| then you'd attach another probe...
| sroussey wrote:
| Focus is still an issue today. We had a version of Firebug that
| would let you debug Firebug. It was great! A bit buggy though,
| so you can see where this is going...
|
| That said, even today when debugging Chrome DevTools with
| Chrome DevTools, window placement is key!!! Ideally, different
| screens. That keeps the mind clear.
| elischleifer wrote:
| Working at Microsoft back in the early 00s I spent a lot of
| unfriendly hours with windbg. On one particular project we hunted
| for a terrible crash for months until it was uncovered that we
| were compiling against the single thread CRT when using threads
| extensively...whoops
| zubairq wrote:
| Really nice read. Does anyone know any other good articles or
| videos about how to write a debugger?
| jansommer wrote:
| I wrote this: https://ja.nsommer.dk/articles/x86-debugger-for-
| windows-and-...
|
| It's a debugger for Windows (and Wine), like the one in the
| article, written in C. It uses software breakpoints (infinite
| breakpoints)
| mrazomor wrote:
| I really liked https://blog.tartanllama.xyz/writing-a-linux-
| debugger-setup/
|
| I learned a lot.
| wila wrote:
| How about a book?
|
| I like "Advanced Windows Debugging" by Mario Hewardt and Daniel
| Pravat.
| HansLambda wrote:
| Take a look heere: https://eli.thegreenplace.net/tag/debuggers
| It was eye-opening for me.
| Modified3019 wrote:
| Not what you asked for, but if that's your interest you'll
| probably appreciate https://justine.lol/blinkenlights/
| emmanueloga_ wrote:
| I was asking this myself this while reading the book "Crafting
| Interpreters". I posted a few resources I found on an issue
| about implementing debuggers [1] -- although honestly I still
| haven't gotten down to read all of them (or to implement a
| debugger! :-/).
|
| --
|
| 1:
| https://github.com/munificent/craftinginterpreters/issues/92...
| a1o wrote:
| Besides breakpoints, any ideas on inspecting the value of a
| variable in each step, figuring out what variables are in
| scope, for the case of an interpreter?
| tibbydudeza wrote:
| For the want of a decent JS React debugger.
| uxp8u61q wrote:
| Debugging JS with vscode is probably the nicest debugging
| experience I've had, bar maybe C# with the latest VS.
___________________________________________________________________
(page generated 2023-09-27 23:02 UTC)