Remote debugging - no assembler code displayed and no code execution pointer lazarus / 12-Debugger ---------------------------------------------------------------------- ccrause · 2017-07-01 20:33 When debugging a remote AVR simulation running on simavr I don't see the assembler code in the assembler debug view.  I also don't see the location of the current line of code to be debugged.  See screen shot of a debugging session attached.  I can view the values of variables in the watch list and see the values update as I step through the code, so the basic debugger session works. Anyone with ideas where I can start digging? ---------------------------------------------------------------------- Ñuño_Martínez · 2017-07-03 08:20 I didn't do such debugging, but since Lazarus is compatible with GDB may be you can use such debugger. If I remember correctly it has a quite stable remote debugging system. ---------------------------------------------------------------------- ccrause · 2017-07-03 09:18 Quote from: Ñuño_Martínez on July 03, 2017, 10:20:33 amI didn't do such debugging, but since Lazarus is compatible with GDB may be you can use such debugger. If I remember correctly it has a quite stable remote debugging system. Some more detail (I realize I was a bit cryptic in the first post): I'm using avr-gdb to debug a remote session with simavr as simulator to run the AVR code.  Lazarus is configured to connect to a gdbserver connection.  Lazarus stops at break points and display simple data types and register contents correctly. However Lazarus doesn't show the current execution point in the source file and also doesn't show code in the Assembler view. My suspicion is that the GDB interface in Lazarus doesn't take into account the architecture difference between the debug target (embedded avr) and the host (linux x86-64), but I don't know how to check that. ---------------------------------------------------------------------- Martin_fr · 2017-07-03 20:34 I will try to add more later / travelling now. check the communication between IDE and gdb / window "debug output" or better via define when compiling the IDE. http://forum.lazarus.freepascal.org/index.php/topic,37353.msg250966.ht ml#msg250966 GDB Should report the location when stopping. (reach breakpoint). IIRC in theIDE the function ProcessStopped. there is a wiki page about this, and how to provide a logfile with all the gdb output. ---------------------------------------------------------------------- ccrause · 2017-07-09 12:06 I've tested debugging for both AVR (via simavr for gdb server) and Linux using the same test project code: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1_linux_x86_64; var  a, b, c: word; function test(const a1, a2: word): word;begin  if a > a2 then    result := a1  else    result := a2;end; begin  a := 250;  b := 5;  c := test(a, b);end. This project was then compiled either for Linux x86_64 (development environment, FPC 3.0.2) and debugged with gdb or compiled for AVR (FPC 3.1.1 cross compiler) and debugged with avr-gdb. A breakpoint was inserted at line 17, the project was run up to the breakpoint and the debug output was saved as project1_linux_x86_64-debug.txt or project1_avr-debug.txt respectively. The native Linux test obviously debugged correctly, with disassembled code and watches populated and the code execution line highlighted.  The AVR test doesn't show disassembled code, the watch (variable a) shows an incorrect value and no highlighting of the current execution point in the source code. The debug output for the native Linux project shows that the break point has been reached, then calls -data-disassemble a couple of times: Code: [Select]=breakpoint-modified,bkpt={number="8",type="breakpoint", disp="keep",enabled="y",addr="0x000000000040021b",func="main",file="pr oject1_linux_x86_64.pp",fullname="/home/christo/fpc/fpc- avr/src/examples/test/project1_linux_x86_64.pp",line="17",thread- groups=["i1"],times="1",original-location="/home/christo/fpc/fpc- avr/src/examples/test/project1_linux_x86_64.pp:17"} ~"\n" ~"Breakpoint 8, main () at project1_linux_x86_64.pp:17\n" ~"17\t  c := test(a, b);\n" *stopped,reason="breakpoint-hit",disp="keep",bkptno="8",frame={addr="0 x000000000040021b",func="main",args=[],file="project1_linux_x86_64.pp" ,fullname="/home/christo/fpc/fpc- avr/src/examples/test/project1_linux_x86_64.pp",line="17"},thread- id="1",stopped-threads="all",core="0" (gdb) <-data-disassemble -s 4194736 -e 4194737 -- 0> Similarly the AVR project debug output shows that it reached the breakpoint, but then no disassemble calls, only calls to determine the type and value of variable "a" which is in the watch list: Code: [Select]=breakpoint-modified,bkpt={number="6",type="breakpoint", disp="keep",enabled="y",addr="0x0000011a",func="main",file="project1_a vr.pp",fullname="/home/christo/fpc/fpc- avr/src/examples/test/project1_avr.pp",line="17",thread- groups=["i1"],times="1",original-location="/home/christo/fpc/fpc- avr/src/examples/test/project1_avr.pp:17"} ~"\nBreakpoint " ~"6, main () at project1_avr.pp:17\n" ~"17\t  c := test(a, b);\n" *stopped,reason="breakpoint-hit",disp="keep",bkptno="6",frame={addr="0 x0000011a",func="main",args=[],file="project1_avr.pp",fullname="/home/ christo/fpc/fpc- avr/src/examples/test/project1_avr.pp",line="17"},thread- id="1",stopped-threads="all" &"ptype A\n" ~"type = WORD\n" ^done (gdb) &"whatis A\n" ~"type = WORD\n" ^done (gdb) <-data-evaluate-expression A> ^done,value="12090" (gdb) <-thread-info> To me it seems as if the debugger communication with avr-gdb works fine, but for some reason the gdbmiserverdebugger doesn't request the same amount of information as gdbmidebugger. Attached please find detailed debugger logs and Lazarus IDE output logs for th two test cases. ---------------------------------------------------------------------- Martin_fr · 2017-07-09 12:54 Is it correct for avr that the exe is loaded at very low addressed? 0x011a ? ---------------- * Ensure you compile with -O- (optimization 0 / none). Even -O1 can screw up. * You may want to try to switch between dwarf(2) and stabs, if there are issues, this may help (project options/debug) * If avail, older gdb.  (on some platforms (win) gdb got worse above version 7.7) ---------------- My expertise is only about the IDE integration (communication to gdb). * If the error is broken debug info by fpc or the linker, then that needs to be discussed with someone from the fpc team (fpc mail list). So it will be hard to tell if it is fpc or gdb. * If the error is in gdb, then all you can try is a diff version of gdb In both cases stabs vs dwarf may help. ---------------- As for the watch: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i <-data- evaluate-expression A>^done,value="12090"(gdb)If gdb is returning the wrong value, then there is nothing the IDE can do. Either gdb is at fault, or fpc generated bad debug info, or the linker messed it up. -------- Disassembler: How or why this affects the disassembler is not clear. Please try (avr only) http://wiki.lazarus.freepascal.org/GDB_Debugger_Tips#Log_info_for_debu g_session This includes the state of the IDE, so it gives some more info. Change the --debug-enable option to:  --debug-enable=DBG_CMD_ECHO,DBG_STATE,DBG_DATA_MONITORS,DBGMI_QUEUE_D EBUG,DBGMI_TYPE_INFO,DBG_DISASSEMBLER,DBG_LOCATION_INFO,DBG_BREAKPOINT ,DBGMI_WITH_DISASS_OVERFLOW,DBG_EVENTS ---------------------------------------------------------------------- ccrause · 2017-07-09 15:24 Quote from: Martin_fr on July 09, 2017, 02:54:01 pmIs it correct for avr that the exe is loaded at very low addressed? 0x011a ? Yes, see attached disassembled source (project1_avr.lss.zip) for memory addresses. Also attached a debug log of a Lazarus debugging session for AVR (lazarus_avr.log.zip). The only thing that seems strange is that after the -exec-continue command, there appears to be an extra "(gdb)" in the output.  See from around line 1003 in the log: Code: [Select]  DebuggerState: Finished dsRun   << TCmdLineDebugger.ReadLn "(gdb) "   ERROR: Got NO stop params at all, but was running The (what I think) is the expected output << TCmdLineDebugger.ReadLn "=breakpoint-modified,bkpt= only gets read on line 1057. A similar debug test with target Linux reads the breakpoint message immediately after issuing the -exec-continue command: Code: [Select]  DebuggerState: Finished dsRun   << TCmdLineDebugger.ReadLn "=breakpoint-modified,bkpt= Could this perhaps mean that avr-gdb is emmitting and extra (gdb) line of text that isn't expected and should be ignored? ps. Thanks for the link to the wiki page for debugger tips Martin. ---------------------------------------------------------------------- Martin_fr · 2017-07-09 18:06 The extra gdb prompt is normal. Remote debug uses gdb in async mode. Normally if the exe in gdb is running, gdb does not accept any commands until the exe stops/pauses. But over remote this is necessary in order to send an "interrupt" (pause request) command. (In none remote, the IDE itself (bypassing gdb) sends a signal to the exe). In Async mode after a run command, the 1st "(gdb)" prompt acknowledges that the run command was accepted. The 2nd "(gdb)" prompt is sent, when the exe is stopped/paused. ----------- The problem in your log is, the order in which data comes from the debugger. The IDE expects the "*stopped,reason=" before the prompt. (And that may be wrong of the IDE). In your example the "(gdb)" prompt happens first, the "*stopped,reason=" later (and therefore ignored). So the IDE does not know where the app stopped, and that explains the lack of location and asm. (You may be able to get asm, by right clicking a stackframe in the stack win, and choose disasm from the context menu) Please report a bug: " remote debug fails to get break location, if gdb sends "*stopped,reason="  late. See http://forum.lazarus- ide.org/index.php/topic,37405.msg252081.html#msg252081 " And attach the log. ------------------------ I have no clue what causes the wrong watch value. Have you tried stabs/dwarf? And -O- Otherwise I suggest to post a question on the fpc mail list. ---------------------------------------------------------------------- ccrause · 2017-07-10 16:32 Bug report: issue 0032130 ---------------------------------------------------------------------- Martin_fr · 2017-07-30 20:27 Could you test? See comment on bug report. ---------------------------------------------------------------------- ccrause · 2017-07-31 22:35 I've tested r.55619 with RequirePID patch from 32084.  It seems as if the IDE now stops and highlights the break point, but then a whole string of disassembler instructions gets generated without end.  I have to stop the debugger to stop the dissembler instruction flood.  From the Lazarus log output the following line seems to  indicate the start of attempted disassembly on line 907 (see attachment): Code: [Select]INFO: TDBGDisassembler.FindRange: Address not found 282 wanted-before=15 wanted-after=17 in map with count=0 Address 282 is the address of the assembler instructions for the line of code at the break point.  A bit further down the log the following seems suspicious: Code: [Select]    >> TCmdLineDebugger.SendCmdLn "-data-disassemble -s 42 -e 466 -- 0"     << TCmdLineDebugger.ReadLn "^done,asm_insns=[{address="0x0080002a" ,inst="nop"},{address="0x0080002c",inst="nop"}, It seems as if avr-gdb added 0x800000 to the address, perhaps this is caused by this known issue?  I will investigate further at a later stage. ---------------------------------------------------------------------- Martin_fr · 2017-07-31 23:26 good news for the breakpoint. About the dis-assembler: there are 2 issues (from your description / I will need to find time to dive through the log later) 1) gdb adding 0x80000 2) the IDE trying forever. 1) is a gdb issue, and probably not solvable from the IDE. It may also lead to the IDE not understanding the result, and discarding it.  (that is unlikely to get fixed in the IDE) 2) this should be fixed in the IDE. the IDE should detect it is not getting sensible data, and should stop. ----------------------- In general the disasm code of the IDE is complicated. Because the IDE attempts to show disass code from after and BEFORE the current location. Disassembling from the current location is easy. The address is known. But disassembling from before is tricky. The byte length of each asm statement is different. That means there is no way of knowing if current_location -0x10 is the begin of a statement or not. But even if it is not, you can start disassembling there, but you get wrong code. So the IDE tries to look for begin of a function, or it tries from different locations and uses some heuristics. That is probably where it currently ends up in a loop. ---------------------------------------------------------------------- Michael Collier · 2017-07-31 23:31 I was trying out the avr-gdb command line debugger from within a windows shell, I asked avr-gdb for the memory location of global variable (it gave 0x800200 which I thought looked strange). When I peeked in the address I received error. Here is shell output: Code: DOS  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i (gdb) p &my_global$3 = (int *) 0x800200(gdb) x 8002000xc35c8:        Ignoring packet error, continuing... I googled for "0x800000 atmel avr memory" and some pages were found striaght away: http://www.atmel.com/webdoc/avrlibcreferencemanual/FAQ_1faq_ramoverlap .html http://www.atmel.com/webdoc/avrlibcreferencemanual/mem_sections_1sec_d ot_data.html Note: As I am writing this I got notification of new post, so maybe martin_fr can help better, hope this helps.. ---------------------------------------------------------------------- ccrause · 2017-08-03 06:29 Quote from: Martin_fr on August 01, 2017, 01:26:05 am1) is a gdb issue, and probably not solvable from the IDE. It may also lead to the IDE not understanding the result, and discarding it.  (that is unlikely to get fixed in the IDE) I'm busy compiling gdb 8.0 with this patch for the avr_integer_to_address internal function which should correct the 0x800000 offset in program memory.  Will test again once I get gdb running. ---------------------------------------------------------------------- ccrause · 2017-08-04 05:10 I think the first issue of detecting that gdb stopped is fixed.  Using a patched gdb the disassembling works and Lazarus displays the disassembled code correctly.  Will log a separate issue to stop the disassemble process eventually to cater for the normal behaviour of gdb for AVR. Thanks Martin! ---------------------------------------------------------------------- ccrause · 2017-08-05 13:03 I've done some more testing with different versions of gdb compiled from source for target AVR to see which versions work with Martin's latest patches: gdb 6.6 - Doesn't work because async mode is not supportedgdb 7.0.1 - Works Sometimes work, see note below.gdb 8.0 - Doesn't work as is, but works with patch.So to ensure that debugging remote target AVR in Lazarus works one needs to apply the patch to gdb versions 7.1 to 8.0 else the disassemble command doesn't parse the correct code space addresses. Edit note: After I upgraded Linux Mint and retested gdb 7.0.1 I noticed the following warning while testing a debug session: Code: DOS  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i warning: (Internal error: pc 0xa2 in read in psymtab, but not in symtab.)  GDB would sometimes not work properly and debugging from Lazarus using this version of gdb could hang the IDE. ---------------------------------------------------------------------- Martin_fr · 2017-08-05 13:55 Thanks. Maybe add this here: http://wiki.lazarus.freepascal.org/Remote_Debugging ---------------------------------------------------------------------- ccrause · 2017-08-05 19:57 Quote from: Martin_fr on August 05, 2017, 03:55:18 pmThanks. Maybe add this here: http://wiki.lazarus.freepascal.org/Remote_Debugging Done! ---------------------------------------------------------------------- avra · 2017-08-08 11:27 Quote from: ccrause on August 05, 2017, 03:03:40 pmgdb 7.0.1 - Worksgdb 8.0 - Doesn't work as is, but works with patch.It would be nice to provide downloadable binaries, at least for your os and bitness ---------------------------------------------------------------------- ccrause · 2017-08-09 19:22 Quote from: avra on August 08, 2017, 01:27:56 pmIt would be nice to provide downloadable binaries, at least for your os and bitness  Download link... ---------------------------------------------------------------------- avra · 2017-08-10 19:13 Quote from: ccrause on August 09, 2017, 09:22:24 pmQuote from: avra on August 08, 2017, 01:27:56 pmIt would be nice to provide downloadable binaries, at least for your os and bitness  Download link... Thank you!