Needed advice to debug leaks shown only after switching to new Laz&FPC lazarus / 12-Debugger ---------------------------------------------------------------------- avra · 2017-12-11 11:42 I have a medium sized Win32 application running 24/7 in heavy industry for more then a year. It has GUI and 2 communication threads, writing data to Oracle. Threads are updating GUI in thread safe way, and I have also implemented thread safe logging for main thread and communication threads. Application is usually running for months without stopping, and I do not have any memory leaks reported on application exit, as you can see in this heap trace output: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i D:\Data\MyUserName\My Documents\Lazarus Projects\L1toOracle\L1toOracle.exe Heap dump by heaptrc unit277512 memory blocks allocated : 97836608/98786600277512 memory blocks freed   : 97836608/987866000 unfreed memory blocks : 0True heap size : 327680True free heap : 327536Should be : 327680 The problem is that when I want to upgrade to any later FPC/Lazarus I get leaks as reported here (even if I just start application and exit it immediately): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i d:\-\-\-\-\-\L1toOracle.exe Heap dump by heaptrc unit18330 memory blocks allocated : 2193384/224837618327 memory blocks freed     : 2193316/22482963 unfreed memory blocks : 68True heap size : 327680 (80 used in System startup)True free heap : 327280Should be : 327328Call trace for block $00081170 size 28  $0070F4B6  $0070E2C5  $004107E6  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00DCall trace for block $0007B370 size 12  $0070E2C5  $004107E6  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00DCall trace for block $00081100 size 28  $006E1D1C  $004107E6  $004107E6  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D Compiling Lazarus with full debug info and running backtrace does not show anything useful, as you can see here: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i GNU gdb (GDB) 7.9Copyright (C) 2015 Free Software Foundation, Inc.License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law.  Type "show copying"and "show warranty" for details.This GDB was configured as "i686-pc-mingw32".Type "show configuration" for configuration details.For bug reporting instructions, please see:.Find the GDB manual and other documentation resources online at:.For help, type "help".Type "apropos word" to search for commands related to "word".This binary was built by Equation Solution ...Reading symbols from L1toOracle.exe...done.(gdb) runStarting program: d:\-\-\-\-\-\L1toOracle.exe[New Thread 7780.0x1778][New Thread 7780.0x10d4][Thread 7780.0x10d4 exited with code 0][New Thread 7780.0x136c][New Thread 7780.0x1aac][New Thread 7780.0x1670][New Thread 7780.0x210][Thread 7780.0x136c exited with code 0][Thread 7780.0x1aac exited with code 0][Thread 7780.0x1670 exited with code 0][Thread 7780.0x210 exited with code 0][Inferior 1 (process 7780) exited normally](gdb) btNo stack.(gdb) In earlier cases when leaks were detected it was mostly my fault that could be located because line numbers and name hints were giving me enough info to get an idea where things went wrong. But this time I have no hints and I do not know where to look. I expected to get more info when I rebuilt Lazarus with "Debug IDE" profile ("-gw -gl -godwarfsets -gh -gt -Co -Cr -Ci -Sa"), but that was not the case. Did I make some error in this step? I would really appreciate any idea or shared experience of the best strategy to start hunting this kind of leaks. Btw, project was initially done in CodeTyphon 5.60 and I faced the problem when trying to make it a Lazarus project. I have tried latest Lazarus stable, Lazarus trunk, and both Lazarus and FPC trunk, and leaks were always present (trunk was from about a month ago). I have also tried to update CodeTyphon and that was only partially successful. The latest CT version where I do not have any leaks is 5.80 (with all updates installed it reports FPC 3.1.1 SVN 51943 and Lazarus from 2016-04-02). Any later CT version also introduces leaks in my application. ---------------------------------------------------------------------- Martin_fr · 2017-12-11 13:51 I never saw BaadFood on a stacktrace. That might mean there is a deeper issue. Maybe you are writing to already freed memory. Try -gh and set environment HEAPTRC="keepreleased" As for the trace. Load in gdb, and try   info line *0x004107E6 I dont know why the top of your trace has addresses in the 00070.... range. Do you use libraries written in pascal? Then the leak is in one of those libs, try the info line on the address 007... The 004... should be in your main app. ---------------------------------------------------------------------- Thaddy · 2017-12-11 14:06 I don't know if it is the case here, but usually $BADF00D indicates heap memory that is allocated in a local routine and not freed: IOW: possibly a locally created class that is not freed. ---------------------------------------------------------------------- avra · 2017-12-11 14:17 Update: Problem solved. No more leaks!    After playing with project options, I have finally managed to get a meaningfull heaptrace: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i d:\-\-\-\-\-\L1toOracle.exe Heap dump by heaptrc unit17635 memory blocks allocated : 2052749/210581617632 memory blocks freed     : 2052681/21057363 unfreed memory blocks : 68True heap size : 393216 (80 used in System startup)True free heap : 392816Should be : 392864Call trace for block $00081248 size 28  $0070D496  TIDTHREADSAFE__CREATE,  line 253 of ./source/IdThreadSafe.pas  $0070C2A5  IDTHREAD_$$_init$,  line 730 of ./source/IdThread.pas  $004107E6  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00DCall trace for block $0007B448 size 12  $0070C2A5  IDTHREAD_$$_init$,  line 730 of ./source/IdThread.pas  $004107E6  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00DCall trace for block $000811D8 size 28  $006DFCFC  IDSTACK_$$_init$,  line 1225 of ./source/IdStack.pas  $004107E6  $004107E6  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D  $BAADF00D It was Indy unit that is not used at all, but somehow it was forgotten and Indy was still in project's packages list. It was enough to delete Indy and leaks have gone. I will test more, but it seams that I can finally use newer Lazarus and FPC. @martin: Thank you so much for looking into this problem. Although I solved the problem before applying your suggestions, I find them very valuable. This has been an uncharted terithory for me, and I can learn a lot from them. Thank you very much! @thaddy: I did not have leaks with old Laz/FPC, so it was not a local class not being freed. As you can see from the latest update, leaks have gone once I deleted not really used Indy package reference from the project. Thank you for looking into this! ---------------------------------------------------------------------- Thaddy · 2017-12-11 14:23 I recall Indy has some expected memory leaks registered (at least under Delphi) ---------------------------------------------------------------------- Remy Lebeau · 2017-12-11 20:24 Quote from: Thaddy on December 11, 2017, 03:23:30 pmI recall Indy has some expected memory leaks registered (at least under Delphi) That is correct.  The 3 leaks in the above report are from Indy. The IdThread unit has a global TIdThreadSafeInteger object, which has an internal TCriticalSection object. The IdStack unit has a global TCriticalSection object. These two global objects are intentionally leaked by default, but the leaks can be disabled by re-compiling Indy with FREE_ON_FINAL enabled (but do read the warnings documented at the very bottom of those units' source code). In Delphi, these 3 leaks are registered with the memory manager so they do not appear in leak reports.  FreePascal does not have that option in the HeapTrc unit (see this discussion). ---------------------------------------------------------------------- avra · 2017-12-11 22:12 Quote from: Remy Lebeau on December 11, 2017, 09:24:16 pmThe 3 leaks in the above report are from Indy. ... These two global objects are intentionally leaked by default, but the leaks can be disabled by re-compiling Indy with FREE_ON_FINAL enabled (but do read the warnings documented at the very bottom of those units' source code).It would really be nice if this knowledge ends up in a wiki: http://wiki.freepascal.org/Indy_with_Lazarus. If no one else with more knowledge of the issue wants to do it, I will. ---------------------------------------------------------------------- avra · 2017-12-26 14:48 QuoteIt would really be nice if this knowledge ends up in a wikiHere it is. Please review and correct if needed. http://wiki.lazarus.freepascal.org/Indy_with_Lazarus#Memory_leaking_is sue btw1. In last 2 weeks my application was running 24/7 and I was lucky to have one network down event and several oracle server down events. Exceptions are logged and properly handled so on application exit there were no leaks. Problem definitely solved! btw2. If anyone is interested, I have contributed thread safe TMemoChannel to MultiLog and assisted author with making MultiLog fully thread safe. All communication threads from my mentioned application log data to the same log file and to the same Memo component without problems. I am not aware of any other logging library that can do this at the moment. ---------------------------------------------------------------------- Thaddy · 2017-12-26 16:46 Remy, Avra, The leaks may disappear if you wrap the leaking classes in something like my smartpointer code I borrowed from an idea by Marco Cantu: http://forum.lazarus.freepascal.org/index.php/topic,33095.msg221076.ht ml#msg221076 http://forum.lazarus.freepascal.org/index.php/topic,37524.msg252382.ht ml#msg252382 Until smartpointers are implemented as standard, that is. Threadsafe. Note my code is cross-platform and tested on nixes (arm too...) and win32/64. E.g.: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program smartpointersCS;{$ifdef fpc}{$mode delphi}{$endif}uses  classes,smartptrs,SyncObjs;var  L:TCriticalSection;  PL:Auto;begin  // Auto create  L:= PL.Value;  L.Acquire; try ... worker.  finally   L.Release; end;  // Auto cleanupend. This uses the smartptrs unit from the second link. ---------------------------------------------------------------------- Remy Lebeau · 2021-12-15 17:03 Quote from: Thaddy on December 26, 2017, 05:46:47 pmThe leaks may disappear if you wrap the leaking classes in something like my smartpointer code I borrowed from an idea by Marco Cantu: http://forum.lazarus.freepascal.org/index.php/topic,33095.msg221076.ht ml#msg221076 http://forum.lazarus.freepascal.org/index.php/topic,37524.msg252382.ht ml#msg252382 The leaks are not a matter of not being able to free the objects (obviously, they can be, as seen when FREE_ON_FINAL is defined), but more a matter of unit dependency.  The objects in question are shared with other units, and it has been noticed that sometimes units may not always be finalized in the order you are expecting, and that has been observed in the wild.  So rather than freeing these objects by default, they are leaked instead, in case they are still accessed later.  The leak is minor (only a few bytes), and only during unit finalization, which 9x% of the time will only be during app shutdown (unless you are using Indy in a DLL that gets unloaded dynamically during runtime).