Sqlite3 Memory Leak lazarus / 16-Databases ---------------------------------------------------------------------- jdp · 2016-07-22 22:34 Hi all. I hope somebody can help me. I have a time on a form that fires every 20 seconds to update a totals table in a sqlite3 db. The memory will go up with about 4 bytes every time. I am releasing the memory.  What am I doing wrong for this to happen ? Regards Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  //Create the DB Connectionfunction CreateConnection:TSQLite3Connection;begin  result := TSQLite3Connection.Create(nil);  result.HostName := '';  result.DatabaseName := 'Solar.db';  result.UserName := '';  result.Password := '';end; procedure update_daily_totals();var   //Declare the DB Vars  DBConnection : TSQLite3Connection;  DBTransaction : TSQLTransaction;  DBQuery : TSQLQuery; begin  try     //Create a connection    DBConnection := CreateConnection;    //Create a transaction    DBTransaction := TSQLTransaction.Create(DBConnection);    //Point to the database instance    DBTransaction.DataBase := DBConnection;     //Open the DB    DBConnection.Connected := True ;    //Create a query to return data    DBQuery := TSQLQuery.Create(DBConnection);    //Point the query to DB and Transaction    DBQuery.DataBase := DBConnection;    DBQuery.Transaction := DBTransaction;     //Insert some data (Primary Key is the date and On Conflict Replace has bee set    DBQuery.SQL.Text := 'INSERT INTO DAILY_TOTALS (RECORD_DATE,GRID_KWH, KWH_CONSUMED, SOLAR_KWH_PRODUCED, MAX_PV_WATTS, MAX_LOAD_WATTS) values (:P1,:P2,:P3,:P4,:P5,:P6)';     //Set the parameters    DBQuery.Params.ParamByName('P1').AsDate := Date;    DBQuery.Params.ParamByName('P2').AsFloat := GridTotalDayExportCount;  DBQuery.Params.ParamByName('P3').AsFloat := GridTotalDayCount;    DBQuery.Params.ParamByName('P4').AsFloat := SolarTotalDayCount;    DBQuery.Params.ParamByName('P5').AsFloat := 0;    DBQuery.Params.ParamByName('P6').AsFloat := 0;      //Execute    DBQuery.ExecSQL;    //Commit the record    DBTransaction.Commit;    //Close the query    DBQuery.Close;    DBTransaction.CloseDataSets;  DBConnection.Connected := False;     finally     //Release the memory     DBTransaction.Free;     DBQuery.Free ;     DBConnection.Free ;    end; end; ---------------------------------------------------------------------- Leledumbo · 2016-07-23 02:09 You can't really know unless you use some leak tracer. So activate -gh along with -gl first, then run your program until it terminates. The result should tell you if there are really memory leaks and their respective stacktrace. ---------------------------------------------------------------------- jdp · 2016-07-23 04:00 Hi. I did do a trace and it shows that all blocks are freed. Yet the memory is growing. ---------------------------------------------------------------------- jdp · 2016-07-23 04:59 Ok this is very strange. If I take this code on its own and place it in a form and I call the code from a timer every sekond then I have no memory leak. But if I use the same code in my bigger app I do. So it seems that some componants or library is interfering with this code. ---------------------------------------------------------------------- rvk · 2016-07-23 07:31 So your memory leak is somewhere else. FYI. if your create an object with an owner you should free it yourself. Only objects created with owner nil should be freed by your code. So either create DBTransaction and DBQuery with nil or remove the .Free lines for those. (It won't cause the memory leak but internally it's very inefficient to free objects if they are "managed" by other objects.) See the warning at the end of this page http://delphi.about.com/od/kbcurt/a/dynamiccreation.htm And http://delphi.about.com/od/kbcurt/ss/dynamiccreateno.htm ---------------------------------------------------------------------- jdp · 2016-07-23 09:29 Hi rvk. Thanks for the reply. What I did was to disable all the threads and timers. Then I enabled them one at a time until I saw the memory grow. The memory stands still until I enable the timer that fires this code in the bigger app. Yet if I run that code on its own it does not grow. And the memory profiler shows all blocks are relaesed. I have tried cmem as well. What I also see is that if I include ssl_openssl in the uses then the debugger will crash. It compiles the app but it wont debug. so can it be that some libraries dont play well together ? ---------------------------------------------------------------------- Thaddy · 2016-07-23 10:34 Quote from: jdp on July 23, 2016, 11:29:19 amWhat I also see is that if I include ssl_openssl in the uses then the debugger will crash. It compiles the app but it wont debug. so can it be that some libraries dont play well together ? Are you compiling for windows 32 bit? That makes a huge difference..... for shared libraries compiled in another language. You need to enable win32 SEH. What we need is platform, compiler version. Both are really important to give good advice in this case. Btw. You narrowed the problem already down to the code that gets executed in the timer. There are two strategies that I would try: - never allocate and free objects nin the timer code, but simply reset them - Within the timer code, use freeandnil. ---------------------------------------------------------------------- jdp · 2016-07-23 14:25 Thanks Thaddy. I am developing an app that works with my solar system on the Raspberry Pi. So the OS is Rasbian Jessie. Lazarus 1.6 (Date 2016-05-23) FPC 2.6.4 arm-linux-gtk 2 When you say reset the variable are you saying I should declare them as globals and reuse them all the time ? ---------------------------------------------------------------------- rvk · 2016-07-23 14:50 Quote from: jdp on July 23, 2016, 04:25:03 pmLazarus 1.6 (Date 2016-05-23) FPC 2.6.4 arm-linux-gtk 2 You didn't install Lazarus 1.6 with FPC 3.0 ?? If you only see the memory leak with the procedure in a timer you could try to create a small test-project with this procedure in a timer. If you still can't simulate it with a small test-project there must be something wrong elsewhere in your code. If you can simulate it with a small test-project then you can zip it and place it here in a post so we can look what's going wrong. ---------------------------------------------------------------------- jdp · 2016-07-23 16:07 Hi. This does not make any scenes at all. If I use this code on its own in a little test app then it works 100%. But If I call it from a timer or a button in the Inverter app then the memory will go up. This is the code I am calling now. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure update_daily_totals();var   //Declare the DB Vars  DBConnection : TSQLite3Connection;  DBTransaction : TSQLTransaction;  DBQuery : TSQLQuery; begin  try     //Create a connection    DBConnection := CreateConnection;    //Create a transaction    DBTransaction := TSQLTransaction.Create(nil);    //Point to the database instance    DBTransaction.DataBase := DBConnection;     //Open the DB    DBConnection.Connected := True ;    //Create a query to return data  DBQuery := TSQLQuery.Create(nil);    //Point the query to DB and Transaction    DBQuery.DataBase := DBConnection;    DBQuery.Transaction := DBTransaction;     //Insert some data (Primary Key is the date and On Conflict Replace has bee set    DBQuery.SQL.Text := 'INSERT INTO DAILY_TOTALS (RECORD_DATE,GRID_KWH, KWH_CONSUMED, SOLAR_KWH_PRODUCED, MAX_PV_WATTS, MAX_LOAD_WATTS) values (:P1,:P2,:P3,:P4,:P5,:P6)';     //Set the parameters    DBQuery.Params.ParamByName('P1').AsDate := Date;    DBQuery.Params.ParamByName('P2').AsFloat := GridTotalDayExportCount;  DBQuery.Params.ParamByName('P3').AsFloat := GridTotalDayCount;    DBQuery.Params.ParamByName('P4').AsFloat := SolarTotalDayCount;    DBQuery.Params.ParamByName('P5').AsFloat := 0;    DBQuery.Params.ParamByName('P6').AsFloat := 0;     DBQuery.Params.Clear;     //Execute    DBQuery.ExecSQL;    //Commit the record    DBTransaction.Commit;    //Close the query    DBQuery.Params.Clear;    DBQuery.Close;    DBConnection.Connected := False;     finally      //Release the memory     freeandnil(DBTransaction);     freeandnil(DBQuery);     freeandnil(DBConnection);    end; end; ---------------------------------------------------------------------- wp · 2016-07-23 16:32 How do you measure that memory usage is growing? In my opinion the only relevant test is heaptrc. If there are zero unfree'ed memory blocks at program end then your program does not have any memory leak whatever you own measuring routine tells. It depends on how the memory manager works and how the balance is set between speed and heap fragmentation. It is hard to predict from the outside how memory usage will evolve while a program is running. Not every allocation is balanced by an equal amount of released memory. Sometimes the memory manager will decide to allocate more or release less memory than requested - what's not released this time may be released the next time. This is out of your control. You only can make sure that every allocation has a matching release, but you cannot control when the release occurs. Ultimately all blocks marked for release are released at program end, and here heaptrc tells you whether any previously allocated blocks are left. ---------------------------------------------------------------------- marcov · 2016-07-23 19:36 Quote from: wp on July 23, 2016, 06:32:13 pmIf there are zero unfree'ed memory blocks at program end then your program does not have any memory leak whatever you own measuring routine tells. That depends on your definition of a memory leak. In the strictest sense (a block without reference in the program outside the heapmanager), it might be. But if something gets added to some global resource that is freed on shutdown, you get all behaviour of a program with a memleak, without the heaptrc leaks to prove it.  Examples that I found out to my shame: - memo's used for logging that due to circumstances got fuller than expected - in a web framework that we used, some resource was not freed from some list, but that list was freed on shutdown. - Several cases with queues that were not drained by their consumers as intended (in various circumstances) but all were deallocated happily on shutdown. Basically having excess references where you don't expect them is the same as a memleak. ---------------------------------------------------------------------- jdp · 2016-07-23 19:59 How we check the memory is with htop. When I start the app this is the memory (first image). You can also see all the threads Then 20 minutes later you can see how much it grew. Now if I leave the app to run for a day it will grow 5mb per day. If I don't start the timer that runs the DB code it does not grow at all.