[SOLVED] Memory leak - how can I locate it?
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
lennit · 2016-02-19 19:17

Hello,
My program gets a huge memory leak "out of nowhere"/no idea from
where. I singled the options out to this piece here. It seems to me I
clear all instances after use, but I obviously don't since it takes
few loops for the memory usage to start piling up. My program usually
uses up to 13MB RAM, after several hours of operation it goes to
250MB+. What is the thing that I miss? This code here is the procedure
(executed over timer every 30sec) that leads to the memory pile. The
arrays used here are not dynamic so the leak can't come from them,
since I use arrays with pretty short length. All *temp here are
constants - html templates as used in Internet Tools.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i StringRes
:= TStringList.Create;  if HttpGetText('192.168.1.111:8080',
StringRes) then      memResponse.Lines.AddStrings(StringRes);    for i
:= 1 to memresponse.Lines.Count do      content := content +
memresponse.Lines[i - 1];    if content <> '' then    begin     
process(content, c1temp);      danni[11, j, 1] :=
strtofloatdef(processedVariables.get(0).toString, 0);     
processedvariables.Clear;      process(content, c2temp);     
danni[11, j, 2] := strtofloatdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;      process(content, iatemp);    
danni[11, j, 3] := strtofloatdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;      process(content, attemp);    
danni[11, j, 4] := strtofloatdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;      process(content, rhtemp);    
danni[11, j, 5] := strtofloatdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;       process(content, a1temp);   
 pvvisual11.Value := strtointdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;      process(content, a2temp);    
hevisual11.Value := strtointdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;      process(content, a3temp);    
covisual11.Value := strtointdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;       process(content, r1temp);   
 relaystate[1] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r2temp);     
relaystate[2] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r3temp);     
relaystate[3] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r4temp);     
relaystate[4] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r5temp);     
relaystate[5] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r6temp);     
relaystate[6] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r7temp);     
relaystate[7] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r8temp);     
relaystate[8] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r9temp);     
relaystate[9] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r10temp);     
relaystate[10] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r11temp);     
relaystate[11] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r12temp);     
relaystate[12] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r13temp);     
relaystate[13] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r14temp);     
relaystate[14] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r15temp);     
relaystate[15] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r16temp);     
relaystate[16] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;       content := '';     
memresponse.Clear;      StringRes.Clear;    end;

----------------------------------------------------------------------
wp · 2016-02-19 19:21

I don't know the context of these lines, but what strikes my eyes is
that the "StringRes" created in the first line does not have a
matching "Destroy" at the end.

----------------------------------------------------------------------
lennit · 2016-02-19 19:26

I use StringRes.Clear as I use this StringList throughout all the
runtime.
StringRes := TStringList.Create; was added now in my post so that it's
clear what StringRes is. I call it at startup. Isn't this enough to
clear the cache:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i      
processedvariables.Clear;      content := '';      memresponse.Clear; 
    StringRes.Clear; 
Or should I create and destroy the StringList at every run of the
timer?

----------------------------------------------------------------------
lennit · 2016-02-19 19:34

In short this procedure here does the following:
Using HTTPSend by synapse strips a web page of it's body into a
StringListThe StringList is processed to a string, as the template
comparison works only with stringsA search for certain variables in
the web page is performedInstances are cleared for the next run in 30
seconds.Maybe that's not the smartest way to process form data, but
it's the one I learned and the only I handle as of now.

----------------------------------------------------------------------
wp · 2016-02-19 19:43

Clearing and destroying are different things. When you "clear" a
stringlist, it is empty but the object TStringList still exists, you
can assign new strings to it without creating it again. "Destroy", on
the other hand, removes the instance of the stringlist, you can no
longer use it unless you create it again. If you don't "destroy" an
object, you end up with a memory leak because the memory occupied by
it is not released. Don't confuse this memory with the string items
removed by "Clear".

In total: every call of .Create must have a matching .Free (which
calls .Destroy). The only (?) exception are classes administrated by
the LCL by means of the owner-mechanism.

----------------------------------------------------------------------
lennit · 2016-02-19 19:50

So if I understood you correctly, the fact that I called
StringList.Clear and I guess clear of the memo as well, did nothing to
clear the memory used.
I thought though, that since they use the same (or pretty much the
same) amount of data, it will somehow keep the same memory usage.
Now I'll make a version that destroys the instance to test it.
Thanks for the explanation!

----------------------------------------------------------------------
JuhaManninen · 2016-02-19 20:06

Quote from: lennit on February 19, 2016, 08:50:52 pmSo if I understood
you correctly, the fact that I called StringList.Clear and I guess
clear of the memo as well, did nothing to clear the memory used.

I guess you come from Java or .NET.
Pascal does not have garbage collection.

----------------------------------------------------------------------
wp · 2016-02-19 20:08

For learning it may be better to run a small project because your
large project may contain other memory leaks.

See attached demo which creates a stringlist (L) and populates it with
100 random strings after clicking the button. After that the
StringList is cleared, but not destroyed - this is what you are doing.
When you close the application you'll see a report with a long list of
memory leaks (this is because the option "Use heaptrc" is checked in
the debugging options of the project inspector). The report tells you
that the memory leaks are created in the procedure
TForm1.Button1Click, line 48 of unit1.pas.

Then uncomment the "L.Free" at the end of the button click procedure.
Now the StringList is properly destroyed, and heaptrc reports "0
unfreed memory blocks" - which is perfect.

----------------------------------------------------------------------
lennit · 2016-02-19 20:27

Thank you @wp!
I guess I have another problem (or another problem too), because the
leak continues even with StringList.Free. Tomorrow is Saturday, so
I'll be able to start working on it early on, as it is quite late
night where I am now.
@JuhaManninen The most theory I've read is Java-related, that's why I
suppose I expect Clear to get me all cleaned. Otherwise I am an
economist, working in financial administration, and part-time writing
software for the family business, because this is what I love. As you
can see for now I am a poor coder

----------------------------------------------------------------------
wp · 2016-02-19 20:50

Quote from: lennit on February 19, 2016, 09:27:16 pmI guess I have
another problem (or another problem too), because the leak continues
even with StringList.Free.There are probably more of this kind. Look
at the heaptrc report, some lines are not resolved, but normally
you'll find a line number and a procedure name soon. You must know
that this list is upside down, i.e. the latest calls are higher in the
list. Go to the first (top-most) memory location which you can
identify by line number. If you look at your source code you will
usually see a "Create", "New", "GetMem" or some other memory
alloation. As I said - "Create" must have a matching "Free", "New"
must have a matching "Dispose", "GetMem" must have a matching
"FreeMem" somewhere, etc.

Some cases are a bit tricky though: For example, if you use the
Objects property available for StringGrids to store additional data
with each string item you must know that this data memory is not freed
automatically when you delete an item, clear or destroy the list.
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i type 
TMyData = object    ID: Integer;    Value: Double;  end; var  L:
TStringList; procedure AddToStringList(Sometext:String, ID: Integer;
Value: Double);var  data: TMyData;begin  data := TMyData.Create;  //
"Create" --> There MUST be a "Free" somewhere!!!  data.ID := ID; 
data.value := Value;  data.City := ACity;  L.AddObject(SomeText,
data);end;  // When you Delete a list item you MUST free the TMyData
object! procedure DeleteItem(AIndex: Integer);var  data:
TMyObject;begin  data := L.Objects[AIndex];  data.Free;   // <--- This
is the "Free" corresponding to the "Create" above! 
L.Delete(AIndex);end;
etc. etc...

----------------------------------------------------------------------
JuhaManninen · 2016-02-19 21:15

Quote from: lennit on February 19, 2016, 09:27:16 pm@JuhaManninen The
most theory I've read is Java-related, that's why I suppose I expect
Clear to get me all cleaned.

Then you should know the concept "garbage collection". Java has it,
Pascal not.
Garbage collection makes things easier for a programmer but IMO is
overrated considering the extra resources used by a virtual machine.
Most memory management is straightforward and easy also manually.

----------------------------------------------------------------------
lennit · 2016-02-20 12:48

Now it's working. This is the code that it transformed to after some
tries.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TForm1.checkWeb;var  content:string;  StringRes:TStringList;begin  j
:= (HH * 6) + (MM div 10) + 1;  if pingres[1] = True then  begin   
StringRes := TStringList.Create;    if
HttpGetText('192.168.1.111:8080', StringRes) then begin      if
StringRes.Count>0 then        for i := 1 to StringRes.Count do       
 content := content + StringRes.Strings[i - 1];    if content <> ''
then    begin      process(content, c1temp);      danni[11, j, 1] :=
strtofloatdef(processedVariables.get(0).toString, 0);     
processedvariables.Clear;      process(content, c2temp);     
danni[11, j, 2] := strtofloatdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;      process(content, iatemp);    
danni[11, j, 3] := strtofloatdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;      process(content, attemp);    
danni[11, j, 4] := strtofloatdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;      process(content, rhtemp);    
danni[11, j, 5] := strtofloatdef(processedVariables.get(0).toString,
0);      processedvariables.Clear;       process(content, r1temp);   
 relaystate[1] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r2temp);     
relaystate[2] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r3temp);     
relaystate[3] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r4temp);     
relaystate[4] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r5temp);     
relaystate[5] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r6temp);     
relaystate[6] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r7temp);     
relaystate[7] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r8temp);     
relaystate[8] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r9temp);     
relaystate[9] := strtointdef(processedVariables.get(0).toString, 0); 
   processedvariables.Clear;      process(content, r10temp);     
relaystate[10] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r11temp);     
relaystate[11] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r12temp);     
relaystate[12] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r13temp);     
relaystate[13] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r14temp);     
relaystate[14] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r15temp);     
relaystate[15] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      process(content, r16temp);     
relaystate[16] := strtointdef(processedVariables.get(0).toString, 0); 
    processedvariables.Clear;      content := '';     
StringRes.Free;      freeThreadVars; // that thing here was the
culprit :)))    end; end;end;end;   
Looks like the processedvariables from Internet Tools needed a
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
freeThreadVars; added to the
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
processedvariables.Clear; at the end of the cycle.
Thank you @wp and @JuhaManninen for giving me such a great explanation
and examples

----------------------------------------------------------------------
wp · 2016-02-20 13:08

One more tipp: If you request resources (memory) from the OS you
should protect your code by means of a try-finally block. This makes
sure that the "Free" is reached even if an exception occurs somewhere
after "Create":
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i begin 
StringRes := TStringList.Create;  try    // do something  finally   
StringRes.Free;  end;end;

----------------------------------------------------------------------
lennit · 2016-02-20 13:15

Yeah. I was actually reading about this in order to see how to make it
and found a good article here in the forum this morning. Before that I
didn't realize I need it, so I have a software that controls a whole
production facility without a single "try...finally..." there 
Thanks for the advice! I am working on it now.