Possible memory leak in TCSVDataset standard class
lazarus / 16-Databases

----------------------------------------------------------------------
kveroneau · 2020-03-28 21:48

I have been trying to eliminate this memory leak heaptrc keeps
complaining about in my code.  I have tried almost everything I can
think of, and refactored my code multiple times this afternoon to no
avail.  I am starting to think that either I am not doing something I
am suppose to, or there is an issue with TCSVDataset where it is
constantly leaking 726 bytes of memory.  The interesting part too, is
if I call this procedure from a Button instead of a menu, it no longer
says the leak is from any of my modules...  So, I suspect the
TCSVDataset class is not correctly freeing something.  Here's my code
where the issue is originating from:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TScheduleForm.FetchScheduleClick(Sender: TObject);begin 
FFileName:=InputBox('Fetch Schedule from Server...', 'Remote Schedule
to fetch?', FFileName);  if FFileName = '' then    Exit;  if
CSVDataset.Active then    CSVDataset.Close;  if FCSVStream <> Nil
then    FCSVStream.Clear  else    FCSVStream:=TMemoryStream.Create; 
if not GetSchedule(FCSVStream, FFileName) then  begin   
ShowMessage('Unable to perform fetch!');   
StatusBar.SimpleText:='Fetch failed for: '+FFileName;   
SpecialMenu.Enabled:=False;    SendSchedule.Enabled:=False;    Exit; {
If this is called, and I just exit the program, no memory leaks are
reported. }  end;  CSVDataset.LoadFromCSVStream(FCSVStream); { Memory
leak only occurs if this is called. }  StatusBar.SimpleText:='Working
with Schedule: '+FFileName;  SpecialMenu.Enabled:=True; 
SendSchedule.Enabled:=True;end; procedure
TScheduleForm.FormDestroy(Sender: TObject);begin  WriteLn('Running
Destroy...');  if CSVDataset.Active then    CSVDataset.Close;  if
FCSVStream <> Nil then    FCSVStream.Free;end; function
TScheduleForm.GetSchedule(dst: TMemoryStream; fname: String):
Boolean;begin  with TFPHTTPClient.Create(Nil) do    Try     
Get(ENDPOINT_URL, dst);      if dst.Size = 3 then       
Result:=False      else        Result:=True;    finally      Free;   
end;end; 
The memory leak only happens if I use the LoadFromCSVStream procedure
call.  The loading seems to work as expected, as my TDataSource, and
my TDBGrid are perfectly able to display the CSV data.

I should note that this CSV file is headless, meaning there are no
Column headers in the file.  I have the following options set on
TCSVDataset:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   object
CSVDataset: TCSVDataset    FieldDefs = <          item        Name =
'Time'        Attributes = [faRequired]        DataType = ftString   
   Precision = -1        Size = 6      end          item        Name =
'Type'        Attributes = [faRequired]        DataType = ftString   
   Precision = -1        Size = 5      end          item        Name =
'Message'        Attributes = [faRequired]        DataType = ftString 
      Precision = -1        Size = 255      end>   
CSVOptions.FirstLineAsFieldNames = False   
CSVOptions.DefaultFieldLength = 255    CSVOptions.Delimiter = ','   
CSVOptions.QuoteChar = '"'    CSVOptions.LineEnding = #10   
CSVOptions.IgnoreOuterWhitespace = False   
CSVOptions.QuoteOuterWhitespace = True    left = 32    top = 80  end 
Here is the full stacktrace from heaptrc output when I use the menu
item selection mode:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Running
Destroy...Heap dump by heaptrc unit231177 memory blocks allocated :
14243823/14729128231165 memory blocks freed     : 14243097/1472838412
unfreed memory blocks : 726True heap size : 327680True free heap :
325056Should be : 325400Call trace for block $00007FFFE17C9440 size
33  $0000000000649554 line 83 of include/menuitem.inc 
$0000000000649EAA line 293 of include/menuitem.inc  $0000000000435C6F 
$0000000000649554 line 83 of include/menuitem.inc  $0000000000649EAA
line 293 of include/menuitem.inc  $0000000000435C6F  $0000000000602D3F
line 4959 of include/wincontrol.inc  $0000000000602B0A line 4871 of
include/wincontrol.incCall trace for block $00007FFFF7E02900 size 27 
$0000000000649554 line 83 of include/menuitem.inc  $0000000000649EAA
line 293 of include/menuitem.inc  $0000000000435C6F  $0000000000649EAA
line 293 of include/menuitem.inc  $0000000000435C6F  $000000000056AD08
line 434 of include/lclintf.inc  $0000000000706032 line 338 of
include/messagedialogs.inc  $000000000070608E line 344 of
include/messagedialogs.incCall trace for block $00007FFFE17A90C0 size
192  $0000000000649554 line 83 of include/menuitem.inc 
$0000000000649EAA line 293 of include/menuitem.inc  $0000000000435C6F 
$0000000000682C4C line 67 of include/buttonglyph.inc 
$00000000006816A2 line 24 of include/bitbtn.inc  $000000000070EFAA
line 177 of buttonpanel.pas  $0000000000710133 line 539 of
buttonpanel.pas  $000000000070F0BC line 194 of buttonpanel.pasCall
trace for block $00007FFFF7E5BC00 size 80  $000000000079F95B 
$0000000000649554 line 83 of include/menuitem.inc  $0000000000649EAA
line 293 of include/menuitem.inc  $0000000000435C6F  $00000000004677E8
line 71 of sched.pas  $0000000000649554 line 83 of
include/menuitem.inc  $0000000000649EAA line 293 of
include/menuitem.inc  $0000000000435C6FCall trace for block
$00007FFFF7E5CAE0 size 80  $000000000079F95B  $0000000000649554 line
83 of include/menuitem.inc  $0000000000649EAA line 293 of
include/menuitem.inc  $0000000000435C6F  $0000000000467892 line 80 of
sched.pas  $0000000000649554 line 83 of include/menuitem.inc 
$0000000000649EAA line 293 of include/menuitem.inc 
$0000000000435C6FCall trace for block $00007FFFE17C4940 size 32 
$000000000079F95B  $0000000000649554 line 83 of include/menuitem.inc 
$0000000000649EAA line 293 of include/menuitem.inc  $0000000000435C6F 
$00000000007B66C5  $0000000000467892 line 80 of sched.pas 
$0000000000649554 line 83 of include/menuitem.inc  $0000000000649EAA
line 293 of include/menuitem.incCall trace for block $00007FFFF7E5C920
size 80  $000000000079F95B  $0000000000649554 line 83 of
include/menuitem.inc  $0000000000649EAA line 293 of
include/menuitem.inc  $0000000000435C6F  $0000000000467892 line 80 of
sched.pas  $0000000000649554 line 83 of include/menuitem.inc 
$0000000000649EAA line 293 of include/menuitem.inc 
$0000000000435C6FCall trace for block $00007FFFE17C4AC0 size 24 
$0000000000649554 line 83 of include/menuitem.inc  $0000000000649EAA
line 293 of include/menuitem.inc  $0000000000435C6F 
$00000000007B643E  $00000000007B66C5  $0000000000467892 line 80 of
sched.pas  $0000000000649554 line 83 of include/menuitem.inc 
$0000000000649EAA line 293 of include/menuitem.incCall trace for block
$00007FFFF7F6EFC0 size 88  $0000000000649554 line 83 of
include/menuitem.inc  $0000000000649EAA line 293 of
include/menuitem.inc  $0000000000435C6F  $00000000007B643E 
$00000000007B66C5  $0000000000467892 line 80 of sched.pas 
$0000000000649554 line 83 of include/menuitem.inc  $0000000000649EAA
line 293 of include/menuitem.incCall trace for block $00007FFFF7E09B80
size 32  $000000000049960F  $0000000000498C82  $000000000049842D 
$0000000000496F03  $000000000049842D  $00000000004977B9 
$000000000049787C  $0000000000499275Call trace for block
$00007FFFF7E09700 size 29  $000000000049960F  $0000000000498C82 
$000000000049842D  $0000000000496F03  $000000000049842D 
$00000000004977B9  $000000000049787C  $0000000000499275Call trace for
block $00007FFFF7E09C40 size 29  $000000000049960F  $0000000000498C82 
$000000000049842D  $0000000000496F03  $000000000049842D 
$00000000004977B9  $000000000049787C  $0000000000499275 
For completion, in case anyone asks, here are the lines in my code
which are being referenced by the stacetrace:

Line 71: FFileName:=InputBox('Fetch Schedule from Server...', 'Remote
Schedule to fetch?', FFileName);

Line 80: if not GetSchedule(FCSVStream, FFileName) then

Again, I want to reiterate that if I call the FetchScheduleClick
method from a button instead of my menu, the stacktrace no longer
references my code, and otherwise the stracktrace is the same:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Running
Destroy...Heap dump by heaptrc unit41039 memory blocks allocated :
2235068/233935241027 memory blocks freed     : 2234342/233860812
unfreed memory blocks : 726True heap size : 327680True free heap :
325056Should be : 325400Call trace for block $00007FFFE17C9C80 size
33  $0000000000612D22 line 2913 of include/control.inc 
$0000000000641ADA line 55 of include/buttoncontrol.inc 
$0000000000642323 line 169 of include/buttons.inc  $00000000006419C6
line 21 of include/buttoncontrol.inc  $0000000000435C6F 
$0000000000603F7D line 5419 of include/wincontrol.inc 
$000000000068F506 line 5747 of qt/qtwidgets.pas  $0000000000690419
line 6120 of qt/qtwidgets.pasCall trace for block $00007FFFF7E03080
size 27  $0000000000612D22 line 2913 of include/control.inc 
$0000000000641ADA line 55 of include/buttoncontrol.inc 
$0000000000642323 line 169 of include/buttons.inc  $00000000006419C6
line 21 of include/buttoncontrol.inc  $0000000000435C6F 
$0000000000603F7D line 5419 of include/wincontrol.inc 
$000000000068F506 line 5747 of qt/qtwidgets.pas  $0000000000690419
line 6120 of qt/qtwidgets.pasCall trace for block $00007FFFE17BA0C0
size 192  $0000000000612D22 line 2913 of include/control.inc 
$0000000000641ADA line 55 of include/buttoncontrol.inc 
$0000000000642323 line 169 of include/buttons.inc  $00000000006419C6
line 21 of include/buttoncontrol.inc  $0000000000435C6F 
$0000000000603F7D line 5419 of include/wincontrol.inc 
$000000000068F506 line 5747 of qt/qtwidgets.pas  $0000000000690419
line 6120 of qt/qtwidgets.pasCall trace for block $00007FFFF7E5C220
size 80  $000000000079F95B  $0000000000612D22 line 2913 of
include/control.inc  $0000000000641ADA line 55 of
include/buttoncontrol.inc  $0000000000642323 line 169 of
include/buttons.inc  $00000000006419C6 line 21 of
include/buttoncontrol.inc  $0000000000435C6F  $0000000000603F7D line
5419 of include/wincontrol.inc  $000000000068F506 line 5747 of
qt/qtwidgets.pasCall trace for block $00007FFFF7E5D100 size 80 
$000000000079F95B  $0000000000612D22 line 2913 of include/control.inc 
$0000000000641ADA line 55 of include/buttoncontrol.inc 
$0000000000642323 line 169 of include/buttons.inc  $00000000006419C6
line 21 of include/buttoncontrol.inc  $0000000000435C6F 
$0000000000603F7D line 5419 of include/wincontrol.inc 
$000000000068F506 line 5747 of qt/qtwidgets.pasCall trace for block
$00007FFFE17C4F40 size 32  $000000000079F95B  $0000000000612D22 line
2913 of include/control.inc  $0000000000641ADA line 55 of
include/buttoncontrol.inc  $0000000000642323 line 169 of
include/buttons.inc  $00000000006419C6 line 21 of
include/buttoncontrol.inc  $0000000000435C6F  $0000000000603F7D line
5419 of include/wincontrol.inc  $000000000068F506 line 5747 of
qt/qtwidgets.pasCall trace for block $00007FFFF7E5CF40 size 80 
$000000000079F95B  $0000000000612D22 line 2913 of include/control.inc 
$0000000000641ADA line 55 of include/buttoncontrol.inc 
$0000000000642323 line 169 of include/buttons.inc  $00000000006419C6
line 21 of include/buttoncontrol.inc  $0000000000435C6F 
$0000000000603F7D line 5419 of include/wincontrol.inc 
$000000000068F506 line 5747 of qt/qtwidgets.pasCall trace for block
$00007FFFE17C50C0 size 24  $0000000000612D22 line 2913 of
include/control.inc  $0000000000641ADA line 55 of
include/buttoncontrol.inc  $0000000000642323 line 169 of
include/buttons.inc  $00000000006419C6 line 21 of
include/buttoncontrol.inc  $0000000000435C6F  $0000000000603F7D line
5419 of include/wincontrol.inc  $000000000068F506 line 5747 of
qt/qtwidgets.pas  $0000000000690419 line 6120 of qt/qtwidgets.pasCall
trace for block $00007FFFF7F6F3C0 size 88  $0000000000612D22 line 2913
of include/control.inc  $0000000000641ADA line 55 of
include/buttoncontrol.inc  $0000000000642323 line 169 of
include/buttons.inc  $00000000006419C6 line 21 of
include/buttoncontrol.inc  $0000000000435C6F  $0000000000603F7D line
5419 of include/wincontrol.inc  $000000000068F506 line 5747 of
qt/qtwidgets.pas  $0000000000690419 line 6120 of qt/qtwidgets.pasCall
trace for block $00007FFFF7E09DC0 size 32  $000000000049960F 
$0000000000498C82  $000000000049842D  $0000000000496F03 
$000000000049842D  $00000000004977B9  $000000000049787C 
$0000000000499275Call trace for block $00007FFFF7E09C40 size 29 
$000000000049960F  $0000000000498C82  $000000000049842D 
$0000000000496F03  $000000000049842D  $00000000004977B9 
$000000000049787C  $0000000000499275Call trace for block
$00007FFFF7E09E80 size 29  $000000000049960F  $0000000000498C82 
$000000000049842D  $0000000000496F03  $000000000049842D 
$00000000004977B9  $000000000049787C  $0000000000499275 
Thank you so much for any help you can provide.  Oh, I am using FPC
3.0.4 with Lazarus 2.0.0.

----------------------------------------------------------------------
eljo · 2020-03-29 04:02

Could you please provide a simple (the smallest possible) sample that
shows the memory leak?
Your current trace logs show blocks in windget files but I see
nowhere  the csvdocument units mentioned which makes think that the
problem is not in the TCSVDataset at all.

To test for that create a simple console application create a
TCSVDataset dynamical through code load any data you have tested so
far and the destroy the TCSVDataset and exit to see the the trace log.

----------------------------------------------------------------------
avra · 2020-03-29 10:59

You could also try QT with trunk FPC/LAZ, try with something other
then QT, and report FPC and LAZ versions used in each try.

----------------------------------------------------------------------
kveroneau · 2020-03-29 20:33

Quote from: eljo on March 29, 2020, 06:02:22 amCould you please
provide a simple (the smallest possible) sample that shows the memory
leak?
Your current trace logs show blocks in windget files but I see
nowhere  the csvdocument units mentioned which makes think that the
problem is not in the TCSVDataset at all.

To test for that create a simple console application create a
TCSVDataset dynamical through code load any data you have tested so
far and the destroy the TCSVDataset and exit to see the the trace log.

I created a basic console application using TCustomApplication, I
assigned the Owner of the created TCSVDataset to the application
class.  I copied and partially modified both my GetSchedule and
FetchSchedule methods only removing the GUI dependent portions.  At
first I thought there was no memory leak, but then I realized that I
entered the wrong filename to fetch...  After using the correct
filename I did indeed see a similar exception.  I will post the entire
test program here minus any endpoints of course.

New stacktrace:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Remote
Schedule to fetch?  * CSV Data loaded successfully.Running
Destroy...Heap dump by heaptrc unit516 memory blocks allocated :
146833/149216513 memory blocks freed     : 146581/1489523 unfreed
memory blocks : 252True heap size : 196608True free heap :
195872Should be : 195960Call trace for block $00007FFFF7FEBD80 size
33  $0000000000400A4F line 90 of leaktest.lpr  $00000000004765E9 
$000000000047D0AE  $000000000047D335  $0000000000400804 line 62 of
leaktest.lpr  $0000000000400A4F line 90 of leaktest.lpr 
$00000000004765E9  $00000000004765E9Call trace for block
$00007FFFF7FEBCC0 size 27  $0000000000400A4F line 90 of leaktest.lpr 
$00000000004765E9  $000000000047C932  $000000000047D0AE 
$000000000047D335  $0000000000400804 line 62 of leaktest.lpr 
$0000000000400A4F line 90 of leaktest.lpr  $00000000004765E9Call trace
for block $00007FFFF7F6C220 size 192  $0000000000400A4F line 90 of
leaktest.lpr  $00000000004765E9 
Here is the code here, should be pretty simple to upload a CSV file
somewhere or spin up a local web server via python2 -m
SimpleHTTPServer to try the test on your own.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
leaktest; {$mode objfpc}{$H+} uses  {$IFDEF UNIX}{$IFDEF UseCThreads} 
cthreads,  {$ENDIF}{$ENDIF}  Classes, SysUtils, CustApp, csvdataset,
fphttpclient  { you can add units after this }; type   {
TMyApplication }   TMyApplication = class(TCustomApplication) 
private    CSVDataset: TCSVDataset;    FCSVStream: TMemoryStream;   
function GetSchedule(dst: TMemoryStream; fname: String): Boolean;   
procedure FetchSchedule;  protected    procedure DoRun; override; 
public    constructor Create(TheOwner: TComponent); override;   
destructor Destroy; override;    procedure WriteHelp; virtual;  end; {
TMyApplication } function TMyApplication.GetSchedule(dst:
TMemoryStream; fname: String): Boolean;begin  with
TFPHTTPClient.Create(Nil) do    Try      AddHeader('X-Filename',
fname);      Get('*Redacted*', dst);      if dst.Size = 3 then       
Result:=False      else        Result:=True;    finally      Free;   
end;end; procedure TMyApplication.FetchSchedule;var  fname:
string;begin  Write('Remote Schedule to fetch? ');  {ReadLn(fname);} 
fname:='*Redacted*';  if fname = '' then    Exit;  if
CSVDataset.Active then    CSVDataset.Close;  if FCSVStream <> Nil
then    FCSVStream.Clear  else    FCSVStream:=TMemoryStream.Create; 
if not GetSchedule(FCSVStream, fname) then  begin    WriteLn(' **
Unable to perform fetch!'); { Again, the memory does not occur if it
cannot fetch the CSV file. }    Exit;  end; 
CSVDataset.LoadFromCSVStream(FCSVStream);  WriteLn(' * CSV Data loaded
successfully.');end; procedure TMyApplication.DoRun;var  ErrorMsg:
String;begin  // quick check parameters  ErrorMsg:=CheckOptions('h',
'help');  if ErrorMsg<>'' then begin   
ShowException(Exception.Create(ErrorMsg));    Terminate;    Exit; 
end;   // parse parameters  if HasOption('h', 'help') then begin   
WriteHelp;    Terminate;    Exit;  end;   FetchSchedule;   // stop
program loop  Terminate;end; constructor
TMyApplication.Create(TheOwner: TComponent);begin  inherited
Create(TheOwner);  StopOnException:=True; 
CSVDataset:=TCSVDataset.Create(Self);end; destructor
TMyApplication.Destroy;begin  WriteLn('Running Destroy...');  if
CSVDataset.Active then    CSVDataset.Close;  if FCSVStream <> Nil
then    FCSVStream.Free;  inherited Destroy;end; procedure
TMyApplication.WriteHelp;begin  { add your help code here } 
writeln('Usage: ', ExeName, ' -h');end; var  Application:
TMyApplication;begin  Application:=TMyApplication.Create(nil); 
Application.Title:='My Application';  Application.Run; 
Application.Free;end. 
I will continue to investigate further myself and post an update here
if I find anything new out.  Thank you so far for your assistance.

UPDATE: So, I tried to compile my FCL with debugging enabled so I can
run my program and see additional output, and now when I run my
program I get a weird Runtime Error 201, which to my understanding is
a range check error.  However, it is occurring in the most oddest
place, within the ssockets unit in the following line:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
addr.sin_addr.s_addr := HostToNet(a.s_addr); 
I did what most might do and did a full Step through all the running
code, and in my Local Variables window, I do not see any issues with
any of the variables in this specific stack frame.  Really odd.  I'll
keep investigating.

UPDATE 2: So, this weird RunTime 201/Range Check Error only occurs if
I compile the packages directory in my source code with make debug to
compile the FCL with debugging enabled, not sure why, if I recompile
the same source after doing a clean using make clean && make, the
Runtime 201/Range Check Error disappears.  Is there a particular
reason why I cannot use the ssockets unit when built with DEBUG
enabled?

UPDATE 3: So, I managed to get this test project to compile with a
DEBUG enabled RTL, and now the stacktrace is pointing to something
related to an ansistring:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Remote
Schedule to fetch?  * CSV Data loaded successfully.Running
Destroy...Heap dump by heaptrc unit516 memory blocks allocated :
146833/149216513 memory blocks freed     : 146581/1489523 unfreed
memory blocks : 252True heap size : 196608True free heap :
195872Should be : 195960Call trace for block $00007FFFF7FEBD80 size
33  $000000000041DC82 line 342 of ../inc/heap.inc  $000000000040F8CF
line 787 of ../inc/astrings.inc  $000000000040E779 line 266 of
../inc/astrings.inc  $000000000047FC29 line 76 of
../objpas/sysutils/sysstr.inc  $00000000004DB5AD  $00000000004DB3B5 
$00000000004DBB66  $00000000004ACCF1Call trace for block
$00007FFFF7FEBCC0 size 27  $000000000041DA5E line 276 of
../inc/heap.inc  $000000000040E33E line 115 of ../inc/astrings.inc 
$000000000040F816 line 771 of ../inc/astrings.inc  $000000000040F00A
line 510 of ../inc/astrings.inc  $00000000004DAED1  $00000000004DB02B 
$00000000004DB6D2  $00000000004AD204Call trace for block
$00007FFFF7F6C220 size 192  $000000000041DA5E line 276 of
../inc/heap.inc  $000000000041724D line 355 of ../inc/objpas.inc 
$00000000004DB68B  $00000000004AD204  $00000000004D77AF 
$00000000004D0AFE  $00000000004DF471  $00000000004E0A37 
As noted, it is still showing the same sizes of memory being leaked. 
I am starting to make some slight progress in figuring out where this
leak is coming from.  I will try and press on and not lose faith.  I
am going to try and see if I can replicate without using any
Networking, so I can use a DEBUG FCL to get a full stracktrace for you
guys. 

UPDATE 4: And may I present a full stacktrace of my use and leaking of
TCSVDataset from FPC 3.0.4!  I will continue investigating as far as
my abilities go, but I made it this far, I'm sure I can figure out
where this leak is now. 

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Remote
Schedule to fetch?  * CSV Data loaded successfully.Running
Destroy...Heap dump by heaptrc unit261 memory blocks allocated :
354082/356968258 memory blocks freed     : 353058/3559363 unfreed
memory blocks : 1024True heap size : 491520True free heap :
489984Should be : 490104Call trace for block $00007FFFF7FBD5B0 size
805Found addr = $0000000000400000  $000000000041D822 line 342 of
../inc/heap.incFound addr = $0000000000400000  $000000000040F46F line
787 of ../inc/astrings.incFound addr = $0000000000400000 
$000000000040E319 line 266 of ../inc/astrings.incFound addr =
$0000000000400000  $000000000047F7C9 line 76 of
../objpas/sysutils/sysstr.incFound addr = $0000000000400000 
$00000000004DC1FD line 409 of fcl-base/src/csvreadwrite.ppFound addr =
$0000000000400000  $00000000004DBFE4 line 367 of fcl-
base/src/csvreadwrite.ppFound addr = $0000000000400000 
$00000000004DC968 line 515 of fcl-base/src/csvreadwrite.ppFound addr =
$0000000000400000  $00000000004AD9C1 line 134 of fcl-
db/src/base/csvdataset.ppCall trace for block $00007FFFF7FADE80 size
27Found addr = $0000000000400000  $000000000041D5FE line 276 of
../inc/heap.incFound addr = $0000000000400000  $000000000040DEDE line
115 of ../inc/astrings.incFound addr = $0000000000400000 
$000000000040F3B6 line 771 of ../inc/astrings.incFound addr =
$0000000000400000  $000000000040EBAA line 510 of
../inc/astrings.incFound addr = $0000000000400000  $00000000004DBA0C
line 282 of fcl-base/src/csvreadwrite.ppFound addr =
$0000000000400000  $00000000004DBB90 line 295 of fcl-
base/src/csvreadwrite.ppFound addr = $0000000000400000 
$00000000004DC347 line 421 of fcl-base/src/csvreadwrite.ppFound addr =
$0000000000400000  $00000000004AE028 line 185 of fcl-
db/src/base/csvdataset.ppCall trace for block $00007FFFF7F7B220 size
192Found addr = $0000000000400000  $000000000041D5FE line 276 of
../inc/heap.incFound addr = $0000000000400000  $0000000000416DED line
355 of ../inc/objpas.incFound addr = $0000000000400000 
$00000000004DC300 line 420 of fcl-base/src/csvreadwrite.ppFound addr =
$0000000000400000  $00000000004AE028 line 185 of fcl-
db/src/base/csvdataset.ppFound addr = $0000000000400000 
$00000000004D633F line 3250 of fcl-db/src/base/bufdataset.pasFound
addr = $0000000000400000  $00000000004CB7B9 line 1240 of fcl-
db/src/base/bufdataset.pasFound addr = $0000000000400000 
$00000000004E13EF line 405 of fcl-db/src/base/dataset.incFound addr =
$0000000000400000  $00000000004E36A5 line 960 of fcl-
db/src/base/dataset.inc 
So to my understanding on how to check which line is leaking is to
check the final file in each reported leak, correct?  So, there should
be 192 bytes being leaked from fcl-db/src/base/dataset.inc on line
number 960?

Update 5: It seems that the TCSVParser might not be getting freed
correctly, I can see line 185 in csvdataset being referenced multiple
times in the above heaptrc trace.  This is line 185 from my local
sources:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
FParser:=TCSVParser.Create; 
I am going to modify the TCSVDataPacketReader class to FreeAndNil
FParser... and that worked.  So, yeah, there is officially a memory
leak here in the csvdataset unit as I thought there was.  I am not
sure how to submit patches or official bug reports.  But here is what
I did in the TCSVDataPacketReader.Destroy destructor to resolve the
leak:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i destructor
TCSVDataPacketReader.Destroy;begin  If FOwnsOptions then   
FreeAndNil(FOPtions);  FreeAndNil(Fline);  if FParser <> Nil then   
FreeAndNil(FParser);  inherited Destroy;end; 
Hopefully all this troubleshoot is helpful to someone who might also
be facing an odd memory leak within the Free Component Library.  I'm
actually surprised nobody noticed this leak until now.  Perhaps I am
doing something wrong in my code to properly free FParser, if I am,
please let me know how I should be using this class to free the
parser.

FINAL UPDATE(I swear):  So, I was correct that there was a known
memory leak in TCSVDataset, well more like a memory leak in a class it
uses, the TCSVPacketReader class.  This leak which I mentioned here is
resolved in the latest release candidate 3.2.0 FPC version.  Here is
their updated code to resolve the leak:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i destructor
TCSVDataPacketReader.Destroy;begin  FreeAndNil(FCreateFieldDefs);  If
FOwnsOptions then    FreeAndNil(FOPtions);  FreeAndNil(Fline); 
FreeAndNil(FParser);  inherited Destroy;end; 
Wish I knew this was an issue and will be resolved in the next FPC
release, or I would not have wasted an entire afternoon
troubleshooting it...  At least I learned a lot more about the
internals of FPC, and how to build the RTL and FCL in debug modes.

----------------------------------------------------------------------
avra · 2020-03-30 09:56

I did not go through compilation and testing of your code, but at
first look I wonder if leak would disappear if you simply move
creation of FCSVStream into constructor.

----------------------------------------------------------------------
PascalDragon · 2020-03-30 10:04

Quote from: kveroneau on March 29, 2020, 10:33:42 pmWish I knew this
was an issue and will be resolved in the next FPC release, or I would
not have wasted an entire afternoon troubleshooting it...  At least I
learned a lot more about the internals of FPC, and how to build the
RTL and FCL in debug modes.

That's why we usually suggest to check against trunk when reproducing
a bug.  In this case the corresponding bug report would have been hard
to find as well cause it originally occurred together with fcl-report.

----------------------------------------------------------------------
kveroneau · 2020-03-30 17:03

Quote from: PascalDragon on March 30, 2020, 12:04:48 pmThat's why we
usually suggest to check against trunk when reproducing a bug.  In
this case the corresponding bug report would have been hard to find as
well cause it originally occurred together with fcl-report.

Just read the bug report, thank you.  Seems to have been reported back
in 2018.  I guess FPC does not release incremental patch versions? 
May I suggest separating out the versions of both the compiler/RTL and
the FCL into separately versioned and released components?  This will
allow bug fixes in the FCL to be released before a newer compiler/RTL
version.  Because, currently my options are to get around this bug, is
to either start using a potentially unstable version of FPC(3.2.0), or
to patch the bug myself in my local FCL.  I did patch the FCL myself
before when I noticed a bug in the Blowfish implementation.  The
encrypted data was not properly using the host processor's
Endianness.  I never opened a bug report for this, nor submitted my
patched unit file to a developer.  The patch is however available on
the forums, if you are curious on what I did, and if the issue has
been resolved.