Is TCSVDataSet fixed? lazarus / 63-Database ---------------------------------------------------------------------- egsuh · 2022-03-01 00:54 In the older versions, there were memory leak when I created and destroyed TCSVDataSet, which does not happen in the recent version. Has the bug been fixed? ---------------------------------------------------------------------- egsuh · 2022-03-01 03:47 No. I think this is not related with version. I found the memory blocks are not freed when I try to save it to stream.  Following code is to send CSVDataSet to web server. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function Taq_r_6.UploadSampleList(tpid: string; tsid: integer; filename:string   ): Boolean;var   AStrm: TStream;   AStrl : TStringList;   turl: string; begin   AStrl := TStringList.Create;   AStrm := TMemoryStream.Create;    CSVDSet.CSVOptions.FirstLineAsFieldNames:= True;   CSVDSet.LoadFromCSVFile(filename);   CSVDSet.SaveToCSVStream(AStrm);  // <== Memory leak is reported here.     turl:= URL + Format('SetDataSet?sid=%d', [tsid]);   Result := HttpPostFile(turl,'SampleList', tpid + '.csv', AStrm, AStrl);   AStrm.Position:= 0;   AStrl.Free;   AStrm.Free;end; But following codes, which use TBufDataSet instead of TCSVDataSet, does not cause memory leak. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function Taq_r_6.UploadQuestionnaire(tpid: string;DataSet:TDataSet): Boolean;var   turl: string;   AStrm: TStream;   AStrl : TStringList;begin   Result:= false;   DataSet.First;   AStrm := TMemoryStream.Create;   AStrl := TStringList.Create;   try      (DataSet as TBufDataSet).SaveToStream(AStrm);      turl := url + Format('SetDataSet?pid=%s', [tpid]);      Result := HttpPostFile(tURL, 'qdef', tpid + '.bds', AStrm, AStrl);      Result:= uppercase(trim(astrl.Text)) = uppercase(BoolToStr(true, true));   finally     AStrl.Free;     AStrm.Free;   end;end;  What could be possible reason? ---------------------------------------------------------------------- egsuh · 2022-03-01 03:56 Well, following gives me peace ^^. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function Taq_r_6.UploadSampleList(tpid: string; tsid: integer; filename:string   ): Boolean;var   AStrm: TMemoryStream;   AStrl : TStringList;   turl: string; begin   AStrl := TStringList.Create;   AStrm := TMemoryStream.Create;    AStrm.LoadFromFile(filename);    turl:= URL + Format('SetDataSet?sid=%d', [tsid]);   Result := HttpPostFile(turl,'SampleList', tpid + '.csv', AStrm, AStrl);                        // (URL, FieldName, FileName, Data, ResultData)   AStrm.Position:= 0;   AStrl.Free;   AStrm.Free;end; ---------------------------------------------------------------------- PascalDragon · 2022-03-01 08:02 Quote from: egsuh on March 01, 2022, 04:47:48 amNo. I think this is not related with version. I found the memory blocks are not freed when I try to save it to stream. Would you please provide a simple, self contained example that shows this leak and report it? ---------------------------------------------------------------------- Thaddy · 2022-03-01 08:36 The free's are called in the wrong order:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i // mind the creation order:   AStrl := TStringList.Create;   AStrm := TMemoryStream.Create; // and here you do the free in the wrong order   AStrl.Free;   AStrm.Free;// which should be in the correct order   AStrm.Free;   AStrl.Free;First create should be last free'd. IOW LIFO modeled, not FIFO Since the stream has a dependency on the list that may cause your problem. It may very well be that there is another bug elsewhere, but you should adhere to Last created should be first to free. Your code contains a programmer error in the above code. (By the way,) I would use two try finally's in the above code:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i    AStrl := TStringList.Create;  try    AStrm := TMemoryStream.Create;    try.....    finally      Astrm.Free;    end;  finally    Astrl.free;  end; ---------------------------------------------------------------------- egsuh · 2022-03-01 09:46 QuoteWould you please provide a simple, self contained example that shows this leak and report it? Please unzip the attached project, and run it. Following codes are all. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);var   AStream: TStream;begin   od1.filter := 'CSV files|*.csv';   if od1.execute then begin      CSVDataSet1.CSVOptions.FirstLineAsFieldNames:= True;      CSVDataSet1.LoadFromCSVFile(od1.filename);      CSVDataSEt1.Open;     AStream:= TMemoryStream.Create;      CSVDataSet1.SaveToCSVStream(AStream);      // CSVDataSet1.SaveToStream(AStream); // this works fine, but following file is not readable.       AStream.Position := 0;      (AStream as TMemoryStream).SaveToFile(ChangeFileExt(od1.filename, '_2.csv'));    AStream.Free;      ShowMessage('Done');   end;end; ---------------------------------------------------------------------- wp · 2022-03-01 10:30 Here is a simpler version of the test project. It confirms that there is a memory leak. It seems to be in the line with TCSVDataset.SaveToCSVFile - when this is removed the memory leak is gone. Please file a bug report in the FPC bug tracker and add my test program (along with a simple data file). Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program csvdataset_memleak; uses  db, csvdataset; var  dataset: TCSVDataset;begin  dataset := TCSVDataset.Create(nil);  try    dataset.CSVOptions.FirstLineAsFieldNames:= True;    dataset.LoadFromCSVFile('test.csv');    dataset.SaveToCSVFile('test-2.csv');    WriteLn('done');  finally    dataset.Free;  end;end. ---------------------------------------------------------------------- Thaddy · 2022-03-01 11:19 It seems that the underlying auto-created stream (for SaveToFile) in TCSVDataPacketReader is not free'd. ---------------------------------------------------------------------- wp · 2022-03-01 11:42 Quote from: Thaddy on March 01, 2022, 12:19:26 pmIt seems that the underlying auto-created stream (for SaveToFile) in TCSVDataPacketReader is not free'd. No. It is the internal TCSVBuilder created by TCSVDataPacketReader.StoreFieldDefs. Adding it to TCSVDatapackageReader.Destroy fixes the memory leak: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i destructor TCSVDataPacketReader.Destroy;begin  FreeAndNil(FBuilder);  FreeAndNil(FCreateFieldDefs);  If FOwnsOptions then    FreeAndNil(FOPtions);  FreeAndNil(Fline);  FreeAndNil(FParser);  inherited Destroy;end; I'll submit this as a patch. [EDIT] Done: https://gitlab.com/freepascal.org/fpc/source/-/issues/39607 ---------------------------------------------------------------------- PascalDragon · 2022-03-01 12:39 Quote from: wp on March 01, 2022, 12:42:01 pmDone: https://gitlab.com/freepascal.org/fpc/source/-/issues/39607 Thank you. ---------------------------------------------------------------------- wp · 2022-03-01 12:40 Michael already fixed it. ---------------------------------------------------------------------- PascalDragon · 2022-03-01 12:41 Quote from: wp on March 01, 2022, 01:40:28 pmMichael already fixed it. Even better. ---------------------------------------------------------------------- egsuh · 2022-03-02 02:52 Is there anything more than adding following line to csvdataset.pp file to fix this? FreeAndNil(FBuilder); I opened csvdataset.pp file, added the above line (and commented out sqldb from uses clause), and built file and then compiled and ran my previous application but the problem still exists. ---------------------------------------------------------------------- dsiders · 2022-03-02 03:20 Quote from: egsuh on March 02, 2022, 03:52:38 amIs there anything more than adding following line to csvdataset.pp file to fix this? FreeAndNil(FBuilder); I opened csvdataset.pp file, added the above line (and commented out sqldb from uses clause), and built file and then compiled and ran my previous application but the problem still exists. There were 2 commits: https://gitlab.com/freepascal.org/fpc/source/- /commit/21326cfb9ab8b66889c9620ea99a14f435b58d5d https://gitlab.com/freepascal.org/fpc/source/- /commit/56acf11ec227af677531fc5950a7a28420746467 ---------------------------------------------------------------------- PascalDragon · 2022-03-02 08:11 Quote from: egsuh on March 02, 2022, 03:52:38 amI opened csvdataset.pp file, added the above line (and commented out sqldb from uses clause), and built file and then compiled and ran my previous application but the problem still exists. How did you build the file? Did you copy it to your own project or did you make sure that the PPU that is provided either with Lazarus (on Windows) or with the FPC installer was updated? ---------------------------------------------------------------------- egsuh · 2022-03-02 08:31 QuoteHow did you build the file? Did you copy it to your own project or did you make sure that the PPU that is provided either with Lazarus (on Windows) or with the FPC installer was updated? I installed Lazarus 2.2.0 / FPC 3.2.2, Windows 11 64 bit. I opened csvdataset.pp file in Lazarus IDE, and ran Run>Build File menu. Now csvdataset.ppu file is more recent than csvdataset.pp file. But the problem still exists --- memory leak is reported at lresources.pp file. Lines 4274,3153, 3180, if exclude control related messages. ---------------------------------------------------------------------- egsuh · 2022-03-02 08:35 Please note that I'm using UTF-8 (without BOM) encoded file. ---------------------------------------------------------------------- wp · 2022-03-02 09:51 Quote from: dsiders on March 02, 2022, 04:20:01 amI opened csvdataset.pp file, added the above line (and commented out sqldb from uses clause), and built file and then compiled and ran my previous application but the problem still exists. The csvdataset unit belongs to the FCL, and thus you must rebuild the entire FPC to activate this change. Since this is a bit more tricky than rebuilding the Lazarus IDE, I'd recommend that you simply copy the file csvdataset.pp into your project folder. Later, when a new version with the modified csvdataset has been released you can remove the local copy of the unit again. ---------------------------------------------------------------------- egsuh · 2022-03-02 10:06 QuoteI'd recommend that you simply copy the file csvdataset.pp into your project folder. Okay. This works. Thanks a lot !!! ---------------------------------------------------------------------- egsuh · 2025-10-01 05:13 This has not been finally fixed yet. ---------------------------------------------------------------------- Thaddy · 2025-10-01 06:53 Do you expect fixes for fpc 3.2.2? These fixes are in 3.2.3 and 3.2.4rc or in trunk.