TCSVDataset duplicates modified records on Post lazarus / 16-Databases ---------------------------------------------------------------------- Martin Lowry · 2022-09-26 14:25 Hi All, As the subject says   Somewhere in the call to Post the record is duplicated and thus two copies appear in the output file, the unmodified original AND a modified copy. TCSVDataset is a TBufDataset descendent so can't trace where it's happening as the write logic is quite complicated. Also the sources are part of FCL. EDIT: Actually Post does not write to file since the dataset components are disabled. Updates are postponed until Close is called, and then the modified records are duplicated. Can somebody suggest what I'm missing? Cheers ---------------------------------------------------------------------- egsuh · 2022-09-27 11:02 Please post some codes. ---------------------------------------------------------------------- Thaddy · 2022-09-27 12:34 Plz provide real code. Your edit is correct: TCsvDataSet, - most if not all datasets - only persist when closed or forced to update. ---------------------------------------------------------------------- Martin Lowry · 2022-09-27 16:30 Okay, I eventually traced the issue into a procedure within TBufDataset from which TCSVDataset descends. Here's the codestack: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i #0 GETDATASETPACKET(0x153bb10, 0xa925010) at ..\..\Extras\SQLITE\bufdataset.pas:3423#1 SAVETOSTREAM(0x153bb10, 0x14f3030, DFDEFAULT) at ..\..\Extras\SQLITE\bufdataset.pas:3490#2 SAVETOFILE(0x153bb10, 0x15031c8 'C:\Users\Martin\Documents'..., DFDEFAULT) at ..\..\Extras\SQLITE\bufdataset.pas:3522#3 DOBEFORECLOSE(0x153bb10) at ..\..\Extras\SQLITE\bufdataset.pas:1440#4 SETACTIVE(0x153bb10, false) at ..\..\Extras\SQLITE\dataset.inc:1102#5 CLOSE(0x153bb10) at ..\..\Extras\SQLITE\dataset.inc:1544#6 FORMCLOSE(0x1546a90, 0x1546a90, CAFREE) at main.pas:168  ... and the procedure extracted from bufdataset.pas. See my embedded comments marked "//ML". Commenting out line 76 gives my expected behaviour - only the modified data is stored. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TCustomBufDataset.GetDatasetPacket(AWriter: TDataPacketReader);   procedure StoreUpdateBuffer(AUpdBuffer : TRecUpdateBuffer; var ARowState: TRowState);  var AThisRowState : TRowState;      AStoreUpdBuf  : Integer;  begin    if AUpdBuffer.UpdateKind = ukModify then      begin      AThisRowState := [rsvOriginal];      ARowState:=[rsvUpdated];      end    else if AUpdBuffer.UpdateKind = ukDelete then      begin      AStoreUpdBuf:=FCurrentUpdateBuffer;    if GetRecordUpdateBuffer(AUpdBuffer.BookmarkData,True,False) then      repeat          if CurrentIndexBuf.SameBookmarks(@FUpdateBuffer[FCur rentUpdateBuffer].NextBookmarkData, @AUpdBuffer.BookmarkData) then          StoreUpdateBuffer(FUpdateBuffer[FCurrentUpdateBuffer], ARowState);        until not GetRecordUpdateBuffer(AUpdBuffer.BookmarkData,True,True);      FCurrentUpdateBuffer:=AStoreUpdBuf;      AThisRowState := [rsvDeleted];      end    else // ie: UpdateKind = ukInsert      ARowState := [rsvInserted];     FFilterBuffer:=AUpdBuffer.OldValuesBuffer;    // OldValuesBuffer is nil if the record is either inserted or inserted and then deleted    if assigned(FFilterBuffer) then      FDatasetReader.StoreRecord(AThisRowState,FCurrentUpdateBuffer);  end;   procedure HandleUpdateBuffersFromRecord(AFindNext : boolean; ARecBookmark : TBufBookmark; var ARowState: TRowState);  var StoreUpdBuf1,StoreUpdBuf2 : Integer;  begin    if not AFindNext then ARowState:=[];    if GetRecordUpdateBuffer(ARecBookmark,True,AFindNext) then      begin    if FUpdateBuffer[FCurrentUpdateBuffer].UpdateKind=ukDelete then      begin        StoreUpdBuf1:=FCurrentUpdateBuffer;        HandleUpdateBuffersFromRecord(True,ARecBookmark,ARowState);        StoreUpdBuf2:=FCurrentUpdateBuffer;        FCurrentUpdateBuffer:=StoreUpdBuf1;        StoreUpdateBuffer(FUpdateBuffer[StoreUpdBuf1], ARowState);        FCurrentUpdateBuffer:=StoreUpdBuf2;        end      else        begin        StoreUpdateBuffer(FUpdateBuffer[FCurrentUpdateBuffer], ARowState);        HandleUpdateBuffersFromRecord(True,ARecBookmark,ARowState);        end;      end  end; var ScrollResult   : TGetResult;    SavedState   : TDataSetState;    ABookMark      : PBufBookmark;    ATBookmark     : TBufBookmark;    RowState       : TRowState; begin  FDatasetReader := AWriter;  try    //  CheckActive;    ABookMark:=@ATBookmark;    FDatasetReader.StoreFieldDefs(FAutoIncValue);     SavedState:=SetTempState(dsFilter);    ScrollResult:=CurrentIndexBuf.ScrollFirst;    while ScrollResult=grOK do      begin      RowState:=[];      CurrentIndexBuf.StoreCurrentRecIntoBookmark(ABookmark);      // updates related to current record are stored first  // ML - This actually causes the ORIGINAL record to be saved  (see line 26-29 above)      HandleUpdateBuffersFromRecord(False,ABookmark^,RowState);      // now store current record  // ML - Now the modified data is stored      FFilterBuffer:=CurrentIndexBuf.CurrentBuffer;      if RowState=[] then        FDatasetReader.StoreRecord([])      else  // ML - TCSVDataPacketReader.StoreRecord ignores parameters  (see csvdataset.pp line 271)        FDatasetReader.StoreRecord(RowState,FCurrentUpdateBuffer);       ScrollResult:=CurrentIndexBuf.ScrollForward;      if ScrollResult<>grOK then        begin        if getnextpacket>0 then        ScrollResult := CurrentIndexBuf.ScrollForward;        end;      end;    // There could be an update buffer linked to the last (spare) record    CurrentIndexBuf.StoreSpareRecIntoBookmark(ABookmark);    HandleUpdateBuffersFromRecord(False,ABookmark^,RowState);     RestoreState(SavedState);     FDatasetReader.FinalizeStoreRecords;  finally    FDatasetReader := nil;  end;end;  Having discovered the source it makes me think the duplication was intended as there doesn't appear to be a work around without affecting other functionality. ---------------------------------------------------------------------- wp · 2022-09-27 16:56 Without seeing your code, I do not believe that there is such a bug in TBufDataset/TCSVDataset. Please run the attached demo project which I posted for some other thread. There are no duplicated records. Find out what you are doing differently. Modify the demo to show the bug and post it here again. ---------------------------------------------------------------------- Martin Lowry · 2022-09-27 17:39 Hi wp, I was able to reproduce the 'bug' by adding the following method to the form: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.FormClose(Sender: TObject; var CloseAction: TCloseAction);begin  CSVDataset1.Close;  CloseAction:=caFree;end;  I ran the test, modified 2 fields in record 2, and closed the form without doing anything further. The original file was rewritten and now has an extra record duplicating the one I modified. Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i FirstName,LastName,City,Address,BirthdateWalter,Mellon,Oklahoma City,"1261, Main Street",01/01/1980Mario,Speedwagon,Hollywood,"1500, Hollywood Blvd",17/12/1982Mario2,Speedwagon2,Hollywood,"1500, Hollywood Blvd",17/12/1982Anna,Mull,Los Angeles,"2202, Capitol Square",17/12/1982  This appears to happen because the update buffers are not cleared by calling MergeChangeLog as is done in TCustomCSVDataset.SaveToCSVStream. As a workaround the duplication can prevented by adding the MergeChangeLog call in a TCustomCSVDataset.OnBeforeClose event. ---------------------------------------------------------------------- wp · 2022-09-27 20:33 Now that I know what to do I can also reproduce it even without your change. Each time when the app is closed after a record has been edited and posted, the un-edited record remains in the database file. Condition is that the database file already exists when the application starts; the way my application is written this is true at the second start, or later. No idea, why this makes a difference... Replaced the TCSVDataset by a TBufDataset, but I do not see the issue here (attached as "bufdataset_duplicate_records.zip") Could you file a bug report? Add the test project, describe the issue as well as the steps to reproduce it. Mention the results of your own analysis of the issue. ---------------------------------------------------------------------- wp · 2022-09-27 20:49 Here is a simpler test project which shows the issue in a pure FPC program without user interaction: it creates the dataset with a few records, edits and posts one record, closes the dataset and reopens it - after reopening the recordcount has increased by one, and the original version of the edited record is still there. ---------------------------------------------------------------------- Martin Lowry · 2022-09-28 13:28 Bug report submitted. See https://gitlab.com/freepascal.org/fpc/source/-/issues/39925