tbufdataset.CopyfromDataset - strange behaviour in Lazarus trunk version lazarus / 63-Database ---------------------------------------------------------------------- kjteng · 2021-11-01 09:07 I noticed that some my code could not copy data properly using copyfromdataset method. The same code works perfectly if I compile it using the stable version of Lazarus but not in the trunk version.  Please see attached pix and sample code. What are the new changes/requirement for tBufdataset? ---------------------------------------------------------------------- wp · 2021-11-01 09:13 Quote from: kjteng on November 01, 2021, 10:07:58 amPlease see attached pix and sample code. Sample code seems to be missing. OK - you added it. Since the code of TBufDataset belongs to the FCL it is probably the FPC version which makes the difference. Please specify your Lazarus and, most of all, FPC versions. I do not understand the screenshots. What is their difference? At first trying your project with Laz/main+fpc3.2.2 and with Laz 2.0.12/fpc3.2 I saw significant differences between the left and right dbgrids, just like in your second screenshot. But then I noticed that the left grid, upon copying the data, has scrolled to the end while the right grid is at the top of the data. Bringing them in sync removes the differences. ---------------------------------------------------------------------- kjteng · 2021-11-02 01:18 Sorry for not being clear in my previous post. The first pix (screenshot (27)) is the result from the trunk version whereas the other pix is from the stable version. In the trunk version, 'PART' column does not display the correct numbers. For example, the original part number was 0 (in the querydataset and table), but shown as -72057594037927934 in the bufdataset. I am using the following:    Trunk version2.3.0 date 2021-10-30, FPC ver 3.3.1    Stable version2.0.12 date 2021-02-21,  FPC ver 3.2.0, SVN 64642 ---------------------------------------------------------------------- wp · 2021-11-02 08:53 Moving to FPC 3.3.1 I see the issue now, and you should report the issue. But the sample project is still too complex for a report, it should be simplified: Remove the dependence on ZEOS, or maybe even move to a simple memory-based dataset as data source. Ideally even the dependence on Lazarus should be removed. Normally, when doing so, additional information about the bug is revealed. ---------------------------------------------------------------------- wp · 2021-11-02 09:07 I now replaced ZEOS by TSQLite3Connection/TSQLTransaction/TSQLQuery which come with Lazarus, and - voila - the error is gone... Therefore I'd say that the issue is not an issue of TBufDataset.CopyFromDataset, but an issue somewhere in ZEOS. Please report it to the ZEOS forum. ---------------------------------------------------------------------- kjteng · 2021-11-03 09:21 Thank you for your reply. Your answer led me to further test using other memorydataset. The results are  as follows: 1. zmQueryDataSet (a descendant of TBufDataset) - same issue 2. rxmemdataset - No issue 3. zMemTable (new component in the zeoslib) - No issue. Based on the above, I think the problem comes from TBufDataset. Is anyone maintaining this component (to whom we can highlight this issue)  currently? ---------------------------------------------------------------------- wp · 2021-11-03 22:52 Quote from: kjteng on November 03, 2021, 10:21:51 amIs anyone maintaining this component (to whom we can highlight this issue)  currently? Most of the database-related bugs are handled by Michel Van Canneyt. But I am rather sure that he will not feel responsible for the issue when it can only be demonstated on the basis of a ZEOS database... I tried to further simplify the problem and modified your demo so that it creates a simple ZEOS-DB with a single ftLargeInt column. And this one cannot be copied to a TBufDataset either, the field does appear but is empty; and when the program ends, there is a memory access error. These are indications that the buffers are not set up correctly in the BufDataset. Two observations: (1) When I replace the ZQuery by a TMemoryDataset the issue is gone. (2) My program can use the TBufDataset of FPC 3.2.2 or FPC 3.3.1 (I copied the sources to subdirs of the project). Both versions show the same error with FPC 3.3.1. But, when I compile with FPC 3.2 the error does not appear, neither with the BufDataset of 3.3.1 nor with 3.2.2. This seems to indicate that the issue does not originate in the TBufdataset sources, but somewhere else, probably in unit db or related, or even completely somewhere else in FPC. ---------------------------------------------------------------------- wp · 2021-11-04 19:08 I think I found the issue. It's the FieldNo property. It seems to be that ZEOS is counting the FieldNo starting at 0, but LCL documentation says that it is a 1-based property (https://www.freepascal.org/docs- html/fcl/db/tfield.fieldno.html), and in fact, there are lots of lines in the FPC database code in which Field.FieldNo-1 is calculated. Please copy the following code into your test program and replace the  bf1.CopyFromDataset(ZReadOnlyQuery1,true) by CopyFromDataset(ZReadOnlyQuery1, bf1, true). This modified code detects whether the FieldNo is 0-based and increments it in this case (the changed lines are highlighted). Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure CopyFromDataset(ASource, ADest: TDataSet; CopyData: Boolean);const  UseStreams = ftBlobTypes;var  I: Integer;  F,F1,F2: TField;  L1,L2: TList;  N: String;  OriginalPosition: TBookMark;  S: TMemoryStream;  minFieldNo: Integer;  fieldNo: Integer;begin  ADest.Close;  ADest.Fields.Clear;  ADest.FieldDefs.Clear;   minFieldNo := MaxInt;  for I:=0 to ASource.FieldCount-1 do  begin    F := ASource.Fields[i];    if F.FieldNo < minFieldNo then      minFieldNo := F.FieldNo;  end;   for I:=0 to ASource.FieldCount-1 do  begin    F := ASource.Fields[I];    fieldNo := F.FieldNo;    if minFieldNo = 0 then inc(fieldno);    TFieldDef.Create(ADest.FieldDefs, F.FieldName, F.DataType, F.Size, F.Required, fieldno);  end;  if (ADest is TBufDataset) then    TBufDataset(ADest).CreateDataset  else    raise Exception.Create('Destination dataset type not supported.');  L1 := nil;  L2 := nil;  S := nil;  if CopyData then    try      L1:=TList.Create;      L2:=TList.Create;      ADest.Open;      for I:=0 to ADest.FieldDefs.Count-1 do      begin        N := ADest.FieldDefs[I].Name;        F1 := ADest.FieldByName(N);        F2 := ASource.FieldByName(N);        L1.Add(F1);        L2.Add(F2);      If (ADest.FieldDefs[I].DataType in UseStreams) and (S=Nil) then        S := TMemoryStream.Create;      end;      OriginalPosition := ASource.GetBookmark;      try        ASource.Open;        ASource.First;        while not ASource.EOF do        begin          ADest.Append;          for I:=0 to L1.Count-1 do          begin          F1 := TField(L1[i]);            F2 := TField(L2[I]);            if not F2.IsNull then              case F1.DataType of                ftFixedChar,                ftString   : F1.AsString:=F2.AsString;              ftFixedWideChar,                ftWideString : F1.AsWideString:=F2.AsWideString;                ftBoolean  : F1.AsBoolean:=F2.AsBoolean;                ftFloat    : F1.AsFloat:=F2.AsFloat;                ftAutoInc,                ftSmallInt,                ftInteger  : F1.AsInteger:=F2.AsInteger;              ftLargeInt : F1.AsLargeInt:=F2.AsLargeInt;                ftDate     : F1.AsDateTime:=F2.AsDateTime;                ftTime     : F1.AsDateTime:=F2.AsDateTime;                ftTimestamp,              ftDateTime : F1.AsDateTime:=F2.AsDateTime;                ftCurrency : F1.AsCurrency:=F2.AsCurrency;                ftBCD,                ftFmtBCD   : F1.AsBCD:=F2.AsBCD;              else                if (F1.DataType in UseStreams) then                begin                S.Clear;                  TBlobField(F2).SaveToStream(S);                S.Position := 0;                  TBlobField(F1).LoadFromStream(S);                end                else                  F1.AsString := F2.AsString;              end;        end;          try            ADest.Post;          except          ADest.Cancel;            raise;          end;          ASource.Next;        end;      finally        ASource.GotoBookmark(OriginalPosition); //Return to original record    end;    finally      L2.Free;      l1.Free;      S.Free;    end;end; After fixing the FieldNo this way the ZEOS file can be copied successfully. ---------------------------------------------------------------------- wp · 2021-11-04 22:44 Reading some comments in the ZEOS code I found that they intentionally switch from 1-based to 0-based indexes. I reported this thread to the ZEOS forum as an example that this change still has issues (https://sourceforge.net/p/zeoslib/tickets/539/). ---------------------------------------------------------------------- kjteng · 2021-11-08 04:04 wp, Thank you for the fix. Now it copydata correctly. I have removed the line 'if CopyData then...' because it wont compile with this line. What would be the effect without this line? ---------------------------------------------------------------------- wp · 2021-11-08 08:58 What's wrong with it? It compiles here... Anyway, when "if copyData" is removed you have the same effect as if CopyData were true, i.e. you will copy the fielddefs of the dataset as well as the data. If CopyData were false you would only copy the the fielddefs and have an empty table. ---------------------------------------------------------------------- kjteng · 2021-11-08 11:50 Ok ok. I didn't notice that "copyfromdataset" is a new procedure with parameter. I just copied the body to an event handler. My bad.