[Solved] TMemDataSet, TBufDataSet file saving problem
lazarus / 16-Databases

----------------------------------------------------------------------
egsuh · 2017-03-29 15:33

Hi,

I defined a Memo / Blob type field in TMemDataSet and TBufDataSet, and
stored data in following ways. The data to be stored are long text
file, e.g. TStrings.Text.

1)     mds1.fieldbyname('memofield').AsString := StringtoStore;
2)     mds1.fieldbyname('memofield').Value := StringtoStore;
3)     (mds1.fieldbyname('memofield') as TBlobField).LoadfromStream
(AStream);


All of the above three methods works fine with both mds and bds. I can
see the content of the field as I entered. That is, data is stored
correctly. 

Problem is that once I save it and load later, the content of
memo/blob fields are not contained.

I tried both "Savetofile" + "Loadfromfile", and "savetostream" +
"loadfromstream".

I think I had no problem in saving and loading the database file as
other fields work fine, only Blob / Memo type fields data are lost ---
not sure which one of storing or retrieving is not working correctly.

Can anyone help me?

----------------------------------------------------------------------
wp · 2017-03-29 16:38

Sorry - I am not going to answer your question, but I want to put your
attention to another issue with TBufDataset which I learned recently:
If you save a few thousand TBufDatset records using its SaveToFile or
SaveToStream methods the process gets extremely slow. To avoid this
you must call MergeChangeLog always after posting a few hundred
records and/or before the call to SaveToFile/Stream. In the following
demo, comment the line "TBufDataset(FExportDataset).MergeChangeLog" to
see the difference:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
project1;  {$mode objfpc}{$H+}  {.$DEFINE USE_MEM_DATASET}  uses  
SysUtils, classes, db, memds, bufdataset;  const   TABLENAME =
'people'; //name for the database table, extension will be added  
DATADIR = 'data'; //subdirectory where database is stored  const  
NUM_RECORDS = 5000;  var   FExportDataset: TDataset;  procedure
CreateDatabase; var   i: Integer;   fn: String;   stream:
TMemoryStream;   t: TDateTime; begin   ForceDirectories(DATADIR);   
fn := DATADIR + DirectorySeparator + TABLENAME + '.db';  
DeleteFile(fn);    {$IFDEF USE_MEM_DATASET}   FExportDataset :=
TMemDataset.Create(nil);   {$ELSE}   FExportDataset :=
TBufDataset.Create(nil);   {$ENDIF}   
FExportDataset.FieldDefs.Add('Last name', ftString, 15);  
FExportDataset.FieldDefs.Add('First name', ftString, 10);  
FExportDataset.FieldDefs.Add('City', ftString, 15);  
FExportDataset.FieldDefs.Add('Birthday', ftDate);  
FExportDataset.FieldDefs.Add('Salary', ftCurrency);  
FExportDataset.FieldDefs.Add('Work begin', ftDateTime);  
FExportDataset.FieldDefs.Add('Work end', ftDateTime);  
FExportDataset.FieldDefs.Add('Size', ftFloat);   {$IFNDEF
USE_MEM_DATASET}   TBufDataset(FExportDataset).CreateDataset;  
{$ENDIF}    FExportDataset.Open;    // Random data   for i:=1 to
NUM_RECORDS do begin     if (i mod 100 = 0) then begin     
WriteLn(Format('Adding record %d...', [i]));       {$IFNDEF
USE_MEM_DATASET}      TBufDataset(FExportDataset).MergeChangeLog;    
{$ENDIF}    end;    FExportDataset.Insert;     // just dummy data... 
 FExportDataset.FieldByName('Last name').AsString := 'A';    
FExportDataset.FieldByName('First name').AsString := 'B';    
FExportDataset.FieldByName('City').AsString := 'C';    
FExportDataset.FieldByName('Birthday').AsDateTime := 0;    
FExportDataset.FieldByName('Salary').AsFloat := 0;    
FExportDataset.FieldByName('Size').AsFloat := 0;    
FExportDataSet.FieldByName('Work begin').AsDateTime := 0;    
FExportDataSet.FieldByName('Work end').AsDateTime := 0;    
FExportDataset.Post;   end;    WriteLn('Saving...');   t := now;  
stream := TMemoryStream.Create;   try     {$IFDEF USE_MEM_DATASET}   
TMemDataset(FExportDataset).SaveToStream(stream);     {$ELSE}    
TBufDataset(FExportDataset).SaveToStream(stream);     {$ENDIF}    
stream.Position := 0;     WriteLn('Written to memory stream: ',
FormatFloat('0.000 s', (now - t) * SECONDS_PER_DAY));    
stream.SaveToFile(fn);   finally     stream.Free;   end;  
Writeln('Done. Total time needed for saving: ', FormatFloat('0.000 s',
(now - t) * SECONDS_PER_DAY));    FExportDataset.Close;   
writeLn(Format('Created file "%s" in folder "data".', [    
ExtractFileName(fn), ExtractFileDir(fn)   ]));   FExportDataset.Free;
end;  begin   CreateDatabase;    WriteLn;   WriteLn('Press ENTER to
close.');   ReadLn; end.

----------------------------------------------------------------------
egsuh · 2017-03-29 16:50

Thank you wp,

This is much help, even though I would not make 5,000 record tables.
Anyway does TMemDataSet show no difference in saving time?

Regards,

----------------------------------------------------------------------
wp · 2017-03-29 16:54

Sorry, I forgot this: TMemDataset does not have this issue. It saves
immediately fast without any further calls. (You can activate the
devine USE_MEM_DATASET in the demo to work with TMemDataset instead of
TBufDataset).

----------------------------------------------------------------------
LacaK · 2017-03-31 17:37

Quote from: egsuh on March 29, 2017, 05:33:07 pmProblem is that once I
save it and load later, the content of memo/blob fields are not
contained.

I tried both "Savetofile" + "Loadfromfile", and "savetostream" +
"loadfromstream".

I think I had no problem in saving and loading the database file as
other fields work fine, only Blob / Memo type fields data are lost ---
not sure which one of storing or retrieving is not working correctly.

TMemDataSet is lightweight in-memory dataset and saving/loading BLOB
field is not implemented there.
Use TBufDataset.

----------------------------------------------------------------------
egsuh · 2017-04-01 14:52

Quote from: LacaK on March 31, 2017, 07:37:03 pmTMemDataSet is
lightweight in-memory dataset and saving/loading BLOB field is not
implemented there.
Use TBufDataset.

Thank you. I'll try it again, but TBufDataSet has the same problem. 
SQLite + sqlite3dataset seems good enough as alternative.

----------------------------------------------------------------------
egsuh · 2017-04-01 15:15

LacaK,

I did a quick test and TBufDataSet seems to work fine.
I must have mistaken something in previous tests.

Thank you very much.