[SOLVED] Emptying a BufDataset
lazarus / 16-Databases

----------------------------------------------------------------------
iru · 2014-04-24 04:45

Gentlefolk,

Somewhere I saw an explanation how to empty all data out of a
tBufDataset. I can no longer find it.

It was something along the lines of:

Save off the tFieldDefs.
Somehow empty/clear the data.
Restore the tFieldDefs.

Can anyone explain how to empty a tBufDataset or point me to the
explanation?

Thanks, Ian

----------------------------------------------------------------------
mangakissa · 2014-04-24 07:41

TBufdataset.close

----------------------------------------------------------------------
janasoft · 2014-04-24 07:44

Perhaps this is what you are looking for:

http://wiki.lazarus.freepascal.org/How_to_write_in-memory_database_app
lications_in_Lazarus/FPC#Saving_MemDatasets_to_persistent_files

There, you can find the method EmptyMemDataSet(DataSet:TMemDataSet);

----------------------------------------------------------------------
iru · 2014-04-24 12:23

Gentlefolk,

Thank you for the responses.

Janasoft. In the link you provided I see something using tMemdataset
which looks like "save the Fielddefs, drop the data, restore the
Fielddefs", I will persue this on tBufDataset.

Mangakissa. This does not work. I have 4000+ records in the
BufDataset. Close and then open the BufDataset and there are still
4000+ records.
I suppose I could Free/Destroy the BufDataset and then create it again
but this seems a touch ugly.

Ian

----------------------------------------------------------------------
taazz · 2014-04-25 05:15

How about Close and then CreateDataset instead of Open?

----------------------------------------------------------------------
mangakissa · 2014-04-25 07:39

Quote from: iruMangakissa. This does not work. I have 4000+ records in
the BufDataset. Close and then open the BufDataset and there are still
4000+ records.That's strange. This example show exactly the behavior
of TBufdataset. Or you do something else to fill the dataset.

----------------------------------------------------------------------
iru · 2014-05-04 00:31

Mangakissa,

Apologies for the delay in responding but other aspects of life get in
the way.....

You are correct. I  miss-read/understood what I was looking at. A
close does clear the count.Thank you.

However the original problem (and why I miss understood things) is
still there.

I have the following code

 Code: [Select]Data.Active := true;
 Databuf.Edit;
 for ctr := 0 to <input-record-count> do
 Begin
      Databuf.append;
      <move data from input-data[ctr] to Databuf record
fields>.AsString
     <Log time, ctr, etc>
  end;
Databuf.Close;

If I execute the above code the log time stamps show that the loading
of 4497 records takes 1-2 seconds. The following  log entries show the
first three records with a time per record of 0-2 millie seconds per
record.

Code: [Select]10:13:29:847 AthsDataModule.Loop: 1 0 202549 0
10:13:29:849 AthsDataModule.Loop: 1 1 157224 1
10:13:29:850 AthsDataModule.Loop: 1 2 202501 2

Similar times per record  at the end of the input.

Code: [Select]10:13:30:718 AthsDataModule.Loop: 1 4494 201967 4494
10:13:30:718 AthsDataModule.Loop: 1 4495 3965 4495
10:13:30:718 AthsDataModule.Loop: 1 4496 203608 4496

If I execute the code a second time  things are different. Note the
times!

Code: [Select]10:14:20:746 AthsDataModule.Loop: 2 0 202549 0
10:14:30:551 AthsDataModule.Loop: 2 1 157224 1
10:14:31:257 AthsDataModule.Loop: 2 2 202501 2
Ten seconds for the first record, about 700 for the second

Things are consistently slow as the record count increases with each
record taking around 65 millie seconds compared to around 1 for the
first load.

Code: [Select]10:18:57:970 AthsDataModule.Loop: 2 4490 200786 4490
10:18:58:035 AthsDataModule.Loop: 2 4491 203276 4491
10:18:58:103 AthsDataModule.Loop: 2 4492 203055 4492
10:18:58:169 AthsDataModule.Loop: 2 4493 203599 4493
10:18:58:234 AthsDataModule.Loop: 2 4494 201967 4494
10:18:58:300 AthsDataModule.Loop: 2 4495 3965 4495
10:18:58:361 AthsDataModule.Loop: 2 4496 203608 4496
Over 4 minutes to process 4497 records. So where is all the time
going? What resource is not available? Memory?

I will create a test project and see if I can re-create the problem in
that, the current project is to large/complex to upload.

Any suggestions, help appreciated?

Ian

----------------------------------------------------------------------
iru · 2014-05-05 13:01

Gentlefolk,

I have tried to create a simplified version of my program which could
be uploaded and tested without special code.

Used a simple text file and a TimeToStrO(now) for logging.
Used string constants to load fields rather than data from a .CSV
file.

Checking the results in the logfile "BufLogFile.txt" I HAVE NOT
reproduced my problem. Times on the Try-1 and Try-2  Open-load-close
Bufdataset are similar.

No great delays on the second Bufdataset load.

I chase on..... Ian

----------------------------------------------------------------------
iru · 2014-05-06 06:43

Gentlefolk,

I am not having any success with the test  program, speeds are good.
Disconnecting the DBgrid from the Datasource improved things a lot.

I have turned back to the original program and placed additional log
entries througout the main loop that loads the BufDataset.

The long delays are in caused by the  BufDataset.APPEND.

More  chasing, Ian.

----------------------------------------------------------------------
cdbc · 2014-05-06 06:59

Hmmmm, thinking beginUpdate -> EndUpdate on dataset and definitely NO
visual controls being updated simultanously 

Regards Benny

----------------------------------------------------------------------
LacaK · 2014-05-06 13:30

Quote from: iru on May 06, 2014, 08:43:00 amThe long delays are in
caused by the  BufDataset.APPEND.
do you have any Indexes defined on TBufDataset ?
Do you mean only Append ? or Append followed by Post ?

----------------------------------------------------------------------
iru · 2014-05-07 07:42

Gentlefolk,

I organised things in the IDE/Project so that I could update and
compile units DB.pas,Dataset.inc and BufDataset.pas.

I progressively added time stamped logging at selected locations. I
could then compare the flow and timing of the first pass through the
code  with the second pass.

Wherever I found a delay in the second pass I would add more logging
points. The problem was found in the following BufDataset.pas code:

Code: [Select]function TCustomBufDataset.GetRecord(Buffer:
TRecordBuffer; GetMode: TGetMode; DoCheck: Boolean): TGetResult;

var Acceptable : Boolean;
    SaveState : TDataSetState;

begin
  Result := grOK;
  Logger.Send('BufDataset.pas: GetRecord');
  with FCurrentIndex do
    begin
    repeat
    Acceptable := True;
    Logger.Send('BufDataset.pas: GetRecord.Case GetMode=?');
    case GetMode of
      gmPrior : Result := ScrollBackward;
      gmCurrent : Result := GetCurrent;
      gmNext : begin
               if not CanScrollForward and (getnextpacket = 0) then
result := grEOF
               else
                 begin
                 result := grOK;
                  Logger.Send('BufDataset.pas:
GetRecord.DoScrollForward');
                 DoScrollForward;
                 end;
               end;
    end;

    if Result = grOK then
      begin
      CurrentRecordToBuffer(buffer);

      if Filtered then
        begin
        FFilterBuffer := Buffer;
        SaveState := SetTempState(dsFilter);
         Logger.Send('BufDataset.pas: GetRecord.DoFilterRecord');
        DoFilterRecord(Acceptable);                                  
     <<<<<<<<<<<<<<<<<<<<<<<<<<<<< Here
        if (GetMode = gmCurrent) and not Acceptable then
          begin
          Acceptable := True;
          Logger.Send('BufDataset.pas: GetRecord.Set acceptable =
true');
          Result := grError;
          end;
        RestoreState(SaveState);
        end;
        Logger.Send('BufDataset.pas: GetRecord.Between ends');
      end
    else if (Result = grError) and doCheck then
      DatabaseError('No record');
    Logger.Send('BufDataset.pas: until Acceptable');
    until Acceptable;    
end;

Woops, 'DoFilterRecord', what filter?????

The problem is me!

During the data load to the BufDataset the input is checked for errors
and a flag is set in an errored record.
At the end of the load if there are any errors a small form with a
grid displays the errors. The user can then cancel the load and
discard the input or accept the input. The 'error' form is closed.

A filter is applied to the BufDataset so only the errored records are
displayed.

I forgot to set BufDataset.Filtered := False when closing the error
display form. Therefore on the second load a filter was doing
something while records were being appended to the BufDatset.

Turn the filter off and the second load is as fast as the first load.

So, learnt some things, now to tidy up and get on with the
application......

Ian