TStringList Slow
lazarus / 21-General

----------------------------------------------------------------------
dbannon · 2021-07-01 01:26

I have some code that takes user text and sprinkles some XML tags were
appropriate and saves to disk. Works OK.  But I decided I'd like to
tidy up the XML and already having a unit capable of doing that, would
like to be able to re-use it.

But, my existing code that applies the xml tags writes to  a memory
stream and then saves that stream to disk. My "pretty up" code takes a
stringlist. So, easy to convert the first unit to write to a
stringlist instead of (in fact, as well as) a memory stream.

Before I went any further, thought I'd do some performance checks, at
least some of this is happening while a user is typing and I have some
fast typists!

I found that converting my text to XML in a memory stream takes about
the same time as putting it into a stringlist. Thats good. But saving
the content was a whole different story -

Saving MemoryStream - 1 mS
StringList.SaveToFile    - 20 mS
StringList.Text writeln(TextFile) 5mS

The third data point above involves opening a TextFile and writing
stringlist.text directly to it. That is substantially faster than the
StringList's built in model but still slower than the Stream.

I am surprised at the difference to be honest !  Anyone have any
thoughts ?

Davo

EDIT : the numbers above were averages of five runs, in some cases a
bit noisy but always about the same ratio between the different
stringlist save models.  Content was about 100K charcters and run on
Linux, FPC 3.2.0, Lazarus Trunk.

----------------------------------------------------------------------
440bx · 2021-07-01 02:52

Quote from: dbannon on July 01, 2021, 03:26:48 amAnyone have any
thoughts ?
This is a guess but, it's possible saving the stream only takes one
call to the O/S API that saves a block of memory to a file whereas
saving a string list is likely done per string, i.e, one call to the
O/S API per string.

That would explain the significant difference in time between the two.

----------------------------------------------------------------------
marcov · 2021-07-01 09:42

There are many small writes. Using a buffered (file)stream might help.

----------------------------------------------------------------------
dbannon · 2021-07-01 09:49

Without looking at the code, yes, I guessed it was doing some loop
through the strings.  But I have to ask why that is ?  As my third
data point indicates, its so much faster to just grab the text
property and write once ? 

My experiments indicate I get exactly the same output in a quarter of
the time ?

Davo

----------------------------------------------------------------------
marcov · 2021-07-01 09:51

Quote from: dbannon on July 01, 2021, 11:49:33 amWithout looking at
the code, yes, I guessed it was doing some loop through the strings. 
But I have to ask why that is ?  As my third data point indicates, its
so much faster to just grab the text property and write once ? 

The text property makes a copy of all data in a single string. Yes,
the write will be faster, but the copy costs time, and more
importantly takes double the memory, limiting the maximum size
stringlist that you can write, specially on 32-bit.  E.g. 1GB
raspberry pis and the like.

I suggest you have a look at unit "bufstream".

----------------------------------------------------------------------
dbannon · 2021-07-01 11:51

OK thanks Marcov, doubling the memory use is not a great idea. (Mind
you, in practice my app is limited to about 100K max data due to other
reasons.)

> I suggest you have a look at unit "bufstream".

Hmm, no wiki page, only a line or two in the F1 help app. Help says
don't use directly, choose either TWriteBufStream or TReadBufStream.
No sure how I can do that, you are suggesting I write to XBuffStream
from the StringList (?) but its not clear to me how I direct that to
the file it needs to end up in ?

Davo

----------------------------------------------------------------------
marcov · 2021-07-01 12:02

Create a write buffered stream to a file.

Then use tstringlist.savetostream() to write to it.

I don't know the unit that well, but have something analogous on
Delphi.

----------------------------------------------------------------------
dbannon · 2021-07-01 13:00

OK, TWriteBufStream's constructor takes TStream as an argument. So, is
the idea that I create a TFileStream with its FileName set to the file
I want to write to and pass that TFileStream to TWriteBufStream's
constructor ?

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
varWBufStream : TWriteBufStream;FileStream :
TFileStream;....FileStream := TFileStream.Create('somefile.text',
fmCreate);WBufStream := TWriteBufStream.Create(FileStream);StringList.
SaveToStream(WBufStream);  // probably a good idea to free some of the
above ...
To my surprise, it compiles !  I will have to leave this for now,
getting late but will report back tomorrow !

Thanks Marcov !

----------------------------------------------------------------------
loaded · 2021-07-01 13:00

Hi,bufstream is really fast. I am using it as follows.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Function  
SpeedReader(file_path: String):
Boolean;varStream:TFileStream;sreader:TReadBufStream;   // add uses
bufstream,filealltext:string;begin   
stream:=TFileStream.Create(file_path,fmOpenRead or  fmShareDenyNone);
 stream.Position:=0;    sreader:=TReadBufStream.Create(stream);
sreader.Position:=0;    setLength(filealltext,stream.Size);   
sreader.ReadBuffer(pointer(filealltext)^, stream.Size);   
////////////// filealltext;    // < ---------------------- actions you
want to do    sreader.free;    stream.Free;end;

----------------------------------------------------------------------
dbannon · 2021-07-01 13:09

Right, had to try it out !

It writes just as fast as writing stringlist.text does, but as Marcov
pointed out, much kinder on memory ! So, my big test file, 5mS

Pretty nice way to work !   Thanks Marcov

And thanks Loaded too, looks like you got there ahead of me !

Davo

----------------------------------------------------------------------
dbannon · 2021-07-02 00:01

I have doc'ed what I just learnt !

https://wiki.freepascal.org/TBufStream

Thanks folks !

----------------------------------------------------------------------
lucamar · 2021-07-02 01:09

For completeness sake, is there any difference when using a
TBufferedFileStream? That should be equivalent but using only one
stream instead of two (at least in the application itself).

----------------------------------------------------------------------
dbannon · 2021-07-02 08:34

> TBufferedFileStream

Who is to know ?  Again, its not mentioned in the wiki, F1 Help does
not know about it. Remember Rumpstein's "Things we know, things we
know we don't know, things we just don't know ?  Until you just
mentioned it, for me, TBufferedFileStream was in the last category.  

Mind you, TBufStream was also a "don't know" for me until Marcov
mentioned it.

Davo

----------------------------------------------------------------------
lucamar · 2021-07-02 08:50

Quote from: dbannon on July 02, 2021, 10:34:59 amWho is to know ? 
Again, its not mentioned in the wiki, F1 Help does not know about it.
Remember Rumpstein's "Things we know, things we know we don't know,
things we just don't know ?  Until you just mentioned it, for me,
TBufferedFileStream was in the last category.
Sorry, since it's in the same unit as TBufStream, TReadBufStream, etc.
(and documented: TBufferedFileStream) I thought you might know.

I'll make my own tests then

----------------------------------------------------------------------
dbannon · 2021-07-02 12:32

Quote from: lucamar on July 02, 2021, 10:50:17 am
Sorry, since it's in the same unit as TBufStream, TReadBufStream, etc.
(and documented: TBufferedFileStream) I thought you might know.


Oh, is it ?   I could not find it with F1 when I tried, just tried
again and there it is !  So, no, I am the one to apologise !   

No docs but its use it pretty obvious and using my same test file,
produces almost exactly the same results as the combination
WriteBufStream and FileStream. And saves a couple of lines of code.

It comes from a different family than  WriteBufStream and FileStream
but does, as you say, appear in the same unit.  Not mentioned in the
docs for bufStream, does not get listed on
https://wiki.freepascal.org/fcl-base - so I wonder how you even know
it exits ?  Anyway, it does exist, is an easy way to achive a good end
and deserves a mention on the wiki, and a minimal (cos thats all thats
needed) example.

Hmm, where ?

Davo

EDIT : OK, added a bit about BufferedFileStream.  But I am wondering
now about the buffer size. https://www.freepascal.org/docs-
html/current/fcl/bufstream/tbufferedfilestream.html says that
BufferFileStream allocates 32K, thats possibly larger than I would
choose but it cannot be changed.  The pages for BufStream indicate its
'16', is that 16 bytes or 16k ?   It is user set-able.  I was sure I
read somewhere that it was 16k but I cannot find it again. Sigh....

----------------------------------------------------------------------
lucamar · 2021-07-02 15:22

Quote from: dbannon on July 02, 2021, 02:32:13 pmBut I am wondering
now about the buffer size. https://www.freepascal.org/docs-
html/current/fcl/bufstream/tbufferedfilestream.html says that
BufferFileStream allocates 32K, thats possibly larger than I would
choose but it cannot be changed.
Doesn't InitializeCache do that?

----------------------------------------------------------------------
dbannon · 2021-07-03 02:16

>Doesn't InitializeCache do that?

Yes, I'd say so but you need to call it after the initial 32K has been
allocated.  In my case, where very large files are rare, a 4K to 8K
cache would be about optimum.  Its important because while most notes
I save are in the 8K to 16K range, some people have a lot open at the
same time. If they close the application, all those notes will save
themselves, all at once. So, they would each allocate 32k then reduce
it down.  I guess with 20 notes open its not too bad, still less than
a Gig of ram. But with more than 20 notes on a 32bit system ....

With BufferedFileStream, the docs are clear, InitializeCache is
expressed in bytes.

But TBufStream, not clear.  I followed the code and believe its in
bytes. But the docs say the default is '16', manually setting a 16
byte cache from Create() results in worse performance than un-buffered
so '16' is definitely not being used as default.  Again, checking the
code, that default of 16 is multiplied by 1024 but a number supplied
to the create method is not. So, thats a doc error, should say default
is 16K.

At least I know know whats happening.

My performance testing numbers are also interesting, with my test
data, 1K of buffer was enough to reach a flat part of the speedup
curve with BufStream. But BufferedFileStream needed at least four
times that before adding more buffer space did not improve things.

It would be quite OS, hardware and data dependent. And I am only
testing on linux ...

Davo

----------------------------------------------------------------------
jamie · 2021-07-03 15:12

Some time back I write a TLinesString class that is based from
Tstrings as a replacement for TStringList in a couple of projects as
it started with something simple needed at the time ended as being one
of my main work horses.

 I have attached the PAS file for anyone to play with, pretty much
like the TstringList, all items are included. Some extra items are
included and I deemed corrected behavior for other items that fail
with TstringList, like the delimited text operations with Quotes and
so on.

 This class does not support NULLS, CR,LFs as part of a string when
returned so if you are embedding these items as valid entities in the
string then don't use this otherwise this works like a TMemo, Note pad
would work. It contants Lines.. not strings and allocates chucks of
memory within so  you don't have massive pointers all over the place..


P.S.
 The INC file is not needed, that was just a test file that was in the
project folder I created as a load test file..