extreamly slow text file loading (55,000 lines text) into listbox lazarus / 13-General ---------------------------------------------------------------------- luthepa1 · 2009-03-19 01:40 Hi all, I was hoping someone could help me understand why and/or advise the best way to carry out my task at hand.  I have an instrument I service for my company that creates text log files of every device command sent to and reply from and starts a new log once the actual working log file reaches 4MB in size.  Well each field is comma separated so I guess you would call this a csv file.  The instrument records roughly 6-10 command per millisecond.  I have software on windows to review log files and filter them which I use to track down the root cause of when there are instrument errors/failure.  Ok so I am writing my own log reviewer app in Lazarus because I want extra features to help speed up troubleshooting and think I can make a better app. I am developing under Linux Ubuntu 8.1.  My problem I am having is that loading a 4MB text file takes for ever!  Using a textfile variable, assgnfile, reset, readln and closefile method is very slow!!!  I just tried this method to open a 4MB text file (this file is 56,000 lines) and it took 27mins to get to the 26,000th line.  I added a counter to update a label object to tell me what line it was up to.  Also a listbox.items.loadfromfile is painfully slow.  I have also found an example on delphi3000 for a line by line read (http://www.delphi3000.com/articles/article_1363.asp?SK=) but this is still to slow for me (but it was faster).  I run a asus mobo with a core 2 duo E8400 3GHz with 4GB ram and sata 2 hdd.  Then my laptop which is a centrino 1.7Ghz, 500MB ram and maybe sata hdd, using the service software log reviewer app can read this 4MB text file, into its stringlist/stringgrid in 2sec!!!!  Way, way, way faster. Can anyone shed some light as to what my problem could be or the method more suitable to perform my task?  I started to look a memory mapped file methods but my programming experience is not the high and would require me to do some more learning.  But i am up for learning!  Below is the fastest routine I have made so far.  But I want a 4MB text file loading in seconds no minutes to hours.   AssignFile(F, fName); //Open up the text file as an untyped file   Reset(F, 1);   fSize := FileSize(fName);    //Get the file size   pb.Max := fSize;   NewBuffer := '';   buffer := '';   lineCount := 0;   //while FilePos(F) <> fSize do   Repeat         //read in up to an 8K block         BlockRead(F, buf, BlockSize, BlockCount);    // Allocate the new buffer    GetMem(NewBuffer, BlockSize + 1);    // Copy the New Data Into The Buffer    StrPLCopy(NewBuffer, buf, BlockCount);    // Concat the buffers    Buffer := Buffer + NewBuffer;    // Return the buffer memory    FreeMem(NewBuffer);    // Step 2. Chop the data into a stringlist    while (Pos(#10, buffer) <> 0) do     begin              //showmessage('ok');              Application.ProcessMessages;         CutAt := Pos(#10, buffer);         Line := Copy(buffer, 1, CutAt);         Line := Trim(Line);         ListBox1.Items.Add(Line);         Delete(buffer, 1, CutAt);              Inc(lineCount);              Label1.Caption := 'Total Lines: ' + IntToStr(lineCount);    end;         //showmessage(IntToStr(FilePos(F)));         Inc(logCount);         pb.Position := BlockSize * logCount;   Until (BlockCount=0); //(logCount=20);   CloseFile(F);                              //close the file, you're done   showmessage(IntToStr(logCount)); } Thanks in advance, Paul. ---------------------------------------------------------------------- Leledumbo · 2009-03-19 04:01 Some people recommends using TFileStream together with TMemoryStream. I'm not very familiar with them though. ---------------------------------------------------------------------- arnoldb · 2009-03-19 05:48 I have had a similar problem in the past...  It turned out that updating screen display labels & the listbox takes a lot of time. A good way to increase speed is to hide the listbox when loading, and not to call processmessages and a label update for every line, but on every -- say 1000th line.  Then once loaded, show the listbox again. ---------------------------------------------------------------------- luthepa1 · 2009-03-19 06:21 Thanks for the feedback guys.  Good points about the label update and Application.Processmessages.  However I originally did not have these lines.  I later added them to get some kind of indication that my app was actually doing something because before adding these line the program just appeared to be frozen, but I could see the CPU at full throttle.  But the thing that really interests me is the comment about hiding the listbox until loaded.  I had been thinking of this and in Delphi there is a command to disable component updating, and then you later re-enable updating once the task complete.  But lazarus does not recognise this command.  Anyway I just tried running again with out these commands and the listbox.visible := false.  2-3 mins pass and still going.  Still too slow. I am thinking I will need to do something more technical like Tfilestream and Tmemorystream.  I keep thinking I should install Delphi to compare but I dont want to because I think Lazarus has some real potential, especially since it runs on so many OS's.  Also the link to that Delphi300 site where there is the code to load line by line in 8K chunks uses Tfilestream.  Thats still not as quick as I want or as quick as I think it can be, although it is quicker like I mentioned originally. More help please for anyone. Thanks ---------------------------------------------------------------------- Vincent Snijders · 2009-03-19 06:51 If just reading is slow (without adding the lines to the listbox), which I don't think, then use SetTextBuf. If the problem is with the listview, make sure you use ListBox.Items.BeginUpdate. I am not sure if Listbox.Items.BeginUpdate has been implemented for gtk, I am pretty sure is implemented for win32. Two examples: Easy coding: Code: [Select]begin   Listbox1.Items.BeginUpdate;   try     ListBox1.Items.LoadFromFile(fName)   finally     Listbox1.Items.EndUpdate;   end; Possibly faster code: Code: [Select]const   BufSize = 1024 * 1024; // 1 MB buffer, experiment with it, bigger than 64 kB doesn't probably show any speedup. var   F: Text;   Buffer: array[1..BufSize] of byte; begin   AssignFile(F,FName);   Reset(F);   SetTextBuf(F, Buffer);   Listbox1.Items.BeginUpdate;   try     repeat        readln(F, line);        ListBox1.Items.Add(line);     until eof(F);     CloseFile(F);   finally     Listbox1.Items.EndUpdate;   end; Instead of assigning directly to the listbox, you can first read it into a stringlist and then do ListBox1.Items.Assign(MyStringList). If you try any of these methods, please post timings. I am interested to see what is fastest. ---------------------------------------------------------------------- Vincent Snijders · 2009-03-19 06:53 Quote from: arnoldb on March 19, 2009, 06:48:49 amA good way to increase speed is to hide the listbox when loading, and not to call processmessages and a label update for every line, but on every -- say 1000th line.  Then once loaded, show the listbox again. Hiding the listbox should not be necessary, if you use BeginUpdate. Application.ShowMessages and the Label updates are a waste of time, indeed. ---------------------------------------------------------------------- luthepa1 · 2009-03-19 11:29 ok well the simple code with the recommendation takes 9:15 (m:sec).  slow. The faster code option recommendation not sure yet.  Keep getting compile error that SettextBuf has "wrong number of parameters specified for call to SetTextBuf".  I even tried a 3rd param for SetTextBuf with the size of the buffer.  I dont understand whats wrong.  Also I had to use "System.Text" for the F variable.  Does not like "Text". ---------------------------------------------------------------------- Vincent Snijders · 2009-03-19 12:01 System.Text or TextFile. Aparently you use this code in a method of an object that has the text property. ---------------------------------------------------------------------- Vincent Snijders · 2009-03-19 13:08 Try Jump to declaration to see what Lazarus (not the compiler) thinks the number of parameters should be. ---------------------------------------------------------------------- chrnobel · 2009-03-19 21:48 A little off topic, but my experience is that text reading as such is very, very fast. I have some programs where I read line by line typically like this: Code: [Select]  Reset(F);   While SeekEof(F)=false do   begin      Readln(F, S);         ....         Do a lot of validating of S, and store the result in a MySQL database         ...   end   CloseFile(F); And it is very fast, typically a few seconds for 50.000+ lines Also I have made an XML reader (which basically is the same), and it takes around 15 seconds to read a 12M XML file and save into a database on a virtual machine on my laptop. So my conclusion is the the file read as such is not a problem, it has to caused by something inside your loop. /Chris ---------------------------------------------------------------------- luthepa1 · 2009-03-19 22:18 Firstly, thanks for all your help so far Vincent.  I appreciate it greatly. Second: I am not sure how to "Jump" to the declaration to see what Lazarus thinks.  I am coming back to pascal coding after years of not coding.  Although I probably still would not know how to do this.  Sorry. So what I had is: SetTextBuf(F, Buffer); This would give the error that there is the wrong number of parameters for the call. Also it give a hint of "Found Declaration: TControl.SetTextBuf(PChar)".  If I click this hint I get a message dialog stating "control.inc" cant be found and to check my search paths for other units.  Not sure if I have screwed Lazarus up a bit with trying to add components to which all of them always error(HistoryFiles, multilog). But if I change it to System.SetTextBuf(F, Buffer); there are no errors and it compiles.  But not sure if its working because my laod times are now worse. This code is executed from the Open item of a menubar component (TMenuItem). So using this code example with 1MB buffer I found it is still slow and I ended up terminating the app after 11mins.  I then changed the buffer size to 64Kb and then 8KB but still the same scenario and I ended up terminating after 11mins.  So the fastest so far is the Listbox1.items.loadfromfile function at just over 9mins. Do you agree that this should load a text file of 55,000 lines in under a minute, even seconds? Thanks. ---------------------------------------------------------------------- luthepa1 · 2009-03-19 22:21 @ chrnobel. I was so glad to hear your reply because you confirm what I would have thought.  A few seconds to load such a large text file.  I think I will start a new project and just simply code to perform this task of text file loading.  Then I can figure out (if there is a sudden speed up) if something else in my app is slowing things down. I will post up my code and results of the new project/app for the purpose of troubleshooting this issue I am having. Cheers! ---------------------------------------------------------------------- Vincent Snijders · 2009-03-19 22:43 Use System.SetTextBuf, so you don't use the method of TControl. If you post your application, with a test data generator, I will run it on windows. ---------------------------------------------------------------------- luthepa1 · 2009-03-19 23:08 Ok, well found something interesting.  Looks like Ubuntu 8.1 maybe the issue.  Below I will put the demo app code for this troubleshooting. I run this on Ubuntu 8.1 64bit OS with Lazarus.  I am wondering also if 64bit is the difference causing probs here.  After 7min I terminated since it again looked like it would take for ever to load a simple text file. So I then took this code/project to my laptop which runs WinXP Pro 32bit with Lazarus.  When I go click open on the file menu in the app I get error message popup "Project Raised exception class 'External: SIGSEGV'.".  So Ithen comment out the variable "Buffer : Array[1..BufferSize] of byte;" as I am not even using it anyway.  The project then runs and loads the text file of 55,000 lines in 20sec!!!! Massive improvement! I tried again on Ubuntu x64 with the buffer variable commented out but no difference. So looks like an OS thing? The Code ----------------- unit main; {$mode objfpc}{$H+} interface uses   Classes, SysUtils, FileUtil, LResources, Forms, Controls, Graphics, Dialogs,   StdCtrls, Menus; const      BufferSize = 1024 * 1024; //1MB buffer, experiment with it, bigger than                                //64Kb probably doesn't show any speedup.. type   { TfrmMain }   TfrmMain = class(TForm)     lstBox: TListBox;     MainMenu1: TMainMenu;     mnuOpen: TMenuItem;     mnuMain: TMenuItem;     OpenDiag: TOpenDialog;     procedure mnuOpenClick(Sender: TObject);   private     { private declarations }   public       { public declarations }   end; var   frmMain: TfrmMain; implementation { TfrmMain } procedure TfrmMain.mnuOpenClick(Sender: TObject); var   fName, line: string;   F: System.Text;   //Buffer : Array[1..BufferSize] of byte; begin   openDiag.Filter:='Tigris Log Files (*.txt)|*.txt';   if openDiag.Execute then   begin      fName := openDiag.FileName;      AssignFile(F, fName);      Reset(F);      try         lstBox.Items.BeginUpdate;         While SeekEOF(F)=False do         begin              ReadLn(F, line);              lstBox.Items.Add(line);         end;         CloseFile(F);      finally         lstBox.Items.EndUpdate;      end;   end; end; initialization   {$I main.lrs} end. ---------------------------------------------------------------------- Vincent Snijders · 2009-03-19 23:11 My guess: BeginUpdate is not implemented in the gtk widgetset interface. ---------------------------------------------------------------------- Vincent Snijders · 2009-03-19 23:13 Ah, that buffer is on the stack, 1 MB on the stack may be too large. Make the buffer a global variable. ---------------------------------------------------------------------- luthepa1 · 2009-03-19 23:20 Ah, yes!  Good point.  Ok on my windows XP side I ran the app again but commented out the try.. finally block and commented out the BeginUpdate and EndUpdate.  It open the text file in 25 seconds.  Only 5 sec slower.   So I am convinced its something else.  Maybe I need to change compiler options on Ubuntu?  Or maybe windows has better caching or something? ---------------------------------------------------------------------- luthepa1 · 2009-03-19 23:28 Ah, ok.  Global. Also out of interest I tried a LoadFromFile funciton and it can load the text file in 21sec.  Slight improvement.  Considering the finished app will be loading many 4MB text files (up to 200MB) a 20-25sec time to load 4MB is still too slow for me but all the other code I had been trying to speed things up should probably work in that regards. So I guess the remaining question for this topic is, why is it so slow on Ubuntu 8.1 x64?  If anyone can answer this you will be the king! Thanks Guys. ---------------------------------------------------------------------- luthepa1 · 2009-03-20 00:00 Just tested on a Ubuntu 8.1 x32 system I have.  Same issue. ---------------------------------------------------------------------- Vincent Snijders · 2009-03-20 09:02 Listboxes are not designed to handle 1000's of items and MB's of data. ---------------------------------------------------------------------- luthepa1 · 2009-03-20 11:20 HOLY CRAP!!! I FIXED IT! It was Lazarus 0.9.26 and possible a gtk combination. What I did was to perform complete removal of the Lazarus that I installed from Synaptic in Ubuntu 8.1 x64.  I had Lazarus 0.9.26 installed with fpc 2.2.2.  Removed Lazarus and fpc from synaptic (purge).  Then in terminal type "sudo apt-get autoremove" to remove all the fpc dependencies. Then I followed this post from this forum. http://forum.lazarus.freepascal.org/index.php/topic,5647.0.html But instead of downloading the version in post I downloaded the latest obviously which was ; fpc-2.2.5-20090320.x86_64.rpm fpc-src-2.2.5-20090320.x86_64.rpm lazarus-0.9.27.19037-20090320.x86_64.rpm Also these rpms are the GTK2 builds, so after creating the deb detailed in the post (link above) I did not have to worry about rebuilding lazarus to GTK2.  So yeah, once you got the rpm packages, convert to deb with alien then install.  Then when loading Lazarus I got message that the fpcscr/2.2.5 didn't look correct.  Clicked OK and all is working. Now the good news! Using the same demo code for the purpose of troubleshooting i got a incredible nice surprise.  My app loaded all the 55,000+ lines into the listbox in 3 seconds!!!! On windows I am getting 30seconds and previously on Ubuntu with the older Lazarus 0.9.26 /w fpc2.2.2 I still had nothing loaded after 11mins.  So I am not sure whether its a bug fix in 0.9.27 /w fpc 2.2.5 or GTK2 support now, or both.  But I am happy now that I got it loading in 3 seconds.  Now to go back to my original log reviewer app I am building to see if that works. Thanks for your help guys.