Tchecklistbox slowdown lazarus / 31-LCL ---------------------------------------------------------------------- jaz2014 · 2022-01-31 11:22 I'm adding a large number of file paths to a checklist box. I have sort set to false. It is the standard lazarus checklistbox. The source (MySQL file) just feeeds each record from a table out in sequence and puts it into the checklistbox (items.add())  without changes My problem is that as I add lines the process gets slower and slower. This makes the task impossible when I get over about 150,000 lines. By the time it reaches 500,000 it's only adding 2 lines a second at most. (Ideally I'd like to get to 1.5m lines)  but this is far too slow. If I can't speed it up I'm going to have to re-write  a large amount of code. (I can't work record by record for this process) Can anyone explain why it is slowing down? Is there anything I might have missed in the settings? thanks ---------------------------------------------------------------------- wp · 2022-01-31 11:35 Did you add a CheckListBox.Items.BeginUpdate before and a CheckListbox.Items.EndUpdate after the loop which adds the items? They prevent redrawing the listbox with every item added. ---------------------------------------------------------------------- jaz2014 · 2022-01-31 17:24 Thanks wp No I didn't know about that but the checklistbox is set not visible. I'll add your suggestion too and see if that helps..... found it: Reading the Tstrings.beginUpdate  it looks like it may just add extra code in which may not help.  I'll give it a go now though. ---------------------------------------------------------------------- winni · 2022-01-31 18:12 Hi! Visual components were never meant to carry such a huge amount of data. This is not only true for Lazarus but also Delphi and the visual M$ languages. This is just a wrong design. Put all your items in a StringList and use the CheckListbox to copy a part of them as a windows in the StringList. Winni ---------------------------------------------------------------------- jaz2014 · 2022-02-02 14:18 Some quite sweeping statements there winni. The design is what I need. Once loaded to the checklistbox a Tstringlist is indeed used (as a name=value pair and custom sorted) Incidently Delphi-5 (which I run in virtual mode) has a line limit of 32768 lines which I expect has more to do with bus width and integer use than the purpose of objects. Ok so - Loading 700,000 file paths into the checklistbox takes 9 hours and 8 minutes. (Just 38MB of text) Save the list to a text file and it loads in just 4 minutes and 3.6 seconds. Doubling the file to 1.4M lines loads from file in 16 minutes 38 seconds. (admittedly the file is on an m.2 disc but then it isn't mySQL that is slowing down so lline by line is effectively ram to ram) wp's suggestion of begin/endupdate is in place for this - it didnt seem to make much difference in either method. Interestingly closing the program takes about 7 seconds (click the window x) - presumably free and nil - or something windows itself does causes that.  The difference indicates a sharp efficiency difference in how the two sources are handled. If I had to guess I'd say line by line is allocating line buffers individually while loadfromfile allocates a block (being a file the size is known)  The difference is really stark though and highlights something very interesting. Visually - handling of 700,000 lines is quite fast as is seaching,checking and so on. For what it's worth - I then do 3.5million file reads on the list which takes 4 minutes and 50 seconds mixing read/seek and compares on each file including the time to sort the stringlinst and remove duplicate entries. For now it looks like my best bet is to develop using a file write from mySQL and read it back in. Perhaps I can create a custom checklist box at a later date. (Assuming streams won't do the trick) ---------------------------------------------------------------------- Martin_fr · 2022-02-02 15:03 Quote from: jaz2014 on February 02, 2022, 03:18:54 pmLoading 700,000 file paths into the checklistbox takes 9 hours and 8 minutes. (Just 38MB of text) Save the list to a text file and it loads in just 4 minutes and 3.6 seconds. QuoteThe difference indicates a sharp efficiency difference in how the two sources are handled. If I had to guess I'd say line by line is allocating line buffers individually while loadfromfile allocates a block (being a file the size is known)  1) Did you recreate the entire control (or even form) between the 2 tests? Or did you just clear the check-listbox, and load the new content? 2) Try Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i CheckListBox1.Items.Capacity := CountOfItemsToBeAdded; ---------------------------------------------------------------------- jaz2014 · 2022-02-02 15:54 Hi Martin I ran multiple tests - reloading the entire application with different amounts of data. (I did consider OS/buffer issues too) CountOfItems to be added looks useful - I'll certainly check that out as the count is known beforehand ( as the text size could be calculated easy enough - hence considering a custom box) I have just discovered a problem with Tstringlist.endUpdate  that stops following things in a FINALLY block from running - I havn't had time to check that out yet but If anyone is interested I'm running FPC 3.3.1 and the issue seems to be if an update doesn't happen (nothing checked in my case) and exit is called instead. There is also an EXCEPT loop inside the TRY FINALLY. I'll try and make time and create a forum thread if I can identify an actual issue - (it may be my ignorance of the function again) ---------------------------------------------------------------------- balazsszekely · 2022-02-02 16:33 @jaz2014 I agree with @winni, there is no point to show 1.5M lines to the user, you should filter the list first. However if you still want to proceed, choose VirtualStringTree, it can handle more then 1.5 M lines without any issues. If you load the data from a database, make sure you set TSQLQuery.UniDirectional property to true, otherwise the bottleneck will be the query itself. Can you upload the 38 MB file somewhere? ---------------------------------------------------------------------- jaz2014 · 2022-02-02 21:53 I've just tried loading the data directly into an array of strings and it looks like the issue may be mySQL after all - Maybe I've done something some time ago when I built my first demo that has changed on my system since I'll need some time to check this out. I'll also double check the query object - but yes GetMen - I do need to display the data. I wouldn't be doing it otherwise. ---------------------------------------------------------------------- wp · 2022-02-02 22:01 Quote from: jaz2014 on February 02, 2022, 03:18:54 pmLoading 700,000 file paths into the checklistbox takes 9 hours and 8 minutes. (Just 38MB of text) Save the list to a text file and it loads in just 4 minutes and 3.6 seconds. Instead of a TCheckListbox I would use a TListView and operate it in virtual mode. Write all your data to an array or StringList and tell the Listview the count of items to be displayed. Write a handler of OnData to pass the data from the storage to the list item. Finally set the Listview's OwnerData to true to activate the virtual mode. Simulating the checkmarks is a bit more complex because I found the property Checkboxes to have no effect. You could add two images to an ImageList, one for the checked and one for the unchecked state. Assign it to the StateImages of the ListView. Then, in order to display an item as checked you set the item's StateIndex to the image index of the checked image, likewise for the unchecked state. And how to store the checked state? You could simply put an additional at the front of the stored string, say a dot ('.') for checked and a space for unchecked. See the attached demo, it should work, at least basically. Note that you can load 1 million items into this virtual listview within a fraction of a second. Of course, in reality, you will experience longer times because data must be read from a file or from a database, or a directory must be iterated for the filename, etc. But for this at least the listview cannot be blamed. ---------------------------------------------------------------------- jaz2014 · 2022-02-04 01:21 Problem solved. It turns out the DbGrid on another tab page was still being updated even though it was dissabled You need to remove the datasource aswell on all DBcomponents or they still get updated or at least cause this massive slowdown.  deliberate feature/bug or just oversight I dont know - but it does seem wrong. ( I also had a DBNavigator in the program connected but that doesnt seem to cause problems if just dissabled)  I havn't had much time to verify everything else but its run a couple of times now. Time to load now 6 minues 5 seconds. (somewhat better than 9 hours I think) And those file operations I do really demonstrate how good FPC is. (There's more to that but I'm impressed anyway) Thank you all for the attempts to help - it is appreciated. as for the rest - I just love the way you are all jumping in redesigning something you have absolutley no knowledge about. Classic (and for once I will ...  ) ---------------------------------------------------------------------- balazsszekely · 2022-02-04 07:05 @jaz2014 QuoteTime to load now 6 minues 5 seconds. (somewhat better than 9 hours I think) And those file operations I do really demonstrate how good FPC is. (There's more to that but I'm impressed anyway) Please take it as a constructive criticism, I'm not trying to be mean or anything, but 6 minutes is still a lot. You almost install an OS nowadays. Create a separate query without a datasource, set unidirectional property, then populate a listview in virtual mode or a virtual string tree. I just ran a few tests, I have a firebird database with a relatively large table(650, 000 records). It took 4 seconds to loop through the records and populate the vst. Even a table with 1.5 m records, shouldn't take more then 15 seconds.