MPHexEditor, large files scrolling lazarus / 5-Lazarus Extra Components ---------------------------------------------------------------------- DIMM_bl4 · 2021-01-31 20:27 Hello to all, my first message here. Have a some trouble with MPHexEditor module, when open large files scrolling became very slow. This is slightly noticeable from 16-32MB size and go worst with increasing size. First of all, I go to example project, and see exactly the same. Lazarus is 2.0.10, FPC 3.2.0, MPHexEditor from here https://github.com/michalgw/mphexeditor I tried to solve by myself, but looks like I'm not strong enough ) Found that things: There are used two ways for scrolling 1. Scroll with keyboard/mousewheel do with changing of 'TopRow' property, and this way is slow. 2. Scroll with side scrollbar is totally differrent and uses 'WMHScroll(var message: TLMHScroll)' and then 'TrySmoothScrollBy(Col, Row)' from module Grids. This way work always fast, with any file size. Would appreciate any help, or thoughts ) ---------------------------------------------------------------------- korba812 · 2021-02-02 17:35 This is a known issue. The problem is with TCustomGrid, from which TMPHexEdit inherits. I have attached a sample project to illustrate the problem. Other problems with using TCustomGrid are excessive memory consumption and row limit. This is the main reason why I stopped further development of TMPHexEdit. It should probably be redesigned from scratch. ---------------------------------------------------------------------- DIMM_bl4 · 2021-02-02 18:28 Thanks for reply. I made a 'temporary' solution - modified procedures KeyDown, DoMouseWheelDown, DoMouseWheelUp from MPHexEditor.pas to also use TrySmoothScrollBy(aCol, aRow) from module Grids (as a scrollbar scrolling). But for this I had to also modify module Grids.pas, for possibility to call TrySmoothScrollBy from outside. I moved it from 'private' declaration to 'protected', same as for some procedures, called from outside. I'm not 100% sure about any side effects, but looks like working good. I attach modified example .exe from MPHexEditor, now it can scrolls up to ~512MBytes files, original example freezed to death (with 100% load to one thread) with constant scrolling at about 32Mbytes files. https://yadi.sk/d/7LU_sUxo1Zvk2Q Quote from: korba812 on February 02, 2021, 06:35:19 pmThe problem is with TCustomGrid, from which TMPHexEdit inherits ... I can not totally agree, because I also tryed to use 'parent' versions of procedures DoMouseWheelDown, DoMouseWheelUp, and scrolling became fast (no freeze)...but at one line per wheel's 'tick'. ---------------------------------------------------------------------- korba812 · 2021-02-02 20:12 IMHO, a better solution will be to expand functionality of TCustomGrid component. The TCustomGrid.UpdateCachedSizes method is called quite often. This function sums height of all rows, so it takes a while for a large number of rows. In many cases, all rows have the same height and there is no point in storing it. I would suggest adding a mode where all rows (or at least non-fixed) have the same height and not store their height values. This will eliminate the problem with slowdown and high memory consumption. ---------------------------------------------------------------------- Mastropiero · 2022-08-24 13:35 Hello, first of all, sorry for the thread necromancy. I have recently faced this issue. It seems that the problem has been inherited from TCustomGrid. Is there any chance of having someday a new version of this component that is not based on TCustomGrid so the problem does not exist anymore? Could the MPHexEditor component be redesigned so it only handles the rows that are actually shown on the screen instead of loading the whole file into RAM? ---------------------------------------------------------------------- korba812 · 2022-08-24 19:02 Quote from: Mastropiero on August 24, 2022, 03:35:20 pmI have recently faced this issue. It seems that the problem has been inherited from TCustomGrid. Is there any chance of having someday a new version of this component that is not based on TCustomGrid so the problem does not exist anymore? As I wrote, it requires a lot of work. Unfortunately, this is not in my plans. But maybe someone will? Quote from: Mastropiero on August 24, 2022, 03:35:20 pmCould the MPHexEditor component be redesigned so it only handles the rows that are actually shown on the screen instead of loading the whole file into RAM? Theoretically yes - MPHexEdit uses TMemoryStream internally and theoretically could handle any stream. But it won't be easy, e.g. inserting or deleting data at the beginning of the file. ---------------------------------------------------------------------- Mastropiero · 2022-08-28 00:17 Thank you for your comments. I have just wrote a small proof-of-concept of buffered reading that uses only 3MB of RAM and it keeps constant and fast even when scrolling >5GB sized files.  I use a simple TMemo to print the HEX data in read-only mode but I think the concept could be applied to any descendant of TCustomGrid. This would solve the speed and memory consumption problem, but the own component's scrollbars wouldn't work anymore. I think this can be easily solved by disabling scrolling and adding an external TScrollBar component synced with the grid. However, I'm wondering how to handle data selection, tracking of modified bytes, undo operations, etc. with this approach. Tons of analysis and work to do... ---------------------------------------------------------------------- korba812 · 2022-08-28 09:58 There is a component that can display (read only) huge files (> 2GB) in hex: https://github.com/Alexey-T/ATBinHex-Lazarus You can see how Alexey handled scroll bars in his component. CNPack has a hex editor component which is based on TCustomControl: https://github.com/cnpack/cnvcl/blob/master/Source/Graphics/CnHexEdito r.pas ---------------------------------------------------------------------- AlexTP · 2022-08-28 12:13 If the problem is inside LCL Grids code, can you take the Grids file, rename it + rename its classes, and include changed file into MPHexEditor? ---------------------------------------------------------------------- Mastropiero · 2022-08-28 20:32 Quote from: korba812 on August 24, 2022, 09:02:44 pmTheoretically yes - MPHexEdit uses TMemoryStream internally and theoretically could handle any stream. But it won't be easy, e.g. inserting or deleting data at the beginning of the file. WinHex (the best hex editor for Windows) does insert operations directly on a RAM buffer. If the file does not fit the buffer, it creates a temporary file. It also offers an in-place edit mode for big objects. ---------------------------------------------------------------------- zeljko · 2022-08-30 12:44 IMO, you should try TVirtualStringTree instead of TCustomGrid or any Grids derivative. ---------------------------------------------------------------------- Mastropiero · 2022-08-30 14:02 Quote from: zeljko on August 30, 2022, 02:44:28 pmIMO, you should try TVirtualStringTree instead of TCustomGrid or any Grids derivative. Do you mean Grids is THAT bad?  Is there any plans to rewrite the Grids unit or something? TCustomGrid is probably one of the most used classes in the original Delphi and it can't understand why it isn't optimized on Lazarus. By the way, I have never worked with the TVirtualStringTree but I'll give it a chance. Looks promising. ---------------------------------------------------------------------- zeljko · 2022-08-30 14:07 Grids is ok for it's purpose, but it's not suitable for big data scenario. I'm using grids in my projects too, but for huge data I'm using TVirtualStringTree (eg 1 million items). ---------------------------------------------------------------------- korba812 · 2022-08-30 18:01 TVirtualStringTree does not solve problem of large files as it can only handle 2^32 rows, so maximum file size is 4GB x 16 bytes per line = ~68GB. It's not much nowadays. We need something that has a row count of (at least) Int64 type. ---------------------------------------------------------------------- zeljko · 2022-08-31 12:52 In that case you should implement your own control (if such beast does not exist already) ---------------------------------------------------------------------- Mastropiero · 2022-08-31 16:16 Quote from: korba812 on August 30, 2022, 08:01:03 pmTVirtualStringTree does not solve problem of large files as it can only handle 2^32 rows, so maximum file size is 4GB x 16 bytes per line = ~68GB. It's not much nowadays. We need something that has a row count of (at least) Int64 type. I think there is something wrong in that approach. Most modern hex editors do not manage zillions of rows. They only store in the control a few rows (the ones being shown on screen and maybe a few more for buffering and fast scrolling) so the control only shows a small portion of the file. The rows are dynamically loaded into the control as the user navigates the whole file. This approach is very fast and uses a very small memory footprint. Size-modifying operations (insert, delete, concatenation...) are solved by using a temporary buffer (or a temporary file for huge amounts of data). EDIT: but, of course, I think that a modern Pascal compiler should use primarily 64bit integers when possible. so maybe many of the classes/controls should be updated. ---------------------------------------------------------------------- korba812 · 2022-09-01 16:28 Quote from: Mastropiero on August 31, 2022, 06:16:53 pmQuote from: korba812 on August 30, 2022, 08:01:03 pmTVirtualStringTree does not solve problem of large files as it can only handle 2^32 rows, so maximum file size is 4GB x 16 bytes per line = ~68GB. It's not much nowadays. We need something that has a row count of (at least) Int64 type. I think there is something wrong in that approach. Most modern hex editors do not manage zillions of rows. They only store in the control a few rows (the ones being shown on screen and maybe a few more for buffering and fast scrolling) so the control only shows a small portion of the file. The rows are dynamically loaded into the control as the user navigates the whole file. This approach is very fast and uses a very small memory footprint. Size-modifying operations (insert, delete, concatenation...) are solved by using a temporary buffer (or a temporary file for huge amounts of data). TCustomGrid and VirtualStringTree run "virtually" - they don't store any data. They only store basic information, e.g. total number of rows, current cell number, first and last visible cell, etc... Incase of TCustomGrid, user has to implement appropriate methods to handle data or draw a cell. As I said before, TMPHexEditor stores data in TMemoryStream. The problem with TCustomGrid is that it stores height of each row separately, which is unnecessary in the case of TMPHexEditor. This can be easily fixed by modifying TCustomGrid code as Alex suggested. A bigger problem is the ability to handle only MaxInteger row count. And that won't be easy to change in TCustomGrid. Quote from: Mastropiero on August 31, 2022, 06:16:53 pmEDIT: but, of course, I think that a modern Pascal compiler should use primarily 64bit integers when possible. so maybe many of the classes/controls should be updated. I don't think it is necessary. The case of hex editor is special. Usually, no such scope is needed. ---------------------------------------------------------------------- Mastropiero · 2022-09-02 19:28 Quote from: korba812 on September 01, 2022, 06:28:21 pm TCustomGrid and VirtualStringTree run "virtually" - they don't store any data. They only store basic information, e.g. total number of rows, current cell number, first and last visible cell, etc... Incase of TCustomGrid, user has to implement appropriate methods to handle data or draw a cell. As I said before, TMPHexEditor stores data in TMemoryStream. The problem with TCustomGrid is that it stores height of each row separately, which is unnecessary in the case of TMPHexEditor. This can be easily fixed by modifying TCustomGrid code as Alex suggested. So, if I used a TCustomGrid with only 64 rows (the visible ones), that wouldn't be a problem anymore. Isn't it? ---------------------------------------------------------------------- korba812 · 2022-09-02 19:47 But scrollbar will only match 64 rows. ---------------------------------------------------------------------- Mastropiero · 2022-09-03 14:03 Quote from: korba812 on September 02, 2022, 09:47:38 pmBut scrollbar will only match 64 rows. In my previous proof-of-concept (as I described some posts ago) I fixed that by hiding the control's scrollbar and putting a TScrollBar component next to the grid. I did the test with a standard TMemo instead of a TCustomGrid, but it did the trick perfectly.  In any case, not only the scrollbars should be fixed; other features such as data range selection should also have to be redone with a different approach. So, in my opinion, it is too much work for a complete MPHexEditor reworking, but I still think that a new (made from scratch) Hex component based on TCustomGrid is feasible. [rant mode on] Edit: never mind... I have just discovered that the min and max values of TScrollBar are also 32bit, so as a workaround I could use the ScrollBar to store the offset at the row level (so if I edit 16 columns, I can scroll files up to 2^32 * 16 bytes). The limit in that case would be 64GB. Still 'small' nowadays. So, at this point, I see that most components in Lazarus have severe limitations to be used in a modern system (and for modern purposes). Definitely, if Lazarus/Free Pascal have become a niche language is for a reason. Lazarus is not a RAD environment anymore if you have to build from scratch even a simple Scroll Bar component. Of course, in the case of the scroll bar there are third-parties alternatives such as ATControls (that I am in fact using for tabs, and I think it's a brilliant suite of components), but in my opinion, the stock classes should be usable without having to use external packages. [rant mode off]