No decent tutorial for the Tdrawgrid - need to output millions of data lazarus / 61-Beginners ---------------------------------------------------------------------- teochris · 2016-05-31 16:15 Hello, I need to output nearly 25 million rows and of 6 columns data from array cell calculations in a Tstringgrid - but it only manages to output only 4 million rows and then the form hangs, I believe because of Ram leaking. Is there a way to output such amount of data in a Tdrawgrid and everything would work properly? I have not found a decent tutorial about how can i output data in a Tdrawgrid can someone please point to me such an example project or send me a sample code? If i export data first on a txt file then maybe the Tdrawgrid or even better the Tstringgrid could output them properly without any problem - an example project would be really helpful. Thanks ---------------------------------------------------------------------- balazsszekely · 2016-05-31 16:25 You ask this question before: http://forum.lazarus.freepascal.org/inde x.php/topic,32646.msg210593.html#msg210593 . Do you have issues with VST? ---------------------------------------------------------------------- teochris · 2016-05-31 16:29 I have not managed to make virtualtree output my data properly so now I am trying to use Tdrawgrid or Tstringgrid to output them properly ---------------------------------------------------------------------- wp · 2016-05-31 16:42 You'll need a 64-bit machine probably. And a good time reading through 150,000,000 cells for the rest of your life  Anyway, here is the code for a TDrawGrid. I had to go down to 2.5 millions rows since I work with Laz-32bit. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);const  n = 6;  m = 2500000;var  i,j: Integer;begin  SetLength(data, n, m);  for i:=0 to n-1 do    for j:=0 to m-1 do      data[i,j] := 10*i+j;  DrawGrid1.DefaultColWidth := 80;  DrawGrid1.RowCount := m+1;  DrawGrid1.ColCount := n+1;  DrawGrid1.Invalidate;end; procedure TForm1.DrawGrid1DrawCell(Sender: TObject; aCol, aRow: Integer;  aRect: TRect; aState: TGridDrawState);var  s: String;  grid: TDrawGrid;begin  if Length(data) = 0 then    exit;  if aCol = 0 then    s := Format('Row %d', [aRow])  else  if aRow = 0 then    s := Format('Col %d', [aCol])  else    s := FloatToStr(data[aCol-1, aRow-1]);  grid := TDrawGrid(Sender);  grid.Canvas.TextOut(ARect.Left + constCellpadding, ARect.Top + ConstCellPadding, s);end; ---------------------------------------------------------------------- teochris · 2016-05-31 17:04 Quote from: wp on May 31, 2016, 06:42:59 pmYou'll need a 64-bit machine probably. And a good time reading through 150,000,000 cells for the rest of your life  Identifier not found "data" what type is it? ---------------------------------------------------------------------- wp · 2016-05-31 17:18 Sorry, should have posted complete project... Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i type  TDataArray = array of array of Double; ---------------------------------------------------------------------- teochris · 2016-05-31 17:25 Quote from: wp on May 31, 2016, 07:18:16 pmSorry, should have posted complete project... Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i type  TDataArray = array of array of Double; I do not only feel newbie but I know I am! After I add this declaration in my form still I am getting the same error Identifier not found "data" ---------------------------------------------------------------------- wp · 2016-05-31 17:51 Then declare the variable: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  Data: TDataArray; But here's the full project. ---------------------------------------------------------------------- balazsszekely · 2016-05-31 18:27 If you choose VST... ---------------------------------------------------------------------- teochris · 2016-05-31 18:35 Quote from: wp on May 31, 2016, 07:51:59 pmThen declare the variable: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  Data: TDataArray; But here's the full project. Thanks for your time and help! When I try to output 2500000 rows it hangs and then raises exception - I have win 10 - 64bit Do you know if I first write all calculations to a txt file and then write this file data to a Tstringgrid or Tdrawgrid If I would have better chances? ---------------------------------------------------------------------- mas steindorff · 2016-05-31 19:01 I ran into this TString limit as well on my 32 bit OS. my limit at the time was around 2G. What I did was to not use the .saveToFile() and coded my own version.  I move a .string[] into a local var and save it one line at a time.  it even run faster that the .SavetoFile() probably due to the  simpler code but it also has worked on 10G data sets. ---------------------------------------------------------------------- teochris · 2016-05-31 19:11 Quote from: mas steindorff on May 31, 2016, 09:01:57 pmI ran into this TString limit as well on my 32 bit OS. my limit at the time was around 2G. What I did was to not use the .saveToFile() and coded my own version.  I move a .string[] into a local var and save it one line at a time.  it even run faster that the .SavetoFile() probably due to the  simpler code but it also has worked on 10G data sets. Can you please post here your solution or even better post the certain part of your project? ---------------------------------------------------------------------- wp · 2016-05-31 19:13 Win 10 64bit is one part of the story. The other one is Lazarus, it must be 64 bit as well. I only have an old Laz 1.44 installation with 64 bit, because I normally work with 32 bit. But with this 64-bit system my demo runs without a crash even with 25 million rows. It's no fun working with it, though, because scrolling is very slow. Part of it is certainly due to the fact that these 150 millions of double values bring reach the limits of my system memory. Buf even an empty drawgrid with that many cells has already halfe the memory consumptions and does not scroll smoothly either. I don't know about the performance of VirtualTreeview, but I would not expect it to be substantially faster. We are at the limits of today's PC's performance here. And no: writing to a file and loading into any visual control will not be faster. But to be honest: why do you want to show that many data? Nobody will look at them, not even you. You must write the results of these calculations to file which you evaluate with some dedicated analysis to extract the information that you need. Since I do not know what you want to achieve I cannot give you more specific hints. ---------------------------------------------------------------------- teochris · 2016-05-31 19:28 Quote from: wp on May 31, 2016, 09:13:30 pmWin 10 64bit is one part of the story. The other one is Lazarus, it must be 64 bit as well. I only have an old Laz 1.44 installation with 64 bit, because I normally work with 32 bit. But with this 64-bit system my demo runs without a crash even with 25 million rows. It's no fun working with it, though, because scrolling is very slow. Part of it is certainly due to the fact that these 150 millions of double values bring reach the limits of my system memory. Buf even an empty drawgrid with that many cells has already halfe the memory consumptions and does not scroll smoothly either. I don't know about the performance of VirtualTreeview, but I would not expect it to be substantially faster. We are at the limits of today's PC's performance here. And no: writing to a file and loading into any visual control will not be faster. But to be honest: why do you want to show that many data? Nobody will look at them, not even you. You must write the results of these calculations to file which you evaluate with some dedicated analysis to extract the information that you need. Since I do not know what you want to achieve I cannot give you more specific hints. These data will not have any real use, I just want them to be shown to users because this will add a feature that similar programs like mine do not have at all and users of my app requested it from me ---------------------------------------------------------------------- tk · 2016-05-31 20:00 I would implement some paging mechanism here to display reasonable amount of data at a time in a grid. And I would not even try to load all those data into memory, rather I would store them in a large file on disk and then read the given chunk into memory. TFileStream should be able to do it, or see here: http://stackoverflow.com/questions/4615286/what-is-the-fastest-way- for-reading-huge-files-in-delphi ---------------------------------------------------------------------- mas steindorff · 2016-05-31 20:23 I did not find the code I had created... too bad, I had some other cool stuff in it..  But fundamentally it is: (done by hand so may have error) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i fhandle :textfile;TStr: Tstringlist or like...procedure fmain.mySaveToFile(filename:string);var str:string  i:interger;begin   AssignFile(fhandle,filename);    if fileExist(Filename) then       // does user wish to append or overwrite...   // Append or ReWrite(fhandle) ... // add headers...//  main save   for i:= 0 to Tstr.count-1 begin      str := Tstr.string[i];         // get the line      writeln(fhandle,str);       // write the line      // if this is some sort of a back ground task...      if (i mod 1000) then        Application.process Messages;      // update  progress bar if you use one ...       end; // of for loop   closefile(fhandle);end; ---------------------------------------------------------------------- teochris · 2016-06-01 09:46 Quote from: tk on May 31, 2016, 10:00:09 pmI would implement some paging mechanism here to display reasonable amount of data at a time in a grid. And I would not even try to load all those data into memory, rather I would store them in a large file on disk and then read the given chunk into memory. TFileStream should be able to do it, or see here: http://stackoverflow.com/questions/4615286/what-is-the-fastest-way- for-reading-huge-files-in-delphi Your idea sounds really good but I am afraid I do not have such experience to do this kind of coding and I cannot find detailed tutorial for it! If you have something similar ready and you wish to shere it with us I maybe able to replicate it for my needs. I have a huge question about Lazarus knowledge that some of the expert users have - I do not know how I can gain this knowledge for Lazarus since there are no detailed tutorials for the most functions - I do know If I should have read more before started using Lazarus - I want to become an expert in Lazarus also BUT I do not know how I can do so! ---------------------------------------------------------------------- howardpc · 2016-06-01 10:46 Quote from: teochris on May 31, 2016, 09:28:35 pmThese data will not have any real use, I just want them to be shown to users because this will add a feature that similar programs like mine do not have at all and users of my app requested it from me What advantage is a useless feature? Similar programs omit this functionality because it is absurd. No one will waste time scrolling painfully slowly amongst 150 million cells in a grid to find data of interest, even the person(s) who asked for it. Far better would be to implement a filter or search function that would locate data of interest without labouring to possibly also display tens of millions of unwanted cells. ---------------------------------------------------------------------- tk · 2016-06-01 12:24 Quote from: teochris on June 01, 2016, 11:46:08 am If you have something similar ready and you wish to shere it with us I maybe able to replicate it for my needs. These were only my ideas, have never had the need to do something like this. More common is to use a database engine for storing such amount of data records and a dbgrid to display them. Paging mechanism could be then made with simple SQL commands. If you are an absolute beginner and don't know the basics I would start with reading some book. Personally I've started with book Mastering Delphi long time ago, you could start with similar newer book specific to Lazarus (see http://wiki.freepascal.org/Pascal_and_Lazarus_Books_and_Magazines). Not everything you will find in a book but believe me it is a good start. ---------------------------------------------------------------------- teochris · 2016-06-01 14:48 Quote from: tk on June 01, 2016, 02:24:48 pmQuote from: teochris on June 01, 2016, 11:46:08 am If you have something similar ready and you wish to shere it with us I maybe able to replicate it for my needs. These were only my ideas, have never had the need to do something like this. More common is to use a database engine for storing such amount of data records and a dbgrid to display them. Paging mechanism could be then made with simple SQL commands. If you are an absolute beginner and don't know the basics I would start with reading some book. Personally I've started with book Mastering Delphi long time ago, you could start with similar newer book specific to Lazarus (see http://wiki.freepascal.org/Pascal_and_Lazarus_Books_and_Magazines). Not everything you will find in a book but believe me it is a good start. Your book suggestions seems a big help for me - very soon I will start reading them - never done so in the past - I just learned pascal because I am a good mathematician and in the beginning of my programming I was looking for code in google to help me solve mathematical problems with pascal!