Sigsegv error during execution lazarus / 21-General ---------------------------------------------------------------------- daringly · 2021-05-25 17:12 I'm running a straightforward program that initially is just reading a lot of data. I'm reading and storing about a billion data points into record types. After about 14 million points (each record is about 80 bytes), I get a Sigsegv error on an array lookup that normally has no problem. When I look at the array right before the crash, the string entries are no longer readable. I have plenty of memory left -- Lazarus at its peak is using less than 8% of my memory (128G). If I run the program as an application instead of in the debugger, it still crashes at the same spot. Is there something you need to do to allow Lazarus to hoard memory? ---------------------------------------------------------------------- Handoko · 2021-05-25 17:39 Maybe it does not help but you can try. I had a computer that almost always crashed when it used a lot of memory. It had 8GB memory (2 x 4GB). I rarely run programs more than 4GB of memory. But weird things often happened if the memory usage was over 4GB. After some inspections, I suspected the second piece of the memory was the reason of the issue. Later I was sure after tested it using RAM testing software. There are many free RAM testing tools can be download from the Internet. Could it be any hidden bug in the program? You can enable runtime checks, recompile the program and test it. The program will run much slower if all the checks are enabled, so they are disabled by default. Lazarus main menu > Project > Project Options > left panel: Compiler Options > Debugging ---------------------------------------------------------------------- howardpc · 2021-05-25 18:18 Presumably your array is constructed on the heap, and not on the stack? ---------------------------------------------------------------------- daringly · 2021-05-25 18:51 var LookupArrays:array[1..4,1..1000] of string; The data I read has a string description. I convert the string into an integer so it won't take up as much space. It's fine at start, and for the first 10 million data points. At exactly datapoint 14,201,639, everything goes to hell. ---------------------------------------------------------------------- daringly · 2021-05-25 18:55 Here's the datastructure eating all the memory: type datap = record        x, y, z: longint;        desc1, desc2, desc3, desc4, desc5: real;        obs1, obs2, obs3, obs4, obs5: longint;      end;  var entries: array[1..20000000] of datap; If I make it much larger than 20 million, it will no longer run. It will create an application, but give me an error. Trying to run the app gives the error "This app won't run in Windows 10". If I try to run it in the debugger, I get a debugger error: "Oops, the debugger entered the error state Save your work now!" This isn't as critical as the other error, but it's annoying. Why am I limited, when I have so much more memory (128G) available? ---------------------------------------------------------------------- daringly · 2021-05-25 18:57 Quote from: Handoko on May 25, 2021, 07:39:31 pmCould it be any hidden bug in the program? You can enable runtime checks, recompile the program and test it. The program will run much slower if all the checks are enabled, so they are disabled by default. Lazarus main menu > Project > Project Options > left panel: Compiler Options > Debugging Thanks for the suggestion. I had tried this earlier and got no new insights. I tried again, and the Sigsegv error simply vanished. I'm baffled. ---------------------------------------------------------------------- daringly · 2021-05-25 19:20 The data structure seems to be limited to 2 gigs max. Is there a compiler option that prevents an array from being larger than that? ---------------------------------------------------------------------- loaded · 2021-05-25 19:33 If you are working with big data, compile your application in 64 bit. Of course, your operating system must be 64 bit. To remove the data limitation; Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$SetPeFlags $0020}   Type the command. ---------------------------------------------------------------------- daringly · 2021-05-25 19:59 Quote from: loaded on May 25, 2021, 09:33:58 pmIf you are working with big data, compile your application in 64 bit. Of course, your operating system must be 64 bit. To remove the data limitation; Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$SetPeFlags $0020}   Type the command. Thanks. I tried this, but any variable larger than 2 gigs still won't compile. Do I need another compiler directive? ---------------------------------------------------------------------- loaded · 2021-05-25 20:23 Quote from: daringly on May 25, 2021, 09:59:26 pmThanks. I tried this, but any variable larger than 2 gigs still won't compile. Do I need another compiler directive? No, scroll down under implementation, save and recompile. You should work smoothly afterwards, because it works for me. With this method, you can easily use up to 4 GB of memory, which has a 32-bit limit. Also ; You can instantly view how much memory your project consumes through process explorer. Attached is the 32 bit application I wrote and the amount of memory it uses. ---------------------------------------------------------------------- daringly · 2021-05-25 20:37 Quote from: loaded on May 25, 2021, 10:23:21 pmNo, scroll down under implementation, save and recompile. You should work smoothly afterwards, because it works for me. With this method, you can easily use up to 4 GB of memory, which has a 32-bit limit. I'm not sure what "implementation" is... but I saved and recompiled. I tried clean up and build also, but still stuck at the 2 Gig limit. 4 is better, but how to I use 64 Gigs at once? ---------------------------------------------------------------------- 440bx · 2021-05-25 21:45 Quote from: daringly on May 25, 2021, 10:37:22 pm I tried clean up and build also, but still stuck at the 2 Gig limit. A program is a PE file.  The PE file format, whether describing a 32bit or 64bit program, is limited to 2GB.  To have a really large data structure, use VirtualAlloc or VirtualAllocEx.  Using those APIs, you can have blocks hundreds of gigabytes large (on 64bit, obviously not on 32bit.) IOW, don't declare an array[gazillion] of somerecord.  Instead declare a pointer to an array of that type, allocate a block of memory using VirtualAlloc for the array then access the array elements.  The best, as well as fastest and easiest, is to access the array elements using pointer arithmetic.  IOW, you don't even declare the array, instead declare the record type, allocate memory for a gazillion records, e.g, recptr := VirtualAlloc (recordtype * gazillionrecords) then use recptr to access any record from 0 to gazillion - 1. HTH. ---------------------------------------------------------------------- PascalDragon · 2021-05-26 07:04 Quote from: 440bx on May 25, 2021, 11:45:03 pmIOW, don't declare an array[gazillion] of somerecord.  Instead declare a pointer to an array of that type, allocate a block of memory using VirtualAlloc for the array then access the array elements.  The best, as well as fastest and easiest, is to access the array elements using pointer arithmetic.  IOW, you don't even declare the array, instead declare the record type, allocate memory for a gazillion records, e.g, recptr := VirtualAlloc (recordtype * gazillionrecords) then use recptr to access any record from 0 to gazillion - 1. One doesn't need to go through this manually. Declaring a dynamic array (entries: array of datap) and then setting its length (SetLength(entries, 20000000)) is enough. Not everyone is against using what the language provides, 440bx. Please note: you need to compile as 64-bit to allocate more than 2 GB (or 3 GB with the PE directive mentioned by loaded). This also applies to the suggestion by 440bx of doing it manually. ---------------------------------------------------------------------- loaded · 2021-05-26 08:05 I have prepared a sample application for you so you can see the memory usage. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit Unit1; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, FileUtil, LResources, Forms, Controls, Graphics, Dialogs,  StdCtrls, jwapsapi, jwawinbase, jwawinnt,math; type   { TForm1 }   TForm1 = class(TForm)    Button1: TButton;    Memo1: TMemo;    procedure Button1Click(Sender: TObject);  private   public   end; var  Form1: TForm1; implementation   {$R *.lfm}{$SetPeFlags $0020} { TForm1 } //https://stackoverflow.com/a/29998974function ConvertBytes(Bytes: Int64): string;const  Description: Array [0 .. 8] of string = ('Bytes', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB');var  i: Integer;begin  i := 0;  while Bytes > Power(1024, i + 1) do Inc(i);  Result := FormatFloat('###0.##', Bytes / Power(1024, i)) + #32 + Descr iption[i];end; //https://forum.lazarus.freepascal.org/index.php?topic= 11992.0procedure Imem;var  psmemCounters: _PROCESS_MEMORY_COUNTERS;  ThisApp: LongWord;begin  ThisApp := GetCurrentProcessID();  ThisApp := OpenProcess(PROCESS_QUERY_INFORMATION or PROCESS_VM_READ, FALSE, ThisApp );  if( ThisApp = LongWord(0)) then    begin    Application.Messagebox('OpenProcess Error', 'Error', 0);    exit;    end;  if(GetProcessMemoryInfo(ThisApp, psmemCounters, sizeof(_PROCESS_MEMORY_COUNTERS)) <> LongBool(0)) then    begin    form1.Memo1.Clear;    form1.Memo1.lines.add('WorkingSetSize :' + ConvertBytes(psmemCounters.WorkingSetSize));    form1.Memo1.lines.add('PageFileUsage :' +  ConvertBytes(psmemCounters.PagefileUsage));    form1.Memo1.lines.add('PeakWorkingSetSize :' + ConvertBytes(psmemCounters.PeakWorkingSetSize));    form1.Memo1.lines.add('PeakPageFileUsage :' + ConvertBytes(psmemCounters.PeakPagefileUsage));    end  else    begin    Application.Messagebox('GetProcessMemoryInfo Error', 'Error', 0);  end;  CloseHandle(ThisApp);end; procedure TForm1.Button1Click(Sender: TObject);var  tmem:array of Double;  ti:integer=999999;  err:Boolean=false;begin  repeat    try      SetLength(tmem,Length(tmem)+ti);      tmem[high(tmem)]:=2021;      Imem;      Application.ProcessMessages;    except      on EOutOfMemory do err:=true;    end;  until err;  Imem;  ShowMessage('The memory limit available for use has been reached.');end;end.  If the application is stuck at 1.2 gb memory limit; There is a very interesting behavior I noticed, I don't know why, but it works.    Move the  Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$SetPeFlags $0020}  parameter above the  Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$R *.lfm} parameter, save the project, recompile and run it. ---------------------------------------------------------------------- 440bx · 2021-05-26 12:32 Quote from: PascalDragon on May 26, 2021, 09:04:25 amOne doesn't need to go through this manually. Declaring a dynamic array (entries: array of datap) and then setting its length (SetLength(entries, 20000000)) is enough. Not everyone is against using what the language provides, 440bx. I'm not against what the language provides.  I'm against not knowing exactly what is going on whenever reasonably possible.  Using VirtualAlloc, I can use an external memory viewer, easily identify the block of memory and inspect it to my heart's content.  Using what the language provides, I have to worry about what the language might decide to do that may be less than ideal. Fortunately, I believe we both live in free countries.  You can do it your way, which is completely fine with me while I keep it doing it my way. I'm not a diplomat but, I know how to compromise ---------------------------------------------------------------------- daringly · 2021-05-26 14:41 Thanks for all the help. The simplest solution worked nicely -- array of entry, followed by PascalDragon's solution of SetLength. I'm now doing data analytics on my desktop that shouldn't be possible. Setlength seemed willing to let me use more memory than I have. Will that cause page faulting later? I sometimes have to laugh at the absurdity of life. I'm using a desktop with lazarus on a big data project because it's so much faster than SQL or Python, and the ability to tightly control what I'm doing means I can turn massive projects around in hours or days instead of months. I started with Pascal 35 years ago, and it's wild how that one (not so refined) skill has pushed me along a single direction in life. ---------------------------------------------------------------------- 440bx · 2021-05-26 14:58 Quote from: daringly on May 26, 2021, 04:41:01 pmThanks for all the help. The simplest solution worked nicely -- array of entry, followed by PascalDragon's solution of SetLength. I'm now doing data analytics on my desktop that shouldn't be possible. Setlength seemed willing to let me use more memory than I have. Will that cause page faulting later? Allocating a larger block than there is memory in the system shouldn't be a problem as long as there is enough page file storage to back the allocation and enough address space to map the memory request. ---------------------------------------------------------------------- PascalDragon · 2021-05-27 07:00 Quote from: daringly on May 26, 2021, 04:41:01 pmI sometimes have to laugh at the absurdity of life. I'm using a desktop with lazarus on a big data project because it's so much faster than SQL or Python, and the ability to tightly control what I'm doing means I can turn massive projects around in hours or days instead of months. I started with Pascal 35 years ago, and it's wild how that one (not so refined) skill has pushed me along a single direction in life. If only more people would appreciate that instead of complaining about the syntax... ---------------------------------------------------------------------- tetrastes · 2021-05-27 08:49 Quote from: 440bx on May 26, 2021, 02:32:20 pmUsing VirtualAlloc, I can use an external memory viewer, easily identify the block of memory and inspect it to my heart's content.  Using what the language provides, I have to worry about what the language might decide to do that may be less than ideal. But with VirtualAlloc you are limited to Windows. And you always have sources of compiler if you "worry about what the language might decide to do". ---------------------------------------------------------------------- 440bx · 2021-05-27 12:49 Quote from: tetrastes on May 27, 2021, 10:49:45 amBut with VirtualAlloc you are limited to Windows. And you always have sources of compiler if you "worry about what the language might decide to do". You have a valid point but, generic/general solutions are rarely optimal.  Doing things "manually" has some disadvantages, portability being one of them but, it does have many advantages too.