Out of memory ?
lazarus / 34-Windows

----------------------------------------------------------------------
Nitorami · 2023-10-05 10:23

Hi,
I am on windows 10 version 22H2 with 32GB RAM, and working with a
large data file. Approx 2.4GB of IEEE double values. I can view and
process all data using a Python viewer, no problem.

But when reading the data with my tool written in FPC, I get an "out
of memory" exception whenever trying to read more than approximately
2GB into dynamic arrays. It does not matter how I split the data into
individual arrays, the total memory seems to be limited. I have
terminated all other memory hungry processes, and the task manager
reports 22 GB available memory, but I still get the exception.

FPC 3.2.2, textmode IDE, 32bit. Is there a global 2GB memory limit for
the 32 bit version ? I thought a limitation only applies to the size
of a single memory block ?

----------------------------------------------------------------------
BlueIcaro · 2023-10-05 11:07

Hi, can you show the code or a simple example?.

/blueIcaro

----------------------------------------------------------------------
Zvoni · 2023-10-05 11:23

Quote from: Nitorami on October 05, 2023, 12:23:58 pmHi,
I am on windows 10 version 22H2 with 32GB RAM, and working with a
large data file. Approx 2.4GB of IEEE double values. I can view and
process all data using a Python viewer, no problem.

But when reading the data with my tool written in FPC, I get an "out
of memory" exception whenever trying to read more than approximately
2GB into dynamic arrays. It does not matter how I split the data into
individual arrays, the total memory seems to be limited. I have
terminated all other memory hungry processes, and the task manager
reports 22 GB available memory, but I still get the exception.

FPC 3.2.2, textmode IDE, 32bit. Is there a global 2GB memory limit for
the 32 bit version ? I thought a limitation only applies to the size
of a single memory block ?
AFAIK, 32-Bit can only address max. 4GB Memory.
It doesn't matter if you have a shitload of RAM.
Your program itself can only "use" max. 4GB (in reality way less than
4GB).
Recompile/rewrite your App as 64-Bit

----------------------------------------------------------------------
mika · 2023-10-05 11:27

could not reproduce on linux 64bit and  fpc322 32bit.
my test program
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
getmem4Gb ; const size = 1024*1024*102;var p : pbyte;     q : qword; 
  d : dword;     n : dword;begin     writeln;     d:=size;     n:=0;  
   {$ifdef cpu32 } writeln('it is 32bit cpu indeed');{$endif}   
 q:=0;     if d>0 then     while q < size*100 do     begin         
inc(n);          q:=q+d;          write(n:4,'  ');         
getmem(p,d);          writeln('Got in total ',q,' bytes
',(q/1024/1024):4:2,' mb   ',(q/1024/1024/1024):4:2,' gb');         
fillchar(p^,d,6);     end;     if d>0 then freemem(p,d);   
 writeln('ok');     writeln;end.
output
Quote...
  36  Got in total 3850371072 bytes 3672.00 mb   3.59 gb
  37  Got in total 3957325824 bytes 3774.00 mb   3.69 gb
  38  Got in total 4064280576 bytes 3876.00 mb   3.79 gb
  39  Got in total 4171235328 bytes 3978.00 mb   3.88 gb
  40  Runtime error 203 at $080624FE
  $080624FE
maybe you are using some integer data type that messes things up over
2gb?

----------------------------------------------------------------------
Nitorami · 2023-10-05 11:32

I just made a similar test program, and it excepts at 1.79GB. Same as
your program does. Interesting.

----------------------------------------------------------------------
Jorg3000 · 2023-10-05 11:57

I once created a Delphi 7 program 10 years ago, that also ran out of
memory at about 1.7 GB.
I then googled that Windows 32-bit only provides a maximum of 1.7 GB
heap.
If you need more, you can't avoid 64 bit.

----------------------------------------------------------------------
Nitorami · 2023-10-05 12:22

But why are 3.9GB allocated then for Mika ?

----------------------------------------------------------------------
440bx · 2023-10-05 12:23

A 32bit Windows program is limited to 2GB of address space.  That is
the standard address space Windows gives a 32bit process.

IF (note the big IF) you mark the 32bit program as "Large address
space aware" AND the 32bit program is running on a 64bit version of
Windows then Windows will give the process 4GB of address space
instead of the standard 2GB.

Very important note: that 4GB is NOT contiguous address space because
the 32bit Windows dll that implement the 32bit subsystem remain at the
standard addresses thereby splitting the 4GB into two (2) large areas.

Summary: you have two solutions if running on a 64bit version of
Windows: 1. mark the 32bit program as "large address space aware" and
deal with the address space discontinuity or 2. compile the program to
be a 64bit program.

if you want the program to use that much data while remaining a 32bit
program and still be able to run on a 32bit version of Windows then
the solution is to map partial sections of the file.  Based on the
2.4GB size you mentioned, a couple of 1.2GB sections might do the
trick for you.  Note that you can only map _one_ section at a time
(since there isn't enough address space to map it as a single
section.)

HTH.

ETA: corrected a typo.

----------------------------------------------------------------------
Nitorami · 2023-10-05 12:25

Thanks for the competent explanation.

----------------------------------------------------------------------
440bx · 2023-10-05 12:30

Quote from: Nitorami on October 05, 2023, 02:25:32 pmThanks for the
competent explanation.
You're welcome and follow this link to see example code on of how to
deal with the problem https://forum.lazarus.freepascal.org/index.php/t
opic,57142.msg425272.html#msg425272