Blockread 32-bit limit lazarus / 21-General ---------------------------------------------------------------------- dsbw · 2025-06-27 23:30 Here's some code I've used many times over the years, but which gives me a disk read error now: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);var f: File;   fs: Int64;    p: Pointer;begin   AssignFile(f, 'megagoals.txt');   reset(f, 1);   fs := FileSize(f);   p := GetMem(fs);   BlockRead(f, p^, fs);   Button1.Caption := IntToStr(fs)+ ' Bytes Read!';end;  If after getting the filesize, I do this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i    fs :=  2147483647;     It works. If I do this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i    fs :=  2147483648;     I get the error. This would seem to indicate there's a 32-bit signed integer in the codepile. But looking at it, it should use the 64-bit everything, as I've specified. Thoughts? I've looked for switches or some indication of something dropping this to 32-bits but haven't seen anything leap out at me. ---------------------------------------------------------------------- jamie · 2025-06-27 23:44 is your target 32 bit? ---------------------------------------------------------------------- dsbw · 2025-06-28 01:00 || is your target 32 bit? Presumably. But I can't figure out how. I looked at my GUI app and saw Win32 GUI app targeted so I unchecked that. Still got the same Disk Read error when I crossed that threshhold. Then I dropped out of the GUI builder to build a straight console app, targeted Win64. Same thing: 7 works, 8 fails. ---------------------------------------------------------------------- Handoko · 2025-06-28 01:47 I tested your code on Windows 11 64-bit Lazarus 3.8 64-bit with 2 files. If the file size was 2130180486, everything worked correctly. But if the file size was 2243830707, I got the error below: ---------------------------------------------------------------------- d2010 · 2025-06-28 02:59 Quote from: dsbw on June 28, 2025, 01:30:09 amThoughts? I've looked for switches or some indication of something dropping this to 32-bits but haven't seen anything leap out at me. You cannot overdraft the memory DDR3 with too-big Buffer, because anyone need work of buffer each-each by time, each each max20MB  per 01sec. If you increase the buffer upload to 1000MB, then You slow-down the speed of buffer, speed of 20MB, down to per60sec, from per01sec. Always, You must use fix buffer, not dynamically of size.(e.g. 2MB), even my files have the sizes 2MB, 1.79MB, 99MB, 50MB.. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TMainForm.Cd_or_DVdToIso(Const SrceFile,DestFile:string;Var cancel:boolean);var  src, dst: Integer;  buf: Pointer;  n, bytes, bytestocopy:Isscint23dll;  lastpart: array [0..4] of Integer;begin  if Tag = 0 then    begin      Tag := 1;      cancel := False;      src := INVALID_HANDLE_VALUE;      dst := INVALID_HANDLE_VALUE;      try      GetMem(buf, (1 shl 20)+(1 shl 20)); // 20 Megabyte ???        src := CreateFile(PChar(SrceFile), GENERIC_READ, FILE_SHARE_READ, nil, OPEN_EXISTING, 0, 0);        if (src = INVALID_HANDLE_VALUE) then        raise EInOutError.Create('Calendar cu texte din opera părintelui Arsenie Boca ');        dst := CreateFile(PChar(DestFile), GENERIC_WRITE, FILE_SHARE_READ, nil, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0);        if (dst = INVALID_HANDLE_VALUE) then          raise EInOutError.Create('Părintele Arsenie Boca cinstit ca Sfântul Cuvios Mărturisitor ...');        bytestocopy := SetFilePointer(src, 0, nil, FILE_END);        SetFilePointer(src, -sizeof(lastpart), nil, FILE_END);        if (not ReadFile(src, lastpart, sizeof(lastpart), n, nil)) then          raise EInOutError.Create('Sfântul Arsenie De La Prislop');..... You search ??? inside the code. ---------------------------------------------------------------------- Jorg3000 · 2025-06-28 04:14 Hi! I don't think it's a problem with GetMem, but that the file functions are not consistently 64 bit. BlockRead() internally calls the Do_Read() function, whose len parameter is only 32 bits. I think this is a bug. Procedure BlockRead(var f:File;var Buf;Count:Int64;var Result:Int64); function do_read(h:thandle; addr:pointer; len: longint): longint; ---------------------------------------------------------------------- MarkMLl · 2025-06-28 06:53 OP: you're asking a specific question in a general forum while giving absolutely no information on the platform(s) you're using. There's a whole lot of problems with various APIs on different OSes and processors which limit the size of transfers. The main issue is that they tend to be poorly-understood and worse-documented. MarkMLl ---------------------------------------------------------------------- marcov · 2025-06-28 08:20 Quote from: Jorg3000 on June 28, 2025, 06:14:10 amI don't think it's a problem with GetMem, but that the file functions are not consistently 64 bit. File pointers are 64-bit. Memory array sizes are a totally different angle. ---------------------------------------------------------------------- Hartmut · 2025-06-28 09:27 I can confirm that this is a bug at least in FPC 3.0.4 + 3.2.0. It occurs also on Linux in a 64-bit program. And blockwrite has the same bug! If you use {$I+} then an exception occurs. If you use {$I-} then function ioresult will return 100 or 101. Please file a bug report in https://gitlab.com/freepascal.org/fpc/source/-/issues and post it's link here, so that it can be found easily by people who have reached this topic. Thanks. ---------------------------------------------------------------------- jamie · 2025-06-28 09:59 ReadFile in widows uses a DWORD for count, not  Int64. Don't know if that helps at all but I do know that FPC likes to convert things to signed integers when it shouldn't. Jamie ---------------------------------------------------------------------- jamie · 2025-06-28 10:18 has anyone tried using a "Cardinal" instead for "fs" ? The limit is still DWORD and last time I knew, Cardinal was supposed to work for that, or maybe it don't in fpc. ---------------------------------------------------------------------- Warfley · 2025-06-28 10:48 If you want to read large files, map them into memory instead. On unix you can use mmap (fpmmap). On windows there is CreateFileMapping. With both of those you'll get a pointer to the raw data of the file you can access as if you read the whole file using a Filestream. But there are a two main advantages: 1. Multiple processes that map the same file as readonly will share the same virtual memory space for that file meaning the data is only read once 2. The file is not read as once but whenever you access a part of the file which was not loaded the OS will load it If you insist on reading a large file, you should split the read up into multiple smaller reads, preferrably not larger than a page size (4K), this way the OS level call will not block for as long reducing the chance of hitting a signal during the read, and also if a signal hits, which interrupts the read, you don't need to redo the whole read just the block. ---------------------------------------------------------------------- marcov · 2025-06-28 12:47 Quote from: Warfley on June 28, 2025, 12:48:49 pmIf you want to read large files, map them into memory instead. On unix you can use mmap (fpmmap). On windows there is CreateFileMapping. With both of those you'll get a pointer to the raw data of the file you can access as if you read the whole file using a Filestream. But there are a two main advantages: 1. Multiple processes that map the same file as readonly will share the same virtual memory space for that file meaning the data is only read once 2. The file is not read as once but whenever you access a part of the file which was not loaded the OS will load it That is an often made point, but there are also downsides to memory mapping, I actually once tried to build a MUD like game on top of memory mapping technology (admittedly in a time when 64MB was a lot of memory), but reverted to a more traditional caching (with a maximum amount of memory dedicated to the cache) to avoid my main datastructures being paged out and suffer bad response to other users (other than the one that did the massive query). ---------------------------------------------------------------------- PascalDragon · 2025-06-28 14:08 Quote from: dsbw on June 28, 2025, 01:30:09 amHere's some code I've used many times over the years, but which gives me a disk read error now: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);var f: File;   fs: Int64;    p: Pointer;begin   AssignFile(f, 'megagoals.txt');   reset(f, 1);   fs := FileSize(f);   p := GetMem(fs);   BlockRead(f, p^, fs);   Button1.Caption := IntToStr(fs)+ ' Bytes Read!';end;  If after getting the filesize, I do this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i    fs :=  2147483647;     Why would you even want to read such a big file at once? While storages become faster (~300 MB/s for fast HDDs, ~ 600 MB/s for S-ATA SSDs and >> 3 GB/s for current NVMe drives with PCIe 4.0 or newer) do you really not want to provide feedback for the user or give them the chance to abort an operation (depending on the use case)? The operating system will have to split up the operation anyway, cause the devices are not able to handle such sizes (e.g. USB devices are often limited to 1 or 2 MB per transfer and even NVMe drives have a speed cap at around 8 MB). Thus it would be better anyway to use smaller transfer sizes (e.g. 2 or 4 MB is rather nice) and read the file in multiple parts. Thus you can not only provide feedback to the user (especially on slower drives), but also won't have to deal with such size limits. ---------------------------------------------------------------------- dsbw · 2025-06-28 23:47 Thanks for the feedback, guys. It's much appreciated. Let me address some of the points raised and clarify what I'm doing at the bottom: QuoteOP: you're asking a specific question in a general forum while giving absolutely no information on the platform(s) you're using. So, pro forma, I should say I'm running on Windows 10 or 11. On the other hand, shouldn't a call with 64-bit integers be reliably callable across platforms? (Or it should raise an error where it can't be done.) Quoted2010 I'm sorry, I don't really understand what you're getting at. The mission is: Read the entire file into memory in one call. 25 years ago I could read a 1GB file into memory on a machine with 1GB, no problem, I feel like 35GB should be well within reach now for a machine with 64GB. (It's not, of course, which is why I'm here. ) Quote from: Jorg3000 on June 28, 2025, 06:14:10 amBlockRead() internally calls the Do_Read() function, whose len parameter is only 32 bits. I think this is a bug. I think it's a bug, too! I mean, maybe for whatever reason it can't be done, but I think it should be indicated somehow, somewhere. Quote from: jamie on June 28, 2025, 12:18:24 pmhas anyone tried using a "Cardinal" instead for "fs" ? I just did. It does not error out. However, it manages this by being completely wrong about the filesize, returning 1 billion something. Quote from: Warfley on June 28, 2025, 12:48:49 pmIf you want to read large files, map them into memory instead. No! Or, more politely, no, thank you!  (I'll explain at the bottom.) Quote from: Warfley on June 28, 2025, 12:48:49 pmWith both of those you'll get a pointer to the raw data of the file you can access as if you read the whole file using a Filestream. But there are a two main advantages: 1. Multiple processes that map the same file as readonly will share the same virtual memory space for that file meaning the data is only read once 2. The file is not read as once but whenever you access a part of the file which was not loaded the OS will load it 1. There are no multiple processes. 2. I am accessing it all at once. Serially, from top-to-bottom every time. Quote from: Warfley on June 28, 2025, 12:48:49 pmIf you insist on reading a large file, you should split the read up into multiple smaller reads, preferrably not larger than a page size (4K), this way the OS level call will not block for as long reducing the chance of hitting a signal during the read, and also if a signal hits, which interrupts the read, you don't need to redo the whole read just the block. I mean, I can stream the stupid thing in. I can break it into chunks. I'm coming from the modern dynamic language world which would love nothing more than to put it in a big bloated tree of some kind. I will elaborate further: Quote from: PascalDragon on June 28, 2025, 04:08:54 pmWhy would you even want to read such a big file at once? I'm glad you asked that!  Quote from: PascalDragon on June 28, 2025, 04:08:54 pmdo you really not want to provide feedback for the user or give them the chance to abort an operation (depending on the use case)? There is no user. There is no operation. There is only analysis. Quote from: PascalDragon on June 28, 2025, 04:08:54 pmThus it would be better anyway to use smaller transfer sizes (e.g. 2 or 4 MB is rather nice) and read the file in multiple parts. Thus you can not only provide feedback to the user (especially on slower drives), but also won't have to deal with such size limits. So, here's the situation: I'm slicing up a database table to learn about its contents. I say to my program "Here's a file. Figure out what's REALLY in it—because DB specs are lazy and sloppy and abused—and give me a report." Generally, I'm able to go through an entire table at once. This file is big enough to where I can only do individual fields, but those are in the tens of gigabytes. By far the most time consuming part of this process is reading from the disk. It's the difference between seconds or minutes (depending on filesize) and instantaneous. It's the difference between being able to run something dozens of times very quickly to figure out how to tweak parameters and filter out garbage, and having to wait ten minutes and then pick up context. I might have to go to C++.  Though, if it is an underlying OS issue, I guess that won't help. (I've been doing this so long, by the way, that I first had to read in 64KB blocks because THAT was the largest value you could blockread/blockwrite. It was needlessly complicated because the data generally comes in as a text file, and there's block boundaries don't fall neatly on line ends. I swore, with God As My Witness, I would never go back to trying read chunked files, once the 32-bit addressing came around.) ---------------------------------------------------------------------- 440bx · 2025-06-29 01:29 Quote from: dsbw on June 29, 2025, 01:47:58 amI might have to go to C++.  Though, if it is an underlying OS issue, I guess that won't help. You mentioned you're running under Windows.  Under Windows, ReadFile takes a 32bit value as the number of bytes to read, therefore Windows will force you to read "small chunks" of 4GBs.  NtReadFile which is what ReadFile uses, also uses a 32bit value to specify the number of bytes to read, therefore even using the native API forces the maximum size of a read to be 4GBs. Since file mapping is page based, I don't see that as a way to circumvent that limitation.  IOW, file mapping will make the O/S break I/Os in chunks of 1 page which may usually be 4K or 4MB, not anything greater than that. It doesn't look like there is a way around performing multiple reads to read files larger than 4GBs.  BlockRead's int64 count parameter won't work under Windows and any value greater than high(DWORD) is invalid and will not deliver the expected result. ETA: Actually, to be precise, the 32bit value limits the size of the read buffer.  Consequently it is not possible to read more than 4GB because the maximum buffer size is limited to 4GB.  That's imposed by NtReadFile. ---------------------------------------------------------------------- ASerge · 2025-06-29 02:20 Quote from: dsbw on June 29, 2025, 01:47:58 amThis file is big enough to where I can only do individual fields, but those are in the tens of gigabytes. You can use memory mapping in Windows. This example is for a 27 GB file. The reading is very fast (less than 1 second): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit Unit1; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls; type  TForm1 = class(TForm)    btnTest: TButton;    procedure btnTestClick(Sender: TObject);  private  public  end; var  Form1: TForm1;                implementation {$R *.lfm} uses RtlConsts, Windows; type  TMemMapRead = class(TCustomMemoryStream)  strict private    FMapHandle: THandle;    procedure CloseMapping;  public    constructor Create(const FileName: string; ShareWrite: Boolean = False);    function Write(const Buffer; Count: LongInt): LongInt; override;    destructor Destroy; override;  end; { TMemMapRead } constructor TMemMapRead.Create(const FileName: string; ShareWrite: Boolean);const  ShareFlag: array[Boolean] of DWORD = (FILE_SHARE_READ, FILE_SHARE_READ or FILE_SHARE_WRITE);var  FileSize: LARGE_INTEGER;  HFile: THandle;  P: Pointer;begin  inherited Create;  HFile := CreateFileW(PWideChar(UTF8Decode(FileName)), GENERIC_READ,    ShareFlag[ShareWrite], nil, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);  if HFile = INVALID_HANDLE_VALUE then    RaiseLastOSError;  FileSize.LowPart := GetFileSize(HFile, @FileSize.HighPart);  if FileSize.QuadPart > 0 then  begin    FMapHandle := CreateFileMapping(HFile, nil, PAGE_READONLY, 0, 0, nil);    CloseHandle(HFile);    if FMapHandle = 0 then      RaiseLastOSError;    P := MapViewOfFile(FMapHandle, FILE_MAP_READ, 0, 0, 0);    if P = nil then      RaiseLastOSError;    SetPointer(P, FileSize.QuadPart);  end  else  begin    CloseHandle(HFile);    SetPointer(nil, 0);  end;end; procedure TMemMapRead.CloseMapping;begin  if Memory <> nil then  begin    UnmapViewOfFile(Memory);    SetPointer(nil, 0);  end;  if FMapHandle <> 0 then  begin    CloseHandle(FMapHandle);    FMapHandle := 0;  end;end; destructor TMemMapRead.Destroy;begin  CloseMapping;  inherited;end; function TMemMapRead.Write(const Buffer; Count: LongInt): LongInt;begin  raise EStreamError.CreateRes(@SWriteError);end; { TForm1 } procedure TForm1.btnTestClick(Sender: TObject);var  Mem: TMemMapRead;begin  Mem := TMemMapRead.Create('D:\Users\Serge\VirtualBox VMs\Work Stations\MSEdge - Win10\MSEdge - Win10-disk002.vmdk');  try    Mem.Position := 4*1024*1024*1024 + 1;    Caption := Mem.ReadDWord.ToString;  finally    Mem.Free;  end;end; ---------------------------------------------------------------------- Hartmut · 2025-06-29 05:26 Quote from: Warfley on June 28, 2025, 12:48:49 pmIf you want to read large files, map them into memory instead. On unix you can use mmap (fpmmap). On windows there is CreateFileMapping. A cross-platform alternative to this is to use package mORMot2 which runs on Windows/Linux/BSD/MacOS. For memory mapping of files use type 'TMemoryMap' in unit mormot.core.os = https://synopse.info/files/doc/api/mormot.core.os.html#TMEMORYMAP For blockread and blockwrite I found out 2 additional important things: 1) If you want to use blockread for files >= 2 GB, you can use a loop and read chunks of up to 2 GB. But do not use a maximum size of 2,147,483,647 = $7FFFFFFF bytes! Instead use a maximum size of 2,147,479,552 = $7FFFF000. Depending of the low 12 bits of your buffer address bigger values may fail. Same for blockwrite. Tested with FPC 3.2.0 on Linux in a 64-bit program. 2) For blockread and blockwrite there exist alternative calls with a 'Result': Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure BlockRead(var f: File; var Buf; Count: Int64; var Result: Int64);procedure BlockWrite(var f: File; const Buf; Count: Int64; var Result: Int64);Usually  programmers use this alternative for reading, if they don't know exactly, how many bytes are left to read. And if 'Result' is < 'Count' they stop reading, thinking they have reached the EOF. But this is wrong if 'Count' is >= 2 GB. In this case 'Result' will never be > $7FFFF000. If you then stop reading, you miss the complete rest of the file. Same for blockwrite. Tested with FPC 3.2.0 on Linux in a 64-bit program. ---------------------------------------------------------------------- jamie · 2025-06-29 13:04 Let the poster do there C++ version if they are so confident, they will find the same limitations! Jamie ---------------------------------------------------------------------- Thaddy · 2025-06-29 13:45 Yes, pretty funny that he can dp better elsewhere. As you remark, same problem is C++. ---------------------------------------------------------------------- dsbw · 2025-06-30 01:39 QuoteLet the poster do there C++ version if they are so confident, they will find the same limitations! QuoteYes, pretty funny that he can dp better elsewhere. As you remark, same problem is C++. Guys, it's literally on the same line I wrote: QuoteI might have to go to C++.  Though, if it is an underlying OS issue, I guess that won't help. The first things I saw implied it was an FPC issue. It appears not to be! Quote from: ASerge on June 29, 2025, 04:20:41 amYou can use memory mapping in Windows. This example is for a 27 GB file. The reading is very fast (less than 1 second): Thank you very much for this. That code is fast because you're making a single call to a random position. Which is cool. But if you read the entire file serially, you'll find it's quite slow. (My 36GB file takes over 2 minutes.) Lacking a solution, I broke down and read the file in 2GB chunks, stitched the stragglers at the end of every block to the beginning of the next block, and reading the file 3 times took less than 20 seconds. Then I discovered that was due to a bug that broke strings.   :'( It was sorta funny. If I set my block size to: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i BlockSize: Int64 = 2048 * 1024 * 1024; It came back blazingly fast, but for some reason started capping the strings it was finding at 255 bytes. If I do this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i BlockSize: Int64 = 2047 * 1024 * 1024; // 64MB chunks instead of 2GB It works but alas, is nowhere near as fast. Final test: Reading the file in 2GB chunks is only about 20% faster (120 seconds to 100 seconds). I have to decide whether that's worth the code overhead. ---------------------------------------------------------------------- 440bx · 2025-06-30 03:04 Quote from: dsbw on June 30, 2025, 03:39:40 amThank you very much for this. That code is fast because you're making a single call to a random position. Which is cool. But if you read the entire file serially, you'll find it's quite slow. (My 36GB file takes over 2 minutes.) You can combine @ASerge's solution with ReadFile. Basically, using ReadFile you can read as much as 4GB in one shot into the file mapping (at least in theory.)  IOW, you don't need to read one page at a time (which will have a tendency to be slow because every time a page is accessed for the first time it causes an exception handled in ring-0 to associate physical memory with the page.) Doing it that way, the only thing you have to bother yourself with is calculating the correct offset in the mapping for every read.   Quite simple actually and extremely quick. If you want to maximize speed, you could use ReadFileEx and asynchronously read blocks into the mapping and process already read blocks while new ones are being read.  Disclaimer: I know it can be done but, I've never indulged into it because I didn't run into a case where the performance gain would be worth the additional code complexity. ETA: I forgot to mention one thing. The first time you allocate a large block of memory, Windows normally allocates NO memory at all, it only gives the program address space.  if the space is committed then Windows will automatically associate a page of real memory with that address whenever the address is touched (read or written.)  The net effect of that is, the first time a page is accessed a transition to ring-0 takes place for the O/S to map physical memory to that virtual address.  This will have one very noticeable side effect: the first time the file is read will take noticeably longer than the subsequent times because the first time incurs all those transitions for the O/S to associate real memory with a virtual address. You want to make it as fast as ring-3 code can make it ? ... use VirtualAllocEx to map a fairly large buffer (multi-gigabyte) while specifying MEM_LARGE_PAGES.  That causes the O/S to associate physical memory with the entire range of address space requested.   It also means the allocation will take significantly longer than a normal 4K page allocation because the O/S is actually managing memory to satisfy the request.  If that sounds good to you, there are significant downsides to that method: the first one is that enough physical memory in contiguous blocks of 4MB need to be available to satisfy the request.  IOW, it is a very real possibility that the request may fail due to insufficient memory because there aren't enough 4MB contiguous blocks to cover the requested range.  The other significant problem is that the memory used is NOT paged, it is locked and used exclusively by your process.  IOW, you can very easily make one of today's fastest machine be slower than a 64KB original IBM PC by allocating memory that way.  Succinctly, it is the easiest way to starve the system of memory.   Fortunatley, in addition to all that, you need SeLockMemoryPrivilege to do that. ---------------------------------------------------------------------- dsbw · 2025-06-30 18:41 That's interesting. Yeah, I did see the whole issue of Windows (particularly 11?) being sort of haphazard with regard to memory and making it hard to allocate huge chunks. Thanks for the feedback! ---------------------------------------------------------------------- Thaddy · 2025-07-01 08:32 It is not a Windows only issue, although 440bx answer is Windows centric: the same issue happens on Unixes. On such big allocations, one usually uses windowed streams for - sequential - access. Otherwise an SQL or NOSQL database which has planar access. These can be allocated in memory. ---------------------------------------------------------------------- creaothceann · 2025-08-04 17:07 Raymond Chen has commented on this issue: https://devblogs.microsoft.com/oldnewthing/20250804-00/?p=111432 ---------------------------------------------------------------------- Thaddy · 2025-08-04 19:54 ---------------------------------------------------------------------- Warfley · 2025-08-04 21:00 Quote from: dsbw on June 30, 2025, 03:39:40 amFinal test: Reading the file in 2GB chunks is only about 20% faster (120 seconds to 100 seconds). I have to decide whether that's worth the code overhead. This should not be the case. The slowest thing about reading a file is of course the IO, if you got an SSD, there should be absolutely no difference between a one time read and many small reads. The only difference between small and big reads are 1. the context switch when calling a kernel function (around a few milliseconds), which in terms of computer speeds is a lot (normal function calls are in the order of nanoseconds) 2. The speed of your code when doing the read. Both of which is negligable compared to the IO speed, once you hit a certain threshold. I quickly written a chunked read function, which just takes a file and reads the whole contents into a continuous block of memory (byte array for convinience): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses  Classes, SysUtils, Math; function ReadFile(const FName: String; ChunkSize: Integer): TBytes;var  fs: TFileStream;  head: SizeInt;begin  Result:=nil;  fs:=TFileStream.Create(FName, fmOpenRead);  try    SetLength(Result, fs.Size);    head:=0;    while head 0 do begin  if Size > MaxFileSize32 then    CurrentSize := MaxFileSize32  else    CurrentSize := Size;  ReadFile(Buffer, CurrentSize);  Size -= CurrentSize;  Buffer += CurrentSize;end; File mapping would only be more effective in case of lack of memory. Why? Because mapped file isn't swapped. It works as swapfile by itself. ---------------------------------------------------------------------- 440bx · 2025-08-04 22:25 Another thing that greatly affects read speed is file fragmentation.  If the file is in a single contiguous block in adjacent tracks that will provide good performance.  If the file sectors are all over the place, that's a recipe for poor performance no matter the buffer size.  Of course, SSDs are not nearly as sensitive to the equivalent problem. Regardless of O/S the underlying drive hardware also has a significant impact on read/write performance and, both the hardware and the algorithms used are optimized for the common case, which doesn't include reading 4GB (or much less for that matter) of data in a single I/O. ---------------------------------------------------------------------- PascalDragon · 2025-08-05 19:43 Quote from: 440bx on August 05, 2025, 12:25:12 amRegardless of O/S the underlying drive hardware also has a significant impact on read/write performance and, both the hardware and the algorithms used are optimized for the common case, which doesn't include reading 4GB (or much less for that matter) of data in a single I/O. And most hardware will have to split up such transfers anyway. E.g. many USB sticks and bridges only really support up to 2 MiB per transfer and above that might even become unstable or throw errors.