TFileStream file size limit lazarus / 64-FPC development ---------------------------------------------------------------------- lagprogramming · 2021-10-04 08:45 Hi!    When trying to save a file in a linux-x86_64 environment, an application returns an error. I've noticed that the error appears when the file exceeds the maximum value of a longint. Looking at the assembler window of Lazarus I've assumed that the error was related to the procedure TStream.WriteBuffer(const Buffer; Count: Longint); that is found in rtl/objpas/classes/streams.inc. Because the situation appears to be simmilar I've modified both TStream.WriteBuffer(const Buffer; Count: Longint) and TStream.ReadBuffer(var Buffer; Count: Longint); from Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TStream.ReadBuffer(var Buffer; Count: Longint); Var  r,t : longint; begin  t:=0;  repeat    r:=Read(PByte(@Buffer)[t],Count-t);  inc(t,r);  until (t=Count) or (r<=0);  if (tmaxlongint then r:=maxlongint;    r:=Read(PByte(@Buffer)[t],r);    inc(t,r);  until (t=Count) or (r<=0);  if (tmaxlongint then r:=maxlongint;     r:=Write(PByte(@Buffer)[t],r);       inc(t,r);    Until (t=count) or (r<=0);    if (t2 or 4GB) in one go The first should be ok, the second is not yet supported. Afaik the variable declaration was considered not worth it, and many 64-bit targets also don't support it. So since the safest blocksize is probably 2GB-1, easiest is to write in one GB increments. p.s. It might you hit some overload issue somewhere with your recompile. ---------------------------------------------------------------------- lagprogramming · 2021-10-04 12:55 The following code fails to write to file Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure writefail;var  x:TFilestream;  p:pointer;begin  p:=getmem(maxlongint+1);  x:=TFilestream.Create('/home/user/bin.txt',fmcreate);  x.WriteBuffer(p^,maxlongint+1);//Fails to write garbage in linux-x86_64  x.Free;end;    If procedure TStream.ReadBuffer(var Buffer; Count: Longint) and procedure TStream.WriteBuffer(const Buffer; Count: Longint) should be left intact then wouldn't be a good idea to modify the following two procedures?   Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TStream.ReadBuffer(var Buffer: TBytes; Offset, Count: NativeInt);begin  ReadBuffer(Buffer[OffSet],Count);end; procedure TStream.WriteBuffer(const Buffer: TBytes; Offset, Count: NativeInt);begin  WriteBuffer(Buffer[Offset],Count);end;    If the above code would be modified adding a repeat/while loop, fpc might have two routines that work as expected. In my point of view it should be better than writing new procedures within local units of developing projects, procedures that would try to do what a developer expects from the above procedure declarations. ---------------------------------------------------------------------- korba812 · 2021-10-04 13:28 Quote from: lagprogramming on October 04, 2021, 02:55:54 pm   The following code fails to write to file Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure writefail;var  x:TFilestream;  p:pointer;begin  p:=getmem(maxlongint+1);  x:=TFilestream.Create('/home/user/bin.txt',fmcreate);  x.WriteBuffer(p^,maxlongint+1);//Fails to write garbage in linux-x86_64  x.Free;end;The result of "MaxLongint + 1" is Int64. Try "MaxLongint - 1". ---------------------------------------------------------------------- marcov · 2021-10-04 14:00 If the function has a type that is limited to 2GB-1, why he would try 2GB or more is a mystery to me.  Note that the upper limit is not 64-bit, but how large a block an application can allocate , which depends on a lot of things. Anyway, afaik there are reports for them, search them and comment on them. Keep in mind that  adding a loop to those functions adds code for people using this for short writes, just to benefit the few people creating a disk/ssd exerciser/benchmark or so doing gigantic writes. ---------------------------------------------------------------------- Thaddy · 2021-10-04 14:31 AFAIK Address space is limited to high(qword), at lot.., but single read/writes are limited to high(nativeint) which is indeed 2G Note that in practise it limits to available memrory otherwise a EOutOfMemory is thrown. Note that for 32 bit WIN there is a PE flag - since windows 7 - that extends available memory to 4G but still with the limitation ffor single read/writes ---------------------------------------------------------------------- Thaddy · 2021-10-04 14:39 Quote from: korba812 on October 04, 2021, 03:28:12 pmThe result of "MaxLongint + 1" is Int64. Try "MaxLongint - 1". In Freepascal simply use High(); ---------------------------------------------------------------------- marcov · 2021-10-04 14:53 Quote from: Thaddy on October 04, 2021, 04:31:14 pmAFAIK Address space is limited to high(qword), at lot.., but single read/writes are limited to high(nativeint) which is indeed 2G size_t does not need to cover the whole address space. If the kernel doesn't hand out such large consecutive blocks to applications, it is only a waste of bits. ---------------------------------------------------------------------- Thaddy · 2021-10-04 15:06 true. btw I should have written High(NativeInt). And that on 32 bit the flag for windows is IMAGE_FILE_LARGE_ADDRESS_AWARE. I don't know if you defined those consts in windows.pas. See MSDN or https://docwiki.embarcadero.com/RADStudio/Alexandria/en/PE _(portable_executable)_header_flags_(Delphi) The value of the flag is $20 ---------------------------------------------------------------------- marcov · 2021-10-04 15:08 Image header structs and constants have been reworked one or two years ago. Doesn't mean it will be perfect, but most should be there, including fairly recent stuff. ---------------------------------------------------------------------- Thaddy · 2021-10-04 15:15 Yes, you are doing a good job there. Tnx ---------------------------------------------------------------------- lagprogramming · 2021-10-04 15:51 Targeting linux-x86_64 we have: sizeof(nativeint)=sizeof(sizeuint)=sizeof(int64)=8 high(nativeint)=high(int64) sizeof(longint)=sizeof(longword)=4    In streams.inc we have: procedure TStream.ReadBuffer(var Buffer: TBytes; Offset, Count: NativeInt); procedure TStream.WriteBuffer(const Buffer: TBytes; Offset, Count: NativeInt);    The purpose of having these two procedures declared in rtl knowing that it's guaranteed to have a failure if you pass a Count value greater than maxlongint remains a mistery to me.    Thank you for your patience. ---------------------------------------------------------------------- marcov · 2021-10-04 15:57 Quote from: lagprogramming on October 04, 2021, 05:51:10 pm   In streams.inc we have: procedure TStream.ReadBuffer(var Buffer: TBytes; Offset, Count: NativeInt); procedure TStream.WriteBuffer(const Buffer: TBytes; Offset, Count: NativeInt); I missed that, it is still longint in 3.2.x.  I don't know who changed that (michael?) Please file a bug for the trunk issue. ---------------------------------------------------------------------- PascalDragon · 2021-10-05 07:15 Quote from: Thaddy on October 04, 2021, 04:31:14 pmAFAIK Address space is limited to high(qword), at lot.., but single read/writes are limited to high(nativeint) which is indeed 2G High(NativeInt) on a 64-bit system is equal to High(Int64). ---------------------------------------------------------------------- Thaddy · 2021-10-05 07:47 In de context of 32 bit, Sarah. That's why I suggested it to express address space. ---------------------------------------------------------------------- PascalDragon · 2021-10-05 11:32 Quote from: Thaddy on October 05, 2021, 09:47:15 amIn de context of 32 bit, Sarah. That's why I suggested it to express address space. You don't mention 32-bit here anywhere except for the PE flag (and the other messages before yours hadn't mentioned 32-bit either): Quote from: Thaddy on October 04, 2021, 04:31:14 pmAFAIK Address space is limited to high(qword), at lot.., but single read/writes are limited to high(nativeint) which is indeed 2G Note that in practise it limits to available memrory otherwise a EOutOfMemory is thrown. Note that for 32 bit WIN there is a PE flag - since windows 7 - that extends available memory to 4G but still with the limitation ffor single read/writes ---------------------------------------------------------------------- lagprogramming · 2021-10-17 14:54 I've stopped using the file related classes like TFileStream and went to a lower level: FileOpen, FileCreate, FileClose, FileSeek and so on.    The following function returns a negative number when passing a 3GB file. The problem is at FileSeek. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Function GetFileSizeUsingFileSeek(FilePath:string):Int64;var  FileH:THandle;begin  FileH:=FileOpen(FilePath,fmOpenRead);  Result:=FileSeek(FileH,0,fsFromEnd);//Also FileSeek returns a wrong result for files greater than 2GB. In linux-x86_64 it may return negative values.  FileClose(FileH);end;    In order to avoid such problems I've decided to modify the file format that is used by the application and avoid file related classes by using low level routines, which means that for me the problem will be solved soon. To be frank, I've never expected such a problem. Probably I'm one of the few who uses files so large in size. In the future I expect other developers to have the same problem. ---------------------------------------------------------------------- AlexTP · 2021-10-17 15:04 Quote The following function returns a negative number when passing a 3GB file. The problem is at FileSeek. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Function FileSeek (Handle : THandle; FOffset, Origin: Longint) : Longint;Function FileSeek (Handle : THandle; FOffset: Int64; Origin: Longint) : Int64; your code calls 1st overload! so it returns nagative res. try to call 2nd overload. Result:=FileSeek(FileH,Int64(0),fsFromEnd); ---------------------------------------------------------------------- Thaddy · 2021-10-17 15:05 The problem is merely the declaration as a signed type instead of unsigned. ---------------------------------------------------------------------- SymbolicFrank · 2021-10-26 21:30 What is the speed increase of using the largest buffer the OS will give you over that 1 GB buffer? Is that measurable? I mean, you need at least an SSD, which write in blocks the size of megabytes. Many random reads will overflow their cache memory, because the individual blocks are too small. But that's just the thing SSDs excel at: a very large amount of random writes. Because there is no mechanical movement. No need to wait for the head to slowly travel to the right location. For the fastest possible, sustained sequential write of a large file, the main thing is to kill all other applications that read or write files, or even use the PCI bus. Kill all other running programs. And at that point, it doesn't matter how large your buffer is, because the tiny overhead is washed away by having all available bandwidth for yourself. Operating systems are quite good at optimizing the available resources. ---------------------------------------------------------------------- PascalDragon · 2021-10-27 07:33 Quote from: SymbolicFrank on October 26, 2021, 11:30:21 pmWhat is the speed increase of using the largest buffer the OS will give you over that 1 GB buffer? Is that measurable? Please note that this will depend on the hardware. If the device is connected by USB for example the transfer size is caped at 1 MiB or at most 2 MiB cause larger sizes tend not to be supported that well by hardware. ATA, SCSI and NVMe also have some limits. The OS will handle that transparently for you of course, but this might result in copying and thus decreased performance compared to manually splitting the transfers (of course one needs to know the best transfer size then...). ---------------------------------------------------------------------- marcov · 2021-10-27 08:12 Quote from: SymbolicFrank on October 26, 2021, 11:30:21 pmWhat is the speed increase of using the largest buffer the OS will give you over that 1 GB buffer? Is that measurable? I mean, you need at least an SSD, which write in blocks the size of megabytes. Many random reads will overflow their cache memory, because the individual blocks are too small. Most SSD DRAM cache are in the single GIGAbyte order too, at best. And some of those many small writes might vacate the cache faster than one large big block (depending on the firmware) post edited, one gigabyte as magnitude, not megabyte of course ---------------------------------------------------------------------- SymbolicFrank · 2021-10-27 09:13 So, the difference between one and many IOPS? Then again, the OS will try to use all free memory as disk cache as well. What is faster, a single, large buffer, managed by the application and only a small cache, managed by the OS, or the other way around? Does the OS limit the cache size for a single IOP? ---------------------------------------------------------------------- marcov · 2021-10-27 12:34 Quote from: SymbolicFrank on October 27, 2021, 11:13:58 amSo, the difference between one and many IOPS? IOPS where? Between application and kernel (iow syscall), between kernel and device controller, or between QuoteThen again, the OS will try to use all free memory as disk cache as well. But if it must hold a part of a large buffer till the controller is ready for it, that is waste too, since the already written part could have been returned to the OS. QuoteWhat is faster, a single, large buffer, managed by the application and only a small cache, managed by the OS, or the other way around? Does the OS limit the cache size for a single IOP? For the last bit: seems win7 at least had a 2GB limit https://community.osr.com/discussion/193831/limitations-on-dma- transfer-size In general, the rule of thumb is that reducing of number of IOPS only improves performance logarithmically, while extremely large buffers might hit all kind of small inefficiencies. Note also that many DMA controllers (even on embedded ARMs and MIPS like PIC32) have scatter/gather DMA, which allow to write multiple buffers in one go. (IOPS). Also most IOPS benchmarks are for large (high core/socket count) transactional system with multiple threads doing independent (non related) I/O, and such benchmarks scale badly to synchonous (one thread) I/O systems because all time waiting while the device to complete a transaction is lost IOPS. ---------------------------------------------------------------------- SymbolicFrank · 2021-10-27 13:06 In other words: do use a buffer and don't write each byte individually, but leave the rest to the OS and hardware? ---------------------------------------------------------------------- marcov · 2021-10-27 13:19 Quote from: SymbolicFrank on October 27, 2021, 03:06:59 pmIn other words: do use a buffer and don't write each byte individually, but leave the rest to the OS and hardware? Do use a buffer, but 32kb to 1mB is quite alright for most things. Maybe if you write a disk clone software higher buffers are more worthwhile, but even that will have decreasing returns while the buffer gets larger. I can't really imagine 1GB buffers adding much. ---------------------------------------------------------------------- SymbolicFrank · 2021-10-27 15:40 Agreed. ---------------------------------------------------------------------- PascalDragon · 2021-10-28 07:21 Quote from: marcov on October 27, 2021, 03:19:26 pmMaybe if you write a disk clone software higher buffers are more worthwhile, but even that will have decreasing returns while the buffer gets larger. Our main product at work is disk cloning software and we see a decrease already with 4 or 8 MiB buffers. ---------------------------------------------------------------------- lagprogramming · 2022-01-22 13:57 I expect this to be related to the subject.    TMemoryStream.SaveToFile silently writes an empty file if the stream size is greater than the maximum value of a longint, at least in a linux-x86_64 environment. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure WriteGarbage(const FileAddr:string; const GarbageSize:longword);var s:TMemoryStream;begin  s:=TMemoryStream.Create;  s.SetSize(GarbageSize);  s.SaveToFile(FileAddr);  s.Free;end; procedure TForm1.Button1Click(Sender: TObject);begin  writegarbage('garbagelongint.bin', high(longint));//Works OK!  writegarbage('garbagelongword.bin', high(longint)+1024*1024*1024);//FAILURE!!!end; ---------------------------------------------------------------------- ASerge · 2022-01-22 14:20 I think this is a question of a separate topic. In Windows, a zero file of 3 GB is written. ---------------------------------------------------------------------- lagprogramming · 2022-01-22 16:27 Quote from: ASerge on January 22, 2022, 03:20:47 pmI think this is a question of a separate topic. In Windows, a zero file of 3 GB is written. I've double checked. It's not a linux file system limitation. Development fpc sources(trunc) have been used, but it's an old problem. If SaveToFile calls something like "WriteBuffer(Memory^,Size);" where Size is int64 on x86_64 targets then the rtl devs might have hit the same problem as I did, meaning that they might have used a routine limited to longints. In this topic I've suggested adding some overloaded functions for 64bit integers. Anyway, the forum moderators and the rtl developers should know about this situation. Maybe this situation is encountered only when targeting linux-x86_64, I don't know. ---------------------------------------------------------------------- AlexTP · 2022-01-22 16:29 Reported to https://gitlab.com/freepascal.org/fpc/source/-/issues/39540 ---------------------------------------------------------------------- ASerge · 2022-01-22 19:59 Quote from: lagprogramming on January 22, 2022, 05:27:30 pmIf SaveToFile calls something like "WriteBuffer(Memory^,Size);" where Size is int64 on x86_64 targets then the rtl devs might have hit the same problem as I did The WriteBuffer function has a limitation, but the SaveToFile function does not. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$MODE OBJFPC} uses Classes; var  M: TMemoryStream;begin  M := TMemoryStream.Create;  try    M.Size := LongWord(3 * 1024 * 1024 * 1024); // Even as LongWord    M.SaveToFile('3Gb.bin');  finally    M.Free;  end;end.File 3Gb.bin has a size of 3 GB. FPC 3.2.2 x64, Windows. ---------------------------------------------------------------------- AlexTP · 2022-01-22 20:42 @ASerge It's strange, because I have the same test code, and I got 0-sized file. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses SysUtils, Classes; procedure WriteGarbage(const fn: string; const aSize: Int64);var  s:TMemoryStream;begin  s:=TMemoryStream.Create;  s.SetSize(aSize);  s.SaveToFile(fn);  s.Free;end; begin  WriteGarbage('3g.bin', Int64(3)*1024*1024*1024);end. ---------------------------------------------------------------------- lagprogramming · 2022-01-24 18:25 Can you take a look at the "Small safety precaution" (042eb7e8) commit. https://gitlab.com/freepascal.org/fpc/source/- /commit/042eb7e8c196b6a0c8f6016af9985b8b3062703b The first silent error that remains undetected if when more bytes than intended are read or written. This one is very nasty because if you write data backwards it may write over the data that has been previously written. The second silent error is when the Read/Write functions return errors(The r<=0 and w<=0 conditions). Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TStream.ReadBuffer(var Buffer; Count: NativeInt);var  r,t: NativeInt;begin  t:=0;  repeat    r:=Count-t;    if r>High(Longint) then r:=High(Longint);    r:=Read(PByte(@Buffer)[t],r);    inc(t,r);  until (t>=Count) or (r<=0);  if (tHigh(Longint) then w:=High(Longint);    w:=Write(PByte(@Buffer)[t],w);    inc(t,w);  until (t>=count) or (w<=0);  if (t