[Solved] Heap overflow error with TFileStream
lazarus / 62-General

----------------------------------------------------------------------
bobonwhidbey · 2022-05-29 22:09

This approach works great for smaller files, but it's crashing with an
heap overflow error at almost the 40,000th 100 byte record. Here's
some streamlined code.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i var 
fstream: TFileStream;  s : stringbegin   fstream :=
TFileStream.Create('MyFileName', fmCreate);  s := GetTitleLine+CRLF;
// CRLF=#13#10  fstream.Write(s[1], length(s));  while not Query.EOF
do begin     s := GetTextofRecord(Query)+CRLF;     fstream.Write(s[1],
length(s));     Query.Next;  end;   fstream.Free;  end; 
Is there an approach I can use to create a large text file and avoid
the heap error?  I'm not wedded to tFileStream. TStringlist also
crashes.

----------------------------------------------------------------------
gucao · 2022-05-29 23:11

I think the query is causing your problem,You can try to let the query
extract 1000 pieces of data each time for testing.

Good luck.

----------------------------------------------------------------------
balazsszekely · 2022-05-30 05:42

Query is a TSQLQuery? If yes, then set Unidirectional property to true
just before opening the query and you are good to go.

----------------------------------------------------------------------
Warfley · 2022-05-30 07:12

On 32 bit systems you only have 4 gigs of address space available. So
the first thing you should do is compile your application as 64 bit.

After that the only limit you might reach is the physically available
memory in your system, if that is a problem look at memory mapped
files (mmap), which will allow you to map file data into a memory
range without it being located in the RAM

----------------------------------------------------------------------
rvk · 2022-05-30 08:52

Quote from: bobonwhidbey on May 30, 2022, 12:09:24 amThis approach
works great for smaller files, but it's crashing with an heap overflow
error at almost the 40,000th 100 byte record. Here's some streamlined
code.
<snip>
Is there an approach I can use to create a large text file and avoid
the heap error?  I'm not wedded to tFileStream. TStringlist also
crashes.
I'm sure there must be something else wrong with your program.
TFileStream can handle very very large files.

Just tried this and it created a file of 498.047KB file !!
Definitely bigger than your 40.000 x 100 bytes files (which is about
3.906KB)

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
Project1;uses Classes; const  GetTitleLine = 'Title Line';  CRLF =
#13#10; function GetTextofRecord(): String;begin  Result := 'This is
just a testline';  while Length(Result) < 100 do Result := Result +
'.';end; var  fstream: TFileStream;  s : string;  Iterations:
integer;begin  fstream := TFileStream.Create('MyFileName', fmCreate); 
s := GetTitleLine+CRLF; // CRLF=#13#10  fstream.Write(s[1],
length(s));  Iterations := 0;  while Iterations < 5000000 do begin   
 s := GetTextofRecord()+CRLF;     fstream.Write(s[1], length(s));   
 Inc(Iterations);  end;   fstream.Free; end.
Show us your GetTextofRecord() function and we might be able to
pinpoint the problem.

----------------------------------------------------------------------
marcov · 2022-05-30 10:15

rvk: the line of text is only stored in memory one at a time.

Getmem & Gucao's remark that the query is somehow gigantic (and a
query is much larger due to the variant use than the resulting bytes),
and of course Warfley's remark to avoid 32-bti in the first place if
you are a heavy memory user.

----------------------------------------------------------------------
Zvoni · 2022-05-30 10:29

Agree with everyone: probably his GetTextOfRecord.
Considering TSQLQuery inherits from TCustomBufDataset, and
TCustomBufDataset has a SaveToFile-Method, which internally creates a
TFileStream........

Show us GetTextOfRecord

----------------------------------------------------------------------
rvk · 2022-05-30 10:35

Quote from: marcov on May 30, 2022, 12:15:03 pmrvk: the line of text
is only stored in memory one at a time.
TFileStream writes out every line to disk. So there is nothing stored
in memory.

So the heap problem isn't related to the TFileStream at all (unlike
what TS expected).
The problem is probably in GetTextOfRecord.

Quote from: marcov on May 30, 2022, 12:15:03 pmGetmem & Gucao's remark
that the query is somehow gigantic (and a query is much larger due to
the variant use than the resulting bytes), and of course Warfley's
remark to avoid 32-bti in the first place if you are a heavy memory
user.The TQuery wouldn't make a difference. It could handle milion and
milion of records. And TFileStream can handle that too.
It's the GetTextOfRecord whic is the problem.

You can see from my example that TFileStream handles this just fine.
It creates a file of 0,5GB. Way more than the 4MB file TS creates when
the heap crash happens.

----------------------------------------------------------------------
marcov · 2022-05-30 11:12

rvk: If the query doesn't prefetch everything.

But yeah, it also could be that something in the procedure is odd.

----------------------------------------------------------------------
Zvoni · 2022-05-30 12:05

Quote from: marcov on May 30, 2022, 01:12:40 pmrvk: If the query
doesn't prefetch everything.

But yeah, it also could be that something in the procedure is odd.
Especially, since the OP passes the whole Object to the Procedure.....
Obviously, not knowing his Project, i'm more surprised, why the OP
doesn't make the Function a Property of his Query-Object, passing, say
the current RecordIndex.....

----------------------------------------------------------------------
bobonwhidbey · 2022-05-30 16:07

Yes - IDList is a TSQLQuery. My problem had nothing to do with saving
to a TFileStream. Setting IDList.Unidirectional := true solved the
problem.

Thank you GetMem and others for weighing in on this.

----------------------------------------------------------------------
rvk · 2022-05-30 16:21

Color me amazed 

QuoteTSQLQuery is a descendant of BufDataset, a dataset that buffers
the data it receives into memory.https://wiki.freepascal.org/Working_W
ith_TSQLQuery#Error_messages

Delphi definitely doesn't have the same problem.

Wonder why this wasn't done the other way around.

----------------------------------------------------------------------
marcov · 2022-05-30 16:25

Quote from: rvk on May 30, 2022, 06:21:31 pm Delphi definitely doesn't
have the same problem.

Using ZEOS on both solves that.

----------------------------------------------------------------------
rvk · 2022-05-30 16:48

I've never had such problems on Delphi.

And from what I can remember, the transaction handling with zeos isn't
all that.
So I kept with IBX on Delphi.

----------------------------------------------------------------------
Thaddy · 2022-05-30 19:26

Using Zeos is a mistake. It should have been removed ions ago.
BTW a heap overflow error? I haven't seen that in 50 years of
programming
Show CODE..

Because an EOutOfMemory should appear first.

Some rot has to be removed.