[Solved] My program freezes after running for several days lazarus / 21-General ---------------------------------------------------------------------- Wilko500 · 2024-10-27 12:34 I have been doing an extended test on my solar PV upload program.  Just when I thought it was all OK the program froze after 3 days.  No errors, No warnings.  I have been able to recreate this by force quoting the program and re-starting, and waiting another 4 days!!!  The program is not doing very much, checking for the existence of a text file (<200 lines), parsing it and updating a SQLite database with a few fields. The actual web upload is not active yet.  The various processing tasks are  controlled by a series of 6 timers (TTimer) and all are frozen as wells the GUI interface.  My program was composed on macOS Monterey and tested on older Catalina Looking at the program details in Mac Activity Monitor I can see some 255 open file references. I don't really understand the stuff in Activity Monitor but it does provide a clue.  The filename referred to is archived, so the program says, so should not be referred to again, but seemingly is. It does seem that there is a coding error in the conversion from VB6.  The VB6 program is running without this error on the same data and on the same Mac (in virtual windows).  Investigations continue but it is rather time consuming. My question is somewhat philosophical.  Does FPC/Lazarus have a maximum open file limit?  And if it does why no error, or crash when it is exceeded? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  launchctl limit maxfiles        maxfiles    256            unlimited     This from the Mac is soft limit, hard limit (I think) Any thoughts? Lazarus 3-7 FPC 3-3-3 macOS Monterey/Catalina ---------------------------------------------------------------------- Thaddy · 2024-10-27 13:23 On Mac, the maximum number of open files per process is indeed 256 (which is extremely st*p** to keep so many files open for weeks, unless it is a system monitor type of process and that should be a Daemon.) You can increase it by using ulimit. On Linux it is - usually - 1024, which is a silly amount over weeks, and can also be increased by using ulimit. On windows, it is not the handles, but the capacity of the filepaths, 260ísh, which can be set in the registry or the group policy. Why would you want so many files actually open per process? That is rare. You can try cat /proc/sys/fs/file-max to see what is the total number the system allows overall.(trillions) [edit] Better set/getrlimit, because ulimit seems to be obsoleted in the future. Still works, though. Close files when not needed or waiting, cache them and opening them when approached/needed. Al this is, I hope explained, OS specific. ---------------------------------------------------------------------- Laksen · 2024-10-27 14:39 How often are you opening the file? How do you open it? ---------------------------------------------------------------------- Wilko500 · 2024-10-27 14:59 Just for clarity I do not intentionally hold any files open. Every 5 minutes I check if a log file exists. If it does I open it read I, parse the data into 4 other files then close them all. That processing is probably less than a second so most of the time my program is just waiting. @Thady,  thank you for your explanation. It is helpful. I have no wish to increase the open file limit but If my program has exceeded the limit causing a complete freeze would have expected some form of error. It is highly likely that I have a coding/logic error but there is no point in posting code until I can identify a suspect proc or function. My best guess for now is that there is a single data file that does not conform to the expected format and this could be due to a power cut affecting the RS485 data transfer. And yes, I do validation when the file is parsed but apparently I have not anticipated every eventuality. Investigations continue and I will post again if/when I discover the solution. ---------------------------------------------------------------------- Josh · 2024-10-27 15:09 hi create a logfile and after every open and close of a file write to the logfile what file is open or closed and where it was called in your app, you should see mismatches pretty quick. ---------------------------------------------------------------------- dseligo · 2024-10-27 15:26 Quote from: Wilko500 on October 27, 2024, 03:59:33 pmJust for clarity I do not intentionally hold any files open. Every 5 minutes I check if a log file exists. If it does I open it read I, parse the data into 4 other files then close them all. That processing is probably less than a second so most of the time my program is just waiting. @Thady,  thank you for your explanation. It is helpful. I have no wish to increase the open file limit but If my program has exceeded the limit causing a complete freeze would have expected some form of error. It is highly likely that I have a coding/logic error but there is no point in posting code until I can identify a suspect proc or function. I propose that you decrease open file limit until you find error in your code - it will freeze your app faster so you don't have to wait several days to see if it's working. ---------------------------------------------------------------------- Wilko500 · 2024-10-27 15:30 Good advice and I do have a log file where I log errors and debug stuff. I have a debug run in progress. Be a few days for next fail ---------------------------------------------------------------------- Aruna · 2024-10-27 15:36 Quote from: Wilko500 on October 27, 2024, 01:34:36 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  launchctl limit maxfiles        maxfiles    256            unlimited     This from the Mac is soft limit, hard limit (I think) Any thoughts? Hi @Wilko500, from what you have described, the program's accumulation of open file references suggests that files aren’t being closed after each access. This could be a classic case of resource leakage, especially with file handles. Increase maxfiles Temporarily: Use launchctl to increase the maxfiles limit temporarily and check if the program runs longer. This can confirm that an open file cap is indeed the issue.Ensure Explicit File Closure: After each file access (both for reading and writing), confirm the files are closed immediately.Ensure that SQLite connections and transactions are also properly managed. Open connections should close after each operation to avoid cumulative open file references.I am not certain but I think why FPC/Lazarus doesn’t throw errors or crash is because as you noted, FPC does not inherently check for file descriptor limits unless an explicit file-open operation fails. Many programs only handle general exceptions, so once the file descriptor limit is met, subsequent file operations can silently fail if not individually checked for errors. So to resolve this I would: Ensure file closures after each access, especially in recurring operations.Temporarily increase maxfiles as a test to validate the open file limit as the cause.Check SQLite connection handling, as these can contribute to the open file count.Hope this helps ---------------------------------------------------------------------- Wilko500 · 2024-10-27 15:38 Quote from: dseligo on October 27, 2024, 04:26:41 pm I propose that you decrease open file limit until you find error in your code - it will freeze your app faster so you don't have to wait several days to see if it's working. What a good idea, thank you ---------------------------------------------------------------------- Thaddy · 2024-10-27 17:10 Quote from: Wilko500 on October 27, 2024, 03:59:33 pm@Thady,  thank you for your explanation. It is helpful. I have no wish to increase the open file limit but If my program has exceeded the limit causing a complete freeze would have expected some form of error. Well, I actually expected that, because if a process is out of its allocated resources, there is no - easy- way to recover unless you have pre-allocated an exception object for that situation beforehand. (Similar to how FPC and Delphi handle EOutOfMemory: with a pre- allocated object) It is simply a fatal exception. Unix like OS's handle that on process level, whereas Windows handles it on system level.(the latter is worse, because you can't recover at all) Keep a simple map with a max of, say 200 handles, and close them when not needed or exceeded. That is per process on Mac. And... really...256 open handles at one time and in the same process? Since everything is a handle on Unix, you also risk losing all your devices.... ---------------------------------------------------------------------- MarkMLl · 2024-10-27 17:43 Quote from: Laksen on October 27, 2024, 03:39:01 pmHow often are you opening the file? How do you open it? More to the point, how do you close it? How are you parsing the directory? Does this require opening a directory which you aren't closing? How are you talking to the RS485 port? Are you explicitly closing it after opening it? Are you doing anything that involves sockets- UDP as much as TCP? Is every open accompanied by a close? MarkMLl ---------------------------------------------------------------------- Wilko500 · 2024-10-27 22:55 Ok. This version of the program is not directly access the RS485, yet.  That will follow in due course.  This is how I open & close files Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   //Check for existing temp files, delete if found  DatFiles:= TStringList.Create;  FindAllFiles(DatFiles, TempPath, '*.dat', False); //find all temp data files  iDat:=datFiles.Count;  If iDat > 0 Then    Begin      //Files found to delete      For i:= 0 To iDat - 1 Do        Begin          DeleteFile(DatFiles[i]);      End;{For}  End;{If}  //Ceate & open files for input and output  Assign(fLog, TempPath + '/' + fDate + '.log'); //Input log file  Assign(fDat, TempPath + '/' + fDate + '-DA.dat'); //Output inverter info file  Assign(fIn, TempPath + '/' + fDate + '-IN.dat'); //Output inverter info file  Assign(fSys, TempPath + '/' + fDate + '-SY.dat'); //Output inverter info file  Assign(fUn, TempPath + '/' + fDate + '-UN.dat'); //Output inverter info file  Reset(fLog);  ReWrite(fDat);  ReWrite(fIn);  ReWrite(fSys);  ReWrite(fUn);  While Not EOF(fLog) Do      Begin      //Parse file one line at a time         //do stuff with lines from file  //And close files  Close(fLog);  Close(fDat);  Close(fIn);  Close(fSys);  Close(fUn);  DatFiles.Free;  Very simple coding. So I started a new debug run with additional debug line in the suspected procedure and the outcome was,  disappointing,  the processing worked as it was supposed to!  Further, watching the Activity Monitor there was no accumulation of open files as I had seen on previous occasions.  I did not change the existing code just added a few WriteLn's and then ran in terminal mode. Tomorrow at daylight my inverter will start provide live data and I will see what happens.  I'll run it for a few days, might even try a power failure to see how it recovers.  It is beginning to look like a case of bad data caused by event unknown (but probably a power failure) that is not being handled correctly.  I'll post again when I get more information from another extended test. ---------------------------------------------------------------------- MarkMLl · 2024-10-28 06:56 Quote from: Wilko500 on October 27, 2024, 11:55:38 pmOk. This version of the program is not directly access the RS485, yet.  That will follow in due course.  This is how I open & close files That's not "a version of the program", it's an incomplete code fragment. Where's FindAllFiles defined? MarkMLl ---------------------------------------------------------------------- Thaddy · 2024-10-28 08:40 A power failure does not keep files open after a restart. A restart initializes all processes and resources from scratch, which means any previously open file handles are closed. So this is unlikely unless you have some recovery mechanism built-in that tries to open too many files at once. Damaged files because of a power failure is another matter, but also does not use handles. Also, long running systems need a PSU and these are not expensive anymore. ---------------------------------------------------------------------- Josh · 2024-10-28 09:03 Hi I do not see any IOERROR checking on any of the file operations. It is not wise to assume that the operations have succeeded. if its a gui app, create a global variable MyOpenFileCount, set it to zero, then after every successful open of file increment it,after every successful close decrement it, create an idle timer and display its value somewhere on your gui, it should match watch the system reports. You can then spot a run away situation, and if one starts you can handle it gracefully. ---------------------------------------------------------------------- krolikbest · 2024-10-28 09:18 QuoteWhile Not EOF(fLog) Do       Begin       //Parse file one line at a time        //do stuff with lines from file      //And close files   Close(fLog);   Close(fDat);   Close(fIn);   Close(fSys);   Close(fUn);   DatFiles.Free; where is the "end"? for this block of code? at the end of this block or somewhere there Quote//And close files Such a loose guesses, could it be that you are trying to open a file when it is already opened (e.g. in last sequence was not closed because some other processes took time and in new sequence it is still open? ---------------------------------------------------------------------- Thaddy · 2024-10-28 09:21 Also note your sqlite db might keep handles open. ---------------------------------------------------------------------- Wilko500 · 2024-11-02 16:26 Update and probable cause identified Quote from: Thaddy on October 28, 2024, 09:40:45 amA power failure does not keep files open after a restart. Yes, I agree but a power failure followed by restart does cause (sometimes) damage to the log file created by the inverter monitoring program (Aurora Communicator, provided by the inverter manufacturer) thus extra initialisation lines may appear in the log file in an unusual order. My code attempts to anticipate and handle this kind of event. While reviewing my code and debug log file following another program freeze of the same kind I did find that in one of my procedures I had unintentionally omitted code in the Exception that would have logged an exception. On both of the "freeze" I ah seen from the macOS Activity monitor that there was a single data file involved.  I have concluded that there was some issue with that file (and my code) that had caused an exception.  That exception had been caught by the Try - Except and the program execution continued WITHOUT any message being output and the file not being closed.  This had then repeated until open file limit exceeded and program freeze. Thank you all for your observations, comments and suggestions. I will mark this thread as solved.