Program creating many threads lazarus / 32-Linux ---------------------------------------------------------------------- Чебурашка · 2023-05-17 10:40 Hello, I have a software that has to periodically send messages over network to a remote listener. Since it can happen that network goes down, listener is dead and many other problems, I decided that my sender program's master cycle has not to be affected by external issues, and basically I work like this when I need to send a message: 1. create a object containing the information to be transmitted 2. create a thread whose only duty is to make the transmission 3. give object to the thread 4. start the thread 5. after some time make sure memory used by thread is released (and if thread is blocked, just kill it). Basically cleanin up all about the thread after he finished is work, and this is done by a "thread manager" object. With this approach I verified that during normal execution there are just few threads runing (one or two, except when transmission fails, that there are some more threads blocked, waiting to be forcibidly killed by the thread manager). Now my problem is that it seems that after a while my program is no longer able to create new threads. It it possible that this is simply Linux that has a hard limit on how many a single process can start in his lifetime? Thank you ---------------------------------------------------------------------- MarkMLl · 2023-05-17 10:50 Quote from: Чебурашка on May 17, 2023, 12:40:10 pmNow my problem is that it seems that after a while my program is no longer able to create new threads. It it possible that this is simply Linux that has a hard limit on how many a single process can start in his lifetime? I don't believe so. Having said that a lot of daemons periodically restart themselves (i.e. kick off a new process and terminate the old one, which would implicitly release any retained resources or state) but the intention of that is usually to avoid memory leaks. Look carefully at your thread creation, make sure that it's not relying on sockets or files being available. In the past I've neglected to close connectionless (UDP) sockets, and each one of those took a handle from the finite per-process pool. MarkMLl ---------------------------------------------------------------------- Чебурашка · 2023-05-17 13:26 Quote from: MarkMLl on May 17, 2023, 12:50:52 pmQuote from: Чебурашка on May 17, 2023, 12:40:10 pmNow my problem is that it seems that after a while my program is no longer able to create new threads. It it possible that this is simply Linux that has a hard limit on how many a single process can start in his lifetime? I don't believe so. Having said that a lot of daemons periodically restart themselves (i.e. kick off a new process and terminate the old one, which would implicitly release any retained resources or state) but the intention of that is usually to avoid memory leaks. Look carefully at your thread creation, make sure that it's not relying on sockets or files being available. In the past I've neglected to close connectionless (UDP) sockets, and each one of those took a handle from the finite per-process pool. MarkMLl I think you are right, my process blocks after about 950 threads started (all except the last one or 2 are already ended). OTOH: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; uses  cthreads, Classes, SysUtils; var  i: uint64;  function Execute(param: Pointer): ptrint;begin  Result := 0;  Inc(i);  Writeln(i);end; begin  i := 0;  while (true) do  begin    BeginThread(@Execute);    Sleep(1);  end;end. it stops at Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i user@machine:~/tmp/test-threads$ ./project1...327393274032741327423274332744 ---------------------------------------------------------------------- MarkMLl · 2023-05-17 14:05 Odd, but I've never tried BeginThread(): only the classical TThread stuff. I've not got time to experiment right now but in your specific example try one of the indivisible increments, and in the general case keep an eye on lsof output and the process's (pseudo-) directory in /proc. MarkMLl ---------------------------------------------------------------------- Чебурашка · 2023-05-17 14:18 Quote from: MarkMLl on May 17, 2023, 04:05:10 pmOdd, but I've never tried BeginThread(): only the classical TThread stuff.   I happened to use the beginThread because is written in the documentation https://www.freepascal.org/docs- html/current/prog/progch10.html#x230-24400010 et seq. Quote from: MarkMLl on May 17, 2023, 04:05:10 pmOdd, but I've never tried BeginThread(): only the classical TThread stuff. I've not got time to experiment right now but in your specific example try one of the indivisible increments, and in the general case keep an eye on lsof output and the process's (pseudo-) directory in /proc. MarkMLl Thanks. For your info and maybe it can help others too, I noticed that the new thread creation stops (in the computer of my interest) when the linux virtual memory totally occupied in 4Gb (it is a ASEM with J1900 with 4Gb), so I believe thers is a relationship. I counted that the threads started where always 973. ---------------------------------------------------------------------- alpine · 2023-05-17 14:48 Quote from: Чебурашка on May 17, 2023, 04:18:47 pmFor your info and maybe it can help others too, I noticed that the new thread creation stops (in the computer of my interest) when the linux virtual memory totally occupied in 4Gb (it is a ASEM with J1900 with 4Gb), so I believe thers is a relationship. I counted that the threads started where always 973. Guess: sounds like thread stacks are not de-allocated? ~1000 threads are usually the Win32 limit (4Gb address space) depending on what else resides in memory (OS, etc.) ---------------------------------------------------------------------- Чебурашка · 2023-05-17 15:08 Quote from: alpine on May 17, 2023, 04:48:47 pmQuote from: Чебурашка on May 17, 2023, 04:18:47 pmFor your info and maybe it can help others too, I noticed that the new thread creation stops (in the computer of my interest) when the linux virtual memory totally occupied in 4Gb (it is a ASEM with J1900 with 4Gb), so I believe thers is a relationship. I counted that the threads started where always 973. Guess: sounds like thread stacks are not de-allocated? ~1000 threads are usually the Win32 limit (4Gb address space) depending on what else resides in memory (OS, etc.) I am under linux, but also in linux the thread stacks must be free up as well. How am I supposed to deallocated them? In the documentation: https://www.freepascal.org/docs- html/current/prog/progch10.html#progse44.html https://wiki.freepascal.org/Multithreaded_Application_Tutorial I don't find anything about that, is there some other source of information? ---------------------------------------------------------------------- alpine · 2023-05-17 15:16 It was just a guess, when I saw the 4Gb number, I've remembered that I had similar problem on win32 once. Never used the BeginThread just TThread. I'll check it. ---------------------------------------------------------------------- Чебурашка · 2023-05-17 15:32 STOP Quote from: PascalDragon on January 12, 2022, 01:43:04 pmYes, one is supposed to use EndThread from the thread function. We should probably mention that in the documentation for BeginThread. (but your hint was game changing, alpine) ---------------------------------------------------------------------- Чебурашка · 2023-05-17 15:44 And the modifed program goes beyond previous count of threads Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; uses  cthreads, Classes, SysUtils; var  i: uint64; function Execute(param: Pointer): ptrint;begin  Result := 0;  Inc(i);  Writeln(i);  EndThread(); // Added call To EndThread as said in other cited forum thread...end; begin  i := 0;  while (true) do  begin    BeginThread(@Execute);    Sleep(1);  end;end. ---------------------------------------------------------------------- alpine · 2023-05-17 15:45 What I've found in rtl/unix/cthreads.pp is that the wrapping ThreadMain() function calls pthread_exit() when the thread function exits, but it seems it is not enough (I'm not a pthreads expert) - looks like it leaves the thread 'joinable' i.e. you should use WaitForThreadTerminate() on it. Or otherwise call the EndThread() explicitly to 'detach' it and then pthread_exit() it. EDIT: Found that one: https://forum.lazarus.freepascal.org/index.php/t opic,57844.msg430664.html#msg430664 Yes, it is the same you've found also ---------------------------------------------------------------------- Чебурашка · 2023-05-17 15:55 I believe that should be added to the various pages https://wiki.freepascal.org/Threads https://www.freepascal.org/docs-html/prog/progse44.html https://wiki.freepascal.org/Multithreaded_Application_Tutorial That EndThread must be called to free the resources used by the thread. ---------------------------------------------------------------------- alpine · 2023-05-17 16:00 Quote from: Чебурашка on May 17, 2023, 05:55:28 pmI believe that should be added to the various pages https://wiki.freepascal.org/Threads https://www.freepascal.org/docs-html/prog/progse44.html https://wiki.freepascal.org/Multithreaded_Application_Tutorial That EndThread must be called to free the resources used by the thread. Yes, it should. But I believe it is not the only option, that is in case you do not care about the returned exit code. The other is WaitForThreadTerminate(), Just read the forum thread posts after the one of PascalDragon. ---------------------------------------------------------------------- cdbc · 2023-05-17 16:11 Hi @Чебурашка: Thank you for doing the research, I just learned a new thing about threads/threadprogramming, and pretty important too  Regards Benny ---------------------------------------------------------------------- MarkMLl · 2023-05-17 18:34 @Alpine et al... I was wondering about (lack of) synchronisation on termination after I posted last, but have been away from my desk. That sort of thing /should/ show up as a ballooning number of entries in /proc, since my understanding is that a Linux thread is a cloned (as distinct from forked) process. I'd throw in that I found myself in a situation a few weeks ago where I wanted to be able to terminate a thread which I knew would be waiting on a syscall with no timeout. I found that a pthread-level SIGHUP worked: until I also tried to handle a SIGHUP at the application level, at which point things started getting exceptionally messy. MarkMLl ---------------------------------------------------------------------- Чебурашка · 2023-05-18 06:31 Quote from: cdbc on May 17, 2023, 06:11:32 pmHi @Чебурашка: Thank you for doing the research, I just learned a new thing about threads/threadprogramming, and pretty important too  Regards Benny Countelss times I been helped by other users of forums. In italian we say "E' una ruota che gira" (literally "It's a spinning wheel") ---------------------------------------------------------------------- alpine · 2023-05-18 06:34 Quote from: MarkMLl on May 17, 2023, 08:34:26 pm@Alpine et al... I was wondering about (lack of) synchronisation on termination after I posted last, but have been away from my desk. That sort of thing /should/ show up as a ballooning number of entries in /proc, since my understanding is that a Linux thread is a cloned (as distinct from forked) process. Please explain: cloned vs forked. AFAIK, forking (of a process) is when it starts like a single one, but eventually: Code: C  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   ...   if (fork() == 0)   {    // do the child work  } else {    // parent continued ...  } ---------------------------------------------------------------------- Чебурашка · 2023-05-18 06:47 Quote from: alpine on May 18, 2023, 08:34:31 amQuote from: MarkMLl on May 17, 2023, 08:34:26 pm@Alpine et al... I was wondering about (lack of) synchronisation on termination after I posted last, but have been away from my desk. That sort of thing /should/ show up as a ballooning number of entries in /proc, since my understanding is that a Linux thread is a cloned (as distinct from forked) process. Please explain: cloned vs forked. AFAIK, forking (of a process) is when it starts like a single one, but eventually: Code: C  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   ...   if (fork() == 0)   {    // do the child work  } else {    // parent continued ...  }  I think MarkMLI is saying that in linux when you call fork, a new process is created by cloning the original = an exact copy of the process descriptor is made and assigned to the forked process (except the pid and ppid) ---------------------------------------------------------------------- MarkMLl · 2023-05-18 06:56 Quote from: Чебурашка on May 18, 2023, 08:47:14 amI think MarkMLI is saying that in linux when you call fork, a new process is created by cloning the original = an exact copy of the process descriptor is made and assigned to the forked process (except the pid and ppid) No, I am not. These are different system calls, and the distinction is critical to the way pthreads are implemented. fork creates a new process with a distinct memory context, subject to COW. clone creates a new process sharing the came memory context, i.e. a thread. The PID situation appears... messy in the TThread context, with the owning process's PID becoming the thread group and the subsidiary threads' PID becoming the thread identifiers. It might be that the BeginThread() return value is significant here. MarkMLl ---------------------------------------------------------------------- alpine · 2023-05-18 07:18 Quote from: MarkMLl on May 18, 2023, 08:56:25 amfork creates a new process with a distinct memory context, subject to COW. clone creates a new process sharing the came memory context, i.e. a thread. I see. But isn't it that the main difference between processes and threads - the former gets a copy of the parent data and the latter - just a private stack space (and CPU/FPU context)? ---------------------------------------------------------------------- MarkMLl · 2023-05-18 07:38 Quote from: alpine on May 18, 2023, 09:18:27 amBut isn't it that the main difference between processes and threads - the former gets a copy of the parent data and the latter - just a private stack space (and CPU/FPU context)? Yes, but the point is that "fork" and "clone" are "terms of art" here and shouldn't be (ab)used lightly. I admit that from the POV of implementing an OS I far prefer the progression coroutine (AKA fiber etc.) -> thread -> process. But the fact of the matter is that in the context of Linux things are enormously constrained by POSIX and (using 1980s terminology) GLX, and changing subsystem architecture is incredibly and increasingly difficult: you've only got to look at the problems caused by the attempted transition from X11 to Wayland. And I'd add at this point that I don't think that things are helped by Lennart Poettering's attempts to introduce complex subsystems like systemd and PulseAudio which operate at multiple levels in parallel with the Linux kernel, rather at distinct layers as is traditional. MarkMLl