Finishing thread does not release Virtual memory lazarus / 62-General ---------------------------------------------------------------------- jollytall · 2022-01-11 10:58 I have an application that makes new threads all the time. They finish normally, no memory leak, etc.. Still the Virtual memory usage of the process is constantly growing. I read many articles and it seems to be the normal process, as they say, because calling the virtual memory deallocation is apparently an expensive process. So what happens, every time when a new thread is launched it reserves memory in the virtual space and when it terminates it releases all physical space, but not the virtual. The next process could maybe use the "abandoned" virtual memory (it is the same structure thread again), but instead it reserves a new virtual address space. Since on x64 the space is large enough, it is not a frequent problem to deal with, but it is alarming to see that the virtual memory usage goes up fast and I also see some small increase in physical memory usage (maybe for the allocation table). I would like to somehow release the virtual memory, even if it is a bit expensive. Somewhere I read a topic about detaching the thread from the process and let it die alone and apparently in such a case the virtual memory reservation "dies" with the left alone thread. I am not sure it is possible, exists, but surely could not find a "DetachThread" procedure (C++ has it). For information: I use Linux/Debian10 with Lazarus 2.0.12, FPC 3.2.0. I know I could use a pool of threads to avoid this problem, but prefer not to rewrite the whole application if I can solve it in the current set-up. ---------------------------------------------------------------------- marcov · 2022-01-11 11:05 The classic cause is memory fragmentation.  A new block is allocated by the application, but it is larger than the empty spaces the heapmanager has, so it allocates it from the OS. The solution is usually to pool large allocations manually, or break them up in smaller pieces. ---------------------------------------------------------------------- Jonas Maebe · 2022-01-11 11:31 If you set FreeOnTerminate of your TThread to true, it will run as a detached thread. Warning: in that case you cannot/must not use TThread.WaitFor on it (partly because in theory the thread could even end before you started waiting on it). You will need to implement your own synchronisation if you need similar functionality (e.g. by setting an event from the thread when it is about to finish). ---------------------------------------------------------------------- jollytall · 2022-01-11 11:57 @marcov, I think it is unlikely. I call the same object type with the same create parameters many times. They should allocate the same memory, so it should fit in to an earlier one. Even if they are slightly different size (for whatever reason), it is unlikely that when I do it for the 100th time, none of the previous 99 is the same size or larger to be reused. I think that the system does not look for virtual memory blocks allocated but not used by the same process, once a new thread is created. @Jonas Thanks, it might be leading to the solution. Two problems: I do not use TThread, but the one level lower BeginThread, so I would need to call whatever has to be called manually. It would not be a problem if I knew what to call. I also have my own Wait implementation, so that is not a problem either. I check the TThread implementation and FFreeOnTermiante I found only once in ThreadProc, and it calls Free (i.e. Destroy), but in Destroy I did not find what to do. Can you help? ---------------------------------------------------------------------- MarkMLl · 2022-01-11 12:10 Quote from: jollytall on January 11, 2022, 11:58:55 amI would like to somehow release the virtual memory, even if it is a bit expensive. On Linux,  swapoff -a  should do it. Yes, it's extreme. But it should have the effect of (a) forcing everything briefly into real memory and (b) in order to do this minimising the amount of demand-loaded executable in memory. You probably want to run  swapon -a  shortly afterwards. Apart from that, listen to Jonas et al. MarkMLl ---------------------------------------------------------------------- jollytall · 2022-01-11 14:25 It does not seem to work, and if I understand it correctly, it not even should. Isn't swapoff/swapon control the allocation of physical memory between RAM and Swap, while it has nothing to do with Virtual memory? ---------------------------------------------------------------------- MarkMLl · 2022-01-11 14:39 Quote from: jollytall on January 11, 2022, 03:25:41 pmIt does not seem to work, and if I understand it correctly, it not even should. Isn't swapoff/swapon control the allocation of physical memory between RAM and Swap, while it has nothing to do with Virtual memory? Swap is normally referred to as virtual memory, or are you talking about something different? MarkMLl ---------------------------------------------------------------------- BrunoK · 2022-01-11 14:41 Maybe try using cmem and see how memory usage vary ? Of course, no heaptrc possible. ---------------------------------------------------------------------- jollytall · 2022-01-11 15:45 Quote from: MarkMLl on January 11, 2022, 03:39:15 pmSwap is normally referred to as virtual memory, or are you talking about something different? MarkMLl Swap (or to be more precise Swap + RAM) is often referred as virtual memory. But this is not what Linux (e.g. in Top) calls Virtual memory. Nowadays VIRT can be much larger than the available physical memory including RAM and swap. I found a good description https://serverfault.com/questions/138427/what-does-virtual-memory- size-in-top-mean explaining how it can happen. Also https://man7.org/linux/man-pages/man1/top.1.html has a paragraph on Linux memory types. The problem to reserve a virtual memory space still reserves (I think) an entry in the allocation table and so the table slowly grows, so over a longer period can cause OOM. (And also the peace of my mind not seeing a number growing permanently beyond any limit.) ---------------------------------------------------------------------- jollytall · 2022-01-11 15:48 Quote from: BrunoK on January 11, 2022, 03:41:35 pmMaybe try using cmem and see how memory usage vary ? Of course, no heaptrc possible. I tried adding cmem, but the VIRT usage grows just the same. ---------------------------------------------------------------------- Martin_fr · 2022-01-11 15:55 Quote from: jollytall on January 11, 2022, 11:58:55 amThey finish normally, no memory leak, etc.. Or none that you know of.... Run your app under valgrind. Compile with debug info enabled: dwarf  (ideally enabled for packages too, use "Additions and Overrides" to add -gw ) Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i valgrind --tool=memcheck  ./project1 ---------------------------------------------------------------------- Martin_fr · 2022-01-11 16:11 Also, its possible that memory is allocated, and only freed when your app shuts down => no memleak, but growing mem usage. E.g. if you add objects to a global list, and you (including any unit used by you) free all of it in a "finalization" block. ---------------------------------------------------------------------- jollytall · 2022-01-11 18:28 Thanks Martin, Honestly, I never used vargrind before and it does not seem to be an easy tool. I get many errors and possibly lost and still reachable bytes. E.g. a summary: ==27332== LEAK SUMMARY: ==27332==    definitely lost: 0 bytes in 0 blocks ==27332==    indirectly lost: 0 bytes in 0 blocks ==27332==      possibly lost: 608 bytes in 2 blocks ==27332==    still reachable: 15,497 bytes in 149 blocks ==27332==         suppressed: 0 bytes in 0 blocks I need more time to understand it, but most of them are in libcrypto (openssl) and libc. I don't know what I can do with them, and whether it can keep the reserved large virtual memory locked for few Kbyte of still reachable memory. It will be a long task to figure it all out... Maybe it is easier to cron a daily restart and the problem is gone :-), but I do not like a solution like that. ---------------------------------------------------------------------- marcov · 2022-01-11 20:44 Quote from: jollytall on January 11, 2022, 12:57:51 pm@marcov, I think it is unlikely. I call the same object type with the same create parameters many times. They should allocate the same memory, so it should fit in to an earlier one Unless a smaller came inbetween and partially filled the  large hole, so that the now smaller hole doesn't fit the next major allocation anymore . Anyway, try to recycle some larger memory allocations e.g. using a tthreadlist), and see if that changes behaviour. ---------------------------------------------------------------------- Thaddy · 2022-01-11 21:02 Wait..wait..waitfor! Use waitfor as already mentioned. ---------------------------------------------------------------------- Martin_fr · 2022-01-11 21:25 Ah, I forgot something, for valgrind you need to use "CMem". Afaik if you did specify the "... for valgrind (-gv)" option, then that is done automatically. Otherwise you have to do that yourself (and then no heaptrc) Quote from: jollytall on January 11, 2022, 07:28:25 pm==27332==    still reachable: 15,497 bytes in 149 blocks ==27332==         suppressed: 0 bytes in 0 blocks Afaik, "still reachable" is memory that was released, but only when the program terminated. Some of this would be expected. Question is, does the number grow, if the program runs for longer? I.e. does your app accumulate more and more memory that it only frees at exit? Mind that leaks can be in libraries too. 10 years ago, I had a webserver with mem issues, turned out openssl was leaking (mind you 10 years, so probably not what you have now). QuoteI need more time to understand it, but most of them are in libcrypto (openssl) and libc. I don't know what I can do with them, and whether it can keep the reserved large virtual memory locked for few Kbyte of still Well, the first issue is, if the grow (in count and accumulated byte- size), if you run the app for longer. It is very well possible that some code "leaks" something when it is initialized. If that memory is indeed needed for the entire run time, and if it is only taken just once, then that is not an issue. But if over time, it gets more... It can be an issue in the library, but it can also be how it is called... ---------------------------------------------------------------------- engkin · 2022-01-11 22:12 Quote from: jollytall on January 11, 2022, 07:28:25 pmI need more time to understand it, but most of them are in libcrypto (openssl) and libc You might have a leak in your usage of libCrypto. Any function that gives a big number, for instance,, you are supposed to free the memory when done with that number. Like BN_bin2bn, SRP_calc_u..etc ---------------------------------------------------------------------- Jonas Maebe · 2022-01-12 07:18 Quote from: jollytall on January 11, 2022, 12:57:51 pm@Jonas Thanks, it might be leading to the solution. Two problems: I do not use TThread, but the one level lower BeginThread, so I would need to call whatever has to be called manually. It would not be a problem if I knew what to call. I also have my own Wait implementation, so that is not a problem either. I check the TThread implementation and FFreeOnTermiante I found only once in ThreadProc, and it calls Free (i.e. Destroy), but in Destroy I did not find what to do. Can you help? We indeed don't have the functional equivalent for detaching a thread with the procedural interface. However, just to make sure, are you calling waitforthreadterminate on all of (non-detached) your threads? Otherwise they will indeed linger on forever. ---------------------------------------------------------------------- jollytall · 2022-01-12 09:06 Thanks for the many useful comments, replies. I try to answer them and report my current progress: Quote from: Martin_fr on January 11, 2022, 10:25:13 pmAh, I forgot something, for valgrind you need to use "CMem". Afaik if you did specify the "... for valgrind (-gv)" option, then that is done automatically. Otherwise you have to do that yourself (and then no heaptrc) Yes, you are right. Specifying -gv adds cmem. Trying to add cmem manually results in a Duplicate Identifier error. While we are at this point, a bit OFF spin-off question, that bothers me. It is always said (and indeed necessary) to put cmem and cthread to the top of the uses lists. I guess it is probably because their Initialization section does some initialization that is needed before other units start to initialize themselves. However I also thought that some of these special units redeclare some methods to provide extra functionality. The issue is that when two units have the same method then always the last one is used not the first. How is it managed for these special units, or they do not overwrite any existing method? Quote from: Martin_fr on January 11, 2022, 10:25:13 pmAfaik, "still reachable" is memory that was released, but only when the program terminated. Some of this would be expected. Question is, does the number grow, if the program runs for longer? I.e. does your app accumulate more and more memory that it only frees at exit? No, this is not the case. Actually, this is only possible if one makes some special efforts, e.g. a dynamic array with pointers or linked objects, but I use none of them. Under "normal" circumstances it cannot happen, because by definition "Still reachable" mean that their pointer (i.e. the variable holding that pointer) still points to the reserved memory area. Having a fix number of variables mean that the amount of Still reachable cannot grow unlimited. To make things even stranger, I analyzed my program a lot and found that when I only start and stop it, it has 13 blocks (5243 bytes) still reachable, but if I use it and exit after that, then this number goes down to 9 (3597). I am trying to check where the difference comes from. Quote from: Martin_fr on January 11, 2022, 10:25:13 pmMind that leaks can be in libraries too. 10 years ago, I had a webserver with mem issues, turned out openssl was leaking (mind you 10 years, so probably not what you have now). The openssl problem I found. What happened was, I had a class level variable to store the SSL Context. When a new thread (object running in the thread) was created, I checked if the class var was already set or not. If not, it was initialized. The problem was that threads were started too close to each other without control to make it thread safe and so the same initialization was called multiple times. Adding a CriticalSection solved it. Quote from: Martin_fr on January 11, 2022, 10:25:13 pmWell, the first issue is, if the grow (in count and accumulated byte-size), if you run the app for longer. It is very well possible that some code "leaks" something when it is initialized. If that memory is indeed needed for the entire run time, and if it is only taken just once, then that is not an issue. But if over time, it gets more... It can be an issue in the library, but it can also be how it is called... I could make a minimal example (see later) to show what grows and how. Quote from: engkin on January 11, 2022, 11:12:02 pmYou might have a leak in your usage of libCrypto. Any function that gives a big number, for instance,, you are supposed to free the memory when done with that number. Like BN_bin2bn, SRP_calc_u..etc Thanks God it was not the case. It is only that it was started multiple times parallel without thread control (see above). Quote from: Thaddy on January 11, 2022, 10:02:24 pmWait..wait..waitfor! Use waitfor as already mentioned. Quote from: Jonas Maebe on January 12, 2022, 08:18:22 amWe indeed don't have the functional equivalent for detaching a thread with the procedural interface. However, just to make sure, are you calling waitforthreadterminate on all of (non-detached) your threads? Otherwise they will indeed linger on forever. Where shall I put it? See the example below that (a) is copied (and further simplified) from the manual (https://www.freepascal.org/docs- html/prog/progse44.html) (b) demonstrates the problem and (c) does not have waitfor in it (maybe this is why it has the same problem). So, the minimal program, I can show the problem with is: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; {$mode objfpc} uses  cthreads,  sysutils; function InThread(p : pointer) : Int64;  begin  Sleep(Random(50));  result:=0;  end; var i : longint;beginfor i:=1 to 100 do  begin  BeginThread(@InThread,nil);  Sleep(Random(50));  end;Sleep(600);writeln('Finished');readln;end.After 100 run it uses 659.4MB VIRT, 2.6MB RES and 1.7MB SHR memory. If I run it 200 times, the memory usage goes up to 1059.8MB/3.4MB/1.7MB. With other words one extra thread blocks 4MB VIRT and 8KB RES. Valgrind shows 100blocks (i.e. the number of threads) and 27200 bytes (i.e. 272byte/thread) possible lost. These numbers bother me. Where shall I put the waitfor? Will it stop this leak? Thanks again, and sorry for the long post. ---------------------------------------------------------------------- jollytall · 2022-01-12 11:46 It seems I have the solution. Thank to all who helped. The solution has two parts. It is true, that there was a memory leak. Although it was reported as a "possibly lost" I think it was really a "still reachable". What happens, when the thread is created it gets a memory area. In the c library they reserve a bit larger space than needed for the array to have enough memory for the array length and reference count (-1, -2 index) and then moves the array pointer to the third block, i.e. element #0. This is why Valgrind reports it as a block that has a pointer pointing inside it (offset 16) and not at the first byte. So it thinks the pointer might have lost it. But it did not. It was worse. The pointer was really there and "in use". As many of you pointed out, although the thread was gone, it left behind its pointer and the allocated memory. If the pointer had pointed to the first byte of the block, Valgrind would report it as "still reachable", clearly indicating that something is not released. To release it, not Waitfor but EndThread was needed. In the above example, placing EndThread between lines 12 and 13 does the trick, and all memory leak is gone. The other part of the story is the Virtual memory usage. It is clear that while the thread did not end correctly (as above), the virtual memory allocated was still reserved. That is also true that an area once reserved is not necessarily released. The good news is though, that it is reused if available. So, what happens is, that the total VIRT memory used is capped by the maximum concurrent threads running at any point in the past. Apparently the OS manages it smart and when I tried to overload it with many threads the VIRT usage temporarily went up high, but then also decreased back to a reasonable level, i.e. the OS freed up some of the reserved memory addresses. ---------------------------------------------------------------------- PascalDragon · 2022-01-12 12:43 Quote from: jollytall on January 12, 2022, 10:06:22 amWhile we are at this point, a bit OFF spin-off question, that bothers me. It is always said (and indeed necessary) to put cmem and cthread to the top of the uses lists. I guess it is probably because their Initialization section does some initialization that is needed before other units start to initialize themselves. However I also thought that some of these special units redeclare some methods to provide extra functionality. The issue is that when two units have the same method then always the last one is used not the first. How is it managed for these special units, or they do not overwrite any existing method? Those don't override any methods, but instead set the managers (TMemoryManager in case of cmem and TThreadManager in case of cthreads) which are in turn used by functions like GetMem or BeginThread. Quote from: jollytall on January 12, 2022, 12:46:13 pmBut it did not. It was worse. The pointer was really there and "in use". As many of you pointed out, although the thread was gone, it left behind its pointer and the allocated memory. If the pointer had pointed to the first byte of the block, Valgrind would report it as "still reachable", clearly indicating that something is not released. To release it, not Waitfor but EndThread was needed. In the above example, placing EndThread between lines 12 and 13 does the trick, and all memory leak is gone. Yes, one is supposed to use EndThread from the thread function. We should probably mention that in the documentation for BeginThread. ---------------------------------------------------------------------- Jonas Maebe · 2022-01-12 20:25 QuoteQuote from: Jonas Maebe on January 12, 2022, 08:18:22 amWe indeed don't have the functional equivalent for detaching a thread with the procedural interface. However, just to make sure, are you calling waitforthreadterminate on all of (non-detached) your threads? Otherwise they will indeed linger on forever. Where shall I put it? You can put it wherever you want. You have to store the result from BeginThread in a TThreadID variable (e.g. an array of TThreadID), and then use WaitForThreadTerminate with those TThreadID values as parameter to reap the threads after they are finished. WaitForThreadTerminate is a blocking call (i.e., if the thread hasn't ended yet, the call will keep waiting until it does), but you can pass a timeout if you want. QuoteWill it stop this leak? It will definitely fix a leak you have when not doing this. ---------------------------------------------------------------------- jollytall · 2022-01-13 07:29 @Jonas, I still do not understand this logic. The one written above is clear. A Thread is started and when it finishes its activity, as a last step after it released all user allocated resources, it also releases the thread management allocated resources with an EndThread call. What would be the purpose of WaitForThreadTerminate in this setup? Surely, it would give an assurance to the caller (typically the main thread) that all threads finished their activity (e.g. closed all files they opened) and it is save to stop the whole process. This is important once in the lifetime of the program, at the very end. Does it also make sure, that when the thread is terminated (joined) its resources are also freed? Reading the source code, it seems (without lengthy analysis) that it only calls the pthread_join system call and in the manual of pthread_join I do not see any resource de- allocation. This would probably not even be possible (I am not sure!), since a terminated thread might still have resources allocated and managed later by the thread it joined to. Also, it would not be practical to have either the main thread blocked during the normal operation of the program (i.e. not at the very end) to see when a thread terminates (hundreds of threads with various running lengths), or to make a separate "thread managing thread" . As written above, the EndThread does the trick perfectly, without any blocking operation in the main thread. (And yes, I have a separate mechanism to wait for all the threads at the very end of the run of the program, not to kill them in the middle of their operation, but that is another topic). Last to note, it is always a blocking operation on *nix operating systems; pthread_join has no timeout parameter. ---------------------------------------------------------------------- Jonas Maebe · 2022-01-13 10:32 Quote from: jollytall on January 13, 2022, 08:29:05 am@Jonas, I still do not understand this logic. The one written above is clear. A Thread is started and when it finishes its activity, as a last step after it released all user allocated resources, it also releases the thread management allocated resources with an EndThread call. What would be the purpose of WaitForThreadTerminate in this setup? Mainly synchronisation between the main thread and the child thread. Additionally, pthread_exit can return a value which can be captured by pthread_join (which is used by the FPC RTL to return the value passed to EndThread to WaitForThreadTerminate). This means the thread must keep state until WaitForThreadTerminate/pthread_join is called. QuoteDoes it also make sure, that when the thread is terminated (joined) its resources are also freed? Reading the source code, it seems (without lengthy analysis) that it only calls the pthread_join system call and in the manual of pthread_join I do not see any resource de-allocation. This would probably not even be possible (I am not sure!), since a terminated thread might still have resources allocated and managed later by the thread it joined to. Also, it would not be practical to have either the main thread blocked during the normal operation of the program (i.e. not at the very end) to see when a thread terminates (hundreds of threads with various running lengths), or to make a separate "thread managing thread" . As written above, the EndThread does the trick perfectly, without any blocking operation in the main thread. (And yes, I have a separate mechanism to wait for all the threads at the very end of the run of the program, not to kill them in the middle of their operation, but that is another topic). It's simply how OS threading interfaces work on either Windows or Unix. This is not an FPC design. And yes, the OS/threading library keeps state around until you join a thread, unless it's a detached one. If you don't care for synchronisation with your threads for whatever reason, you need a detached threads. However, this functionality is indeed not exposed by the procedural FPC threading interface, because it's the exception rather than the rule in most programs. QuoteLast to note, it is always a blocking operation on *nix operating systems; pthread_join has no timeout parameter. Indeed, I forgot about that.