[SOLVED] Interesting Problem when Parallel Processing lazarus / 21-General ---------------------------------------------------------------------- ASBzone · 2021-01-29 07:12 I recently implemented parallel processing for a prime number evaluation utility, and it works properly 95% of the time (at least). However, if I run it with maximum threads for any given system, there is a chance that it will not get the correct totals for highest prime or number of primes found. This happens most often with the max thread count, but I have seen it happen far less frequently with a lower thread count (say, 8 of 16 threads). I have been unable to determine any consistency.  I've run multiple iterations and had 2-4 in a row that were good, and 2 or 3 that were bad consecutively.  I've tested on multiple systems, with multiple processors, both AMD and Intel based.  I have only tested on Windows 10, and Server 2019, with OS level Spectre/Meltdown mitigations off. I stripped down the code entirely to just enough to run in multi- threaded mode, and have attached it here. I'm running the config shown in my signature (re: Windows, FPC, Lazarus) The issue is more likely to manifest for me with numbers greater than 10 Billion. Please see the examples below of correct and incorrect output: CORRECT Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i >GetPrimeInfo64.exe Max Search ......... 50,000,000,000 (0xBA43B7400)Total Primes .......  2,119,609,265 (0x7E56ABB1)Highest Prime ...... 49,999,999,967 (0xBA43B73DF)----------------------------- ----------------------------------------------Start Time ......... 01/29/2021 01:34:36.017End Time ........... 01/29/2021 01:35:07.914Elapsed Time .......             0:00:31.897-------------- -------------------------------------------------------------  >GetPri meInfo64.exeSearching for all prime numbers up through 50000000000 using 16 threads.  Current Time is: 1/29/2021 1:40:06----------------- ---------------------------------------------------------------------- ---------------------------------Max Search ......... 50,000,000,000Total Primes ....... 2,119,654,578Highest Prime ...... 49,999,999,967-------------------------------------------------------- -------------------Start Time ......... 01/29/2021 01:40:06.944End Time ........... 01/29/2021 01:40:34.167Elapsed Time .......           0:00:27.223--------------------------------------------------------- ------------------  >GetPrimeInfo64.exeSearching for all prime numbers up through 50000000000 using 16 threads.  Current Time is: 1/29/2021 1:44:27--------------------------------------------------------------- ---------------------------------------------------------Max Search ......... 50,000,000,000Total Primes ....... 2,119,654,578Highest Prime ...... 49,999,999,967------------------------------------------- --------------------------------Start Time ......... 01/29/2021 01:44:27.508End Time ........... 01/29/2021 01:45:41.045Elapsed Time .......             0:01:13.537--------------------------------------- ------------------------------------  INCORRECT Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i >GetPrimeInfo64.exeSearching for all prime numbers up through 50000000000 using 16 threads.  Current Time is: 1/29/2021 1:38:03----- ---------------------------------------------------------------------- ---------------------------------------------Max Search ......... 50,000,000,000Total Primes ....... 2,119,576,535      <--- ???Highest Prime ...... 49,999,195,979   <--- ???-------------------------------- -------------------------------------------Start Time ......... 01/29/2021 01:38:03.780End Time ........... 01/29/2021 01:38:30.022Elapsed Time .......             0:00:26.242-------------- -------------------------------------------------------------  >GetPri meInfo64.exeSearching for all prime numbers up through 50000000000 using 16 threads.  Current Time is: 1/29/2021 1:39:12----------------- ---------------------------------------------------------------------- ---------------------------------Max Search ......... 50,000,000,000Total Primes ....... 2,119,540,122    <--- ???Highest Prime ...... 49,998,748,771    <--- ???------------------------------- --------------------------------------------Start Time ......... 01/29/2021 01:39:12.332End Time ........... 01/29/2021 01:39:38.467Elapsed Time .......             0:00:26.135-------------- -------------------------------------------------------------  If anyone can verify that this also happens to them, I'd love to hear about it.  I did a couple runs with Range checking and some other settings on, but nothing was flagged, and I turned it back off since that slows things down tremendously. ---------------------------------------------------------------------- Marc · 2021-01-29 08:45 Usualy when this happens in MT applications, it means that an critical section is missing somewhere. One thread reading while the other is updateing. At first sight I cannot spot a trivial point (also due to the naming of variables it is hard to see the scope of them) Anyway, one suggestion to make the wait loop less CPU consuming. First free the the theads manually, no FreeOnTerminate, instead of Code: [Select]  while (CurrentIndex < MaxIndex) do     sleep(100); use the following: Code: [Select]  for I := 0 to ThreadCount - 1 do   begin     Workers[I].Waitfor;     Workers[I].Free;   end; This way the OS waits till the first is finished and so on edit And this may explain the differences. CurrentIndex may reach MaxIndex while other threads are still running, but you exit the MultiThread proc Marc ---------------------------------------------------------------------- ASBzone · 2021-01-29 14:48 Quote from: Marc on January 29, 2021, 09:45:17 am Anyway, one suggestion to make the wait loop less CPU consuming. First free the the theads manually, no FreeOnTerminate, instead of Code: [Select]  while (CurrentIndex < MaxIndex) do     sleep(100); use the following: Code: [Select]  for I := 0 to ThreadCount - 1 do   begin     Workers[I].Waitfor;     Workers[I].Free;   end; This way the OS waits till the first is finished and so on edit And this may explain the differences. CurrentIndex may reach MaxIndex while other threads are still running, but you exit the MultiThread proc Marc Thanks for the feedback, Marc I'll make this change.    It is interesting to me that it does not happen more frequently than it does, given what you have suggested.  Let's see how it plays out, though. EDIT: Well, that doesn't work.   I breaks in short order. Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i An unhandled exception occurred at $0000000000338061:EAccessViolation: Access violation  $0000000000338061  $00000001000090AB  $000000010001C77B  $0000000100001F2E  $00000001000023B2  $0000000100002466  $00000001000102C0  $0000000100001750  $00007FF85BC07034  $00007FF85D49D0D1  BTW, I have previously tested with the SLEEP as low as 10 and as high as 100, and it has had very little impact on the overall behavior, other than a little performance boost. ---------------------------------------------------------------------- Marc · 2021-01-29 14:59 It is not only about performance, but you exit before all threads are finished Are you sure you set FreeOnTerminate to false ? If so, better find out why WaitFor raises an exception ---------------------------------------------------------------------- ASBzone · 2021-01-29 20:10 QuoteAre you sure you set FreeOnTerminate to false ? Actually, it was not. I've made that change and I am testing it some more... So far, so good... I will keep you posted. ---------------------------------------------------------------------- ASBzone · 2021-01-29 22:06 Testing has been very promising thus far.... I'll run some longer tests over the weekend... ---------------------------------------------------------------------- ASBzone · 2021-01-31 22:57 Quote from: Marc on January 29, 2021, 03:59:09 pmIt is not only about performance, but you exit before all threads are finished Are you sure you set FreeOnTerminate to false ? If so, better find out why WaitFor raises an exception @Marc,  I was able to do a great deal of testing today, and I want to thank you for pointing out the issue I was experiencing. It now works flawlessly, and each run has a more consistent time, too.   (This bugged me before that there was so much variance on the runs using the same number of threads.) It is about 3% slower than before, on average, which goes to show that I was getting lucky quite a bit before in not seeing the impact more regularly. Much appreciated.