Mutex v. InterLockExchange question for dummies. lazarus / 21-General ---------------------------------------------------------------------- AlanTheBeast · 2021-04-02 20:13 tl;dr?  go to Section IV) I) I have a program running on a Pi 3B+ (Ultibo environment) processing data from a GPS at 16 'frames' per second (about 800 bytes per frame). There is the 'main' program, and it has started three threads and then loops with a loop sleep of 5ms.  There are various simple message queues between these. II) The three threads that have been created:        One (GPS serial input) runs until the serial input is empty and then computes how long to sleep (about 6 - 10 ms typically).  This one sends 'work' to the other two via message queues.       Those two run until their input queues are exhausted and then sleep for ~30ms.  The busier of these takes about 6ms on average to complete, but can range up to 10 or 12 ms.  Would be nice to schedule these rather than have them sleep but I haven't figured that out yet.  For now they compute their sleep time, but that's likely to drift.  For now that's not an issue. All three send status strings to the main via simple queues (this is where I get timing data to verify the performance).  I have not looked into CPU Affinity (state or controlling it) but the RTL is allegedly optimizing so there is a good chance the threads are on different CPU's often.  (3B+ has 4 cores). These queues are very simple.  Array of strings or array of array of bytes; head and tail pointer and how_many messages are waiting in the queue. III) Only that last variable (in each queue) needs protection so I use InterlockExchange: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     While InterlockExchange (QALock,1)>0 do;       //wait for the lock to clear    inc(QAWait);                                                   //QAWait=number of messages waiting    InterlockExchange (QALock,0);  I believe that's correct.  Seems to work anyway.  I don't protect the head, tail and messages as the queues are  twice as large as the message rate.  (the queues have 32 slots and the peak wait I've seen is 17). Section IV) So.  Now onto my ignorance.  Is there a plain explanation of how mutex works, what are the pre-requisites for it, etc.?  A nice clear example for the simplest of cases.  The threads above are just simple functions.  They are not wrapped in an object or class. Thanks. ---------------------------------------------------------------------- MarkMLl · 2021-04-02 20:38 You don't introduce the term "mutex" until you actually ask the question, which I'd suggest is problematic. I'd normally expect mutex, futex and so on (as with process, thread and fiber) to be an OS-level concept, so they not only describe or protect an entity in your program from local interference but also from external interference. Extrapolating from what you've actually asked to what I think you think you wanted to ask, TCriticalSection protects against entry by multiple threads but not by multiple sequences of instructions in the same thread, while InterlockExchange etc. can be used to protect against possible reentry within the same thread but you assume responsibility for avoiding deadlock and telling the OS to go do something else for a while. Now I know that you're going to ask how a single thread can have multiple sequences of instructions, so I'll C&P this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i (* The choice here is Synchronize() which blocks, PostMessage() which doesn't   *)(* block but needs to be protected against reentry, SendMessage() which doesn't *)(* block but doesn't swap to the GUI thread either, and QueueAsyncCall() which  *)(* is both non-blocking and protects against reentry and is generally the best  *)(* choice if reasonable performance is needed (tens to hundreds of messages per *)(* second).                                                                   *)  I suspect that you've not been treated like a student for some time, but I want you to focus on what happens if a background thread calls PostMessage(), and the handler includes an Application.ProcessMessages. :-) MarkMLl ---------------------------------------------------------------------- 440bx · 2021-04-02 20:53 Quote from: AlanTheBeast on April 02, 2021, 10:13:28 pmIs there a plain explanation of how mutex works, what are the pre-requisites for it, etc.?  A nice clear example for the simplest of cases.  The threads above are just simple functions.  They are not wrapped in an object or class. I'll try to give you an explanation that drives home the important characteristics of a mutex (many of them apply to other kernel objects used for synchronization) The first thing that is important is that a mutex, specifically, a named mutex, can be used by multiple processes to wait for some resource that is shared among them.  The InterlockExchange method cannot, since the memory address being locked and exchanged is local to the process (it is not present/seen in other processes.)  That might very well be the most important difference between a mutex (or other waitable kernel objects) and the lock/exchange method.  A named mutex, unlike some area of memory that is locked/exchanged, exists in multiple processes (at the O/S level) which must open it in order to be able to use it. Another important characteristic is that when using a lock/exchange method, the treads are responsible for not hogging a bunch of CPU while getting the lock(s) they want to acquire.  When using a mutex (or other waitable object), it is the O/S that manages access to the resource and, while a thread is waiting for a resource it wants to acquire, it doesn't consume any CPU because the O/S scheduler won't allocate CPU cycles to it until the resource is available to be acquired (by any thread that is waiting for it.) if your goal is to minimize latency at all costs within a single process that runs multiple threads then the lock/exchange method will do it but, it definitely has the potential to be a CPU hog. Unfortunately, I don't have a simple example. HTH. ---------------------------------------------------------------------- Martin_fr · 2021-04-02 21:04 The code you have is called a spin-lock. (Not sure if InterlockedCompareExchange or InterlockExchange  is the better choice). A spin lock, is similar to a CriticalSection, only that the process keeps hold of the CPU, and keeps it busy. So you should only use it, if you know it will always just be a few spins, until you get it. In your example, where all you protect is a single "inc(n)", which will be a few cpu cycles only, it should be fine to use a spin lock. Assuming that all other threads using that lock, will also do equally little. You do take the minimum risk, that the OS could take the CPU away from a thread while it is in the lock. Then any waiting thread has to spin a bit longer. Also spin locks, only make sense if your code will run on several cpu cores. If you only have one core, or if you set the affinity of your entire process to one single core, then the spin-lock will be real bad. Having said that, afaik on intel InterlockExchange will also do read/write barriers. That may not be the case on all other systems. Very roughly spoken, read/write barriers prevent your CPU from doing nasty pre-fetch work (or other optimizations), and load the variable while still in the spin lock. After all, in a modern CPU the order of execution is not the same as in your code. If all you need to do is to increment a variable: InterlockedIncrement ---------------------------------------------------------------------- MarkMLl · 2021-04-02 21:13 Quote from: Martin_fr on April 02, 2021, 11:04:58 pmHaving said that, afaik on intel InterlockExchange will also do read/write barriers. That may not be the case on all other systems. I think the intention is that it does (I recall discussing this with Jonas), but whether that's robustly tested on all possible platforms is another question entirely. MarkMLl ---------------------------------------------------------------------- AlanTheBeast · 2021-04-02 22:23 Thanks all for your replies.  I'll start digging - but from the replies I don't think a Mutex is needed for my current case.  Why I asked is below... Quote from: 440bx on April 02, 2021, 10:53:22 pmif your goal is to minimize latency at all costs within a single process that runs multiple threads then the lock/exchange method will do it but, it definitely has the potential to be a CPU hog. This particular case is very 'tight'.  For all 3 queues both the message sender and the message receiver only lock their section of code long enough to change the byte(s) as needed then give up the lock.  See the cited code in my first post?  It is essentially the same for the sender as the receiver.  So each end would only lose a few cycles if the lock were on.  Collisions are possible - indeed certain - since I tested the implementation and made errors, those errors manifested themselves messily until I sorted it out. Yes, it's multi core and yes I'm assured by the nice fellow at Ultibo that my several threads will be assigned a CPU - any CPU - when they are woken according to the RTL (which is using a tailored threads which I assume is based on cthreads).  Even if it were single CPU, it is a pre-empt-ing thread manager so collisions are possible. Why I asked?  The very nice fellow who runs Ultibo initially suggested "a Mutex or at least an InterlockExchange".  The latter was pretty easy to implement, the former I get lost everywhere I read.  So I'm missing something essential to understanding it.  But now I see I likely don't need it for this case.  Or want it.  So it goes to my other background catching up on Pascal RTL. Indeed for the queues in question I could do without a lock at all - as long as the consumer keeps the queue from filling - then I could dump the wait counter altogether and the consumer would just need to eat messages until the Tail_index = Head_index. Quote from: Martin_fr on April 02, 2021, 11:04:58 pmThe code you have is called a spin-lock. (Not sure if InterlockedCompareExchange or InterlockExchange  is the better choice). A spin lock, is similar to a CriticalSection, only that the process keeps hold of the CPU, and keeps it busy. So you should only use it, if you know it will always just be a few spins, until you get it. In your example, where all you protect is a single "inc(n)", which will be a few cpu cycles only, it should be fine to use a spin lock. Assuming that all other threads using that lock, will also do equally little. You do take the minimum risk, that the OS could take the CPU away from a thread while it is in the lock. Then any waiting thread has to spin a bit longer. Also spin locks, only make sense if your code will run on several cpu cores. If you only have one core, or if you set the affinity of your entire process to one single core, then the spin-lock will be real bad. If all you need to do is to increment a variable: InterlockedIncrement InterlockedIncrement looks interesting.  I didn't spot it in the code at Ultibo.  I'll take another look. And yes, as per your observations, the code is pretty tight everywhere that the InterlockExchange is used. In the Ultibo environment, the CPU won't be taking much away from program - unless I start really loading up Ultibo with common I/O (Ethernet, USB, etc. and so on).  That's way down the pike. Thanks! ---------------------------------------------------------------------- AlanTheBeast · 2021-04-02 22:52 Quote from: jamie on April 03, 2021, 12:37:44 amyou don't show your code section how you are populating the buffers so I can't say however, I do such coding myself and what I use is a circular buffer of 2's complement. so lets say you have a buffer of 1024 bytes, you have two indexes, a lastStore and nextRead. if the NextRead is the same as the Laststore then that means the buffer is empty.. and when you add one to the buffer you increment the LastStore using a MASK or MOD function so that when it exceeds the 1024 it simply wraps around.. doing this  you can fill the buffer while reading at the same time from different threads. if the two indexes of behind each other then the buffer is full! Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i          UBX_Queue[UBX_QHead] := UBX_In_Mess;         UBX_QHead := (UBX_QHead +1) and $1F;               //32 entries         While InterlockedExchange(UBX_QLock,1)>0 do;         inc(UBX_QWait);         InterlockedExchange(UBX_QLock,0);  The consumer is the same... so, yess, essentially as you say.  (Tail instead of Head). If I get rid of UBX_Wait and have the consumer simply eat until UBX_QTail catches up to UBX_QHead then I really don't need the interlock as long as the consumer is reliably in front of the job (and is to date for all 3 queues...).  I wrote this in the last reply, but edited it out.  So - we're on the same page. That said, the amount of time locked is trivial at both ends, unless as Martin_fr suggests, the thread manager pre-empts while the lock is on.  But it can pre-empt in any case... (Your presentation of it reminded me of the first time I did such a long time ago in 8085 assembler... though I did also maintain a waiting count) ---------------------------------------------------------------------- AlanTheBeast · 2021-04-02 23:01 Quote from: MarkMLl on April 02, 2021, 10:38:32 pmYou don't introduce the term "mutex" until you actually ask the question, which I'd suggest is problematic. I suspect that you've not been treated like a student for some time, but I want you to focus on what happens if a background thread calls PostMessage(), and the handler includes an Application.ProcessMessages. :-) MarkMLl As I said in the other reply to "all and 440bx", my question was borne of someone else's suggestion.  Having explored various articles on Mutex' I'm no further ahead in getting it.  That is I understand the intent, just not the use case / implementation.  As previously stated I'm not an OS expert of any kind and so I get lost with the layers of abstractions that have evolved over time while I wasn't paying much attention... I don't care much for your "focus" suggestion but from the context it has all the makings of some sort of lockup condition. ---------------------------------------------------------------------- Martin_fr · 2021-04-02 23:54 1) Probably not for your/this case, but in case you ever need it TLazMonitor (unit lazcollections), will start a spin lock (though spinning on "TryEnter" of a critical section, but (important) not giving up the cpu). That is, it takes indeed more work to detect, if the lock is open (call a function), but it will not take the big loss of giving up its scheduled cpu time. If it can't acquire in a given time, it will go full waiting for the critical section. 2) Based on the above: TLazThreadedQueue 3) Spin lock and starvation. On more issue on the spinlocks you use. If you have lots of threads trying to acquire the lock, there is no order in which they will receive it. The one who came latest to try and get it, may be the one that gets it first. Pure random. That means, if there are continuously threads trying to acquire the lock, then there may be a thread that can wait forever, but never get it (because there will always be another one getting it first) So the overall count of threads trying to get a particular spinlock should be moderate to low. And each thread should spent sufficient time between any 2 attempts on the lock. Thus each thread eventually gets the lock. 4) Quote from: AlanTheBeast on April 03, 2021, 12:52:55 amCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i          UBX_Queue[UBX_QHead] := UBX_In_Mess;         UBX_QHead := (UBX_QHead +1) and $1F;               //32 entries         While InterlockedExchange(UBX_QLock,1)>0 do;         inc(UBX_QWait);         InterlockedExchange(UBX_QLock,0); So there is ONLY exactly one thread that will write to the queue? And ONLY exactly  one that reads? Because you modify UBX_QHead outside the lock. So only one thread (and only the same thread for the entire time) can be allowed to "push" new items. And I assume there is an UBX_QTail (or Bottom), for the reader? And UBX_QWait means you have something in the queue (not empty) "waiting for someone to get the data from the queue" ? Do you have/need protection against the queue getting a 33rd entry pushed (overwriting data)? If you have more than one thread pushing (or pulling), you might want InterlockedCompareExchange Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   repeat  CurHead := UBX_QHead;  // the current first free slot    NextHead :=(CurHead + 1) and $1f;  until InterlockedCompareExchange(UBX_QHead, NextHead, CurHead);  // You stored the next value, while UBX_QHead was at CurHead, so CurHead is yours  UBX_Queue[CurHead] := UBX_In_Mess;  Note you can simplify this "A:=InterlockedIncrement(UBX_QHead)" The value may become bigger than 31, but you can use "A and $1f" ---------------------------------------------------------------------- AlanTheBeast · 2021-04-03 13:13 Quote So there is ONLY exactly one thread that will write to the queue? And ONLY exactly  one that reads? Yes, (though there is a case I need to look at to be sure.  Coffee hasn't kicked in yet). This is why I could (as discussed with jamie) not even bother with a wait_count and lock.  As long as the tail <> head, then just consume the queue.  Requires assurance that the consumer is always running faster than the producer (which is the case now with 2:1 advantage for the worst case).  And of course that the head is the last thing updated for the queue state. Quote"A:=InterlockedIncrement(UBX_QHead)" The value may become bigger than 31, but you can use "A and $1f"      UBX_QHead := InterlockedIncrement(UBX_Head) and $1F;     ?? I may try this in the one case where there may be 2 producers - but I'm pretty sure there isn't. ---------------------------------------------------------------------- MarkMLl · 2021-04-03 13:45 Even if you only have one producer and one consumer, it is worth adding some form of mutual exclusion if there is any risk whatsoever that they will be subsequently moved into separate threads or put into event handlers that could potentially overlap. And you wonder how I know? Let's put it this way: not because of anything somebody's written in a book or learned article. MarkMLl ---------------------------------------------------------------------- Martin_fr · 2021-04-03 14:37 Quote from: AlanTheBeast on April 03, 2021, 03:13:49 pmAs long as the tail <> head, then just consume the queue.  Yeaaah, no... If you mean in the consumer "if tail <> head then" That won't work. Because the consumer must do 2 memory reads, and the producer may change value in between. (and also: no memory barriers). You can do such an unprotected "if" as a pre-test. If it returns false "tail = head", then you can tread the queue as empty. It may just be changing, but then you get it in the next round (of course that could dampen your consumer speed advantage). If the unprotected "if" says the 2 values differ, then you need to do a protected test. Quote  And of course that the head is the last thing updated for the queue state. Even then, my above answer holds. Even if your code acts in that order (and would theoretically pass / I do not know that), the cpu can execute it out of order. And the it will fail. If you want the deep end of the pool: Wikipedia an "memory barriers" Also keep in mind, that a compiler may optimize code, and hold a variable in a register (not sure if FPC does that for global vars). Search the forum for discussions on the C "volatile" declaration. (or google / Wikipedia) QuoteRequires assurance that the consumer is always running faster than the producer (which is the case now with 2:1 advantage for the worst case). Unless you run on some sort of "real time OS" that is never guaranteed. On your classic Windows, Linux, MacOS, even Android or IOS you are depending on the task scheduler. And there never is a guarantee that your consumer thread will not at some stage get less CPU time than it needs. Does your producer allocate memory for the data it produces? Or does it have statically pre-allocated mem. In the later case, it will be screwed, when the consumer does not keep up. If it allocs mem, then there might be a real nifty trick with linked list (making the queue size indefinitely big). But I need to look into it, if it really will work. ---------------------------------------------------------------------- AlanTheBeast · 2021-04-03 15:23 Hi Martin, This is pretty much an OS-less environment (Ultibo) with Pascal RTL and some background handling.  Not technically an RTOS but closer than anything on Win, Mac OS, Linux.  More OS-y components can be added, but I will be doing the strict minimum of that. From the Ultibo site Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i What is Ultibo?Ultibo core is an embedded or bare metal development environment for Raspberry Pi. It isnot an operating system but provides many of the same services as an OS, things like memory management,networking, filesystems and threading plus much more. So you don’t have to start from scratch just to create your ideas. Globals are not in register AFAI have seen looking at compiled assembler (on Mac FPC -O2 at least, I haven't looked at the ARM assembly output).  IAC: were it the case, the current scheme wouldn't even work as, eg, the QCount in one thread would be different than the QCount in the other.  Fails would indicate. I have chosen the statically allocated queue sizes to be appropriate to the task with a 2:1 oversize.  The consumer threads compute their sleep period at the end of each cycle.  It is generously over requirement. Further: the project I'm doing is not meant to be portable. You have a good point with the out-of-order execution.  I'll leave the locks in place for now because they seem to be working well (ran 24 hours with no issues with a steady input of data).  (Each thread outputs some state info to monitor its critical values). Finally, the processor is far beyond the processing needs - at least so far - 1KHz external interrupt and lovely math and possibly graphics to come and even that shouldn't strain it.  If I implement a Pi Zero version ... well, then things will become more difficult to be sure. I did think about a linked list queue but thought it added complexity and processing for little gain. ---------------------------------------------------------------------- Martin_fr · 2021-04-03 15:24 -- EDIT : overlapped with your last post Just a guess. Would your producer be reading data from some hardware? E.g. serial port, or some other input. And that data comes at a fixed rate, and is not buffered, so your producer must read on a fixed interval? In that case, your number one enemy is the task scheduler. For a user level aff there simply is no way to ensure getting enough CPU time. For that kind of stuff, you need some elevated (kernel level or similar) driver. You also need pre-allocated memory, and that memory must be protected from being swapped to disk (Most OS allow you do do that, but again you may need elevated privileges). Then you probably want a consumer, that runs at highest user level priority, to have the best chance to keep up with emptying the buffer. But ultimately you need error handling, if the buffer overflows. ---------------------------------------------------------------------- Martin_fr · 2021-04-03 15:32 OK, so basically your program has the entire hardware to itself. (potentially less an ssh client, likely on a private network / so no flooding attacks / which means you can ensure it won't take to much resources) There are no "unknown" processes run by user. So, yes that gives you a good starting ground. It also probably means you have enough control over mem alloc, to make sure swapping and similar issues will not come into play. Generally speaking, on many OS, entering a critical section is extremely fast. And using TryEnter (see TLazMonitor) will prevent giving up scheduled CPU time. ---------------------------------------------------------------------- AlanTheBeast · 2021-04-03 15:50 We're having forum reply collisions....  Yes, currently a pretty even rate serial input at about 800 bytes / 100ms (or 63ms depends...).  So, the consumers (1 for an NMEA string, 1 for the binary arrays) have no problem at all keeping up. The serial in is currently sleeping 10ms (it varies based on load) and about 240 bytes to process on each wake.  The consumers every 30ms (and I've initiated them 15ms out of phase, but I doubt that will hold over time). And yep, my code is the only "main" code running.  This is akin to an Arduino - just far more powerful and far more RTL talent from FPC and extras in it. I'll be adding an interrupt for precise timing (GPS) and TBD (interrupt or not) for a motion sensor.  Once all that is together and runs well. ... I tear it all down and design a proper program.  At that point I may add things like Ethernet and/or a LoRa and a display for graphics.  But the Ethernet will be standalone - not web. Current phase is getting bits and pieces to work well... Swapping?  There is nothing to swap to!    ...generous memory on these little boards... ---------------------------------------------------------------------- 440bx · 2021-04-03 16:22 I just want to "plant" an idea in your head that looks like might be usable in the case you are dealing with.  It consists of having two counters.  One is a "WriteCounter" used by the server thread that adds elements to the queue, the other is a "ReadCounter" used by the client that reads elements off the queue. The important thing is the threads don't share writes, IOW, the "WriteCounter" is written only by the server thread and the "ReadCounter" counter is written only by the client thread.  Because of that, they don't have to block.  The server thread can _read_ the "ReadCounter" and figure out if it is getting too far ahead of the client and if so, sleep or do something else.  The client only reads the "WriteCounter" to figure out if a new element has been added to the queue and if not, sleep or do something else. The net effect is, there is no wait latency due to locks since there are no locks involved.  The threads figure out what they should do based on the values of the Read/Write counters.  HTH. ---------------------------------------------------------------------- AlanTheBeast · 2021-04-03 17:12 Quote from: 440bx on April 03, 2021, 06:22:07 pmI just want to "plant" an idea in your head that looks like might be usable in the case you are dealing with.  It consists of having two counters.  One is a "WriteCounter" used by the server thread that adds elements to the queue, the other is a "ReadCounter" used by the client that reads elements off the queue. The important thing is the threads don't share writes, IOW, the "WriteCounter" is written only by the server thread and the "ReadCounter" counter is written only by the client thread.  Because of that, they don't have to block.  The server thread can _read_ the "ReadCounter" and figure out if it is getting too far ahead of the client and if so, sleep or do something else.  The client only reads the "WriteCounter" to figure out if a new element has been added to the queue and if not, sleep or do something else. The net effect is, there is no wait latency due to locks since there are no locks involved.  The threads figure out what they should do based on the values of the Read/Write counters.  HTH. That's smart.  However, this is a real time project and I'd rather it be inefficient in CPU and memory allocation than smart with just enough. The producer (server) cannot sleep off a full queue.  It has to deal with the input without loss - the Serial input (system level) has a fixed buffer size - so to sleep it off would just be kicking the problem deeper into the system.  (Yes, I can increase the buffer size when I open the device, but that's not the solution to the problem [currently 2KiB]). The consumer threads ( at this point ) 'balance' by consuming the entire queue on each wake and it's all designed so there is no possibility of the producer sending more than the consumer can take.  This is not satisfactory from an OS pov, where all cases need to be handled as loads vary greatly - but this is not an OS environment. Producer thread balances by computing a sleep period to satisfy a waiting number of bytes in the serial queue that is at least 128 and no more than 256 bytes.   If less than the former, the sleep interval is increased by 1 ms; if more than the later the sleep is made ¾ of its current value, but at least 1m.  This 'rings' for 10s or so after start then settles to some value (depending on set message rate and baud rate). It is satisfactory where loads are pretty easily predicted, measurable and testable.  Which is my case.  And I'll be adding another device that will produce about 120,000 bits/s in a very stable/consistent manner (I think). All of you fine people are giving me great advice that fits a different generalized OS environment. This is an embedded project.  Things are stripped down bear bones with the advantages of almost all of the FPC RTL and some other things... ---------------------------------------------------------------------- AlanTheBeast · 2021-04-03 23:42 Taking this further, I've implemented CPU affinity for the discussed threads - should pretty much guarantee lock hits!  But also means I don't have to attempt to keep two threads out of phase with each other.  The lost time in the lock exchange is a handful of CPU cycles... ---------------------------------------------------------------------- MarkMLl · 2021-04-04 07:47 Quote from: AlanTheBeast on April 04, 2021, 01:42:37 amTaking this further, I've implemented CPU affinity for the discussed threads - should pretty much guarantee lock hits! No way. That's a recommendation not a mandate, and you can't trust it 100%. MarkMLl ---------------------------------------------------------------------- AlanTheBeast · 2021-04-04 13:33 I get your point, but it's just a nice way to encourage the load levelling - not a critical requirement. Out of curiosity, I just added a sampler such that each thread, at a "wake point" (after its sleep(n) statement) gets its actual (not affinity setting) CPU number and stores it in the sampling array.  Occasionally push a switch to have the sampling array dumped to screen.  They are consistently as set (yes: I'm not saving every possible instance).   Nor can such save pre-empted resumption CPU number. Also added a lockhit counter to the busiest thread's queue out interlock, and it occasionally increases... (once every minute or so). From the Threads unit .ThreadResume function (partial) I gather it's very unlikely that the selected affinity won't be honoured. (Also not 100% sure this applies to resuming a thread after it has done Sleep(n); but I didn't see anything elsewhere ). Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     {Check CPU}    if SCHEDULER_CPU_COUNT > 1 then     begin      if ThreadEntry.CurrentCPU <> ThreadEntry.TargetCPU then       begin      {$IFDEF SCHEDULER_DEBUG}        Inc(SchedulerSelectCPU[CPUGetCurrent]);        {$ENDIF}              {Update CPU}        ThreadEntry.CurrentCPU:=ThreadEntry.TargetCPU;     end      else       begin        {Check Affinity}        if (ThreadEntry.Affinity and (1 shl CPUGetCurrent)) = 0 then         begin          {$IFDEF SCHEDULER_DEBUG}          Inc(SchedulerSelectAffinity[CPUGetCurrent]);          {$ENDIF}                  {Check CPUs 0 to SCHEDULER_CPU_COUNT - 1}          for Count:=0 to SCHEDULER_CPU_COUNT - 1 do           begin            if (ThreadEntry.Affinity and (1 shl Count)) <> 0 then             begin            {Update CPU}              ThreadEntry.CurrentCPU:=Count;            ThreadEntry.TargetCPU:=Count;                            Break;             end;            end;          end;        end;        end; ... though its possible the RTL could change the affinity elsewhere ... that unit is > 25,000 lines ... not fun to wade through.