Subj : Re: Binding a thread to particular processor Linux
To   : comp.programming.threads
From : doug
Date : Sun Mar 27 2005 11:45 am

Thanks for getting back to me, Ben.

So it seems my understanding of the thread affinity stuff is ok. As you 
point out, the existence of this api is unusual in my 2.4 system, but to cut 
a long story short, that's fine in my circumstances.

Now, about the interrupts:
- my investigation on the web reveals that my kernel round-robins 
responsibility for handling interrupts amongst the CPUs.  Each cpu handles 
interrupts for a short period, then this responsibility moves to the next 
CPU.  And so on...
- my threads are each bound to a specific processor.  Some of these threads 
perform lots of i/o.  All threads performs lots of number crunching (this is 
why they've been bound to a specific CPU - so that the data they're 
crunching doesn't get thrashed between cpu caches)
- I/O causes interrupts from the hardware to the CPU.  (Sometimes these get 
back to the process as e.g. SIGPIPE.)  But if these interrupts are sent to a 
processor not running the thread involved in the I/O, then we take a hit as 
we have to communicate the interrupt/signal between CPUs

THE LAST POINT ABOVE MAY BE COMPLETELY WRONG!  If it is, please correct me, 
as i'm chasing a red herring.

(The symptom is that a multithreaded prog runs slower on multi-cpu hardware 
than single-cpu hardware.  I've taken care to seperate data structures so 
they lie on different memory pages, bind all threads working on the same 
memory page to the same cpu, and there are no read/write global data 
structures or semaphores - only global read-only data.
Even if I'd cocked up the allocation of data/pages/etc., binding all threads 
to one processor shows an improvement, but it's still not as good as running 
in uniprocessor mode.  I can only explain this by saying that something in 
my process must be running on another cpu.  Since I bind all my threads to a 
single CPU, the only explanation i can find is interrupts.  I'm likely 
wrong!)

Any help appreciated,

Doug

"Ben Hutchings" <ben-public-nospam@decadentplace.org.uk> wrote in message 
news:slrnd447tg.3k7.ben-public-nospam@decadentplace.org.uk...
> doug wrote:
>> To answer my own question, sched_setaffinity allows you to set processor
>> affinity.  I've managed to find object code on a 2.4 system with this
>> function, and it seems to work.
>
> My manual page says "The affinity syscalls were introduced  in Linux
> kernel 2.5.8."  Maybe your vendor backported them to 2.4, but you had
> better not rely on them being present in other distributions with
> 2.4 kernels.
>
>> It also seems to provide thread affinity - at least the benchmark results
>> change as I assign different threads to different processors, so i seems 
>> to
>> be affecting per-thread, rather than just per process.  Does this 
>> surprise
>> anyone?
>
> Not really, since the threads are actually implemented as separate
> processes in the 2.4 kernel.
>
>> Now I'm stuck on setting interrupt affinity.  Anyone know how to do this?
>> Or does sched_setaffinity do this for me?
> <snip>
>
> Surely interrupts aren't associated with any process, so this wouldn't
> have any effect on them.  Do you mean signals?  Signals run within an
> existing thread so they should, I think, be subject to the affinity of
> that thread.
>
> -- 
> Ben Hutchings
> In a hierarchy, every employee tends to rise to his level of incompetence.

.