Subj : Re: A multithreading benchmark
To   : comp.programming.threads
From : Uenal Mutlu
Date : Wed Jun 08 2005 04:25 am

"doug" <noone@nowhere.co.uk> wrote
> "Uenal Mutlu" wrote
> > "chris noonan" <usenet@leapheap.co.uk> wrote
> >> I am designing a benchmark program for investigating
> >> multithreaded performance. I will use the program
> >> for comparing various heap managers, but it is bound
> >> to be useful for other purposes.
> >>
> >> These are the design criteria I have so far:
> >>
> >> 1. The benchmark will be semi-synthetic, in that
> >> it will be constructed for the stated purpose,
> >> but will perform some useful task, and could be
> >> mistaken for a 'real' application.
> >>
> >> 2. The program will take the server role of a
> >> client-server system, and will run on a
> >> thread-per-connection basis.
> >>
> >> 3. There will be a fixed amount of work, the
> >> number of threads will be parameterisable for
> >> each run of the benchmark, and the workload
> >> will be divided between however many threads
> >> there are.
> >>
> >> 4. The program will be coded in C++ in natural
> >> style.
> >>
> >> 5. The dynamic memory profile of the running
> >> benchmark will conform as far as possible to
> >> empirical measurements of average allocation
> >> size, duration etc. to be found in the
> >> literature.
> >>
> >> Anything else worth considering?
> >>
> >> Initially the program will be targeted to
> >> Microsoft Windows platforms, as there is a
> >> dearth of suitable benchmarks.
> >>
> >> Is there some way of simulating a number of
> >> clients without the complication of having
> >> multiple machines and real network connections?
> >
> > A benchmark for thread performance should be done completely
> > on the same one local machine. Once on a single CPU machine
> > and then on a multiple CPU machine.
> > Adding other factors like network etc. would disturb the benchmark.
> >
>
> It's a tricky one, eh?
>
> Normally, you'd want your benchmark to be useful in real life scenarios.
> But...
> - if you don't do real network I/O then you will miss how the threading
> behaves in the presence of e.g. the TCP/IP network stack (this has bitten me
> before)
> - but you don't really want to count the cost of network time and client,
> now, so maybe you shouldn't do it.
> - or maybe putting the clients on the local machine would work, but you
> don't want to have to account for their activity
>
> How do other people tune things like threadpools, etc?  We normally go for
> option 1 above - use real clients and network travel.

You can simulate both server and clients on the same machine.
Then using network functions (ie. local adress(es) and ports) is even ok.
The next best way would be using machines within 1 hop.
Of course it's up to what you exactly want to measure. For measuring thread
performance I would not go on wire, but using the IP adress(es) of the local
machine would be ok (plus same or different ports (depending on protocol)
on that machine).

.