Subj : Re: A multithreading benchmark
To   : comp.programming.threads
From : David Schwartz
Date : Tue Jun 07 2005 06:54 pm


"Uenal Mutlu" <520001085531-0001@t-online.de> wrote in message 
news:d85dgl$n4g$00$1@news.t-online.com...

> You can simulate both server and clients on the same machine.
> Then using network functions (ie. local adress(es) and ports) is even ok.
> The next best way would be using machines within 1 hop.
> Of course it's up to what you exactly want to measure. For measuring 
> thread
> performance I would not go on wire, but using the IP adress(es) of the 
> local
> machine would be ok (plus same or different ports (depending on protocol)
> on that machine).

    You will probably never see duplicate packets, out of order packets, 
missing packets, consecutive short packets, defective packets, or any number 
of other issues in such a test scenario. If you're trying to get an idea how 
networking code will work on the Internet, this won't give you a clue.

    Just be warned, you may get larger chunks from 'recv' or see fewer 
partial sends. You may get burstier data (where you get lots of data on 
multiple connections and then no data on any rather than continuous bits on 
a small number of connections at a time), which can improve your performance 
by allowing you to do more work per context switch.

    On the other hand, if you're just interested in what your code does 
after 'send' or 'recv' returns, it will give you some idea.

    One other issue, depending upon the protocol you're dealing with, is 
that your test clients may not do a good job of representing real-world 
clients. For example, if your protocol is HTTP, what fraction of real-world 
clients will send an 'If-Modified-Since' header? What fraction will ask for 
HTTP/1.0 and which HTTP/1.1? How many will try to request an favicon file? 
There are similar issues for most protocols.

    Benchmarking is *very* hard, and multi-threading programs are even 
harder to benchmark (especially if they're poorly coded in the first place). 
Good design principles will tend to make your code less fragile, and 
hopefully it will behave in the real world more like it does in the 
benchmarks, but this only applies to your code if you have experience in 
making this happen. Otherwise, beginning multi-threaded programmers get 
nasty surprises. (Take advice from experienced programmers!)

    DS

.