Subj : Is there a deadlock or race condition here?
To   : comp.programming.threads
From : lindahlb
Date : Wed Apr 13 2005 10:26 pm

I've got two processes with some shared memory acting as a ring buffer.
One process is a producer (which produces data on the ring buffer), the
other process is a consumer, which takes data off the ring buffer and
writes it to disk (a logging process). The producer produces pieces 136
bytes at a time. The consumer consumes as much data as possible in 512
byte pieces (O_DIRECT).

The objects, produce and consume, are semaphores. The set method
increases the semaphore count by MAXIMUM, the wait method decreases
(and waits) the semaphore count by MINIMUM. Note that an ERANGE errno
is ignored, because I'm guessing that it can legitimately occur (or am
I wrong?). This basically makes the semaphore objects act like atomic
manual reset events (see Win32). The set method sets the event, the
wait method waits (if necessary) for the event and then resets it on
the way out (atomically).

The producer side will be called many times and should be low overhead,
the consumer side can be acceptably slower (especially since it does
direct file IO). However, in an attempt to cut down on the overhead,
I'm concerned that I may have introduced a possible deadlock or race
condition. I've gone through many different scenarios on paper and
can't come up with any possible deadlocks or race conditions, but I
would really like someone with more experience to take a look at it. It
should be capable of SMP execution. The atomic operations are performed
with the intel 'xadd' instruction, with the lock prefix under SMP and
hyperthreaded environments - it could be done with pseudo-cmpxchg loops
on other target machines as well.

If deadlock or race conditions do exist, how can I elimate them? In the
solution, I'd like to not use pipes (small buffers and extra overhead)
and I'd also like to avoid using semaphore calls for every execution in
the producer - as it is called quite often and extra delays there
should be avoided.

producer:

while(len + 136 >= maxlen)
  consume.wait();

add_data_to_buffer(data);

// atomically adds 136 to len and returns the original value of len
(xadd)
if (atomic_get_add(len, 136) == 0)
  produce.set();

consumer:

while(len < 512)
  produce.wait();

n = write_data_to_disk(len);

// atomically subtracts n from len and returns the original value of
len (xadd)
if (atomic_get_sub(len, n) == maxlen)
  consume.set();

.