From nobody@FreeBSD.org  Sun Aug  4 14:34:30 2002
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 74E6F37B400
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  4 Aug 2002 14:34:30 -0700 (PDT)
Received: from www.freebsd.org (www.FreeBSD.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 2963D43E5E
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  4 Aug 2002 14:34:30 -0700 (PDT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.12.4/8.12.4) with ESMTP id g74LYTOT074372
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 4 Aug 2002 14:34:29 -0700 (PDT)
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.12.4/8.12.4/Submit) id g74LYTM6074371;
	Sun, 4 Aug 2002 14:34:29 -0700 (PDT)
Message-Id: <200208042134.g74LYTM6074371@www.freebsd.org>
Date: Sun, 4 Aug 2002 14:34:29 -0700 (PDT)
From: Antti Louko <alo@iki.fi>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Pthread library open sets O_NONBLOCK flag and causes unnecessary EAGAIN errors especially with /dev/stdout.
X-Send-Pr-Version: www-1.0

>Number:         41331
>Category:       kern
>Synopsis:       Pthread library open sets O_NONBLOCK flag and causes unnecessary EAGAIN errors especially with /dev/stdout.
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-threads
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Aug 04 14:40:01 PDT 2002
>Closed-Date:    Sun Jun 04 23:52:28 GMT 2006
>Last-Modified:  Sun Jun 04 23:52:28 GMT 2006
>Originator:     Antti Louko
>Release:        4.5
>Organization:
>Environment:
FreeBSD xxx.louko.com 4.5-RELEASE FreeBSD 4.5-RELEASE #1: Tue Mar  5 23:48:33 EET 2002     root@xxx.louko.com:/xxx/src/sys/compile/ALO  i386   
>Description:
When using threads, the threads library
(/usr/src/lib/libc_r/uthread/uthread_fd.c) sets O_NONBLOCK for each
new opened file and fails to emulate the blocking write(2) correctly.
This affects eg. when a program opens /dev/stdout for writing and thje
reader process is slower than the writer.  The writer gets EAGAIN
error which shouldn't happen when the open call didn't specify
O_NONBLOCK flag.

>How-To-Repeat:
Run the following program (compiled with gcc -pthread -o writer
writer.c) and feed its output to the shell script:

./writer | sh reader.sh

reader.sh:
----------------------------------------
#! /bin/sh
while :
do
  dd of=/dev/null bs=1k count=1
  sleep 1
done
----------------------------------------

writer.c
----------------------------------------
#include <stdio.h>
#include <errno.h>
#include <fcntl.h>

main(int argc, char **argv)

{
  char  buf[1024];
  int   fd;
  int   i;
  int   total = 0;

  fd = open("/dev/stdout",O_WRONLY|O_CREAT|O_TRUNC,0666);
  if (fd == -1) {
    int e;
    e = errno;
    fprintf(stderr,"open, errno: %d\n",e);
    exit(1);
  }
  for(i = 0; i < sizeof(buf); i++)
    buf[i] = 'a' + (i % 26);
  for(;;) {
    int n;
    n = write(fd,buf,sizeof(buf));
    if (n > 0)
      total += n;
    if (n != sizeof(buf)) {
      int       e;
      e = errno;
      fprintf(stderr,"write, errno: %d\n",e);
      fprintf(stderr,"Total bytes written %d\n",total);
      exit(1);
    }
  }
}
----------------------------------------
Run the following program (compiled with gcc -pthread -o writer
writer.c) and feed its output to the shell script:

./writer | sh reader.sh

reader.sh:
----------------------------------------
#! /bin/sh
while :
do
  dd of=/dev/null bs=1k count=1
  sleep 1
done
----------------------------------------

writer.c
----------------------------------------
#include <stdio.h>
#include <errno.h>
#include <fcntl.h>

main(int argc, char **argv)

{
  char  buf[1024];
  int   fd;
  int   i;
  int   total = 0;

  fd = open("/dev/stdout",O_WRONLY|O_CREAT|O_TRUNC,0666);
  if (fd == -1) {
    int e;
    e = errno;
    fprintf(stderr,"open, errno: %d\n",e);
    exit(1);
  }
  for(i = 0; i < sizeof(buf); i++)
    buf[i] = 'a' + (i % 26);
  for(;;) {
    int n;
    n = write(fd,buf,sizeof(buf));
    if (n > 0)
      total += n;
    if (n != sizeof(buf)) {
      int       e;
      e = errno;
      fprintf(stderr,"write, errno: %d\n",e);
      fprintf(stderr,"Total bytes written %d\n",total);
      exit(1);
    }
  }
}
----------------------------------------
Run the following program (compiled with gcc -pthread -o writer
writer.c) and feed its output to the shell script:

./writer | sh reader.sh

reader.sh:
----------------------------------------
#! /bin/sh
while :
do
  dd of=/dev/null bs=1k count=1
  sleep 1
done
----------------------------------------

writer.c
----------------------------------------
#include <stdio.h>
#include <errno.h>
#include <fcntl.h>

main(int argc, char **argv)

{
  char  buf[1024];
  int   fd;
  int   i;
  int   total = 0;

  fd = open("/dev/stdout",O_WRONLY|O_CREAT|O_TRUNC,0666);
  if (fd == -1) {
    int e;
    e = errno;
    fprintf(stderr,"open, errno: %d\n",e);
    exit(1);
  }
  for(i = 0; i < sizeof(buf); i++)
    buf[i] = 'a' + (i % 26);
  for(;;) {
    int n;
    n = write(fd,buf,sizeof(buf));
    if (n > 0)
      total += n;
    if (n != sizeof(buf)) {
      int       e;
      e = errno;
      fprintf(stderr,"write, errno: %d\n",e);
      fprintf(stderr,"Total bytes written %d\n",total);
      exit(1);
    }
  }
}
----------------------------------------

>Fix:
      
>Release-Note:
>Audit-Trail:

From: Archie Cobbs <archie@packetdesign.com>
To: freebsd-gnats-submit@FreeBSD.org, alo@iki.fi
Cc:  
Subject: Re: misc/41331: Pthread library open sets O_NONBLOCK flag and causes 
 unnecessary EAGAIN errors especially with /dev/stdout.
Date: Mon, 09 Sep 2002 16:06:25 -0700

 This problem doesn't occur for me on 4.6-stable. Does it still occur
 for you? If not, let me know and I'll close this bug.
 
 Thanks,
 -Archie
 
 __________________________________________________________________________
 Archie Cobbs     *     Packet Design     *     http://www.packetdesign.com

From: alo@louko.com (Antti Louko)
To: archie@packetdesign.com
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/41331: Pthread library open sets O_NONBLOCK flag and causes 
 unnecessary EAGAIN errors especially with /dev/stdout.
Date: 10 Sep 2002 04:45:11 -0000

 I just tried it in 4.6 Release and 4.6.2 and it still occurs.  This
 bug makes it for example quite impossible to use /dev/stdout in python
 scripts if python has been compiled with threads support.
 
 Are you sure that you compiled the writer.c with:
 
 gcc -pthread -o writer writer.c
 
 What happens is:
 
 alo@toivoton 123: ./writer | sh reader.sh
 write, errno: 35
 Total bytes written 16384
 1+0 records in
 1+0 records out
 1024 bytes transferred in 0.000032 secs (31932842 bytes/sec)
 1+0 records in
 1+0 records out
 1024 bytes transferred in 0.000035 secs (29217465 bytes/sec)
 .
 . Repetead until:
 .
 0+0 records in
 0+0 records out
 0 bytes transferred in 0.000018 secs (0 bytes/sec)

From: Archie Cobbs <archie@packetdesign.com>
To: Antti Louko <alo@louko.com>
Cc: freebsd-gnats-submit@FreeBSD.org, deischen@freebsd.org
Subject: Re: misc/41331: Pthread library open sets O_NONBLOCK flag and causes 
 unnecessary EAGAIN errors especially with /dev/stdout.
Date: Tue, 10 Sep 2002 08:26:40 -0700

 Antti Louko wrote:
 > I just tried it in 4.6 Release and 4.6.2 and it still occurs. 
 
 I think this is a kernel bug, rather than a pthread bug.
 
 What's happening is that the file descriptor associated with
 /dev/stdout is "inheriting" the flags associated with file
 descriptor 0. This happens on both -stable and -current.
 
 This seems like broken behavior to me: flags should be associated
 with the file descriptor, not the underlying device or entity.
 But I don't know what the "official" semantics of /dev/stdout
 are supposed to be.
 
 The program below demonstrates the problem:
 
   $ cc -g -Wall -o flags flags.c
   $ ./flags 
   O_NONBLOCK is set
 
 -Archie
 
 __________________________________________________________________________
 Archie Cobbs     *     Packet Design     *     http://www.packetdesign.com
 
 #include <stdio.h>
 #include <fcntl.h>
 #include <err.h>
 
 int
 main(int argc, char **argv)
 {
         int flags;
         int fd;
 
         if ((flags = fcntl(0, F_GETFL, 0)) == -1)
                 err(1, "fcntl");
         if (fcntl(0, F_SETFL, flags | O_NONBLOCK) == -1)
                 err(1, "fcntl");
         if ((fd = open("/dev/stdout", O_WRONLY, 0)) == -1)
                 err(1, "open");
         if ((flags = fcntl(fd, F_GETFL, 0)) == -1)
                 err(1, "fcntl");
         printf("O_NONBLOCK is %s\n", (flags & O_NONBLOCK) ? "set" : "not set");
         return (0);
 }

From: alo@louko.com (Antti Louko)
To: archie@packetdesign.com
Cc: freebsd-gnats-submit@FreeBSD.org, deischen@freebsd.org
Subject: Re: misc/41331: Pthread library open sets O_NONBLOCK flag and causes 
 unnecessary EAGAIN errors especially with /dev/stdout.
Date: 10 Sep 2002 16:26:31 -0000

    Sender: archie@dellroad.org
    Date: Tue, 10 Sep 2002 08:26:40 -0700
    From: Archie Cobbs <archie@packetdesign.com>
    Organization: Packet Design
    X-Accept-Language: en
    CC: freebsd-gnats-submit@FreeBSD.org, deischen@freebsd.org
 
    Antti Louko wrote:
    > I just tried it in 4.6 Release and 4.6.2 and it still occurs. 
 
    I think this is a kernel bug, rather than a pthread bug.
 
    What's happening is that the file descriptor associated with
    /dev/stdout is "inheriting" the flags associated with file
    descriptor 0. This happens on both -stable and -current.
 
    This seems like broken behavior to me: flags should be associated
    with the file descriptor, not the underlying device or entity.
    But I don't know what the "official" semantics of /dev/stdout
    are supposed to be.
 
    The program below demonstrates the problem:
 
      $ cc -g -Wall -o flags flags.c
      $ ./flags 
      O_NONBLOCK is set
 
 But the O_NONBLOCK is not set in the kernel.  It is probably set in
 /usr/src/lib/libc_r/uthread/uthread_fd.c (which sets O_NONBLOCK flags
 for fds in range 0..2) or somewhere else in libc_r because it doesn't
 happen without thread library.  But someone who is familiar with the
 pthreads library should check what should happen.
 
 I don't feel competent to fix this.
 
 Regards,
 
 	Antti

From: "Dorr H. Clark" <dclark@applmath.scu.edu>
To: freebsd-gnats-submit@FreeBSD.org, alo@iki.fi
Cc:  
Subject: Re: misc/41331: Pthread library open sets O_NONBLOCK flag and causes 
 unnecessary EAGAIN errors especially with /dev/stdout.
Date: Mon, 05 May 2003 20:37:20 -0700

 42943 is a duplicate of 41331, or at best an extended symptom.
 
 First of all, both 41331 and 42943 occur on purpose, 
 and this is not a regression introduced by some other change,
 it is part of the original pthread code.
 
 We believe that this behavior is a side effect of the FreeBSD
 threads implementation, and that 41331 and 42943 are normal behavior.
 These bugs can only plausibly be addressed by rearchitecting 
 the threads implementation, possibly in the context of 5.x.
 
 In FreeBSD, the pthread implementation is as a user-level thread
 package,
 in other words, all threads execute in the context of a single process, 
 and the thread scheduler is hidden in the threaded version of libc.
 As a side-effect of this implementation, when a blocking system call 
 is executed, the kernel only knows about the process, and 
 this results in all the threads being blocked.
 
 To avoid blocking the entire thread group, the stdin, stdout and stderr 
 file descriptors are made non-blocking.
 
 Note that while these two bugs were filed against the recent 4.x
 releases,
 there is nothing new in this area.  The code governing this behavior
 has 2.x and 3.x labels in the CVS tree.
 
 In 41331, there is a test program, and as part of our investigation
 we tried this program on both Solaris and Linux.  Neither one exhibits
 the 41331 behavior, however readers who are familiar with the threads
 implementation will realize that these are three radically different
 threads implementations, accounting for the discrepancy.
 
 If the 41331 behavior is intolerable, there are two possible approaches,
 either wait for a pthreads implementation on top of the kernel threads 
 in FreeBSD 5.0 or install the linuxthreads port, which simulates the
 Linux 
 way of handling threads as separate processes. 
 
 NOTE:  The following code excerpt is from 4.7-STABLE !
 
 Here are two code snips which control this behavior.
 
 code snip from libc_r/uthread/pthread_private.h :
 /*
  * Standard I/O file descriptors need special flag treatment since
  * setting one to non-blocking does all on *BSD. Sigh. This array
  * is used to store the initial flag settings.
  */
 SCLASS int      _pthread_stdio_flags[3];
 
 code snip from uthread_fd.c :
 
                         /* Check if a stdio descriptor: */
                         if ((fd < 3) && (_pthread_stdio_flags[fd] !=
 -1))
                                 /*
                                  * Use the stdio flags read by
                                  * _pthread_init() to avoid
                                  * mistaking the non-blocking
                                  * flag that, when set on one
                                  * stdio fd, is set on all stdio
                                  * fds.
                                  */
                                 entry->flags = _pthread_stdio_flags[fd];
 
                         /*
                          * Make the file descriptor non-blocking.
                          * This might fail if the device driver does
                          * not support non-blocking calls, or if the
                          * driver is naturally non-blocking.
                         */
                         saved_errno = errno;
                         _thread_sys_fcntl(fd, F_SETFL,
                             entry->flags | O_NONBLOCK);
                         errno = saved_errno;
 
 Rashmi Venkatesh, engineer
 Dorr H. Clark, advisor
 COEN 284 - Operating Systems Case Study 
 Santa Clara University, 
 Santa Clara CA.
Responsible-Changed-From-To: freebsd-bugs->freebsd-threads 
Responsible-Changed-By: kris 
Responsible-Changed-When: Sat Jul 12 18:38:58 PDT 2003 
Responsible-Changed-Why:  
Assign to threads mailing list 

http://www.freebsd.org/cgi/query-pr.cgi?pr=41331 
State-Changed-From-To: open->closed 
State-Changed-By: rodrigc 
State-Changed-When: Sun Jun 4 23:52:05 UTC 2006 
State-Changed-Why:  
New libpthread in FreeBSD 5.x does not exhibit this problem. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=41331 
>Unformatted:
