From lkoeller@koellers.net  Mon Mar 15 12:33:05 2004
Return-Path: <lkoeller@koellers.net>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id A943016A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 15 Mar 2004 12:33:05 -0800 (PST)
Received: from sally.dts-online.net (sally.dts-online.net [212.62.68.35])
	by mx1.FreeBSD.org (Postfix) with ESMTP id CDD5A43D2F
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 15 Mar 2004 12:33:04 -0800 (PST)
	(envelope-from lkoeller@koellers.net)
Received: from door.koellers.net (213-182-122-22.teleos-web.de [213.182.122.22])
	by sally.dts-online.net (Postfix) with ESMTP id EF1C726DEA8
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 15 Mar 2004 21:33:02 +0100 (CET)
Received: from odie.koellers.net (root@odie.koellers.net [192.168.4.2])
	by door.koellers.net (8.12.9p2/8.12.8) with ESMTP id i2FKWhQn065065
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 15 Mar 2004 21:32:43 +0100 (CET)
	(envelope-from lkoeller@koellers.net)
Received: from odie.koellers.net (lkoeller@localhost [127.0.0.1])
	by odie.koellers.net (8.12.9p2/8.12.9) with ESMTP id i2FKWhLd014270
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 15 Mar 2004 21:32:43 +0100 (CET)
	(envelope-from lkoeller@odie.koellers.net)
Received: (from lkoeller@localhost)
	by odie.koellers.net (8.12.9p2/8.12.9/Submit) id i2FKWhOG014269;
	Mon, 15 Mar 2004 21:32:43 +0100 (CET)
	(envelope-from lkoeller)
Message-Id: <200403152032.i2FKWhOG014269@odie.koellers.net>
Date: Mon, 15 Mar 2004 21:32:43 +0100 (CET)
From: Lars Kller <lkoeller@koellers.net>
Reply-To: Lars Kller <lars@koellers.net>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: FreeBSD (OpenBSD) pthread implicit set/unset O_NONBLOCK flag
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         64313
>Category:       kern
>Synopsis:       FreeBSD (OpenBSD) pthread implicit set/unset O_NONBLOCK flag
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-threads
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 15 12:40:21 PST 2004
>Closed-Date:    Fri Nov 16 01:16:16 UTC 2007
>Last-Modified:  Fri Nov 16 01:16:16 UTC 2007
>Originator:     Lars Kller
>Release:        FreeBSD 4.9-RELEASE-p1 i386
>Organization:
Computing Center, University of Bielefeld, Germany
>Environment:
System: FreeBSD odie.koellers.net 4.9-RELEASE-p1 FreeBSD 4.9-RELEASE-p1 #3: Sun Jan 25 19:07:53 CET 2004 root@odie.koellers.net:/usr/src/sys/compile/ODIE i386

>Description:

            /*
	     * Work around a bug in OpenBSD & FreeBSD userspace pthreads
	     * implementations.
	     *
	     * The pthreads implementation under the hood sets O_NONBLOCK
	     * implicitly on all fds. This setting is not visible to the user
	     * application but is relied upon by the pthreads library to prevent
	     * blocking syscalls in one thread from halting all threads in the
	     * process. When a process exit()s or exec()s, the implicit
	     * O_NONBLOCK flags are removed from all fds, EVEN THOSE IT INHERITED.
	     * If another process is still using the inherited fds, there will
	     * soon be trouble.
	     *
	     * apcupsd is bitten by this issue after fork()ing a child process to
	     * run apccontrol.
	     *
	     * select() is conveniently immune to the O_NONBLOCK issue so we use
	     * that to make sure the following read() will not block.
	     */

>How-To-Repeat:

	Happens sometimes in the threaded apcupsd when it calls the apccontrol script.
>Fix:

	Don't know, I only notice the problem in the threaded apcups port which
	I'm the maintainer for.

	I receive the attached patch for the apcupsd port from

	Gary Bajaj <b04@interbaun.com>

	Originally it was written by Adam Kropelin:

###################################################
##### Attached to make the problem more clear #####
##### See description below		      #####
###################################################

--- ./src/apcnis.c	Fri Jul 18 05:32:19 2003
+++ ./apcupsd-3.10.11-debug3/src/apcnis.c	Fri Feb  6 21:19:14 2004
@@ -197,7 +197,6 @@
    int newsockfd, sockfd, childpid;
    struct sockaddr_in cli_addr;       /* client's address */
    struct sockaddr_in serv_addr;      /* our address */
-   socklen_t clilen;
    int tlog;
    int turnon = 1;
    struct s_arg *arg;
@@ -269,11 +268,7 @@
       /*
        * Wait for a connection from a client process.
        */
-       clilen = sizeof(cli_addr);
-       for (tlog=0; (newsockfd = accept(sockfd, (struct sockaddr *)&cli_addr, &clilen)) < 0; tlog -= 5*60 ) {
-	  if (errno == EINTR) {
-	     continue;
-	  }
+       for (tlog=0; (newsockfd = net_accept(sockfd, &cli_addr)) < 0; tlog -= 5*60 ) {
 	  if (tlog <= 0) {
 	     tlog = 60*60;
              log_event(ups, LOG_ERR,  "apcserver: accept error. ERR=%s",
--- ./src/lib/apclibnis.c	Sat Aug  3 18:49:45 2002
+++ ./apcupsd-3.10.11-debug3/src/lib/apclibnis.c
Fri Feb  6 21:38:58 2004
@@ -71,12 +71,50 @@

 static int read_nbytes(int fd, char *ptr, int nbytes)
 {
-    int nleft, nread;
-
+    int nleft, nread, rc;
+
+#if defined HAVE_PTHREADS && (defined HAVE_OPENBSD_OS || defined HAVE_FREEBSD_OS)
+    fd_set fds;
+#endif
+
     nleft = nbytes;
-    errno = 0;
+
     while (nleft > 0) {
+
 	do {
+
+#if defined HAVE_PTHREADS && (defined HAVE_OPENBSD_OS || defined HAVE_FREEBSD_OS)
+            /*
+	     * Work around a bug in OpenBSD & FreeBSD userspace pthreads
+	     * implementations.
+	     *
+	     * The pthreads implementation under the hood sets O_NONBLOCK
+	     * implicitly on all fds. This setting is not visible to the user
+	     * application but is relied upon by the pthreads library to prevent
+	     * blocking syscalls in one thread from halting all threads in the
+	     * process. When a process exit()s or exec()s, the implicit
+	     * O_NONBLOCK flags are removed from all fds, EVEN THOSE IT INHERITED.
+	     * If another process is still using the inherited fds, there will
+	     * soon be trouble.
+	     *
+	     * apcupsd is bitten by this issue after fork()ing a child process to
+	     * run apccontrol.
+	     *
+	     * select() is conveniently immune to the O_NONBLOCK issue so we use
+	     * that to make sure the following read() will not block.
+	     */
+	    do {
+		FD_ZERO(&fds);
+		FD_SET(fd, &fds);
+		rc = select(fd+1, &fds, NULL, NULL, NULL);
+	    } while (rc == -1 && (errno == EINTR || errno == EAGAIN));
+	    if (rc < 0)
+	    {
+		net_errno = errno;
+		return(-1);		 /* error */
+	    }
+#endif
+
 	    nread = read(fd, ptr, nleft);
 	} while (nread == -1 && (errno == EINTR || errno == EAGAIN));
 	if (nread <= 0) {
@@ -100,6 +138,15 @@

     nleft = nbytes;
     while (nleft > 0) {
+#if defined HAVE_PTHREADS && (defined HAVE_OPENBSD_OS || defined HAVE_FREEBSD_OS)
+	/*
+	 * Work around a bug in OpenBSD & FreeBSD userspace pthreads
+	 * implementations. Rationale is the same as described above.
+	 * This seemingly-pointless fcntl() call causes the pthreads
+	 * library to reapply the O_NONBLOCK flag appropriately.
+	 */
+	fcntl(fd, F_SETFL, fcntl(fd, F_GETFL));
+#endif
 	nwritten = write(fd, ptr, nleft);
 	if (nwritten <= 0) {
 	    net_errno = errno;
@@ -225,6 +272,13 @@
 	return -1;
     }
     /* connect to server */
+#if defined HAVE_PTHREADS && (defined HAVE_OPENBSD_OS || defined HAVE_FREEBSD_OS)
+    /*
+     * Work around a bug in OpenBSD & FreeBSD userspace pthreads
+     * implementations. Rationale is the same as described above.
+     */
+    fcntl(sockfd, F_SETFL, fcntl(sockfd, F_GETFL));
+#endif
     if (connect(sockfd, (struct sockaddr *) &tcp_serv_addr, sizeof(tcp_serv_addr)) < 0) {
         sprintf(net_errbuf, "tcp_open: cannot connect to server %s on port %d.\n\
 ERR=%s\n", host, port, strerror(errno));
@@ -243,6 +297,50 @@
     close(sockfd);
 }

+/*
+ * Accept a TCP connection.
+ * Returns -1 on error.
+ * Returns file descriptor of new connection otherwise.
+ */
+int net_accept(int fd, struct sockaddr_in *cli_addr)
+{
+    socklen_t clilen = sizeof(*cli_addr);
+    int newfd, rc;
+
+#if defined HAVE_PTHREADS && (defined HAVE_OPENBSD_OS || defined HAVE_FREEBSD_OS)
+    fd_set fds;
+#endif
+
+    do {
+
+#if defined HAVE_PTHREADS && (defined HAVE_OPENBSD_OS || defined HAVE_FREEBSD_OS)
+	/*
+	 * Work around a bug in OpenBSD & FreeBSD userspace pthreads
+	 * implementations. Rationale is the same as described above.
+	 */
+	do {
+	    FD_ZERO(&fds);
+	    FD_SET(fd, &fds);
+	    rc = select(fd+1, &fds, NULL, NULL, NULL);
+	} while (rc == -1 && (errno == EINTR || errno == EAGAIN));
+	if (rc < 0)
+	{
+	    net_errno = errno;
+	    return(-1);		 /* error */
+	}
+#endif
+
+	newfd = accept(fd, (struct sockaddr*)cli_addr, &clilen);
+    } while (newfd == -1 && (errno == EINTR || errno == EAGAIN));
+
+    if (newfd < 0)
+    {
+	net_errno = errno;
+        return(-1);		 /* error */
+    }
+
+    return newfd;
+}

 int	upserror, syserrno;

--- ./include/apc_nis.h	Tue May 28 09:34:24 2002
+++ ./apcupsd-3.10.11-debug3/include/apc_nis.h
Fri Feb  6 21:19:14 2004
@@ -40,4 +40,7 @@
 /* Close the network connection */
 void net_close(int sockfd);

+/* Wait for and accept a new TCP connection */
+int net_accept(int fd, struct sockaddr_in *cli_addr);
+
 extern int  upserror, syserrno;
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-threads 
Responsible-Changed-By: kris 
Responsible-Changed-When: Mon Mar 15 16:33:11 PST 2004 
Responsible-Changed-Why:  
Assign to threads developers 

http://www.freebsd.org/cgi/query-pr.cgi?pr=64313 

From: Craig Rodrigues <rodrigc@crodrigues.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc: lars@koellers.net
Subject: Re: kern/64313: FreeBSD (OpenBSD) pthread implicit set/unset O_NONBLOCK flag
Date: Tue, 16 Mar 2004 13:10:53 -0500

 Hi,
 
 What is the point of this PR?  It contains
 a patch to what looks like a program in ports.
 
 The O_NONBLOCK issue with libc_r is known.
 Take a look at:
 http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/41331
 
 This problem doesn't occur with the
 new KSE ( http://www.freebsd.org/kse ) libpthread implementation
 of threads in FreeBSD 5.x.
 
 Most likely this problem will not be fixed in libc_r,
 since most efforts these days are focused on
 getting libpthread solid in 5.x.
 
 Dan Eischen has been suspending (and not closing) these kinds of PR's, i.e.
 problems that occur in libc_r but not in libpthread,
 because most likely they will not be fixed in libc_r
 (but they may be one day, if someone takes an interest in fixing up libc_r).
 
 

From: Valentin Nechayev <netch@ivb.nn.kiev.ua>
To: Lars =?iso-8859-1?Q?K=F6ller?= <lars@koellers.net>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/64313: FreeBSD (OpenBSD) pthread implicit set/unset O_NONBLOCK flag
Date: Sun, 21 Mar 2004 17:04:41 +0200

  Mon, Mar 15, 2004 at 21:32:43, lkoeller (Lars Koeller) wrote about "kern/64313: FreeBSD (OpenBSD) pthread implicit set/unset O_NONBLOCK flag": 
 
 LKl> 	     * The pthreads implementation under the hood sets O_NONBLOCK
 LKl> 	     * implicitly on all fds. This setting is not visible to the user
 LKl> 	     * application but is relied upon by the pthreads library to prevent
 LKl> 	     * blocking syscalls in one thread from halting all threads in the
 LKl> 	     * process. When a process exit()s or exec()s, the implicit
 LKl> 	     * O_NONBLOCK flags are removed from all fds, EVEN THOSE IT INHERITED.
 
 This is unavoidable feature of libc_r. It can't detect whether O_NONBLOCK
 was set by itself or by another application, and can't organize thread sleeping
 without O_NONBLOCK.
 If this is critical, use another thread library (e.g. linuxthreads on 4.x,
 libkse or libthr on 5.x).
 Libc_r is unrepairable for your request (unless brain-damaged unix api
 with its per-opened-object O_NONBLOCK will be fixed). So your PR is
 meaningless and should be closed.
 
 
 -netch-

Adding to audit trail from misfiled followup ports/64349:

 >What is the point of this PR?  It contains
 >a patch to what looks like a program in ports.
 
 Yes, I was only hit by this bug cause I'm the apcupsd maintainer, and
 someone sent me the apcupsd workaround patch.
 
 >The O_NONBLOCK issue with libc_r is known. Take a look at:
 >http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/41331
 
 Sorry, I've overseen this when looking for pthread issues.
 
 >This problem doesn't occur with the new KSE ( http://www.freebsd.org/kse )
 >libpthread implementation of threads in FreeBSD 5.x.
 >
 >Most likely this problem will not be fixed in libc_r, since most efforts
 >these days are focused on getting libpthread solid in 5.x.
 >
 >Dan Eischen has been suspending (and not closing) these kinds of PR's,
 >i.e. problems that occur in libc_r but not in libpthread, because most
 >likely they will not be fixed in libc_r (but they may be one day, if
 >someone takes an interest in fixing up libc_r).
 
 Hmm, I understand focussing on 5.X, but there are a lot of production
 machines out there, that will not so soon be updated to 5.X when 4.Y is
 running very stable on them.
 
 So it would be nice if someone could backport the solution.
 
 Lars
State-Changed-From-To: open->suspended 
State-Changed-By: linimon 
State-Changed-When: Mon Mar 29 09:15:36 PST 2004 
State-Changed-Why:  
Incorporating text from misfiled followup kern/64349. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=64313 
State-Changed-From-To: suspended->open 
State-Changed-By: linimon 
State-Changed-When: Mon Mar 29 09:16:56 PST 2004 
State-Changed-Why:  
GNATS is one stupid program ... 

http://www.freebsd.org/cgi/query-pr.cgi?pr=64313 
State-Changed-From-To: open->suspended 
State-Changed-By: linimon 
State-Changed-When: Mon Mar 29 09:17:37 PST 2004 
State-Changed-Why:  
Following the advice of respondants to this PR, mark this as 
suspended until and unless a volunteer is found who is willing 
to work on known libc_r issues.  While this would be nice, it 
is unlikely, considering that 5.3 will probably mark the 
transition to 5-STABLE. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=64313 
State-Changed-From-To: suspended->closed 
State-Changed-By: kmacy 
State-Changed-When: Fri Nov 16 01:15:27 UTC 2007 
State-Changed-Why:  

libc_r is no longer supported 

http://www.freebsd.org/cgi/query-pr.cgi?pr=64313 
>Unformatted:
This is an artifact of libc_r. libc_r is no longer supported.
