From nobody@FreeBSD.org  Thu May 25 17:38:29 2006
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 1B57416ACB2
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 25 May 2006 17:38:29 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 4D06B43D58
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 25 May 2006 17:38:26 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k4PHcQao080549
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 25 May 2006 17:38:26 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id k4PHcQnt080548;
	Thu, 25 May 2006 17:38:26 GMT
	(envelope-from nobody)
Message-Id: <200605251738.k4PHcQnt080548@www.freebsd.org>
Date: Thu, 25 May 2006 17:38:26 GMT
From: Kurt Miller <lists@intricatesoftware.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: close() socket deadlocks blocked threads
X-Send-Pr-Version: www-2.3

>Number:         97921
>Category:       kern
>Synopsis:       [socket] close() socket deadlocks blocked threads
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    rwatson
>State:          analyzed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu May 25 17:40:17 GMT 2006
>Closed-Date:    
>Last-Modified:  Mon Jan 14 04:20:02 UTC 2008
>Originator:     Kurt Miller
>Release:        6.1-release
>Organization:
Intricate Software
>Environment:
FreeBSD freebsd6-1.zonesville.com 6.1-RELEASE FreeBSD 6.1-RELEASE #0: Sun May  7 04:42:56 UTC 2006     root@opus.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP  i386
>Description:
When a thread is blocked waiting for data on a socket and another thread closes the socket, the blocked thread remains blocked indefinitely. Both kse and thr have this issue. c_r returns with -1 errno==EBADF. Solaris
returns with -1 errno==0.

I originally posted this on the freebsd-threads list. Daniel Eischen replied indicating this is not a thread library issue.
>How-To-Repeat:
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <sys/param.h>
#include <netdb.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <pthread.h>

static void *
start_routine(void *arg)
{
  int sock1 = (int)arg;
  struct sockaddr remote_addr;
  char readBuf;
  int n, remote_addr_len = sizeof(struct sockaddr);

  n = recvfrom(sock1, &readBuf, 1, 0, &remote_addr, &remote_addr_len);

  if (n == -1) {
    printf("unblocked with errno = %d\n", errno);
  }

  return (0);
}

void
buildAddr4(struct sockaddr_in *addr4, int address, short port) {
  memset((char *) addr4, 0, sizeof(struct sockaddr_in));
  addr4->sin_port = htons(port);
  addr4->sin_addr.s_addr = (uint32_t) htonl(address);
  addr4->sin_family = AF_INET;
}

int
main() {
  int sock1;
  struct sockaddr addr;
  pthread_t thread1;
  void *value_ptr;

  buildAddr4((struct sockaddr_in *)&addr, 0, 0);

  if ((sock1 = socket(AF_INET, SOCK_DGRAM, 0)) < 0)
    exit(1);
  if (bind(sock1, (struct sockaddr *)&addr, sizeof(struct sockaddr_in)) != 0)
    exit(1);

  pthread_create(&thread1, NULL, start_routine, (void *)sock1);
  sleep(1);

  close(sock1);
  pthread_join(thread1, &value_ptr);

  return (0);
}
>Fix:

>Release-Note:
>Audit-Trail:

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Kurt Miller <lists@intricatesoftware.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/97921: close() socket deadlocks blocked threads
Date: Wed, 7 Jun 2006 14:40:55 +0400

   I have run this testcase on Linux 2.6.16.19. The program blocks,
 but if suspended/resumed it returns. recvfrom() returns -1, errno
 is EBADF.
 
   I think that we should not block (like Solaris), and return EBADF
 (like Linux).
 
 -- 
 Totus tuus, Glebius.
 GLEBIUS-RIPN GLEB-RIPE
Responsible-Changed-From-To: freebsd-bugs->rwatson 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Wed Jun 7 14:30:14 UTC 2006 
Responsible-Changed-Why:  
Grab ownership of this PR.  This is a somewhat complicated issue 
to address, as socket shutdown occurs only once the last user 
space reference to the socket is released.  This occurs when the 
last outstanding system call after close() takes place.  To 
change this, we will need to change the socket state engine some 
to issue a cancellation on last close rather than last reference, 
which is a notion we sort of (but not entirely) support.  Are 
there any standards that specify this or related behavior that 
we should be aware of? 


http://www.freebsd.org/cgi/query-pr.cgi?pr=97921 

From: "Arne H. Juul" <arnej@pvv.ntnu.no>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/97921: [socket] close() socket deadlocks blocked threads
Date: Mon, 11 Dec 2006 17:09:31 +0100 (CET)

 Actually, a cancellation on last close() could still be wrong (too late), 
 there might be other file descriptors referring to the same underlying 
 socket.  The issue here is that the file descriptor used in the read() 
 requests is invalidated because it's closed (or overwritten with something 
 else, with dup2()).  This means the read() must be interrupted somehow; 
 then after that the normal reference counting should be used to trigger 
 socket shutdown.
 
 I also don't quite understand the comment that this isn't thread related, 
 as there's no way for a process to both be in a blocking read() on a 
 file descriptor and close() the same file descriptor at the same time 
 without using threads.
 
    -  Arne H. J.
 

From: Kurt Miller <lists@intricatesoftware.com>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/97921: [socket] close() socket deadlocks blocked threads
Date: Mon, 11 Dec 2006 18:08:08 -0500

 Daniel Eischen's comment "not a thread library issue" was
 to indicate the problem is located in the kernel not the
 kse or thr threading libraries.
 
 This issue may also apply to shutdown() in addition to
 close() and dup2().
State-Changed-From-To: open->analyzed 
State-Changed-By: rwatson 
State-Changed-When: Sun Jan 13 18:19:44 UTC 2008 
State-Changed-Why:  
We seem to have an understanding of the way it does behave (close() doesn't 
start a disconnect when two threads are using the same fd), so mark 
analyzed. 

Could you confirm that shutdown() really doesn't dislodge the thread in 
read()? 


http://www.freebsd.org/cgi/query-pr.cgi?pr=97921 

From: Kurt Miller <lists@intricatesoftware.com>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/97921: [socket] close() socket deadlocks blocked threads
Date: Sun, 13 Jan 2008 23:18:25 -0500

 I just checked on 6.1-release amd64 with both libpthread and libthr and
 shutdown() *does* dislodge the thread in read(). Both close() and dup2() 
 don't. I don't have a more current system to test on at the moment.
>Unformatted:
