From winfried@miranda.expro.pl  Mon Apr 25 12:16:57 2005
Return-Path: <winfried@miranda.expro.pl>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 9E6AC16A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 25 Apr 2005 12:16:57 +0000 (GMT)
Received: from miranda.expro.pl (miranda.expro.pl [193.25.166.12])
	by mx1.FreeBSD.org (Postfix) with ESMTP id B7E2643D5C
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 25 Apr 2005 12:16:56 +0000 (GMT)
	(envelope-from winfried@miranda.expro.pl)
Received: by miranda.expro.pl (Postfix, from userid 1001)
	id 68D6E153CA; Mon, 25 Apr 2005 14:16:50 +0200 (CEST)
Message-Id: <20050425121650.68D6E153CA@miranda.expro.pl>
Date: Mon, 25 Apr 2005 14:16:50 +0200 (CEST)
From: Jan Srzednicki <w@expro.pl>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: TCP socket support broken on a busy port
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         80322
>Category:       kern
>Synopsis:       TCP socket support broken on a busy port
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Apr 25 12:20:26 GMT 2005
>Closed-Date:    Tue Nov 20 06:57:36 UTC 2007
>Last-Modified:  Tue Nov 20 06:57:36 UTC 2007
>Originator:     Jan Srzednicki
>Release:        FreeBSD 4.10-RELEASE i386
>Organization:
Expro
>Environment:
System: FreeBSD miranda 4.10-RELEASE FreeBSD 4.10-RELEASE #19: Tue Jan 4 12:11:01 CET 2005 


	
>Description:

There appears to be a rather weird problem with TCP sockets implementation.
After a server daemon listening on a very busy unprivileged (>1024) port
dies (I'm not able to repeat that with a low-usage port), the port is
closed as it should, sockstat doesn't show anything listening on it,
everything looks fine. But an attempt to bind to that port with UID != 0
results in failure -- EADDRINUSE. Here is a strace snap from a netcat
process trying to bind to that port (UID!=0):

socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)

But, when I swich to root everything is fine:

socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(3, 1)                            = 0

(I can repeat that subsequently, once for root, once for user; it's not
a timeout issue of any kind)

There are no ipfw nor ipf rules on that host. There's no apparent reason
why the 3306 port is treated in a different way, provided that in the
begining the server daemon started succesfully. So, it appears that the
dead server daemon leaves the port in some weird state. Maybe it's some
socket cleaning issue.

I haven't found any way to fix this behaviour except rebooting that
machine. I have checked the CVS tree for RELENG_4, but I haven't found
any relevant changes since 4.10-RELEASE in that area.

>How-To-Repeat:

1. Get a quite busy machine with a very busy port (SQL port is a good
example)
2. Kill the server. Don't let it shutdown cleanly.
3. Try binding to that port first with UID!=0, then with UID==0.
	
>Fix:
	


>Release-Note:
>Audit-Trail:

From: Pawel Malachowski <pawmal-posting@freebsd.lublin.pl>
To: Jan Srzednicki <w@expro.pl>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/80322: TCP socket support broken on a busy port
Date: Mon, 25 Apr 2005 17:42:57 +0200

 On Mon, Apr 25, 2005 at 02:16:50PM +0200, Jan Srzednicki wrote:
 
 > After a server daemon listening on a very busy unprivileged (>1024) port
 > dies (I'm not able to repeat that with a low-usage port), the port is
 > closed as it should, sockstat doesn't show anything listening on it,
 > everything looks fine. But an attempt to bind to that port with UID != 0
 > results in failure -- EADDRINUSE.
 
 Hm, I saw behaviour like this on 4.8 machine running some shoutcast
 server; after killing radio process it was unable to bind to port
 as if it was already open (nothing was listening on it) but after
 a minute or so it was able to bind again. I've never investigated
 this.
 
 
 -- 
 Pawe Maachowski

From: "Matthew D. Fuller" <fullermd@over-yonder.net>
To: FreeBSD-gnats-submit@freebsd.org
Cc:  
Subject: Re: kern/80322: TCP socket support broken on a busy port
Date: Mon, 25 Apr 2005 11:21:28 -0500

 On Mon, Apr 25, 2005 at 03:50:22PM +0000 I heard the voice of
 Pawel Malachowski, and lo! it spake thus:
 >  
 >  Hm, I saw behaviour like this on 4.8 machine running some shoutcast
 >  server; after killing radio process it was unable to bind to port
 >  as if it was already open (nothing was listening on it) but after a
 >  minute or so it was able to bind again. I've never investigated
 >  this.
 
 This is where the SO_REUSEADDR setsockopt() comes into play.
 
 
 -- 
 Matthew Fuller     (MF4839)   |  fullermd@over-yonder.net
 Systems/Network Administrator |  http://www.over-yonder.net/~fullermd/
            On the Internet, nobody can hear you scream.

From: Bill Fenner <fenner@research.att.com>
To: w@expro.pl
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/80322: TCP socket support broken on a busy port
Date: Mon, 25 Apr 2005 09:24:38 -0700

 Are you using SO_REUSEADDR (in both the process that dies and
 the new process that tries to bind)?  Does netstat show the socket
 in TIME_WAIT?

From: Jan Srzednicki <w@expro.pl>
To: Bill Fenner <fenner@research.att.com>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/80322: TCP socket support broken on a busy port
Date: Wed, 4 May 2005 10:49:48 +0200

 On Mon, Apr 25, 2005 at 09:24:38AM -0700, Bill Fenner wrote:
 > 
 > Are you using SO_REUSEADDR (in both the process that dies and
 > the new process that tries to bind)?
 
 Yes.
 
 > Does netstat show the socket in TIME_WAIT?
 
 I'm currently unable to replicate this now, upgraded mysqld doesn't die
 in that way, killing it with -9 doesn't trigger that behaviour.
 
 -- 
 Jan Srzednicki
 w@expro.pl
 

From: Jan Srzednicki <w@expro.pl>
To: Bill Fenner <fenner@research.att.com>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/80322: TCP socket support broken on a busy port
Date: Thu, 12 May 2005 17:01:29 +0200

 On Mon, Apr 25, 2005 at 09:24:38AM -0700, Bill Fenner wrote:
 > 
 > Are you using SO_REUSEADDR (in both the process that dies and
 > the new process that tries to bind)?  Does netstat show the socket
 > in TIME_WAIT?
 
 OK, I finally managed to reproduce that. Yes, netstat indeed shows the
 socket in TIME_WAIT state with one connection, both queues empty. After
 this connection has timeouted, I can bind to the port as normal.
 
 -- 
 Jan Srzednicki
 w@expro.pl
 
State-Changed-From-To: open->feedback 
State-Changed-By: kmacy 
State-Changed-When: Tue Nov 20 06:53:32 UTC 2007 
State-Changed-Why:  

If I understand correctly, you're saying that SO_REUSADDR wasn't working correctly. 
Is that still the case on RELENG_7? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=80322 
State-Changed-From-To: feedback->closed 
State-Changed-By: kmacy 
State-Changed-When: Tue Nov 20 06:57:19 UTC 2007 
State-Changed-Why:  

Mail bounces, no way to confirm. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=80322 
>Unformatted:
