From nobody  Thu Feb 11 07:29:49 1999
Received: (from nobody@localhost)
          by hub.freebsd.org (8.8.8/8.8.8) id HAA01792;
          Thu, 11 Feb 1999 07:29:49 -0800 (PST)
          (envelope-from nobody)
Message-Id: <199902111529.HAA01792@hub.freebsd.org>
Date: Thu, 11 Feb 1999 07:29:49 -0800 (PST)
From: david@inty.net
To: freebsd-gnats-submit@freebsd.org
Subject: TCP problem binding port - address already in use
X-Send-Pr-Version: www-1.0

>Number:         10028
>Category:       kern
>Synopsis:       TCP problem binding port - address already in use
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Feb 11 07:30:00 PST 1999
>Closed-Date:    Sat Feb 13 02:23:41 PST 1999
>Last-Modified:  Thu Feb 25 10:20:05 PST 1999
>Originator:     David Hedley
>Release:        2.2.8 & 3.0
>Organization:
Intelligent Network Technologies Ltd
>Environment:
FreeBSD inty.david.inty.net 3.0-RELEASE FreeBSD 3.0-RELEASE #2: Fri Jan  8 19:02:26 GMT 1999     david@brian.visint.co.uk:/usr/src/sys/compile/NEWINTY  i386

>Description:
I am developing a server program which listens on a port (8080 in this case)
Often when the program exits (usually on a signal), the kernel does not
'unbind' the port (i.e. netstat -na shows the port still in LISTEN state). I
can still make connections to the port, even though there is no process
listening on it. Trying to run the server program again reports 'address
already in use'. There is no way to clear this problem other than rebooting,
which is unacceptable.

The problem occurs with kernel versions 2.2.x and 3.0

>How-To-Repeat:

>Fix:
rebooting. :(
>Release-Note:
>Audit-Trail:

From: David Greenman <dg@root.com>
To: david@inty.net
Cc: freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: kern/10028: TCP problem binding port - address already in use 
Date: Thu, 11 Feb 1999 09:04:05 -0800

 >>Description:
 >I am developing a server program which listens on a port (8080 in this case)
 >Often when the program exits (usually on a signal), the kernel does not
 >'unbind' the port (i.e. netstat -na shows the port still in LISTEN state). I
 >can still make connections to the port, even though there is no process
 >listening on it. Trying to run the server program again reports 'address
 >already in use'. There is no way to clear this problem other than rebooting,
 >which is unacceptable.
 >
 >The problem occurs with kernel versions 2.2.x and 3.0
 
    Hmmm. I've never seen this before. Are you absolutely sure that there isn't
 another process with the socket open?
 
 -DG
 
 David Greenman
 Co-founder/Principal Architect, The FreeBSD Project

From: Bill Fenner <fenner@parc.xerox.com>
To: david@inty.net
Cc: freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: kern/10028: TCP problem binding port - address already in use 
Date: Thu, 11 Feb 1999 09:39:01 PST

 In message <199902111529.HAA01792@hub.freebsd.org>you write:
 >Often when the program exits (usually on a signal), the kernel does not
 >'unbind' the port (i.e. netstat -na shows the port still in LISTEN state).
 
 Have you tried using "lsof" from /usr/ports/sysutils/lsof to figure out
 what process is associated with the LISTEN socket?  Could this server have
 spawned a child which still has the socket open?
 
   Bill

From: David Hedley <david@inty.co.uk>
To: Bill Fenner <fenner@parc.xerox.com>
Cc: freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: kern/10028: TCP problem binding port - address already in use 
Date: Thu, 11 Feb 1999 17:49:30 +0000 (GMT)

 On Thu, 11 Feb 1999, Bill Fenner wrote:
 
 > In message <199902111529.HAA01792@hub.freebsd.org>you write:
 > >Often when the program exits (usually on a signal), the kernel does not
 > >'unbind' the port (i.e. netstat -na shows the port still in LISTEN state).
 > 
 > Have you tried using "lsof" from /usr/ports/sysutils/lsof to figure out
 > what process is associated with the LISTEN socket?  Could this server have
 > spawned a child which still has the socket open?
 > 
 
 On further investigation it looks more like some interaction with
 'routed'. It seems that when the problem happens, there are connections to
 port 8080 held in 'CLOSE_WAIT', and this is causing the kernel to not
 unbind the port. However, if I kill 'routed' (which is running), then the
 CLOSE_WAIT connections disappear and the port is no longer bound to
 anything. This is particularly strange in that the routing table is
 identical before and after 'routed' is killed.
 
 David
 --
 Dr David Hedley,Inty Development,Bristol,UK,0117 9050500,http://www.inty.net/
 
State-Changed-From-To: open->closed 
State-Changed-By: dillon 
State-Changed-When: Sat Feb 13 02:23:41 PST 1999 
State-Changed-Why:  
The bind: address in use error is correct operation.  The SO_RESUSEADDR 
socket opt must be used to defeat this feature.  If this turns out not 
to solve the problem we can reopen the bug report. 

From: Matthew Dillon <dillon@apollo.backplane.com>
To: David Hedley <david@inty.co.uk>
Cc: Bill Fenner <fenner@parc.xerox.com>, freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: kern/10028: TCP problem binding port - address already in use 
Date: Sat, 13 Feb 1999 02:23:01 -0800 (PST)

     The error message is correct.  When there are still connections
     attached to the bound port and the program listening on the port
     goes away, the kernel does not allow reattaching to the port until
     a timeout - which I think should be around 2 minutes.
 
     You can tell the kernel to blow it off and allow reattachments with
     the SO_REUSEADDR socket opt.
 
 	int on = 1;
 	setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, (void *)&on, sizeof(on));
 	listen(lfd, ... );
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>

From: David Hedley <david@inty.co.uk>
To: Matthew Dillon <dillon@apollo.backplane.com>
Cc: Bill Fenner <fenner@parc.xerox.com>, freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: kern/10028: TCP problem binding port - address already in use 
Date: Sat, 13 Feb 1999 16:15:02 +0000 (GMT)

 On Sat, 13 Feb 1999, Matthew Dillon wrote:
 
 >     The error message is correct.  When there are still connections
 >     attached to the bound port and the program listening on the port
 >     goes away, the kernel does not allow reattaching to the port until
 >     a timeout - which I think should be around 2 minutes.
 > 
 >     You can tell the kernel to blow it off and allow reattachments with
 >     the SO_REUSEADDR socket opt.
 > 
 > 	int on = 1;
 > 	setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, (void *)&on, sizeof(on));
 > 	listen(lfd, ... );
 > 
 
 The code in the server does this already - it doesn't make any difference
 as I still get 'address already in use....'
 It also doesn't explain that when I kill 'routed' the problem disappears
 immediately.
 
 David
 --
 Dr David Hedley,Inty Development,Bristol,UK,0117 9050500,http://www.inty.net/
 

From: Matthew Dillon <dillon@apollo.backplane.com>
To: David Hedley <david@inty.co.uk>
Cc: Bill Fenner <fenner@parc.xerox.com>, freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: kern/10028: TCP problem binding port - address already in use 
Date: Sat, 13 Feb 1999 10:56:52 -0800 (PST)

 :> 	int on = 1;
 :> 	setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, (void *)&on, sizeof(on));
 :> 	listen(lfd, ... );
 :> 
 :
 :The code in the server does this already - it doesn't make any difference
 :as I still get 'address already in use....'
 :It also doesn't explain that when I kill 'routed' the problem disappears
 :immediately.
 :
 :David
 :--
 :Dr David Hedley,Inty Development,Bristol,UK,0117 9050500,http://www.inty.net/
 
     Are you trying to bind to a specific IP address?
 
     The only thing routed messes with is the route table, but if routed is 
     screwing up the interface routes it could certainly cause bind() to 
     screw up - but it wouldn't be due to the hanging connections.  Instead it
     would be due to a bad route.
 
     Try running 'netstat -rn' while it is screwing up, and then again after
     you turn off routed and it stops screwing up.  Then compare the two route
     tables.
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>

From: David Hedley <david@inty.co.uk>
To: freebsd-gnats-submit@freebsd.org
Cc:  
Subject: Re: kern/10028: TCP problem binding port - address already in use
Date: Thu, 25 Feb 1999 08:45:41 +0000 (GMT)

 Problem resolved.
 
 It seems that the program in question restarts 'routed' as part of its
 operation. When it execs a new routed, the child process inherits it's
 parents open file descriptors - including any active sockets. Consequently
 when the server process restarts, it can no longer bind to its socket. A
 stupid problem really, and one easily fixed by setting the close-on-exec
 flag on the sockets in question.
 
 My apologies for troubling everyone with this, and thanks for everyone's
 efforts in trying to track it down - I should have known better than to
 have thought it a problem with FreeBSD!
 
 David
 --
 Dr David Hedley,Inty Development,Bristol,UK,0117 9050500,http://www.inty.net/
 
 
>Unformatted:
