From nobody@FreeBSD.org  Wed Aug 23 00:57:21 2006
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 473AC16A4DE
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 23 Aug 2006 00:57:21 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id A4C6B43D45
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 23 Aug 2006 00:57:20 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k7N0vKdi003194
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 23 Aug 2006 00:57:20 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id k7N0vK7q003192;
	Wed, 23 Aug 2006 00:57:20 GMT
	(envelope-from nobody)
Message-Id: <200608230057.k7N0vK7q003192@www.freebsd.org>
Date: Wed, 23 Aug 2006 00:57:20 GMT
From: Daniel Austin <daniel@kewlio.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: kernel panic in 6.1-stable during normal operation
X-Send-Pr-Version: www-2.3

>Number:         102412
>Category:       kern
>Synopsis:       kernel panic in 6.1-stable during normal operation
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    rwatson
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Aug 23 01:00:31 GMT 2006
>Closed-Date:    Wed Nov 29 10:22:06 GMT 2006
>Last-Modified:  Wed Nov 29 10:22:06 GMT 2006
>Originator:     Daniel Austin
>Release:        6.1-STABLE (CVS-20060822)
>Organization:
Kewlio.net Limited
>Environment:
FreeBSD <hostname> 6.1-STABLE FreeBSD 6.1-STABLE #6: Tue Aug 22 21:09:55 BST 2006     dan@<hostname>:/usr/obj/usr/src/sys/kewlio  i386

>Description:
The server is a dedicated IRC server on one of the big 4 networks.
After a few hours of normal operation, the kernel panics.

I compiled the kernel with WITNESS and INVARIANTS enabled after the original kernel page faults.

Crash debug (with WITNESS/INVARIANTS enabled):

Unread portion of the kernel message buffer:
panic: mtx_lock() of destroyed mutex @ /usr/src/sys/netinet/ip_output.c:1193
Uptime: 2h51m41s
Dumping 503 MB (2 chunks)
  chunk 0: 1MB (160 pages) ... ok
  chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7

#0  doadump () at pcpu.h:165
165     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc0655934 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0655bb2 in panic (fmt=0xc087b45c "mtx_lock() of destroyed mutex @ %s:%d") at /usr/src/sys/kern/kern_shutdown.c:565
#3  0xc064d16e in _mtx_lock_flags (m=0xc5c21900, opts=0, file=0xc088b6d5 "/usr/src/sys/netinet/ip_output.c", line=1193)
    at /usr/src/sys/kern/kern_mutex.c:281
#4  0xc06f5390 in ip_ctloutput (so=0x0, sopt=0xe63e8c90) at /usr/src/sys/netinet/ip_output.c:1193
#5  0xc0704b6f in tcp_ctloutput (so=0xc57cfb20, sopt=0xe63e8c90) at /usr/src/sys/netinet/tcp_usrreq.c:1038
#6  0xc068f6f4 in sosetopt (so=0xc57cfb20, sopt=0xe63e8c90) at /usr/src/sys/kern/uipc_socket.c:1563
#7  0xc0694045 in kern_setsockopt (td=0xc4bfd480, s=1244, level=0, name=0, val=0x0, valseg=UIO_USERSPACE, valsize=0)
    at /usr/src/sys/kern/uipc_syscalls.c:1351
#8  0xc0693f8e in setsockopt (td=0xc4bfd480, uap=0x0) at /usr/src/sys/kern/uipc_syscalls.c:1307
#9  0xc080fdcf in syscall (frame=
      {tf_fs = 59, tf_es = -1078001605, tf_ds = -1078001605, tf_edi = -1077941472, tf_esi = 1244, tf_ebp = -1077941512, tf_isp = -432108188, tf_ebx = 1244, tf_edx = 1, tf_ecx = 0, tf_eax = 105, tf_trapno = 0, tf_err = 2, tf_eip = 672523411, tf_cs = 51, tf_eflags = 646, tf_esp = -1077941556, tf_ss = 59})
    at /usr/src/sys/i386/i386/trap.c:981
#10 0xc07fed2f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#11 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

(kgdb) f 3
#3  0xc064d16e in _mtx_lock_flags (m=0xc5c21900, opts=0, file=0xc088b6d5 "/usr/src/sys/netinet/ip_output.c", line=1193)
    at /usr/src/sys/kern/kern_mutex.c:281
281             KASSERT(m->mtx_lock != MTX_DESTROYED,
(kgdb) f 4
#4  0xc06f5390 in ip_ctloutput (so=0x0, sopt=0xe63e8c90) at /usr/src/sys/netinet/ip_output.c:1193
1193                            INP_LOCK(inp);
(kgdb) f 5
#5  0xc0704b6f in tcp_ctloutput (so=0xc57cfb20, sopt=0xe63e8c90) at /usr/src/sys/netinet/tcp_usrreq.c:1038
1038                    error = ip_ctloutput(so, sopt);


Crash debug (without WITNESS/INVARIANTS):

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x78
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc0681f29
stack pointer           = 0x28:0xe63e8ab8
frame pointer           = 0x28:0xe63e8abc
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = resume, IOPL = 0
current process         = 624 (ircd.200608220145)
trap number             = 12
panic: page fault
Uptime: 14h17m13s
Dumping 503 MB (2 chunks)
  chunk 0: 1MB (160 pages) ... ok
  chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7

(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc0661ad6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0661d6c in panic (fmt=0xc08ac642 "%s") at /usr/src/sys/kern/kern_shutdown.c:565
#3  0xc0861034 in trap_fatal (frame=0xe63e8a78, eva=120) at /usr/src/sys/i386/i386/trap.c:836
#4  0xc0860816 in trap (frame=
      {tf_fs = -432144376, tf_es = -994115544, tf_ds = -994115544, tf_edi = 0, tf_esi = -994102144, tf_ebp = -432108868, tf_isp = -432108892, tf_ebx = -994097216, tf_edx = -994097216, tf_ecx = 4, tf_eax = -994102112, tf_trapno = 12, tf_err = 0, tf_eip = -1066918103, tf_cs = 32, tf_eflags = 65543, tf_esp = -994102144, tf_ss = -432108832}) at /usr/src/sys/i386/i386/trap.c:269
#5  0xc084fa8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6  0xc0681f29 in turnstile_setowner (ts=0xc4bf47c0, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:432
#7  0xc0682220 in turnstile_wait (lock=0xc5ef6090, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:591
#8  0xc065842c in _mtx_lock_sleep (m=0xc5ef6090, tid=3300865152, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579
#9  0xc07090ec in ip_ctloutput (so=0xc4bf34a0, sopt=0xe63e8c90) at /usr/src/sys/netinet/ip_output.c:1193
#10 0xc071971f in tcp_ctloutput (so=0xc5121000, sopt=0xe63e8c90) at /usr/src/sys/netinet/tcp_usrreq.c:1038
#11 0xc069bb10 in sosetopt (so=0xc5121000, sopt=0xe63e8c90) at /usr/src/sys/kern/uipc_socket.c:1563
#12 0xc06a0dfd in kern_setsockopt (td=0xc4bf3480, s=2662, level=-994102112, name=-994102112, val=0xc4bf47c0, valseg=UIO_USERSPACE, valsize=4)
    at /usr/src/sys/kern/uipc_syscalls.c:1351
#13 0xc06a0d2e in setsockopt (td=0xc4bf3480, uap=0xc4bf34a0) at /usr/src/sys/kern/uipc_syscalls.c:1307
#14 0xc086134b in syscall (frame=
      {tf_fs = 198377531, tf_es = 223215675, tf_ds = -1078001605, tf_edi = -1077941472, tf_esi = 2662, tf_ebp = -1077941512, tf_isp = -432108188, tf_ebx = 2662, tf_edx = 1, tf_ecx = 0, tf_eax = 105, tf_trapno = 0, tf_err = 2, tf_eip = 672523411, tf_cs = 51, tf_eflags = 646, tf_esp = -1077941556, tf_ss = 59})
    at /usr/src/sys/i386/i386/trap.c:981
#15 0xc084fadf in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#16 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

(kgdb) f 6
#6  0xc0681f29 in turnstile_setowner (ts=0xc4bf47c0, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:432
432             ts->ts_owner = owner;
(kgdb) f 7
#7  0xc0682220 in turnstile_wait (lock=0xc5ef6090, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:591
591                     turnstile_setowner(ts, owner);
(kgdb) f 8
#8  0xc065842c in _mtx_lock_sleep (m=0xc5ef6090, tid=3300865152, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579
579                     turnstile_wait(&m->mtx_object, mtx_owner(m));
(kgdb) print *m
$1 = {mtx_object = {lo_class = 0xc093be44, lo_name = 0xc08d60d8 "inp", lo_type = 0xc08d3871 "tcpinp", lo_flags = 4849664, lo_list = {tqe_next = 0x0,
      tqe_prev = 0x0}, lo_witness = 0x0}, mtx_lock = 6, mtx_recurse = 0}

>How-To-Repeat:
I have not been able to duplicate this problem on another function machine, however I installed and moved the IRC server to different hardware and the same problem occurs.  This happens approximately every 3 hours and seems related to the level of network traffic.

The server has fxp and em-based cards with polling enabled.

I am happy to work with anyone that needs any further information or assistance in diagnosing the problem.  The problem started since updating from CVS recently  (a CVS build from 20060623 did not cause a kernel panic, but did cause a deadlock in the ircd process)

Copies of vmcores/kernels with debugging symbols have been kept.
>Fix:
Unsure.
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->rwatson 
Responsible-Changed-By: glebius 
Responsible-Changed-When: Sun Sep 3 13:52:45 UTC 2006 
Responsible-Changed-Why:  
Robert has fixed this in HEAD and has workarounded this in RELENG_6. 
Let him decide on the status of this PR. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=102412 
State-Changed-From-To: open->feedback 
State-Changed-By: rwatson 
State-Changed-When: Sun Sep 3 14:48:34 UTC 2006 
State-Changed-Why:  
A patch for this (or a related) problem has been committed to RELENG_6 
as src/sys/netinet/ip_output.c:1.242.2.10.  Could you check to see 
what version of the file you're running with?  The build date in uname 
is from after the commit date (2006/07/03) but I don't know when you 
did your source update. 

Thanks! 


http://www.freebsd.org/cgi/query-pr.cgi?pr=102412 

From: Daniel Austin <daniel@kewlio.net>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Sun, 03 Sep 2006 16:11:04 +0100

 <16:07> dan@server:140-src-all>cat checkouts.cvs:RELENG_6 | grep
 src/sys/netinet/ip_output.c
 C src/sys/netinet/ip_output.c,v RELENG_6 .
 2#871#110#11557750236#3649593#444 1.242.2.11 2006.08.10.10.41.50
 2#871#110#11561158595#518333#644
 
 I ran a full cvsup before compiling the kernels before i submitted the
 PR to make sure it wasnt fixed in releng_6 - so the source update
 date/time is the same as the kernel build.
 
 If there's any patches you would like me to test, i can do this no
 problem as it panics the kernel within a few hours everytime.
 
 Thanks,
 
 Daniel.

From: Diane Bruce <db@db.net>
To: bug-followup@FreeBSD.org, daniel@kewlio.net
Cc:  
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Thu, 19 Oct 2006 23:06:41 -0400

 This diff fixes this bug.
 
 --- patch.diff begins here ---
 --- kern/uipc_socket.c.orig	Mon Oct 16 21:07:02 2006
 +++ kern/uipc_socket.c	Mon Oct 16 21:09:49 2006
 @@ -1559,10 +1559,14 @@
 
  	error = 0;
  	if (sopt->sopt_level != SOL_SOCKET) {
 +		int res=ENOPROTOOPT;
 +		SOCK_LOCK(so);
  		if (so->so_proto && so->so_proto->pr_ctloutput)
 -			return ((*so->so_proto->pr_ctloutput)
 +
 +			res= ((*so->so_proto->pr_ctloutput)
  				  (so, sopt));
 -		error = ENOPROTOOPT;
 +		SOCK_UNLOCK(so);
 +		return res;
  	} else {
  		switch (sopt->sopt_name) {
  #ifdef INET
 --- patch.diff ends here ---
 
 Result of live testing on this server:
 
 *** Server Up 1 days, 1:16:00
 *** Highest connection count:  4122 (4121 clients)
 
 We are in the process of testing this fix still, but so far it has been up
 the longest it has ever been.
 
 - Diane
 
 - db@db.net http://www.db.net/~db

From: Robert Watson <rwatson@FreeBSD.org>
To: Diane Bruce <db@db.net>
Cc: bug-followup@FReeBSD.org
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Sat, 21 Oct 2006 02:22:03 +0100 (BST)

 On Fri, 20 Oct 2006, Diane Bruce wrote:
 
 > This diff fixes this bug.
 >
 > --- kern/uipc_socket.c.orig	Mon Oct 16 21:07:02 2006
 > +++ kern/uipc_socket.c	Mon Oct 16 21:09:49 2006
 > @@ -1559,10 +1559,14 @@
 >
 >  	error = 0;
 >  	if (sopt->sopt_level != SOL_SOCKET) {
 > +		int res=ENOPROTOOPT;
 > +		SOCK_LOCK(so);
 >  		if (so->so_proto && so->so_proto->pr_ctloutput)
 > -			return ((*so->so_proto->pr_ctloutput)
 > +
 > +			res= ((*so->so_proto->pr_ctloutput)
 >  				  (so, sopt));
 > -		error = ENOPROTOOPT;
 > +		SOCK_UNLOCK(so);
 > +		return res;
 >  	} else {
 >  		switch (sopt->sopt_name) {
 >  #ifdef INET
 > --
 >
 > Result of live testing on this server:
 >
 > *** Server Up 1 days, 1:16:00
 > *** Highest connection count:  4122 (4121 clients)
 >
 > We are in the process of testing this fix still, but so far it has been up 
 > the longest it has ever been.
 
 Hmm.  However, it likely introduces a lock order between the socket and 
 protocol code.  Normally, the protocol lock falls ahead of the socket lock in 
 the lock order, so if the above is run with WITNESS, it likely gets quite 
 upset.  And without it may well deadlock.
 
 The issue here is that the TCP code allows the PCB to evaporate at any moment, 
 but dereferences the so_pcb pointer unconditionally (as well as keeping 
 potentially stale copies around).  The commit referenced earlier in the PR is 
 a work-around, in that it validates the pcb pointer at points where it may 
 have changed, and doesn't assume that so_pcb is always non-NULL.  I take it 
 that that work-around doesn't fix it for you?  It's not a perfect fix, but 
 should close many of the most common races (for example, when a page fault 
 occurred after so_pcb was dereferenced, the connection was reset, and then the 
 fault handler returned continuing socket option processing).
 
 This is fixed properly in 7.x, where we no longer allow so_pcb to become 
 invalidated while the socket is valid.  Since I'm not ready to MFC those 
 changes yet (as they are fairly sweeping), I've been hoping we can get by with 
 reasonable workarounds that close major races where so_pcb is invalidated 
 until it can be MFC'd.
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge

From: Robert Watson <rwatson@FreeBSD.org>
To: Diane Bruce <db@db.net>
Cc: bug-followup@FReeBSD.org
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Sat, 21 Oct 2006 02:22:03 +0100 (BST)

 On Fri, 20 Oct 2006, Diane Bruce wrote:
 
 > This diff fixes this bug.
 >
 > --- kern/uipc_socket.c.orig	Mon Oct 16 21:07:02 2006
 > +++ kern/uipc_socket.c	Mon Oct 16 21:09:49 2006
 > @@ -1559,10 +1559,14 @@
 >
 >  	error = 0;
 >  	if (sopt->sopt_level != SOL_SOCKET) {
 > +		int res=ENOPROTOOPT;
 > +		SOCK_LOCK(so);
 >  		if (so->so_proto && so->so_proto->pr_ctloutput)
 > -			return ((*so->so_proto->pr_ctloutput)
 > +
 > +			res= ((*so->so_proto->pr_ctloutput)
 >  				  (so, sopt));
 > -		error = ENOPROTOOPT;
 > +		SOCK_UNLOCK(so);
 > +		return res;
 >  	} else {
 >  		switch (sopt->sopt_name) {
 >  #ifdef INET
 > --
 >
 > Result of live testing on this server:
 >
 > *** Server Up 1 days, 1:16:00
 > *** Highest connection count:  4122 (4121 clients)
 >
 > We are in the process of testing this fix still, but so far it has been up 
 > the longest it has ever been.
 
 Hmm.  However, it likely introduces a lock order between the socket and 
 protocol code.  Normally, the protocol lock falls ahead of the socket lock in 
 the lock order, so if the above is run with WITNESS, it likely gets quite 
 upset.  And without it may well deadlock.
 
 The issue here is that the TCP code allows the PCB to evaporate at any moment, 
 but dereferences the so_pcb pointer unconditionally (as well as keeping 
 potentially stale copies around).  The commit referenced earlier in the PR is 
 a work-around, in that it validates the pcb pointer at points where it may 
 have changed, and doesn't assume that so_pcb is always non-NULL.  I take it 
 that that work-around doesn't fix it for you?  It's not a perfect fix, but 
 should close many of the most common races (for example, when a page fault 
 occurred after so_pcb was dereferenced, the connection was reset, and then the 
 fault handler returned continuing socket option processing).
 
 This is fixed properly in 7.x, where we no longer allow so_pcb to become 
 invalidated while the socket is valid.  Since I'm not ready to MFC those 
 changes yet (as they are fairly sweeping), I've been hoping we can get by with 
 reasonable workarounds that close major races where so_pcb is invalidated 
 until it can be MFC'd.
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge

From: Diane Bruce <db@db.net>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: Diane Bruce <db@db.net>, bug-followup@FReeBSD.org, wes@freebsd.org
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Sat, 21 Oct 2006 09:56:43 -0400

 On Sat, Oct 21, 2006 at 02:22:03AM +0100, Robert Watson wrote:
 >
 > On Fri, 20 Oct 2006, Diane Bruce wrote:
 >
 > >This diff fixes this bug.
 > >
 > >--- kern/uipc_socket.c.orig	Mon Oct 16 21:07:02 2006
 > >+++ kern/uipc_socket.c	Mon Oct 16 21:09:49 2006
 > >@@ -1559,10 +1559,14 @@
 > >
 > > 	error = 0;
 > > 	if (sopt->sopt_level != SOL_SOCKET) {
 > >+		int res=ENOPROTOOPT;
 ...
 > >*** Server Up 1 days, 1:16:00
 > >*** Highest connection count:  4122 (4121 clients)
 > >
 > >We are in the process of testing this fix still, but so far it has been up
 > >the longest it has ever been.
 >
 > Hmm.  However, it likely introduces a lock order between the socket and
 > protocol code.  Normally, the protocol lock falls ahead of the socket lock
 
 I sleepily submitted the wrong patch! The patch I attached was a preliminary
 look at the bug and somehow got attached instead of the patch that worked.
 (i.e. a thinking out loud type of patch.)
 
 This is the patch that ran over a day.
 
 --- patch.diff begins here ---
 --- netinet/tcp_usrreq.c.orig	Sat Oct 14 18:10:23 2006
 +++ netinet/tcp_usrreq.c	Sat Oct 14 18:12:52 2006
 @@ -1205,17 +1205,19 @@
  	inp->inp_vflag |= INP_IPV4;
  	tp = tcp_newtcpcb(inp);
  	if (tp == 0) {
 -		int nofd = so->so_state & SS_NOFDREF;	/* XXX */
 +		int nofd = so->so_state & SS_NOFDREF;
 
  		so->so_state &= ~SS_NOFDREF;	/* don't free the socket yet */
 
  		INP_LOCK(inp);
 +		if (nofd & SS_NOFDREF) {
  #ifdef INET6
 -		if (isipv6)
 -			in6_pcbdetach(inp);
 -		else
 +			if (isipv6)
 +				in6_pcbdetach(inp);
 +			else
  #endif
 -		in_pcbdetach(inp);
 +			in_pcbdetach(inp);
 +		}
  		so->so_state |= nofd;
  		return (ENOBUFS);
  	}
 --- patch.diff ends here ---
 
 > in the lock order, so if the above is run with WITNESS, it likely gets
 > quite upset.  And without it may well deadlock.
 
 You are right, it would get quite upset or dead in rather short order.
 
 >
 > The issue here is that the TCP code allows the PCB to evaporate at any
 > moment, but dereferences the so_pcb pointer unconditionally (as well as
 
 Yepp.
 The patch attached (this worked over a day) does not unconditionally
 destroy the PCB. What is amusing is the condition where it panics the
 server, happens when the zone allocator fails to find memory,
 it then tries to re-use the pcb attached the socket already. Unfortunately
 it destroys the attached PCB unilaterally which causes the grief.
 Unfortunately there could be a memory leak introduced here, I'm just
 tracing back the code now to check. It looks like a callout should still
 be active and do a tcp_discardb().
 
 This server is a very busy one, with a lot of RSTs occuring as clients
 connect and immediately get kicked off or disconnect, hence are hittng
 it hard, dropping through your window.
 
 > keeping potentially stale copies around).  The commit referenced earlier in
 > the PR is a work-around, in that it validates the pcb pointer at points
 > where it may have changed, and doesn't assume that so_pcb is always
 > non-NULL.  I take it that that work-around doesn't fix it for you?  It's
 > not a perfect fix, but should close many of the most common races (for
 
 Right, clients drop right through the window on a regular basis
 anywhere from half an hour to an all time high of 24 hours, it would
 depend on server load how many RSTs etc.
 
 > example, when a page fault occurred after so_pcb was dereferenced, the
 > connection was reset, and then the fault handler returned continuing socket
 > option processing).
 >
 > This is fixed properly in 7.x, where we no longer allow so_pcb to become
 > invalidated while the socket is valid.  Since I'm not ready to MFC those
 
 Right. That is what was happening.
 
 > changes yet (as they are fairly sweeping), I've been hoping we can get by
 > with reasonable workarounds that close major races where so_pcb is
 > invalidated until it can be MFC'd.
 
 The proper attached diff *does* fix this bug for us. Unfortunately,
 the machine we are testing on is out for the night until someone
 can walk up to it and hit the reset button.
 
 My apologies for sending the wrong diff.
 
 
 --
 - db@db.net http://www.db.net/~db

From: Robert Watson <rwatson@FreeBSD.org>
To: Diane Bruce <db@db.net>
Cc: bug-followup@FreeBSD.org, wes@freebsd.org
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Sun, 22 Oct 2006 21:58:24 +0100 (BST)

 On Sat, 21 Oct 2006, Diane Bruce wrote:
 
 > The proper attached diff *does* fix this bug for us. Unfortunately, the 
 > machine we are testing on is out for the night until someone can walk up to 
 > it and hit the reset button.
 
 Try the attached variation, which re-validated so_pcb pointers after returning 
 from potentially blocking copyin operations before following (inp).  I've not 
 tested these changes locally.
 
 --- patch.diff begins here ---
 Index: ip_output.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/netinet/ip_output.c,v
 retrieving revision 1.242.2.15
 diff -u -r1.242.2.15 ip_output.c
 --- ip_output.c	6 Oct 2006 20:26:06 -0000	1.242.2.15
 +++ ip_output.c	22 Oct 2006 20:42:02 -0000
 @@ -1190,6 +1190,11 @@
   				m_free(m);
   				break;
   			}
 +			if (so->so_pcb == NULL) {
 +				m_free(m);
 +				error = EINVAL;
 +				break;
 +			}
   			INP_LOCK(inp);
   			error = ip_pcbopts(inp, sopt->sopt_name, m);
   			INP_UNLOCK(inp);
 @@ -1212,6 +1217,10 @@
   			if (error)
   				break;
 
 +			if (so->so_pcb == NULL) {
 +				error = EINVAL;
 +				break;
 +			}
   			switch (sopt->sopt_name) {
   			case IP_TOS:
   				inp->inp_ip_tos = optval;
 @@ -1286,6 +1295,10 @@
   			if (error)
   				break;
 
 +			if (so->so_pcb == NULL) {
 +				error = EINVAL;
 +				break;
 +			}
   			INP_LOCK(inp);
   			switch (optval) {
   			case IP_PORTRANGE_DEFAULT:
 @@ -1328,6 +1341,11 @@
   			req = mtod(m, caddr_t);
   			len = m->m_len;
   			optname = sopt->sopt_name;
 +			if (so->so_pcb == NULL) {
 +				m_free(m);
 +				error = EINVAL;
 +				break;
 +			}
   			error = ipsec4_set_policy(inp, optname, req, len, priv);
   			m_freem(m);
   			break;
 --- patch.diff ends here ---

From: Diane Bruce <db@db.net>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: Diane Bruce <db@db.net>, bug-followup@FreeBSD.org, wes@freebsd.org
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Mon, 23 Oct 2006 21:05:19 -0400

 On Sun, Oct 22, 2006 at 09:58:24PM +0100, Robert Watson wrote:
 >
 > On Sat, 21 Oct 2006, Diane Bruce wrote:
 >
 > >The proper attached diff *does* fix this bug for us. Unfortunately, the
 > >machine we are testing on is out for the night until someone can walk up
 > >to it and hit the reset button.
 >
 > Try the attached variation, which re-validated so_pcb pointers after
 > returning from potentially blocking copyin operations before following
 > (inp).  I've not tested these changes locally.
 
 I redid the diff and reviewed it.
 
 The machine in question has had an uptime of over a day:
 *** Server Up 1 days, 1:42:14
 *** Highest connection count:  5517 (5516 clients)
 
 I'd say this is a definite fix.
 
 --
 - db@db.net http://www.db.net/~db
State-Changed-From-To: feedback->patched 
State-Changed-By: rwatson 
State-Changed-When: Tue Oct 24 13:24:22 UTC 2006 
State-Changed-Why:  
I've committed the latest patch to RELENG_6 as ip_output.c:1.242.2.16. 
An post-commit review or testing would be welcome, especially using the 
forthcoming FreeBSD 6.2-BETA3.  I've left the PR in the 'patched' state 
until I've had feedback confirming all is good.  Also, to remind me to 
merge this to 5.x in a few days. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=102412 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/102412: commit references a PR
Date: Tue, 24 Oct 2006 13:23:27 +0000 (UTC)

 rwatson     2006-10-24 13:23:03 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_6)
     sys/netinet          ip_output.c 
   Log:
   Reduce the size of a number of race windows in the TCP socket options
   processing code: a RST may arrive during a socket option call, causing
   the PCB to be freed, leading to an invalid pointer dereference.  When
   the kernel blocks in a socket option copyin or memory allocation (such
   as during heavy paging), the race window is greatly widened.  This
   change re-validates the PCB pointer after returning from the copy/alloc
   operation.  This does not eliminate the problem, but does narrow the
   window significantly (to the point where it may not be observed at all).
   
   The proper fix is in 7.x, which significantly re-works the socket and
   PCB code so that PCB's are not ripped out from under sockets on reset.
   However, those changes are not appropriate for an MFC during a release
   cycle.  As a result, this is not an MFC, but new code crafted for 6.x.
   
   PR:                     kern/102412
   Reported by:            Daniel Austin <daniel at kewlio dot net>
   Tested by:              Diane Bruce <db at db dot net>
   Reviewed by:            Diane Bruce <db at db dot net>
   Approved by:            re (kensmith)
   
   Revision    Changes    Path
   1.242.2.16  +18 -0     src/sys/netinet/ip_output.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 

From: Diane Bruce <db@db.net>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: Diane Bruce <db@db.net>, bug-followup@FreeBSD.org, wes@freebsd.org,
	Daniel Austin <daniel@kewlio.net>
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Thu, 26 Oct 2006 13:53:47 -0400

 ----- Forwarded message from Daniel Austin <daniel@kewlio.net> -----
 
 X-PGP-Universal: processed;
 	by dandell on Thu, 26 Oct 2006 02:21:51 +0000
 Date: Thu, 26 Oct 2006 02:21:34 +0100
 From: Daniel Austin <daniel@kewlio.net>
 Reply-To: daniel@kewlio.net
 Organization: Kewlio.net Limited
 User-Agent: Thunderbird 1.5.0.7 (Windows/20060909)
 To: db@db.net
 CC: ksaihr@error404.nls.net
 Subject: london.uk coredump
 X-UIDL: P,e"!m0>"![]<!![AG!!
 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.0.3
 
 Hi,
 
 As requested:
 
 Panic String: mtx_lock() of destroyed mutex @
 /usr/src/sys/netinet/ip_output.c:1198
 
 --cut: /usr/src/sys/netinet/ip_output.c--
                         if (so->so_pcb == NULL) {
                                 m_free(m);
                                 error = EINVAL;
                                 break;
                         }
                         INP_LOCK(inp);
 --cut--
 
 Line 1198 is INP_LOCK(inp);
 
 
 
 Thanks,
 
 --
 Daniel Austin MBCS
 Managing Director
 Kewlio.net Limited
 
 
 ----- End forwarded message -----
 
 Keep in mind, no WITNESS code was compiled in on this core.
 
 --
 - db@db.net http://www.db.net/~db

From: Robert Watson <rwatson@FreeBSD.org>
To: Diane Bruce <db@db.net>
Cc: bug-followup@FreeBSD.org, wes@freebsd.org, 
    Daniel Austin <daniel@kewlio.net>
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Thu, 26 Oct 2006 19:04:35 +0100 (BST)

 On Thu, 26 Oct 2006, Diane Bruce wrote:
 
 > As requested:
 >
 > Panic String: mtx_lock() of destroyed mutex @
 > /usr/src/sys/netinet/ip_output.c:1198
 >
 > --cut: /usr/src/sys/netinet/ip_output.c--
 >                        if (so->so_pcb == NULL) {
 >                                m_free(m);
 >                                error = EINVAL;
 >                                break;
 >                        }
 >                        INP_LOCK(inp);
 > --cut--
 >
 > Line 1198 is INP_LOCK(inp);
 
 Questions:
 
 (1) Is there a coredump available for this, and are you running gdb on it.  If
      so, I'd very much like to see "print *so" in the above stack frame, as
      well as print inp, print *inp.
 
 (2) For the sake of consistency, for each reported panic, it would be really
      helpful to see a backtrace, either from DDB or gdb.  If from DDB, then
      having file/line number conversions of the function offsets would be
      useful.  While it sounds like it's the same trace as before based on
      context, it always sucks when you discover later you're tracking the wrong
      bug. :-).
 
 Also, in general, when referencing a file name, please make sure to mention 
 the revision number.  In this case, there's only one revision it can be due to 
 it including the most recent commit, but as a general rule, it's really 
 helpful, especially if someone is going to look back at the PR trail a few 
 months later.
 
 As a comment for everyone following the PR who is not Diane, since Diane and I 
 have been chatting on IRC: what we're trying to do here is narrow the race(s) 
 associated with the pcb/socket life cycle without actually closing them in 6.x 
 for 6.2.  The reason is that the actual change that fixes this is very large 
 and complicated, and is too risky for the 6.2 release cycle.  We should (hope) 
 be able to sufficiently narrow the race to allow it not to occur in practice 
 so as to correct the bug in the release without taking on the risk of the 
 larger change.  I can then schedule the larger MFC for after the release has 
 gone out and settled.
 
 Thanks,
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge

From: Robert Watson <rwatson@FreeBSD.org>
To: Diane Bruce <db@db.net>
Cc: bug-followup@FreeBSD.org, wes@freebsd.org, 
    Daniel Austin <daniel@kewlio.net>
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Thu, 26 Oct 2006 19:22:41 +0100 (BST)

 On Thu, 26 Oct 2006, Diane Bruce wrote:
 
 > Panic String: mtx_lock() of destroyed mutex @ 
 > /usr/src/sys/netinet/ip_output.c:1198
 >
 > --cut: /usr/src/sys/netinet/ip_output.c--
 >                        if (so->so_pcb == NULL) {
 >                                m_free(m);
 >                                error = EINVAL;
 >                                break;
 >                        }
 >                        INP_LOCK(inp);
 > --cut--
 >
 > Line 1198 is INP_LOCK(inp);
 
 Try the attached patch, which uses the pcbinfo lock to prevent so_pcb from 
 changing (ideally).  There's still a race between the two assignments at the 
 head of the function, but this may significantly reduce the chances of the 
 race firing.  Untested, I'm afraid, but likely good.
 
 Index: ip_output.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/netinet/ip_output.c,v
 retrieving revision 1.242.2.16
 diff -u -r1.242.2.16 ip_output.c
 --- ip_output.c	24 Oct 2006 13:23:03 -0000	1.242.2.16
 +++ ip_output.c	26 Oct 2006 18:20:55 -0000
 @@ -1155,6 +1155,7 @@
   	struct sockopt *sopt;
   {
   	struct	inpcb *inp = sotoinpcb(so);
 +	struct	inpcbinfo *pcbinfo = inp->inp_pcbinfo;
   	int	error, optval;
 
   	error = optval = 0;
 @@ -1190,12 +1191,15 @@
   				m_free(m);
   				break;
   			}
 +			INP_INFO_WLOCK(pcbinfo);
   			if (so->so_pcb == NULL) {
 +				INP_INFO_WUNLOCK(pcbinfo);
   				m_free(m);
   				error = EINVAL;
   				break;
   			}
   			INP_LOCK(inp);
 +			INP_INFO_WUNLOCK(pcbinfo);
   			error = ip_pcbopts(inp, sopt->sopt_name, m);
   			INP_UNLOCK(inp);
   			return (error);

From: Diane Bruce <db@db.net>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: Diane Bruce <db@db.net>, bug-followup@FreeBSD.org, wes@freebsd.org,
	Daniel Austin <daniel@kewlio.net>
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Thu, 2 Nov 2006 19:52:23 -0500

 On Thu, Oct 26, 2006 at 07:22:41PM +0100, Robert Watson wrote:
 > On Thu, 26 Oct 2006, Diane Bruce wrote:
 >
 > >Panic String: mtx_lock() of destroyed mutex @
 > >/usr/src/sys/netinet/ip_output.c:1198
 > >
 ...
 
 Now an even more amusing core!
 
 
 Fatal trap 12: page fault while in kernel mode
 fault virtual address   = 0x40
 fault code              = supervisor read, page not present
 instruction pointer     = 0x20:0xc070a54e
 stack pointer           = 0x28:0xe63dab14
 frame pointer           = 0x28:0xe63dab34
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 917 (ircd.200610140041)
 trap number             = 12
 panic: page fault
 
 #0  doadump () at pcpu.h:165
 #1  0xc0662102 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
 #2  0xc0662398 in panic (fmt=0xc087c509 "%s")
     at /usr/src/sys/kern/kern_shutdown.c:565
 #3  0xc08311f0 in trap_fatal (frame=0xe63daad4, eva=64)
     at /usr/src/sys/i386/i386/trap.c:837
 #4  0xc0830f57 in trap_pfault (frame=0xe63daad4, usermode=0, eva=64)
     at /usr/src/sys/i386/i386/trap.c:745
 #5  0xc0830bb5 in trap (frame=
       {tf_fs = -432209912, tf_es = -1065222104, tf_ds = -994115544, tf_edi = -432165744, tf_esi = -432165744, tf_ebp = -432166092, tf_isp = -432166144, tf_ebx = 0, tf_edx = -432165488, tf_ecx = 0, tf_eax = -989738920, tf_trapno = 12, tf_err = 0, tf_eip = -1066359474, tf_cs = 32, tf_eflags = 66182, tf_esp = -432165744, tf_ss = -432166100}) at /usr/src/sys/i386/i386/trap.c:435
 #6  0xc081fb8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
 #7  0xc070a54e in ip_ctloutput (so=0xc501c858, sopt=0xe63dac90)
     at /usr/src/sys/netinet/ip_output.c:1157
 #8  0xc071ae2f in tcp_ctloutput (so=0xc501c858, sopt=0xe63dac90)
     at /usr/src/sys/netinet/tcp_usrreq.c:1038
 #9  0xc069c330 in sosetopt (so=0xc501c858, sopt=0xe63dac90)
     at /usr/src/sys/kern/uipc_socket.c:1563
 #10 0xc06a1645 in kern_setsockopt (td=0xc4bf8900, s=43, level=-989738920,
     name=-989738920, val=0x0, valseg=UIO_USERSPACE, valsize=3862801808)
     at /usr/src/sys/kern/uipc_syscalls.c:1351
 #11 0xc06a1566 in setsockopt (td=0xc4bf8900, uap=0xc501c858)
     at /usr/src/sys/kern/uipc_syscalls.c:1307
 #12 0xc0831507 in syscall (frame=
       {tf_fs = 220987451, tf_es = 59, tf_ds = -1078001605, tf_edi = -1077941472, tf_esi = 43, tf_ebp = -1077941512, tf_isp = -432165532, tf_ebx = 43, tf_edx = 1, tf_ecx = 0, tf_eax = 105, tf_trapno = 0, tf_err = 2, tf_eip = 672523399, tf_cs = 51, tf_eflags = 646, tf_esp = -1077941556, tf_ss = 59})
     at /usr/src/sys/i386/i386/trap.c:983
 #13 0xc081fbdf in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
 #14 0x00000033 in ?? ()
 Previous frame inner to this frame (corrupt stack?)
 #7  0xc070a54e in ip_ctloutput (so=0xc501c858, sopt=0xe63dac90)
     at /usr/src/sys/netinet/ip_output.c:1157
 1157            struct  inpcb *inp = sotoinpcb(so);
 (kgdb) p so
 $1 = (struct socket *) 0xc501c858
 (kgdb) p inp
 $5 = (struct inpcb *) 0x0
 (kgdb) p so->so_pcb
 $6 = (void *) 0x0
 (kgdb) list
 1152    int
 1153    ip_ctloutput(so, sopt)
 1154            struct socket *so;
 1155            struct sockopt *sopt;
 1156    {
 1157            struct  inpcb *inp = sotoinpcb(so);
 1158            int     error, optval;
 1159
 1160            error = optval = 0;
 1161            if (sopt->sopt_level != IPPROTO_IP) {
 ...
 (kgdb) down
 #6  0xc081fb8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
 139             call    trap
 Current language:  auto; currently asm
 
 --
 - db@db.net http://www.db.net/~db

From: Robert Watson <rwatson@FreeBSD.org>
To: Diane Bruce <db@db.net>, Daniel Austin <daniel@kewlio.net>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Sat, 25 Nov 2006 11:00:47 +0000 (GMT)

 On Fri, 3 Nov 2006, Diane Bruce wrote:
 
 > The following reply was made to PR kern/102412; it has been noted by GNATS.
 >
 > From: Diane Bruce <db@db.net>
 > To: Robert Watson <rwatson@FreeBSD.org>
 > Cc: Diane Bruce <db@db.net>, bug-followup@FreeBSD.org, wes@freebsd.org,
 > 	Daniel Austin <daniel@kewlio.net>
 > Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
 > Date: Thu, 2 Nov 2006 19:52:23 -0500
 >
 > On Thu, Oct 26, 2006 at 07:22:41PM +0100, Robert Watson wrote:
 > > On Thu, 26 Oct 2006, Diane Bruce wrote:
 > >
 > > >Panic String: mtx_lock() of destroyed mutex @
 > > >/usr/src/sys/netinet/ip_output.c:1198
 > > >
 > ...
 >
 > Now an even more amusing core!
 
 Could you try the attached patch, which provides a more substantive fix while 
 still avoiding the major MFC of 7-CURRENT structural fixes?
 
 If GNATS has mangled the patch, please try:
 
    http://www.watson.org/~robert/freebsd/netperf/20061124-ip_ctloutput.diff
 
 I've done light testing here, but not with the orts of loads that generates 
 the above class of panics.
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge
 
 Index: ip_output.c
 ===================================================================
 RCS file: /data/fbsd-cvs/ncvs/src/sys/netinet/ip_output.c,v
 retrieving revision 1.242.2.16
 diff -u -r1.242.2.16 ip_output.c
 --- ip_output.c	24 Oct 2006 13:23:03 -0000	1.242.2.16
 +++ ip_output.c	24 Nov 2006 15:47:13 -0000
 @@ -1148,15 +1148,29 @@
 
   /*
    * IP socket option processing.
 + *
 + * There are two versions of this call in order to work around a race
 + * condition in TCP in FreeBSD 6.x.  In the TCP implementation, so->so_pcb
 + * can become NULL if the pcb or pcbinfo lock isn't held.  However, when
 + * entering ip_ctloutput(), neither lock is held, and finding the pointer to
 + * either lock requires follow so->so_pcb, which may be NULL.
 + * ip_ctloutput_pcbinfo() accepts the pcbinfo pointer so that the lock can be
 + * safely acquired.  This is not required in FreeBSD 7.x because the
 + * invariants on so->so_pcb are much stronger, so it cannot become NULL
 + * while the socket is in use.
    */
   int
 -ip_ctloutput(so, sopt)
 +ip_ctloutput_pcbinfo(so, sopt, pcbinfo)
   	struct socket *so;
   	struct sockopt *sopt;
 +	struct inpcbinfo *pcbinfo;
   {
   	struct	inpcb *inp = sotoinpcb(so);
   	int	error, optval;
 
 +	if (pcbinfo == NULL)
 +		pcbinfo = inp->inp_pcbinfo;
 +
   	error = optval = 0;
   	if (sopt->sopt_level != IPPROTO_IP) {
   		return (EINVAL);
 @@ -1190,12 +1204,15 @@
   				m_free(m);
   				break;
   			}
 +			INP_INFO_WLOCK(pcbinfo);
   			if (so->so_pcb == NULL) {
 +				INP_INFO_WUNLOCK(pcbinfo);
   				m_free(m);
   				error = EINVAL;
   				break;
   			}
   			INP_LOCK(inp);
 +			INP_INFO_WUNLOCK(pcbinfo);
   			error = ip_pcbopts(inp, sopt->sopt_name, m);
   			INP_UNLOCK(inp);
   			return (error);
 @@ -1217,10 +1234,14 @@
   			if (error)
   				break;
 
 +			INP_INFO_WLOCK(pcbinfo);
   			if (so->so_pcb == NULL) {
 +				INP_INFO_WUNLOCK(pcbinfo);
   				error = EINVAL;
   				break;
   			}
 +			INP_LOCK(inp);
 +			INP_INFO_WUNLOCK(pcbinfo);
   			switch (sopt->sopt_name) {
   			case IP_TOS:
   				inp->inp_ip_tos = optval;
 @@ -1277,6 +1298,7 @@
   				OPTSET(INP_DONTFRAG);
   				break;
   			}
 +			INP_UNLOCK(inp);
   			break;
   #undef OPTSET
 
 @@ -1295,11 +1317,13 @@
   			if (error)
   				break;
 
 +			INP_INFO_WLOCK(pcbinfo);
   			if (so->so_pcb == NULL) {
   				error = EINVAL;
   				break;
   			}
   			INP_LOCK(inp);
 +			INP_INFO_WUNLOCK(pcbinfo);
   			switch (optval) {
   			case IP_PORTRANGE_DEFAULT:
   				inp->inp_flags &= ~(INP_LOWPORT);
 @@ -1480,6 +1504,15 @@
   	return (error);
   }
 
 +int
 +ip_ctloutput(so, sopt)
 +	struct socket *so;
 +	struct sockopt *sopt;
 +{
 +
 +	return (ip_ctloutput_pcbinfo(so, sopt, NULL));
 +}
 +
   /*
    * Set up IP options in pcb for insertion in output packets.
    * Store in mbuf with pointer in pcbopt, adding pseudo-option
 Index: ip_var.h
 ===================================================================
 RCS file: /data/fbsd-cvs/ncvs/src/sys/netinet/ip_var.h,v
 retrieving revision 1.95
 diff -u -r1.95 ip_var.h
 --- ip_var.h	2 Jul 2005 23:13:31 -0000	1.95
 +++ ip_var.h	24 Nov 2006 15:32:53 -0000
 @@ -144,6 +144,7 @@
 
   struct ip;
   struct inpcb;
 +struct inpcbinfo;
   struct route;
   struct sockopt;
 
 @@ -164,6 +165,8 @@
   extern struct	pr_usrreqs rip_usrreqs;
 
   int	 ip_ctloutput(struct socket *, struct sockopt *sopt);
 +int	 ip_ctloutput_pcbinfo(struct socket *, struct sockopt *sopt,
 +	    struct inpcbinfo *pcbinfo);
   void	 ip_drain(void);
   void	 ip_fini(void *xtp);
   int	 ip_fragment(struct ip *ip, struct mbuf **m_frag, int mtu,
 Index: tcp_usrreq.c
 ===================================================================
 RCS file: /data/fbsd-cvs/ncvs/src/sys/netinet/tcp_usrreq.c,v
 retrieving revision 1.124.2.3
 diff -u -r1.124.2.3 tcp_usrreq.c
 --- tcp_usrreq.c	27 Sep 2006 09:24:44 -0000	1.124.2.3
 +++ tcp_usrreq.c	24 Nov 2006 14:59:41 -0000
 @@ -1035,7 +1035,7 @@
   			error = ip6_ctloutput(so, sopt);
   		else
   #endif /* INET6 */
 -		error = ip_ctloutput(so, sopt);
 +		error = ip_ctloutput_pcbinfo(so, sopt, &tcbinfo);
   		return (error);
   	}
   	tp = intotcpcb(inp);
State-Changed-From-To: patched->feedback 
State-Changed-By: rwatson 
State-Changed-When: Sat Nov 25 11:21:12 UTC 2006 
State-Changed-Why:  
Change to feedback; new patch is attached to this problem report. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=102412 

From: Daniel Austin <daniel@kewlio.net>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: Diane Bruce <db@db.net>
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Sun, 26 Nov 2006 00:21:50 +0000

 Hi Robert,
 
 Testing it out now - will let you know the outcome.
 
 
 Thanks,
 
 Daniel.
 
 Robert Watson wrote:
 > 
 > On Fri, 3 Nov 2006, Diane Bruce wrote:
 > 
 >> The following reply was made to PR kern/102412; it has been noted by 
 >> GNATS.
 >>
 >> From: Diane Bruce <db@db.net>
 >> To: Robert Watson <rwatson@FreeBSD.org>
 >> Cc: Diane Bruce <db@db.net>, bug-followup@FreeBSD.org, wes@freebsd.org,
 >>     Daniel Austin <daniel@kewlio.net>
 >> Subject: Re: kern/102412: kernel panic in 6.1-stable during normal 
 >> operation
 >> Date: Thu, 2 Nov 2006 19:52:23 -0500
 >>
 >> On Thu, Oct 26, 2006 at 07:22:41PM +0100, Robert Watson wrote:
 >> > On Thu, 26 Oct 2006, Diane Bruce wrote:
 >> >
 >> > >Panic String: mtx_lock() of destroyed mutex @
 >> > >/usr/src/sys/netinet/ip_output.c:1198
 >> > >
 >> ...
 >>
 >> Now an even more amusing core!
 > 
 > Could you try the attached patch, which provides a more substantive fix 
 > while still avoiding the major MFC of 7-CURRENT structural fixes?
 > 
 > If GNATS has mangled the patch, please try:
 > 
 >   http://www.watson.org/~robert/freebsd/netperf/20061124-ip_ctloutput.diff
 > 
 > I've done light testing here, but not with the orts of loads that 
 > generates the above class of panics.
 > 
 > Robert N M Watson
 > Computer Laboratory
 > University of Cambridge
 > 
 > Index: ip_output.c
 > ===================================================================
 > RCS file: /data/fbsd-cvs/ncvs/src/sys/netinet/ip_output.c,v
 > retrieving revision 1.242.2.16
 > diff -u -r1.242.2.16 ip_output.c
 > --- ip_output.c    24 Oct 2006 13:23:03 -0000    1.242.2.16
 > +++ ip_output.c    24 Nov 2006 15:47:13 -0000
 > @@ -1148,15 +1148,29 @@
 > 
 >  /*
 >   * IP socket option processing.
 > + *
 > + * There are two versions of this call in order to work around a race
 > + * condition in TCP in FreeBSD 6.x.  In the TCP implementation, so->so_pcb
 > + * can become NULL if the pcb or pcbinfo lock isn't held.  However, when
 > + * entering ip_ctloutput(), neither lock is held, and finding the 
 > pointer to
 > + * either lock requires follow so->so_pcb, which may be NULL.
 > + * ip_ctloutput_pcbinfo() accepts the pcbinfo pointer so that the lock 
 > can be
 > + * safely acquired.  This is not required in FreeBSD 7.x because the
 > + * invariants on so->so_pcb are much stronger, so it cannot become NULL
 > + * while the socket is in use.
 >   */
 >  int
 > -ip_ctloutput(so, sopt)
 > +ip_ctloutput_pcbinfo(so, sopt, pcbinfo)
 >      struct socket *so;
 >      struct sockopt *sopt;
 > +    struct inpcbinfo *pcbinfo;
 >  {
 >      struct    inpcb *inp = sotoinpcb(so);
 >      int    error, optval;
 > 
 > +    if (pcbinfo == NULL)
 > +        pcbinfo = inp->inp_pcbinfo;
 > +
 >      error = optval = 0;
 >      if (sopt->sopt_level != IPPROTO_IP) {
 >          return (EINVAL);
 > @@ -1190,12 +1204,15 @@
 >                  m_free(m);
 >                  break;
 >              }
 > +            INP_INFO_WLOCK(pcbinfo);
 >              if (so->so_pcb == NULL) {
 > +                INP_INFO_WUNLOCK(pcbinfo);
 >                  m_free(m);
 >                  error = EINVAL;
 >                  break;
 >              }
 >              INP_LOCK(inp);
 > +            INP_INFO_WUNLOCK(pcbinfo);
 >              error = ip_pcbopts(inp, sopt->sopt_name, m);
 >              INP_UNLOCK(inp);
 >              return (error);
 > @@ -1217,10 +1234,14 @@
 >              if (error)
 >                  break;
 > 
 > +            INP_INFO_WLOCK(pcbinfo);
 >              if (so->so_pcb == NULL) {
 > +                INP_INFO_WUNLOCK(pcbinfo);
 >                  error = EINVAL;
 >                  break;
 >              }
 > +            INP_LOCK(inp);
 > +            INP_INFO_WUNLOCK(pcbinfo);
 >              switch (sopt->sopt_name) {
 >              case IP_TOS:
 >                  inp->inp_ip_tos = optval;
 > @@ -1277,6 +1298,7 @@
 >                  OPTSET(INP_DONTFRAG);
 >                  break;
 >              }
 > +            INP_UNLOCK(inp);
 >              break;
 >  #undef OPTSET
 > 
 > @@ -1295,11 +1317,13 @@
 >              if (error)
 >                  break;
 > 
 > +            INP_INFO_WLOCK(pcbinfo);
 >              if (so->so_pcb == NULL) {
 >                  error = EINVAL;
 >                  break;
 >              }
 >              INP_LOCK(inp);
 > +            INP_INFO_WUNLOCK(pcbinfo);
 >              switch (optval) {
 >              case IP_PORTRANGE_DEFAULT:
 >                  inp->inp_flags &= ~(INP_LOWPORT);
 > @@ -1480,6 +1504,15 @@
 >      return (error);
 >  }
 > 
 > +int
 > +ip_ctloutput(so, sopt)
 > +    struct socket *so;
 > +    struct sockopt *sopt;
 > +{
 > +
 > +    return (ip_ctloutput_pcbinfo(so, sopt, NULL));
 > +}
 > +
 >  /*
 >   * Set up IP options in pcb for insertion in output packets.
 >   * Store in mbuf with pointer in pcbopt, adding pseudo-option
 > Index: ip_var.h
 > ===================================================================
 > RCS file: /data/fbsd-cvs/ncvs/src/sys/netinet/ip_var.h,v
 > retrieving revision 1.95
 > diff -u -r1.95 ip_var.h
 > --- ip_var.h    2 Jul 2005 23:13:31 -0000    1.95
 > +++ ip_var.h    24 Nov 2006 15:32:53 -0000
 > @@ -144,6 +144,7 @@
 > 
 >  struct ip;
 >  struct inpcb;
 > +struct inpcbinfo;
 >  struct route;
 >  struct sockopt;
 > 
 > @@ -164,6 +165,8 @@
 >  extern struct    pr_usrreqs rip_usrreqs;
 > 
 >  int     ip_ctloutput(struct socket *, struct sockopt *sopt);
 > +int     ip_ctloutput_pcbinfo(struct socket *, struct sockopt *sopt,
 > +        struct inpcbinfo *pcbinfo);
 >  void     ip_drain(void);
 >  void     ip_fini(void *xtp);
 >  int     ip_fragment(struct ip *ip, struct mbuf **m_frag, int mtu,
 > Index: tcp_usrreq.c
 > ===================================================================
 > RCS file: /data/fbsd-cvs/ncvs/src/sys/netinet/tcp_usrreq.c,v
 > retrieving revision 1.124.2.3
 > diff -u -r1.124.2.3 tcp_usrreq.c
 > --- tcp_usrreq.c    27 Sep 2006 09:24:44 -0000    1.124.2.3
 > +++ tcp_usrreq.c    24 Nov 2006 14:59:41 -0000
 > @@ -1035,7 +1035,7 @@
 >              error = ip6_ctloutput(so, sopt);
 >          else
 >  #endif /* INET6 */
 > -        error = ip_ctloutput(so, sopt);
 > +        error = ip_ctloutput_pcbinfo(so, sopt, &tcbinfo);
 >          return (error);
 >      }
 >      tp = intotcpcb(inp);
 
 
 Thanks,
 
 -- 
 Daniel Austin MBCS
 Managing Director
 Kewlio.net Limited

From: Robert Watson <rwatson@FreeBSD.org>
To: Daniel Austin <daniel@kewlio.net>
Cc: Diane Bruce <db@db.net>, bug-followup@FreeBSD.org
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Tue, 28 Nov 2006 15:00:18 +0000 (GMT)

 On Sun, 26 Nov 2006, Daniel Austin wrote:
 
 > Testing it out now - will let you know the outcome.
 
 Any luck?  We'd like to get this fix into 6.2, but time is starting to run 
 low.
 
 Thanks,
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge
 
 >
 >
 > Thanks,
 >
 > Daniel.
 >
 > Robert Watson wrote:
 >> 
 >> On Fri, 3 Nov 2006, Diane Bruce wrote:
 >> 
 >>> The following reply was made to PR kern/102412; it has been noted by 
 >>> GNATS.
 >>> 
 >>> From: Diane Bruce <db@db.net>
 >>> To: Robert Watson <rwatson@FreeBSD.org>
 >>> Cc: Diane Bruce <db@db.net>, bug-followup@FreeBSD.org, wes@freebsd.org,
 >>>     Daniel Austin <daniel@kewlio.net>
 >>> Subject: Re: kern/102412: kernel panic in 6.1-stable during normal 
 >>> operation
 >>> Date: Thu, 2 Nov 2006 19:52:23 -0500
 >>> 
 >>> On Thu, Oct 26, 2006 at 07:22:41PM +0100, Robert Watson wrote:
 >>> > On Thu, 26 Oct 2006, Diane Bruce wrote:
 >>> >
 >>> > >Panic String: mtx_lock() of destroyed mutex @
 >>> > >/usr/src/sys/netinet/ip_output.c:1198
 >>> > >
 >>> ...
 >>> 
 >>> Now an even more amusing core!
 >> 
 >> Could you try the attached patch, which provides a more substantive fix 
 >> while still avoiding the major MFC of 7-CURRENT structural fixes?
 >> 
 >> If GNATS has mangled the patch, please try:
 >>
 >>   http://www.watson.org/~robert/freebsd/netperf/20061124-ip_ctloutput.diff
 >> 
 >> I've done light testing here, but not with the orts of loads that generates 
 >> the above class of panics.
 >> 
 >> Robert N M Watson
 >> Computer Laboratory
 >> University of Cambridge
 >> 
 >> Index: ip_output.c
 >> ===================================================================
 >> RCS file: /data/fbsd-cvs/ncvs/src/sys/netinet/ip_output.c,v
 >> retrieving revision 1.242.2.16
 >> diff -u -r1.242.2.16 ip_output.c
 >> --- ip_output.c    24 Oct 2006 13:23:03 -0000    1.242.2.16
 >> +++ ip_output.c    24 Nov 2006 15:47:13 -0000
 >> @@ -1148,15 +1148,29 @@
 >>
 >>  /*
 >>   * IP socket option processing.
 >> + *
 >> + * There are two versions of this call in order to work around a race
 >> + * condition in TCP in FreeBSD 6.x.  In the TCP implementation, so->so_pcb
 >> + * can become NULL if the pcb or pcbinfo lock isn't held.  However, when
 >> + * entering ip_ctloutput(), neither lock is held, and finding the pointer 
 >> to
 >> + * either lock requires follow so->so_pcb, which may be NULL.
 >> + * ip_ctloutput_pcbinfo() accepts the pcbinfo pointer so that the lock can 
 >> be
 >> + * safely acquired.  This is not required in FreeBSD 7.x because the
 >> + * invariants on so->so_pcb are much stronger, so it cannot become NULL
 >> + * while the socket is in use.
 >>   */
 >>  int
 >> -ip_ctloutput(so, sopt)
 >> +ip_ctloutput_pcbinfo(so, sopt, pcbinfo)
 >>      struct socket *so;
 >>      struct sockopt *sopt;
 >> +    struct inpcbinfo *pcbinfo;
 >>  {
 >>      struct    inpcb *inp = sotoinpcb(so);
 >>      int    error, optval;
 >> 
 >> +    if (pcbinfo == NULL)
 >> +        pcbinfo = inp->inp_pcbinfo;
 >> +
 >>      error = optval = 0;
 >>      if (sopt->sopt_level != IPPROTO_IP) {
 >>          return (EINVAL);
 >> @@ -1190,12 +1204,15 @@
 >>                  m_free(m);
 >>                  break;
 >>              }
 >> +            INP_INFO_WLOCK(pcbinfo);
 >>              if (so->so_pcb == NULL) {
 >> +                INP_INFO_WUNLOCK(pcbinfo);
 >>                  m_free(m);
 >>                  error = EINVAL;
 >>                  break;
 >>              }
 >>              INP_LOCK(inp);
 >> +            INP_INFO_WUNLOCK(pcbinfo);
 >>              error = ip_pcbopts(inp, sopt->sopt_name, m);
 >>              INP_UNLOCK(inp);
 >>              return (error);
 >> @@ -1217,10 +1234,14 @@
 >>              if (error)
 >>                  break;
 >> 
 >> +            INP_INFO_WLOCK(pcbinfo);
 >>              if (so->so_pcb == NULL) {
 >> +                INP_INFO_WUNLOCK(pcbinfo);
 >>                  error = EINVAL;
 >>                  break;
 >>              }
 >> +            INP_LOCK(inp);
 >> +            INP_INFO_WUNLOCK(pcbinfo);
 >>              switch (sopt->sopt_name) {
 >>              case IP_TOS:
 >>                  inp->inp_ip_tos = optval;
 >> @@ -1277,6 +1298,7 @@
 >>                  OPTSET(INP_DONTFRAG);
 >>                  break;
 >>              }
 >> +            INP_UNLOCK(inp);
 >>              break;
 >>  #undef OPTSET
 >> 
 >> @@ -1295,11 +1317,13 @@
 >>              if (error)
 >>                  break;
 >> 
 >> +            INP_INFO_WLOCK(pcbinfo);
 >>              if (so->so_pcb == NULL) {
 >>                  error = EINVAL;
 >>                  break;
 >>              }
 >>              INP_LOCK(inp);
 >> +            INP_INFO_WUNLOCK(pcbinfo);
 >>              switch (optval) {
 >>              case IP_PORTRANGE_DEFAULT:
 >>                  inp->inp_flags &= ~(INP_LOWPORT);
 >> @@ -1480,6 +1504,15 @@
 >>      return (error);
 >>  }
 >> 
 >> +int
 >> +ip_ctloutput(so, sopt)
 >> +    struct socket *so;
 >> +    struct sockopt *sopt;
 >> +{
 >> +
 >> +    return (ip_ctloutput_pcbinfo(so, sopt, NULL));
 >> +}
 >> +
 >>  /*
 >>   * Set up IP options in pcb for insertion in output packets.
 >>   * Store in mbuf with pointer in pcbopt, adding pseudo-option
 >> Index: ip_var.h
 >> ===================================================================
 >> RCS file: /data/fbsd-cvs/ncvs/src/sys/netinet/ip_var.h,v
 >> retrieving revision 1.95
 >> diff -u -r1.95 ip_var.h
 >> --- ip_var.h    2 Jul 2005 23:13:31 -0000    1.95
 >> +++ ip_var.h    24 Nov 2006 15:32:53 -0000
 >> @@ -144,6 +144,7 @@
 >>
 >>  struct ip;
 >>  struct inpcb;
 >> +struct inpcbinfo;
 >>  struct route;
 >>  struct sockopt;
 >> 
 >> @@ -164,6 +165,8 @@
 >>  extern struct    pr_usrreqs rip_usrreqs;
 >>
 >>  int     ip_ctloutput(struct socket *, struct sockopt *sopt);
 >> +int     ip_ctloutput_pcbinfo(struct socket *, struct sockopt *sopt,
 >> +        struct inpcbinfo *pcbinfo);
 >>  void     ip_drain(void);
 >>  void     ip_fini(void *xtp);
 >>  int     ip_fragment(struct ip *ip, struct mbuf **m_frag, int mtu,
 >> Index: tcp_usrreq.c
 >> ===================================================================
 >> RCS file: /data/fbsd-cvs/ncvs/src/sys/netinet/tcp_usrreq.c,v
 >> retrieving revision 1.124.2.3
 >> diff -u -r1.124.2.3 tcp_usrreq.c
 >> --- tcp_usrreq.c    27 Sep 2006 09:24:44 -0000    1.124.2.3
 >> +++ tcp_usrreq.c    24 Nov 2006 14:59:41 -0000
 >> @@ -1035,7 +1035,7 @@
 >>              error = ip6_ctloutput(so, sopt);
 >>          else
 >>  #endif /* INET6 */
 >> -        error = ip_ctloutput(so, sopt);
 >> +        error = ip_ctloutput_pcbinfo(so, sopt, &tcbinfo);
 >>          return (error);
 >>      }
 >>      tp = intotcpcb(inp);
 >
 >
 > Thanks,
 >
 > -- 
 > Daniel Austin MBCS
 > Managing Director
 > Kewlio.net Limited
 >

From: Diane Bruce <db@db.net>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: Daniel Austin <daniel@kewlio.net>, Diane Bruce <db@db.net>,
	bug-followup@FreeBSD.org
Subject: Re: kern/102412: kernel panic in 6.1-stable during normal operation
Date: Tue, 28 Nov 2006 10:07:05 -0500

 On Tue, Nov 28, 2006 at 03:00:18PM +0000, Robert Watson wrote:
 >
 > On Sun, 26 Nov 2006, Daniel Austin wrote:
 >
 > >Testing it out now - will let you know the outcome.
 >
 > Any luck?  We'd like to get this fix into 6.2, but time is starting to run
 > low.
 
 *** Server Up 1 days, 21:58:39
 *** Highest connection count:  5516 (5515 clients)
 
 It's looking good to me.
 
 ... rest elided
 
 --
 - db@db.net http://www.db.net/~db

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/102412: commit references a PR
Date: Tue, 28 Nov 2006 21:41:32 +0000 (UTC)

 rwatson     2006-11-28 21:41:12 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_6)
     sys/netinet          ip_output.c ip_var.h tcp_usrreq.c 
   Log:
   Reformulate ip_ctloutput() and tcp_ctloutput() to work around the fact
   that so_pcb can be invalidated at any time due to an untimely reset.
   Move the body of ip_ctloutput() to ip_ctloutput_pcbinfo(), which
   accepts a pcbinfo argument, and wrap it with ip_ctloutput(), which
   passes a NULL.  Modify tcp_ctloutput() to directly invoke
   ip_ctloutput_pcbinfo() and pass tcbinfo.  Hold the pcbinfo lock when
   dereferencing so_pcb and acquiring the inpcb lock in order to prevent
   the inpcb from being freed; the pcbinfo lock is then immediately
   dropped.  This is required as TCP may free the inppcb and invalidate
   so_pcb due to a reset at any time in the RELENG_6 network stack, which
   otherwise leads to a panic.
   
   This panic might be frequently seen on highly loaded IRC and Samba
   servers, which have long-lasting TCP connections, query socket options
   frequently, and see a significant number of reset connections.
   
   This change has been merged directly to RELENG_6 as the problem does
   not exist in HEAD, where the invariants for so_pcb are much stronger;
   the architectural changes in HEAD avoid the need to acquire a global
   lock in the socket option path.  This change will be merged to
   RELENG_6_2.
   
   PR:             102412, 104765
   Reviewed by:    Diane Bruce <db at db.net>
   Tested by:      Daniel Austin <daniel at kewlio dot net>,
                   Kai Gallasch <gallasch at free dot de>
   
   Revision    Changes    Path
   1.242.2.17  +34 -1     src/sys/netinet/ip_output.c
   1.95.2.1    +3 -0      src/sys/netinet/ip_var.h
   1.124.2.4   +1 -1      src/sys/netinet/tcp_usrreq.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/102412: commit references a PR
Date: Tue, 28 Nov 2006 23:19:35 +0000 (UTC)

 rwatson     2006-11-28 23:19:18 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_6_2)
     sys/netinet          ip_output.c ip_var.h tcp_usrreq.c 
   Log:
   Merge ip_output.c:1.242.2.17, ip_var.h:1.95.2.1, tcp_usrreq.c:1.124.2.4
   from RELENG_6 to RELENG_6_2:
   
     Reformulate ip_ctloutput() and tcp_ctloutput() to work around the fact
     that so_pcb can be invalidated at any time due to an untimely reset.
     Move the body of ip_ctloutput() to ip_ctloutput_pcbinfo(), which
     accepts a pcbinfo argument, and wrap it with ip_ctloutput(), which
     passes a NULL.  Modify tcp_ctloutput() to directly invoke
     ip_ctloutput_pcbinfo() and pass tcbinfo.  Hold the pcbinfo lock when
     dereferencing so_pcb and acquiring the inpcb lock in order to prevent
     the inpcb from being freed; the pcbinfo lock is then immediately
     dropped.  This is required as TCP may free the inppcb and invalidate
     so_pcb due to a reset at any time in the RELENG_6 network stack, which
     otherwise leads to a panic.
   
     This panic might be frequently seen on highly loaded IRC and Samba
     servers, which have long-lasting TCP connections, query socket options
     frequently, and see a significant number of reset connections.
   
     This change has been merged directly to RELENG_6 as the problem does
     not exist in HEAD, where the invariants for so_pcb are much stronger;
     the architectural changes in HEAD avoid the need to acquire a global
     lock in the socket option path.  This change will be merged to
     RELENG_6_2.
   
     PR:             102412, 104765
     Reviewed by:    Diane Bruce <db at db.net>
     Tested by:      Daniel Austin <daniel at kewlio dot net>,
                     Kai Gallasch <gallasch at free dot de>
   
   Approved by:    re (kensmith)
   
   Revision        Changes    Path
   1.242.2.16.2.1  +34 -1     src/sys/netinet/ip_output.c
   1.95.8.1        +3 -0      src/sys/netinet/ip_var.h
   1.124.2.3.2.1   +1 -1      src/sys/netinet/tcp_usrreq.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: feedback->closed 
State-Changed-By: rwatson 
State-Changed-When: Wed Nov 29 10:20:02 UTC 2006 
State-Changed-Why:  
Close PR as the problem is now believed fixed; the final patch has been 
merged to RELENG_6 and RELENG_6_2 for inclusion in FreeBSD 6.2.  Please 
let me know if the problem is not resolved and we can re-open the PR. 

RELENG_6 revisions: 

Revision    Changes    Path 
1.242.2.17  +34 -1     src/sys/netinet/ip_output.c 
1.95.2.1    +3 -0      src/sys/netinet/ip_var.h 
1.124.2.4   +1 -1      src/sys/netinet/tcp_usrreq.c 

RELENG_6_2 revisions: 

Revision        Changes    Path 
1.242.2.16.2.1  +34 -1     src/sys/netinet/ip_output.c 
1.95.8.1        +3 -0      src/sys/netinet/ip_var.h 
1.124.2.3.2.1   +1 -1      src/sys/netinet/tcp_usrreq.c 


http://www.freebsd.org/cgi/query-pr.cgi?pr=102412 
>Unformatted:
