From nobody@FreeBSD.org  Fri Oct  7 21:14:03 2005
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id CBDFF16A41F
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  7 Oct 2005 21:14:03 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 9B6DA43D48
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  7 Oct 2005 21:14:03 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id j97LE3ps085450
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 7 Oct 2005 21:14:03 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id j97LE3mD085449;
	Fri, 7 Oct 2005 21:14:03 GMT
	(envelope-from nobody)
Message-Id: <200510072114.j97LE3mD085449@www.freebsd.org>
Date: Fri, 7 Oct 2005 21:14:03 GMT
From: Chris Ulrich <cdu@ucr.edu>
To: freebsd-gnats-submit@FreeBSD.org
Subject: 5.4 system with SMP and IPFW crashes under load (mbuf underrun)
X-Send-Pr-Version: www-2.3

>Number:         87094
>Category:       kern
>Synopsis:       5.4 system with SMP and IPFW crashes under load (mbuf underrun)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-net
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Oct 07 21:20:08 GMT 2005
>Closed-Date:    Wed Jul 21 22:31:04 UTC 2010
>Last-Modified:  Wed Jul 21 22:31:04 UTC 2010
>Originator:     Chris Ulrich
>Release:        5.4-RELEASE-p7 / intel 386
>Organization:
University of California at Riverside
>Environment:
FreeBSD wg03 5.4-RELEASE-p7 FreeBSD 5.4-RELEASE-p7 #0: Tue Oct  4 15:57:24 PDT 2005     shock@wg04:/usr/obj/usr/src/sys/FW_NO_SMP  i386

>Description:
Firewall system running nat pumping packets between 2 intel EM interfaces.

When the system is running IPFW and has SMP turned on, the system will crash after a random (large) number of packets flow through the system.
>How-To-Repeat:
Run SMP and IPFW and natd on a 5.4 i386 system and run iperf between one interface and the other (server outside the nat interface).

After a minute or so, netstat -m will show something like:
4294266316 mbufs in use
4294737989/25600 mbuf clusters in use (current/max)
3560445 KBytes allocated to network
201 calls to protocol drain routines

After another period of time, the system will eventually reboot.  The behavior is not related to load or firewall rule changes.

>Fix:
              
>Release-Note:
>Audit-Trail:

From: Robert Watson <rwatson@FreeBSD.org>
To: Chris Ulrich <cdu@ucr.edu>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/87094: 5.4 system with SMP and IPFW crashes under load
 (mbuf	underrun)
Date: Sat, 8 Oct 2005 18:47:42 +0100 (BST)

 On Fri, 7 Oct 2005, Chris Ulrich wrote:
 
 > Run SMP and IPFW and natd on a 5.4 i386 system and run iperf between one 
 > interface and the other (server outside the nat interface).
 >
 > After a minute or so, netstat -m will show something like:
 > 4294266316 mbufs in use
 > 4294737989/25600 mbuf clusters in use (current/max)
 > 3560445 KBytes allocated to network
 > 201 calls to protocol drain routines
 >
 > After another period of time, the system will eventually reboot.  The 
 > behavior is not related to load or firewall rule changes.
 
 FYI, statistics errors in the mbuf allocator in 5.x are probably unrelated 
 to the reboot, and occurs due to races in statistics gathering.  6.x 
 corrects this problem, and I'm considering a merge of the changes to the 
 5.x branch following 6.0-RELEASE and some additional in-field testing.
 
 The reboot is, however, concerning.  Do you get a panic and reboot, or 
 just a reboot?  In particular, if you compile in the kernel debugger, does 
 it drop to the debugger at some point, or reboot?
 
 Could you try a 5-STABLE kernel and see if that works better?  There have 
 been a number of synchronization fixes that have not yet been released as 
 errata, which might affect this.
 
 Thanks,
 
 Robert N M Watson
State-Changed-From-To: open->feedback 
State-Changed-By: linimon 
State-Changed-When: Fri Oct 28 06:54:55 GMT 2005 
State-Changed-Why:  
This should have been set to feedback to note that the submitter had been 
asked for more information. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=87094 

Adding to audit trail from misfiled PR kern/90994:

Date: Tue, 27 Dec 2005 18:41:30 -0800
From: cdu@charon.ucr.edu

The issue with a 5.4 i386 SMP system crashing under heavy network
load was not resolved but we were able to work around the problem by
turning off the IPFW nat and using PF for nat.  This has resulted in
a system that is 100% reliable under the same conditions that caused
the old system to fail.
 
 Chris Ulrich
 cdu@ucr.edu
State-Changed-From-To: feedback->suspended 
State-Changed-By: linimon 
State-Changed-When: Wed Dec 28 03:14:37 UTC 2005 
State-Changed-Why:  
Copy text from misfiled PR kern/90994 here.  Mark as 'suspended' since 
the submitter found a workaround, but it does indicate a problem that 
needs to be fixed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=87094 
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: brucec 
Responsible-Changed-When: Wed Jul 21 16:35:28 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=87094 
State-Changed-From-To: suspended->closed  
State-Changed-By: brucec 
State-Changed-When: Wed Jul 21 22:30:29 UTC 2010 
State-Changed-Why:  
FreeBSD 5.x is no longer supported. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=87094 
>Unformatted:
Does this problem still exist with recent releases?
