From nobody@FreeBSD.org  Sun Dec 19 20:03:15 2004
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id A2E4A16A4CF
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 19 Dec 2004 20:03:15 +0000 (GMT)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 910D343D1F
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 19 Dec 2004 20:03:15 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id iBJK3F9H065661
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 19 Dec 2004 20:03:15 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id iBJK3FjJ065659;
	Sun, 19 Dec 2004 20:03:15 GMT
	(envelope-from nobody)
Message-Id: <200412192003.iBJK3FjJ065659@www.freebsd.org>
Date: Sun, 19 Dec 2004 20:03:15 GMT
From: Justin Erenkrantz <justin@erenkrantz.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: netstat -m 'mbufs in use' output appears to be incorrect
X-Send-Pr-Version: www-2.3

>Number:         75277
>Category:       kern
>Synopsis:       netstat -m 'mbufs in use' output appears to be incorrect
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    rwatson
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Dec 19 20:10:14 GMT 2004
>Closed-Date:    Sun Nov 27 18:58:43 GMT 2005
>Last-Modified:  Sun Nov 27 18:58:43 GMT 2005
>Originator:     Justin Erenkrantz
>Release:        5.3-STABLE
>Organization:
Apache Software Foundation
>Environment:
FreeBSD ... 5.3-STABLE FreeBSD 5.3-STABLE #2: Sat Dec 18 14:54:30 PST 2004     root@...:/usr/obj/usr/src/sys/SMP  i386
>Description:
The 'mbufs in use' parameter from 'netstat -m' appears from to be incorrect (or non-sensical).

% netstat -m
4294965753 mbufs in use
621/25600 mbuf clusters in use (current/max)
0/5/6656 sfbufs in use (current/peak/max)
871 KBytes allocated to network
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
34 calls to protocol drain routines

We are using the em(4) driver with RELENG_5 from Dec 18th.  We upgraded to RELENG_5 from 5.3-RELEASE to catch the recent em(4) updates.
>How-To-Repeat:
netstat -m
>Fix:

>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: kris 
State-Changed-When: Wed Jan 5 00:58:34 GMT 2005 
State-Changed-Why:  
Feedback requested 

http://www.freebsd.org/cgi/query-pr.cgi?pr=75277 

From: Kris Kennaway <kris@obsecurity.org>
To: Justin Erenkrantz <justin@erenkrantz.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/75277: netstat -m 'mbufs in use' output appears to be incorrect
Date: Tue, 4 Jan 2005 16:58:51 -0800

 On Sun, Dec 19, 2004 at 08:03:15PM +0000, Justin Erenkrantz wrote:
 
 > The 'mbufs in use' parameter from 'netstat -m' appears from to be incorrect (or non-sensical).
 
 I believe this is a known issue on SMP systems.  Robert Watson posted
 a patch on the current@ mailing list around November in response to a
 similar report by Andrew Gallatin (synchronizing the mbuf counters has
 performance penalties though).  Can you confirm that if you use that
 patch the problem goes away?
 
 Kris

From: Martin Matuska <matuska@firma.zoznam.sk>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/75277: netstat -m 'mbufs in use' output appears to be incorrect
Date: Wed, 02 Mar 2005 17:38:31 +0100

 We encounter the same problem on latest 5-STABLE, on Sun Fire V65x (dual Xeon SMP machine w 4G of RAM)
 
 This system is under moderate network load - an average about 2000 tcp connections, ~350 permanent running processes
 
 FreeBSD 5.4-PRERELEASE i386, kernel built from sources: Sat Feb 26 18:19:57 CET 2005
 
 # netstat -m
 4294886151 mbufs in use
 4294959052/25600 mbuf clusters in use (current/max)
 0/3/6656 sfbufs in use (current/peak/max)
 4157529 KBytes allocated to network
 0 requests for sfbufs denied
 0 requests for sfbufs delayed
 0 requests for I/O initiated by sendfile
 2 calls to protocol drain routines
 
 -- 
 Martin Matuska
 IT Systems Administrator
 
 Zoznam s.r.o., Slovakia

From: Kris Kennaway <kris@obsecurity.org>
To: Martin Matuska <matuska@firma.zoznam.sk>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/75277: netstat -m 'mbufs in use' output appears to be incorrect
Date: Wed, 2 Mar 2005 17:15:03 -0800

 On Wed, Mar 02, 2005 at 04:40:24PM +0000, Martin Matuska wrote:
 > The following reply was made to PR kern/75277; it has been noted by GNATS.
 > 
 > From: Martin Matuska <matuska@firma.zoznam.sk>
 > To: freebsd-gnats-submit@FreeBSD.org
 > Cc:  
 > Subject: Re: kern/75277: netstat -m 'mbufs in use' output appears to be incorrect
 > Date: Wed, 02 Mar 2005 17:38:31 +0100
 > 
 >  We encounter the same problem on latest 5-STABLE, on Sun Fire V65x (dual Xeon SMP machine w 4G of RAM)
 >  
 >  This system is under moderate network load - an average about 2000 tcp connections, ~350 permanent running processes
 >  
 >  FreeBSD 5.4-PRERELEASE i386, kernel built from sources: Sat Feb 26 18:19:57 CET 2005
 >  
 >  # netstat -m
 >  4294886151 mbufs in use
 >  4294959052/25600 mbuf clusters in use (current/max)
 >  0/3/6656 sfbufs in use (current/peak/max)
 >  4157529 KBytes allocated to network
 >  0 requests for sfbufs denied
 >  0 requests for sfbufs delayed
 >  0 requests for I/O initiated by sendfile
 >  2 calls to protocol drain routines
 
 This is documented in the release errata.  The stats are incorrect on
 SMP, and making them correct would impose a performance penalty.
 
 Kris
State-Changed-From-To: feedback->patched 
State-Changed-By: rwatson 
State-Changed-When: Fri Jul 22 02:53:11 GMT 2005 
State-Changed-Why:  
This problem was introduced with the move to using UMA for mbuf 
allocation, as the mbuf allocation stats were no longer properly 
synchronized as of that time.  As a result, busy SMP systems could 
see stat drived. 

This problem is believed to be fixed in 7-CURRENT via the introduction 
of libmemstat(3), as well as changes to uma(9) and malloc(9) statistics 
gathering.  An MFC to 6.x for 6.0 is planned in the near future.  MFC 
to 5.x is being considered, but will require some additional work as 
there have been a number of significant changes in memory allocation 
between 5.x and 6.x.  It would be helpful to confirm that the 6.x 
statistics now render properly, etc.  Leaving this PR open until MFC's 
take place. 



Responsible-Changed-From-To: freebsd-bugs->rwatson 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Fri Jul 22 02:53:11 GMT 2005 
Responsible-Changed-Why:  
This problem was introduced with the move to using UMA for mbuf 
allocation, as the mbuf allocation stats were no longer properly 
synchronized as of that time.  As a result, busy SMP systems could 
see stat drived. 

This problem is believed to be fixed in 7-CURRENT via the introduction 
of libmemstat(3), as well as changes to uma(9) and malloc(9) statistics 
gathering.  An MFC to 6.x for 6.0 is planned in the near future.  MFC 
to 5.x is being considered, but will require some additional work as 
there have been a number of significant changes in memory allocation 
between 5.x and 6.x.  It would be helpful to confirm that the 6.x 
statistics now render properly, etc.  Leaving this PR open until MFC's 
take place. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=75277 
State-Changed-From-To: patched->closed 
State-Changed-By: rwatson 
State-Changed-When: Sun Nov 27 18:57:42 GMT 2005 
State-Changed-Why:  
The changes associated with this PR were merged to RELENG_6 for inclusion 
in 6.0-RELEASE.  Currently, there are no plans to backport these changes 
to RELENG_5, as they change the layout of a number of kernel data 
structures and would disrupt ABI stability for device drivers in the 
branch. 

Thanks for the report! 


http://www.freebsd.org/cgi/query-pr.cgi?pr=75277 
>Unformatted:
