From plosher@isc.org  Tue Jun 17 11:53:50 2003
Return-Path: <plosher@isc.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 7361937B401
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 17 Jun 2003 11:53:50 -0700 (PDT)
Received: from sf1.isc.org (sf1.isc.org [204.152.184.237])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 2196943FBF
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 17 Jun 2003 11:53:50 -0700 (PDT)
	(envelope-from plosher@isc.org)
Received: by sf1.isc.org (Postfix, from userid 10188)
	id F3AAD2846E; Tue, 17 Jun 2003 18:53:49 +0000 (UTC)
Message-Id: <20030617185349.F3AAD2846E@sf1.isc.org>
Date: Tue, 17 Jun 2003 18:53:49 +0000 (UTC)
From: Peter Losher <plosher@isc.org>
Reply-To: Peter Losher <plosher@isc.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc: plosher@isc.org
Subject: kmem_map too small after around 12 hours. (5.1-REL)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         53416
>Category:       kern
>Synopsis:       kmem_map too small after around 12 hours. (5.1-REL, 5.2.1-REL)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jun 17 12:00:30 PDT 2003
>Closed-Date:    Wed Jan 12 02:20:33 GMT 2005
>Last-Modified:  Wed Jan 12 02:20:33 GMT 2005
>Originator:     Peter Losher
>Release:        FreeBSD 5.1-RELEASE i386
>Organization:
ISC
>Environment:
System: FreeBSD sf1.isc.org 5.1-RELEASE FreeBSD 5.1-RELEASE #2: Tue Jun 17 05:12:24 UTC 2003 plosher@sf1.isc.org:/usr/src/sys/i386/compile/ISC_FBSD_51_SMP i386

>Description:

So, I recently put a Quad-Xeon server (w/ 4GB of RAM) into production as
primarily as a master FTP server for ISC as well as for FreeBSD (US/IPv6
side of ftp.freebsd.org).  All has been going well for the past couple of
weeks until 5.1 was released. Traffic spiked, and the server started
panicking every 12 hours (give or take 30 min):

> panic: kmem_malloc(4096): kmem_map too small: 377487360 total allocated

So after looking at the archives, I read that FreeBSD had issues with
installed memory above 2GB, so I set the sysctl kern.maxvnodes=130000, but
to no effect:

> panic: kmem_malloc(16384): kmem_map too small: 293519360 total allocated

I have since removed my NMBCLUSTER setting in the kernel, so it would be
set automatically, (also resetting the kern.maxvnodes sysctl back to default) 
I have even set VM_KMEM_SIZE_SCALE="2" (I haven't messed with the other 
KMEM kernel options, as I was hoping to avoid it)  And 13 hours ago I 
updated the kernel with the latest kern_malloc.c file in the RELENG_5_1 branch.

That too also panicked after 12.5 hours this morning, looking at 'netstat -m'
just before it crashed, it had wired memory reserves of 16M, and the amount 
(percentage being used) in the last hour rose steadily from 68% to about 86% 
before the panic occurs.

>How-To-Repeat:
	
>Fix:

The work around is to reboot the box every 11.5 hours, so the 
kmem_map gets reset.  However that is not a viable solution for a 
production box. :(  Is there any way to check and make sure there is not a
leakage, and if not, make sure that FreeBSD is calculating a large enough
kmem_map?

>Release-Note:
>Audit-Trail:

From: Don Bowman <don@sandvine.com>
To: "'freebsd-gnats-submit@FreeBSD.org'" <freebsd-gnats-submit@FreeBSD.org>,
	"'plosher@isc.org'" <plosher@isc.org>
Cc:  
Subject: Re: kern/53416: kmem_map too small after around 12 hours. (5.1-RE
	L)
Date: Sun, 7 Mar 2004 13:36:49 -0500 

 FYI this is still an issue with 5.2.1 with
 a dual xeon with 4GB of memory.
 I have set VM_KMEM_SIZE_SCALE and VM_KMEM_SIZE_MAX
 to workaround it.
 
State-Changed-From-To: open->feedback 
State-Changed-By: kris 
State-Changed-When: Wed Jan 5 00:54:42 GMT 2005 
State-Changed-Why:  
As suggested, increasing the value of VM_KMEM_SIZE_MAX (from default 
value of 320MB) is expected to fix this.  Is it still an issue after 
tuning that parameter? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=53416 
State-Changed-From-To: feedback->closed 
State-Changed-By: kris 
State-Changed-When: Wed Jan 12 02:20:18 GMT 2005 
State-Changed-Why:  
Submitter confirms problem is resolved. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=53416 
>Unformatted:
