From nobody@FreeBSD.org  Mon Jun 20 05:48:55 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 9046B106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 20 Jun 2011 05:48:55 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 7637C8FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 20 Jun 2011 05:48:55 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p5K5mt1C011803
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 20 Jun 2011 05:48:55 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p5K5mtfL011802;
	Mon, 20 Jun 2011 05:48:55 GMT
	(envelope-from nobody)
Message-Id: <201106200548.p5K5mtfL011802@red.freebsd.org>
Date: Mon, 20 Jun 2011 05:48:55 GMT
From: Pavlo <pigich@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: panic: kmem_malloc(20480): kmem_map too small
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         158063
>Category:       kern
>Synopsis:       [panic] kmem_malloc(20480): kmem_map too small
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jun 20 05:50:09 UTC 2011
>Closed-Date:    Fri Feb 24 07:27:54 UTC 2012
>Last-Modified:  Fri Feb 24 07:27:54 UTC 2012
>Originator:     Pavlo
>Release:        8.1-RELEASE-p1
>Organization:
>Environment:
FreeBSD PPPoE01-localhost 8.1-RELEASE-p1 FreeBSD 8.1-RELEASE-p1 #0: Thu Nov 25 10:11:59 EET 2010     root@localhost:/usr/src/sys/i386/compile/GENERIC  i386
>Description:
Unread portion of the kernel message buffer:
panic: kmem_malloc(20480): kmem_map too small: 532951040 total allocated
cpuid = 0
Uptime: 15d15h30m13s
Physical memory: 1897 MB
Dumping 694 MB: 679 663 647 631 615 599 583 567 551 535 519 503 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7
...

Loaded symbols for /boot/kernel/ng_vjc.ko
Reading symbols from /boot/kernel/ng_tcpmss.ko...Reading symbols from /boot/kernel/ng_tcpmss.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_tcpmss.ko
#0  doadump () at pcpu.h:246
246     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt full
#0  doadump () at pcpu.h:246
No locals.
#1  0xc0794cf7 in boot (howto=260) at ../../../kern/kern_shutdown.c:416
        _giantcnt = Variable "_giantcnt" is not available.
#2  0xc0794f59 in panic (fmt=Variable "fmt" is not available.
) at ../../../kern/kern_shutdown.c:590
590             boot(bootopt);
#3  0xc0995fda in kmem_malloc (map=0xc159008c, size=20480, flags=2)
    at ../../../vm/vm_kern.c:305
305       panic("kmem_malloc(%ld): kmem_map too small: %ld total allocated",
#4  0xc098aba7 in page_alloc (zone=0x0, bytes=20480, pflag=0xf4932aef "\002",
    wait=2) at ../../../vm/uma_core.c:979
979             p = (void *) kmem_malloc(kmem_map, bytes, wait);
#5  0xc098d410 in uma_large_malloc (size=20480, wait=2)
    at ../../../vm/uma_core.c:3003
3003            mem = page_alloc(NULL, size, &flags, wait);
#6  0xc0781c00 in malloc (size=20480, mtp=0xc0bb23cc, flags=2)
    at ../../../kern/kern_malloc.c:391
391                     va = uma_large_malloc(size, flags);
#7  0xc07ca023 in sbuf_new (s=0xc6b16aa0, buf=0x0, length=16385, flags=0)
    at ../../../kern/subr_sbuf.c:193
193             s->s_buf = SBMALLOC(s->s_size);
#8  0xc083e61c in ifioctl (so=0xdf9924d4, cmd=3221776676, data=0xccf1d660 "",
    td=0xc595b780) at ../../../net/if.c:2732
2732            sb = sbuf_new(NULL, NULL, max_len + 1, SBUF_FIXEDLEN);
#9  0xc07d9d42 in soo_ioctl (fp=0xc688ab98, cmd=3221776676, data=0xccf1d660,
    active_cred=0xc511d100, td=0xc595b780) at ../../../kern/sys_socket.c:212
212                             error = ifioctl(so, cmd, data, td);
#10 0xc07d2360 in kern_ioctl (td=0xc595b780, fd=80, com=3221776676,
    data=0xccf1d660 "") at file.h:262
262     file.h: No such file or directory.
        in file.h
#11 0xc07d24d4 in ioctl (td=0xc595b780, uap=0xf4932cf8)
    at ../../../kern/sys_generic.c:678
678             error = kern_ioctl(td, uap->fd, com, data);
#12 0xc0a66343 in syscall (frame=0xf4932d38) at ../../../i386/i386/trap.c:1111
1111                    error = (*sa.callp->sy_call)(td, sa.args);
#13 0xc0a48b10 in Xint0x80_syscall () at ../../../i386/i386/exception.s:261
261             call    syscall
Current language:  auto; currently asm
#14 0x00000033 in ?? ()



sysctl -a | grep kmem
<118>savecore: reboot after panic: kmem_malloc(20480): kmem_map too small: 532951040 total allocated
<118>Jun 16 23:04:58 PPPoE01-G46 savecore: reboot after panic: kmem_malloc(20480): kmem_map too small: 532951040 total allocated
vm.kmem_size_scale: 3
vm.kmem_size_max: 536870912
vm.kmem_size_min: 0
vm.kmem_size: 536870912


# cat /boot/loader.conf
vm.kmem_size_max="512M"
vm.kmem_size="512M"

>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Pavlo <pigich@gmail.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/158063: panic: kmem_malloc(20480): kmem_map too small
Date: Mon, 20 Jun 2011 15:34:44 +0400

   Can you please provide output of the following commands:
 
 > vmstat -z -M vmcore.0
 > vmstat -m -M vmcore.0
 
 -- 
 Totus tuus, Glebius.

From: Gleb Smirnoff <glebius@freebsd.org>
To: Pasha Pigich <pigich@gmail.com>
Cc: bug-followup@freebsd.org
Subject: Re: kern/158063: panic: kmem_malloc(20480): kmem_map too small
Date: Tue, 21 Jun 2011 12:34:43 +0400

 On Mon, Jun 20, 2011 at 05:59:25PM +0300, Pasha Pigich wrote:
 P>      dummynet 33141 397658K       -    33142  512,1024
 P>      dummynet   506   206K       -   205783  256,512
 ...
 P> I know that something eats RAM and then kernel panics, but I don't know how
 P> to find what.?
 
 I suspect dummynet.
 
 Do you have any monitoring facility like zabbix? Can you start to monitor
 this value:
 
 expr $(vmstat -m | grep dummynet | awk '{print $2" +"}') 0
 
 It'll be interesting to see plot for this value. Does it grow steadily
 during machine runtime?
 
 -- 
 Totus tuus, Glebius.

From: Gleb Smirnoff <glebius@freebsd.org>
To: Pasha Pigich <pigich@gmail.com>
Cc: bug-followup@freebsd.org
Subject: Re: kern/158063: panic: kmem_malloc(20480): kmem_map too small
Date: Tue, 21 Jun 2011 13:31:52 +0400

   btw, recently a memleak in dummynet has been fixed:
 
 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/156083
 
 I'd suggest you first to upgrade to recent stable/8, and then
 continue investigation.
 
 -- 
 Totus tuus, Glebius.
State-Changed-From-To: open->feedback 
State-Changed-By: ae 
State-Changed-When: Wed Jun 22 09:41:30 UTC 2011 
State-Changed-Why:  
Can you confirm that the problem is fixed in the fresh stable/8? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=158063 
State-Changed-From-To: feedback->closed 
State-Changed-By: glebius 
State-Changed-When: Thu Jun 23 07:55:53 UTC 2011 
State-Changed-Why:  
Looks like dup of kern/156083 

http://www.freebsd.org/cgi/query-pr.cgi?pr=158063 
State-Changed-From-To: closed->feedback 
State-Changed-By: glebius 
State-Changed-When: Tue Jun 28 09:01:12 UTC 2011 
State-Changed-Why:  
Bug isn't fixed. Set feedback state, since originator is  
asked to upgrade to recent stable/8. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=158063 

From: Gleb Smirnoff <glebius@freebsd.org>
To: Pasha Pigich <pigich@gmail.com>
Cc: bug-followup@freebsd.org
Subject: Re: kern/158063: panic: kmem_malloc(20480): kmem_map too small
Date: Tue, 28 Jun 2011 13:01:01 +0400

 On Tue, Jun 28, 2011 at 11:54:28AM +0300, Pasha Pigich wrote:
 P> After patching and waiting for few days memory usage of dummynet continuing
 P> raising. Need I open new bug report or can   continue here?
 P> I attached img with raising of memory usage
 
 I will re-open the PR.
 
 Can you upgrade to latest stable/8? That would make debugging easier.
 After upgrade, I will send you a patch that tracks memory usage
 of particular dummynet allocations separately, and we will build
 several plots.
 
 -- 
 Totus tuus, Glebius.
State-Changed-From-To: feedback->closed 
State-Changed-By: jh 
State-Changed-When: Fri Feb 24 07:27:54 UTC 2012 
State-Changed-Why:  
Feedback timeout. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=158063 
>Unformatted:
