From jay@redux.oneway.com  Sun Jun 15 09:17:19 2003
Return-Path: <jay@redux.oneway.com>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 8507E37B401
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 15 Jun 2003 09:17:19 -0700 (PDT)
Received: from redux.oneway.com (redux.oneway.com [216.22.46.103])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 9D69543F85
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 15 Jun 2003 09:17:18 -0700 (PDT)
	(envelope-from jay@redux.oneway.com)
Received: from redux.oneway.com (localhost [127.0.0.1])
	by redux.oneway.com (8.12.5/8.12.5) with ESMTP id h5FGH84m028890
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 15 Jun 2003 12:17:08 -0400 (EDT)
Received: (from root@localhost)
	by redux.oneway.com (8.12.5/8.12.5/Submit) id h5FGH7T1028889
	for FreeBSD-gnats-submit@freebsd.org; Sun, 15 Jun 2003 12:17:07 -0400 (EDT)
Message-Id: <200306151558.h5FFwRCK041698@doppel.oneway.com>
Date: Sun, 15 Jun 2003 11:58:27 -0400 (EDT)
From: Jay Kuri <jay@oneway.com>
Reply-To: Jay Kuri <jay@oneway.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: fill up a malloc md-disk on 5.1-R causes panic
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         53350
>Category:       kern
>Synopsis:       fill up a malloc md-disk on 5.1-R causes panic
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jun 15 09:20:03 PDT 2003
>Closed-Date:    Tue Aug 24 12:09:25 GMT 2004
>Last-Modified:  Tue Aug 24 12:09:25 GMT 2004
>Originator:     Jay Kuri
>Release:        FreeBSD 5.1-RELEASE i386
>Organization:
oneway.com
>Environment:
System:         FreeBSD 5.1-RELEASE i386


>Description:
          If you have a 5.1-R system with, say, 256M of ram
and you mdconfig one or several malloc md-devices totalling around 100M,
if you attempt to use them to capacity, the machine will panic.  The
message given is:

 panic: kmem_malloc(4096): kmem_map too small: 108347392 total allocated

I have taken some time to track down how to repeat the problem at will...
so perhaps someone can figure it out.  The problem does not occur if you
have more memory, or at least, you may have to use larger md drives to do it.
I tried it on the exact same configuration, only with 4G
of ram, and there is no panic. I have also tried it on several machines,
with the same results.  I would guess that it has something to do with
caching/buffering of the copied files, but that's just a guess.

I have attached a shell script that will trigger the error.  I've run this
several times, usually in single-user mode, and without fail on a 256M
machine it will panic.  set 'cache_dirs' to any set of directories that
will total 100M or more.  On my specific config I have a default 5.1R
install with a minimal install+catman, the dirs listed in the script total
around 110M.

The script that got me started on this whole thing worked fine on 
5.0-Current (Apr 30), but panic'd the machine as of 5.1R.  (it's a 
different script than the one below, but I removed the non-essential 
parts of the script to make it less confusing)  I should also mention 
that the kernel is identical to 'GENERIC' except that 'options POLLING' 
was included.

Attached is a script that reliably causes the problem. Below are the 
results of running the script on a machine with 4G and on a machine with 256M
RAM.

- 4G log -
# df -k         
Filesystem 1K-blocks   Used  Avail Capacity  Mounted on
/dev/acd0     345120 345120      0   100%    /
devfs              1      1      0   100%    /dev
/dev/da0s1    253888  84416 169472    33%    /opt
# sh -x test.sh
+ PATH=/bin:/sbin:/usr/bin:/usr/sbin
+ cache_dirs=/bin /sbin /usr/bin /usr/sbin /usr/lib
+ sysctl+ awk {print int($2 / 1024) - 100000}
 hw.physmem
+ space_in_k=3928684
+ echo Ram available to us: 3928684
Ram available to us: 3928684
+ totaldirsize=0
+ type=malloc
+ /sbin/mdconfig -a -t malloc -s 1M
+ mddev=md0
+ /sbin/newfs /dev/md0
/dev/md0: 1.0MB (2048 sectors) block size 16384, fragment size 2048
        using 2 cylinder groups of 0.50MB, 32 blks, 64 inodes.
super-block backups (for fsck -b #) at:
 160, 1184
+ /sbin/mount /dev/md0 /mnt
+ mkdir /mnt/copy
+ /bin/cp /bin/cp /mnt/
+ /usr/bin/du+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
 -sk /bin
+ dirsize=12050
+ /bin/expr 0 + 12050
+ totaldirsize=12050
+ [ 12050 -le 3928684 ]
+ echo /bin
+ dirs_mirrored=/bin
+ /sbin/mdconfig -a -t malloc -s 12050k
+ mddevice=md1
+ /sbin/newfs -i 1024 -U /dev/md1
+ /sbin/tunefs -m 2 /dev/md1
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md1 /mnt/copy
+ /mnt/cp -Rp /bin/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/bin
+ /sbin/mount /dev/md1 /mnt/bin
+ /usr/bin/du -sk /sbin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=33913
+ /bin/expr 12050 + 33913
+ totaldirsize=45963
+ [ 45963 -le 3928684 ]
+ echo /bin /sbin
+ dirs_mirrored=/bin /sbin
+ /sbin/mdconfig -a -t malloc -s 33913k
+ mddevice=md2
+ /sbin/newfs -i 1024 -U /dev/md2
+ /sbin/tunefs -m 2 /dev/md2
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md2 /mnt/copy
+ /mnt/cp -Rp /sbin/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/sbin
+ /sbin/mount /dev/md2 /mnt/sbin
+ /usr/bin/du -sk /usr/bin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=34375
+ /bin/expr 45963 + 34375
+ totaldirsize=80338
+ [ 80338 -le 3928684 ]
+ echo /bin /sbin /usr/bin
+ dirs_mirrored=/bin /sbin /usr/bin
+ /sbin/mdconfig -a -t malloc -s 34375k
+ mddevice=md3
+ /sbin/newfs -i 1024 -U /dev/md3
+ /sbin/tunefs -m 2 /dev/md3
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md3 /mnt/copy
+ /mnt/cp -Rp /usr/bin/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/bin
+ /sbin/mount /dev/md3 /mnt/usr/bin
+ /usr/bin/du -sk /usr/sbin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=10420
+ /bin/expr 80338 + 10420
+ totaldirsize=90758
+ [ 90758 -le 3928684 ]
+ echo /bin /sbin /usr/bin /usr/sbin
+ dirs_mirrored=/bin /sbin /usr/bin /usr/sbin
+ /sbin/mdconfig -a -t malloc -s 10420k
+ mddevice=md4
+ /sbin/newfs -i 1024 -U /dev/md4
+ /sbin/tunefs -m 2 /dev/md4
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md4 /mnt/copy
+ /mnt/cp -Rp /usr/sbin/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/sbin
+ /sbin/mount /dev/md4 /mnt/usr/sbin
+ /usr/bin/du -sk /usr/lib
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=49560
+ /bin/expr 90758 + 49560
+ totaldirsize=140318
+ [ 140318 -le 3928684 ]
+ echo /bin /sbin /usr/bin /usr/sbin /usr/lib
+ dirs_mirrored=/bin /sbin /usr/bin /usr/sbin /usr/lib
+ /sbin/mdconfig -a -t malloc -s 49560k
+ mddevice=md5
+ /sbin/newfs -i 1024 -U /dev/md5
+ /sbin/tunefs -m 2 /dev/md5
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md5 /mnt/copy
+ /mnt/cp -Rp /usr/lib/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/lib
+ /sbin/mount /dev/md5 /mnt/usr/lib
# df -k
Filesystem 1K-blocks   Used  Avail Capacity  Mounted on
/dev/acd0     345120 345120      0   100%    /
devfs              1      1      0   100%    /dev
/dev/da0s1    253888  84416 169472    33%    /opt
/dev/md0         846    110    670    14%    /mnt
/dev/md1       10304   9528    570    94%    /mnt/bin
/dev/md2       29414  28644    182    99%    /mnt/sbin
/dev/md3       29812  29088    128   100%    /mnt/usr/bin
/dev/md4        8866   8270    420    95%    /mnt/usr/sbin
/dev/md5       43142  43074   -794   102%    /mnt/usr/lib
# top
last pid:   136;  load averages:  0.01,  0.08,  0.06  up 0+00:05:06    15:40:18
2 processes:   1 running, 1 sleeping

Mem: 56M Active, 17M Inact, 168M Wired, 42M Cache, 69M Buf, 3608M Free
Swap: 


  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
   44 root       8    0   896K   444K wait     0:00  0.00%  0.00% sh
  136 root      96    0  2136K  1300K RUN      0:00  0.00%  0.00% top

- test log 256M -
# top
last pid:    54;  load averages:  0.35,  0.24,  0.10  up 0+00:02:05    16:17:12
2 processes:   1 running, 1 sleeping

Mem: 1184K Active, 268K Inact, 8328K Wired, 3664K Buf, 235M Free
Swap: 


  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
   54 root      96    0  2136K   780K RUN      0:00  0.00%  0.00% top
   48 root       8    0   900K   320K wait     0:00  0.00%  0.00% sh

# df -k  
Filesystem 1K-blocks   Used  Avail Capacity  Mounted on
/dev/acd0     345120 345120      0   100%    /
devfs              1      1      0   100%    /dev
/dev/da0s1    253888  84416 169472    33%    /opt
# sh -x test.sh
+ PATH=/bin:/sbin:/usr/bin:/usr/sbin
+ cache_dirs=/bin /sbin /usr/bin /usr/sbin /usr/lib
+ sysctl+ awk {print int($2 / 1024) - 100000}
 hw.physmem
+ space_in_k=154980
+ echo Ram available to us: 154980
Ram available to us: 154980
+ totaldirsize=0
+ type=malloc
+ /sbin/mdconfig -a -t malloc -s 1M
+ mddev=md0
+ /sbin/newfs /dev/md0
/dev/md0: 1.0MB (2048 sectors) block size 16384, fragment size 2048
        using 2 cylinder groups of 0.50MB, 32 blks, 64 inodes.
super-block backups (for fsck -b #) at:
 160, 1184
+ /sbin/mount /dev/md0 /mnt
+ mkdir /mnt/copy
+ /bin/cp /bin/cp /mnt/
+ /usr/bin/du+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
 -sk /bin
+ dirsize=12050
+ /bin/expr 0 + 12050
+ totaldirsize=12050
+ [ 12050 -le 154980 ]
+ echo /bin
+ dirs_mirrored=/bin
+ /sbin/mdconfig -a -t malloc -s 12050k
+ mddevice=md1
+ /sbin/newfs -i 1024 -U /dev/md1
+ /sbin/tunefs -m 2 /dev/md1
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md1 /mnt/copy
+ /mnt/cp -Rp /bin/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/bin
+ /sbin/mount /dev/md1 /mnt/bin
+ /usr/bin/du -sk /sbin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=33913
+ /bin/expr 12050 + 33913
+ totaldirsize=45963
+ [ 45963 -le 154980 ]
+ echo /bin /sbin
+ dirs_mirrored=/bin /sbin
+ /sbin/mdconfig -a -t malloc -s 33913k
+ mddevice=md2
+ /sbin/newfs -i 1024 -U /dev/md2
+ /sbin/tunefs -m 2 /dev/md2
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md2 /mnt/copy
+ /mnt/cp -Rp /sbin/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/sbin
+ /sbin/mount /dev/md2 /mnt/sbin
+ /usr/bin/du -sk /usr/bin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=34375
+ /bin/expr 45963 + 34375
+ totaldirsize=80338
+ [ 80338 -le 154980 ]
+ echo /bin /sbin /usr/bin
+ dirs_mirrored=/bin /sbin /usr/bin
+ /sbin/mdconfig -a -t malloc -s 34375k
+ mddevice=md3
+ /sbin/newfs -i 1024 -U /dev/md3
+ /sbin/tunefs -m 2 /dev/md3
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md3 /mnt/copy
+ /mnt/cp -Rp /usr/bin/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/bin
+ /sbin/mount /dev/md3 /mnt/usr/bin
+ /usr/bin/du -sk /usr/sbin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=10420
+ /bin/expr 80338 + 10420
+ totaldirsize=90758
+ [ 90758 -le 154980 ]
+ echo /bin /sbin /usr/bin /usr/sbin
+ dirs_mirrored=/bin /sbin /usr/bin /usr/sbin
+ /sbin/mdconfig -a -t malloc -s 10420k
+ mddevice=md4
+ /sbin/newfs -i 1024 -U /dev/md4
+ /sbin/tunefs -m 2 /dev/md4
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md4 /mnt/copy
+ /mnt/cp -Rp /usr/sbin/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/sbin
+ /sbin/mount /dev/md4 /mnt/usr/sbin
+ /usr/bin/du -sk /usr/lib
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=49560
+ /bin/expr 90758 + 49560
+ totaldirsize=140318
+ [ 140318 -le 154980 ]
+ echo /bin /sbin /usr/bin /usr/sbin /usr/lib
+ dirs_mirrored=/bin /sbin /usr/bin /usr/sbin /usr/lib
+ /sbin/mdconfig -a -t malloc -s 49560k
+ mddevice=md5
+ /sbin/newfs -i 1024 -U /dev/md5
+ /sbin/tunefs -m 2 /dev/md5
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md5 /mnt/copy
+ /mnt/cp -Rp /usr/lib/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
panic: kmem_malloc(4096): kmem_map too small: 108347392 total allocated

syncing disks, buffers remaining... 

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x14
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xc02d1ad9
stack pointer           = 0x10:0xd24fb544
frame pointer           = 0x10:0xd24fb578
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 137 (cp)
trap number             = 12
panic: page fault
Uptime: 3m12s
Terminate ACPI
Automatic reboot in 15 seconds - press a key on the console to abort



>How-To-Repeat:

On a 5.1-R machine with 256M of ram, boot single-user and run this script
(ensure that the size of the dirs in cache_dirs is 100M or more) 

#!/bin/sh

## quick script to show the mdconfig -t malloc + fill = panic bug on 5.1-R
PATH=/bin:/sbin:/usr/bin:/usr/sbin

# on a machine with 256M of ram, this is enough to crash it. 112M roughly on
# my machine.  Use any combination of directories that will push the limit
# up over 100M 
cache_dirs="/bin /sbin /usr/bin /usr/sbin /usr/lib"

# say we have 100M less than we actually have, to make sure we don't
# try to use more memory than we have 
space_in_k=`sysctl hw.physmem|awk '{print int($2 / 1024) - 100000}'`

echo Ram available to us: $space_in_k
totaldirsize=0

type="malloc"
## This gives us a writable directory so that we can run this off of the 5.1 
## install or live cd.  
mddev=`/sbin/mdconfig -a -t malloc -s 1M`
/sbin/newfs /dev/${mddev}
/sbin/mount /dev/${mddev} /mnt
mkdir /mnt/copy
/bin/cp /bin/cp /mnt/

for dir in ${cache_dirs}
do
      dirsize=`/usr/bin/du -sk ${dir}|/usr/bin/awk '{print int(1024 + ($1 * 1.20
))}'`
      totaldirsize=`/bin/expr ${totaldirsize} + ${dirsize}`
      if [ ${totaldirsize} -le ${space_in_k} ]; then
           dirs_mirrored=`echo ${dirs_mirrored} $dir`
           mddevice=`/sbin/mdconfig -a -t ${type} -s ${dirsize}k`
           /sbin/newfs -i 1024 -U /dev/${mddevice} >/dev/null
           /sbin/tunefs -m 2 /dev/${mddevice}
           /sbin/mount /dev/${mddevice} /mnt/copy >/dev/null
           /mnt/cp -Rp ${dir}/ /mnt/copy
           /sbin/umount /mnt/copy >/dev/null
           /bin/mkdir -p /mnt${dir}
           /sbin/mount /dev/${mddevice} /mnt${dir} >/dev/null
      fi
done

        
>Fix:

        None that I know of.
>Release-Note:
>Audit-Trail:

From: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
To: Jay Kuri <jay@oneway.com>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic 
Date: Sun, 15 Jun 2003 18:24:17 +0200

 >>Description:
 >          If you have a 5.1-R system with, say, 256M of ram
 >and you mdconfig one or several malloc md-devices totalling around 100M,
 >if you attempt to use them to capacity, the machine will panic.  The
 >message given is:
 >
 > panic: kmem_malloc(4096): kmem_map too small: 108347392 total allocated
 
 It's really very simple:  You ran out of kernel mapped memory ("KVM").
 
 Remedy: Don't Do That.
 
 See also: malloc(9), md(4)
 
 This PR can be closed.
 
 -- 
 Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 phk@FreeBSD.ORG         | TCP/IP since RFC 956
 FreeBSD committer       | BSD since 4.3-tahoe    
 Never attribute to malice what can adequately be explained by incompetence.

From: Jason Kuri <jay@oneway.com>
To: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic 
Date: Sun, 15 Jun 2003 11:32:41 -0500

 Hi there,
 
 That makes sense... but hold on a sec.
 
 I have 256M of ram, and try to use 110M for malloc-disk, and my machine 
 panics?  That seems awfully broken to me.  This is single user mode, 
 nothing else is running.  Why does 110M of MD-disk cause a 256M machine 
 to run out of memory?   I should have 140M or so left for the system to 
 play with...   If nothing else,  shouldn't it at least fail and not 
 panic the system?
 
 This worked fine 1 month ago. (5.0-Current from APR-30)
 
 Jay
 
 
 
 
 On Sunday, June 15, 2003, at 11:24  AM, Poul-Henning Kamp wrote:
 
 >>> Description:
 >>          If you have a 5.1-R system with, say, 256M of ram
 >> and you mdconfig one or several malloc md-devices totalling around 
 >> 100M,
 >> if you attempt to use them to capacity, the machine will panic.  The
 >> message given is:
 >>
 >> panic: kmem_malloc(4096): kmem_map too small: 108347392 total 
 >> allocated
 >
 > It's really very simple:  You ran out of kernel mapped memory ("KVM").
 >
 > Remedy: Don't Do That.
 >
 > See also: malloc(9), md(4)
 >
 > This PR can be closed.
 >
 > -- 
 > Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 > phk@FreeBSD.ORG         | TCP/IP since RFC 956
 > FreeBSD committer       | BSD since 4.3-tahoe
 > Never attribute to malice what can adequately be explained by 
 > incompetence.
 >
 

From: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
To: Jason Kuri <jay@oneway.com>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic 
Date: Sun, 15 Jun 2003 18:35:05 +0200

 In message <FC408487-9F4E-11D7-B102-00039398BAE0@oneway.com>, Jason Kuri writes
 :
 >Hi there,
 >
 >That makes sense... but hold on a sec.
 >
 >I have 256M of ram, and try to use 110M for malloc-disk, and my machine 
 >panics?  That seems awfully broken to me. 
 
 Notice I said "Kernel mapped memory".
 
 That's usually on the order of 1/3 or your RAM.
 
 -- 
 Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 phk@FreeBSD.ORG         | TCP/IP since RFC 956
 FreeBSD committer       | BSD since 4.3-tahoe    
 Never attribute to malice what can adequately be explained by incompetence.

From: Jason Kuri <jay@oneway.com>
To: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic 
Date: Sun, 15 Jun 2003 11:51:39 -0500

 Ok, so speaking practically, how do I deal with this issue?  I have a 
 script that needs to be able to cache directories in ram...  As I said, 
 I was able to do this as recently as April 30.  Is there some sort of 
 tunable parameter I can use to increase kernel's malloc bucket limit?  
 (kernel option or ..?  )  Or, at absolute least, some way to determine 
 what it is, so that I can force the script not to use more than that?   
 and how much must be free leftover to allow the system to run 
 'normally' ?
 
 Shouldn't mdconfig not allow me to reserve more space than it can 
 provide me with?  I'm not '-o force'ing the mdconfig.  It seems to me 
 that if the request is for too much, the mdconfig should fail.
 
 and still if it is too much, shouldn't it deny the allocation rather 
 than panic?   It seems to me like the amount that the system can 
 provide changes between the time I mdconfig the disk, and the time I 
 try to use it.
 
 Thanks for your assistance,
 
 Jay
 
 On Sunday, June 15, 2003, at 11:35  AM, Poul-Henning Kamp wrote:
 
 > In message <FC408487-9F4E-11D7-B102-00039398BAE0@oneway.com>, Jason 
 > Kuri writes
 > :
 >> Hi there,
 >>
 >> That makes sense... but hold on a sec.
 >>
 >> I have 256M of ram, and try to use 110M for malloc-disk, and my 
 >> machine
 >> panics?  That seems awfully broken to me.
 >
 > Notice I said "Kernel mapped memory".
 >
 > That's usually on the order of 1/3 or your RAM.
 >
 > -- 
 > Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 > phk@FreeBSD.ORG         | TCP/IP since RFC 956
 > FreeBSD committer       | BSD since 4.3-tahoe
 > Never attribute to malice what can adequately be explained by 
 > incompetence.
 >
 

From: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
To: Jason Kuri <jay@oneway.com>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic 
Date: Sun, 15 Jun 2003 18:56:28 +0200

 In message <A228C296-9F51-11D7-B102-00039398BAE0@oneway.com>, Jason Kuri writes
 :
 >Ok, so speaking practically, how do I deal with this issue?  I have a 
 >script that needs to be able to cache directories in ram... 
 
 You add more RAM or change the fraction of RAM assigned to KVM.
 I can't remember if this is documented somewhere.
 
 mdconfig(8) would have a hard time being able to tell when to reject
 a request, and for testing purposes, it is a very convenient thing
 to be able to create 1TB devices (and this works, as long as you 
 don't fill it with more data than you have KVM to).
 
 -- 
 Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 phk@FreeBSD.ORG         | TCP/IP since RFC 956
 FreeBSD committer       | BSD since 4.3-tahoe    
 Never attribute to malice what can adequately be explained by incompetence.

From: Jason Kuri <jay@oneway.com>
To: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic 
Date: Sun, 15 Jun 2003 12:35:02 -0500

 Adding ram is not an option, as we need to be able to run this on 
 different machines with different configs.
 
 Is there a way to determine at run-time how much kernel mapped memory 
 is available for use?  I have determined that adding -o reserve will 
 cause the initial mdconfig to fail if the space is not available...  Is 
 there any documentation anywhere on the issues related to kernel mapped 
 memory and allocation?
 
 Is this tunable at run time, and if not, can you point me to some 
 information on how to properly use VM_KMEM_SIZE_SCALE (which I'm 
 assuming is the right option) ?
 
 And one final question, am I right in thinking that increasing this 
 will make less memory available for user processes even if the space is 
 unused by the kernel?
 
 Thanks,
 
 Jay
 
 On Sunday, June 15, 2003, at 11:56  AM, Poul-Henning Kamp wrote:
 
 > In message <A228C296-9F51-11D7-B102-00039398BAE0@oneway.com>, Jason 
 > Kuri writes
 > :
 >> Ok, so speaking practically, how do I deal with this issue?  I have a
 >> script that needs to be able to cache directories in ram...
 >
 > You add more RAM or change the fraction of RAM assigned to KVM.
 > I can't remember if this is documented somewhere.
 >
 > mdconfig(8) would have a hard time being able to tell when to reject
 > a request, and for testing purposes, it is a very convenient thing
 > to be able to create 1TB devices (and this works, as long as you
 > don't fill it with more data than you have KVM to).
 >
 > -- 
 > Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 > phk@FreeBSD.ORG         | TCP/IP since RFC 956
 > FreeBSD committer       | BSD since 4.3-tahoe
 > Never attribute to malice what can adequately be explained by 
 > incompetence.
 >
 

From: "Brian K. White" <brian@aljex.com>
To: <freebsd-gnats-submit@FreeBSD.org>, <jay@oneway.com>
Cc:  
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic
Date: Thu, 13 Nov 2003 18:30:24 -0500

 Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic
 
 I get the same thing on a 1G machine when trying to use a 768 M ramdisk when
 the ramdisk is only 25% full.
 It never fails at exactly the same place, but always within a minute or two
 and it never gets anywhere close to filling up the md-disk.
 
 In my case I was creating the 768 meg malloc disc and trying to copy
 /usr/src into it.
 initially I tried a couple times with midnight commander, and after that
 crashed twice, I tried just using tar
 
 mdconfig -a -t malloc -s 768m
 newfs -U /dev/md0
 mount /dev/md0 /md0
 cd /usr/src ;tar cvf - * |(cd /md0 ;tar xvf - )
 
 /usr/src is only about 350 megs so it should fit easy, but just in case some
 difference in block sizes cause the same data to take up mor room on md0
 than it does on the hard drive, I watched df in another console while it was
 copying.
 
 repeatedly looking at df in another consle showed the disk only reaching
 about 25% full at the time of kernel panic.
 
 I got some different messages each time
 one time I just got this line repeated scrolling up the console untill I
 decided to power off:
   initiate_write_filepage: already started
 
 the last time I got this:
   panic: kmem_malloc(4096) kmem_map too small: 275251200 total allocated
   syncing disks, buffers remaining... 7095
   panic: initiate_write_inodeblock_ufs2: already started
 
 This is the 5.1 that is on the 5.1 iso as of about Nov/03/2003
 
 Brian K. White  --  brian@aljex.com  --  http://www.aljex.com/bkw/
 +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
 filePro BBx  Linux SCO  Prosper/FACTS AutoCAD  #callahans Satriani
 

From: Jason Kuri <jay@oneway.com>
To: "Brian K. White" <brian@aljex.com>
Cc: <freebsd-gnats-submit@FreeBSD.org>
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic
Date: Sat, 15 Nov 2003 17:24:31 -0600

 Hello Brian,
 
 This is not a bug, per se.   You see, though you have 1G of ram in the
 machine, the system only reserves a certain amount for kernel memory
 (max 200M by default).  MD-disks are entirely stored in kernel memory
 and they do not auto-allocate, so you can create a 768M MD disk, but
 the memory in the kernel is not actually allocated for it until you try
 to use it.
 
 The end result of all this is that you can create a large MD disk, and
 it will work... until you try to fill it up... at which point it will
 panic the kernel because the kernel has run out of memory.
 
 The way to test this is to add the -o reserve option to the mdconfig
 command.  It will fail/panic immediately instead of waiting until you
 fill it up.  That option requires the memory to be pre-allocated rather
 than allocated as needed.
 
 The way to fix it is to change two options in your kernel config:
 
 options VM_KMEM_SIZE_MAX    "(800*1024*1024)"
 options VM_KMEM_SIZE_SCALE "2"
 
 These will increase the amount of space allocated to the kernel...
 VM_KMEM_SIZE_SCALE is a scaling factor.... meaning that the kernel will
 get physical-memory/VM_KMEM_SIZE_SCALE up to VM_KMEM_SIZE_MAX... which
 in the above case is 800M.   The above settings on a 1G machine would
 result in you having 512M for the kernel... Please note, however, that
 this memory is what is allocated for EVERYTHING in the kernel... so if
 you fill the memory with an MD-disk of 512M, you would probably crash
 again.
 
 I don't know what will happen if you set VM_KMEM_SIZE_SCALE = 1... I've
 only ever used about 400M of MD-disk on a 2G machine... in which case
 VM_KMEM_SIZE_SCALE=2 is still ok.  You'll have to try it out and see...
   let me know how it goes.
 
 Please read this:
 
 http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2003-06/
 1599.html
 
 to understand how it works a bit more.
 
 And good luck.
 
 Jay
 
 
 
 On Nov 13, 2003, at 5:30 PM, Brian K. White wrote:
 
 > Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic
 >
 > I get the same thing on a 1G machine when trying to use a 768 M
 > ramdisk when
 > the ramdisk is only 25% full.
 > It never fails at exactly the same place, but always within a minute
 > or two
 > and it never gets anywhere close to filling up the md-disk.
 >
 > In my case I was creating the 768 meg malloc disc and trying to copy
 > /usr/src into it.
 > initially I tried a couple times with midnight commander, and after
 > that
 > crashed twice, I tried just using tar
 >
 > mdconfig -a -t malloc -s 768m
 > newfs -U /dev/md0
 > mount /dev/md0 /md0
 > cd /usr/src ;tar cvf - * |(cd /md0 ;tar xvf - )
 >
 > /usr/src is only about 350 megs so it should fit easy, but just in
 > case some
 > difference in block sizes cause the same data to take up mor room on
 > md0
 > than it does on the hard drive, I watched df in another console while
 > it was
 > copying.
 >
 > repeatedly looking at df in another consle showed the disk only
 > reaching
 > about 25% full at the time of kernel panic.
 >
 > I got some different messages each time
 > one time I just got this line repeated scrolling up the console untill
 > I
 > decided to power off:
 >   initiate_write_filepage: already started
 >
 > the last time I got this:
 >   panic: kmem_malloc(4096) kmem_map too small: 275251200 total
 > allocated
 >   syncing disks, buffers remaining... 7095
 >   panic: initiate_write_inodeblock_ufs2: already started
 >
 > This is the 5.1 that is on the 5.1 iso as of about Nov/03/2003
 >
 > Brian K. White  --  brian@aljex.com  --  http://www.aljex.com/bkw/
 > +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
 > filePro BBx  Linux SCO  Prosper/FACTS AutoCAD  #callahans Satriani
 

From: "Brian K. White" <brian@aljex.com>
To: "Jason Kuri" <jay@oneway.com>
Cc: <freebsd-gnats-submit@FreeBSD.org>
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic
Date: Sun, 16 Nov 2003 08:52:14 -0500

 ----- Original Message ----- 
 From: "Jason Kuri" <jay@oneway.com>
 To: "Brian K. White" <brian@aljex.com>
 Cc: <freebsd-gnats-submit@FreeBSD.org>
 Sent: Saturday, November 15, 2003 6:24 PM
 Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic
 
 
 > Hello Brian,
 > 
 > This is not a bug, per se.   You see, though you have 1G of ram in the
 > machine, the system only reserves a certain amount for kernel memory
 > (max 200M by default).  MD-disks are entirely stored in kernel memory
 > and they do not auto-allocate, so you can create a 768M MD disk, but
 > the memory in the kernel is not actually allocated for it until you try
 > to use it.
 > 
 > The end result of all this is that you can create a large MD disk, and
 > it will work... until you try to fill it up... at which point it will
 > panic the kernel because the kernel has run out of memory.
 > 
 > The way to test this is to add the -o reserve option to the mdconfig
 > command.  It will fail/panic immediately instead of waiting until you
 > fill it up.  That option requires the memory to be pre-allocated rather
 > than allocated as needed.
 > 
 > The way to fix it is to change two options in your kernel config:
 > 
 > options VM_KMEM_SIZE_MAX    "(800*1024*1024)"
 > options VM_KMEM_SIZE_SCALE "2"
 > 
 > These will increase the amount of space allocated to the kernel...
 > VM_KMEM_SIZE_SCALE is a scaling factor.... meaning that the kernel will
 > get physical-memory/VM_KMEM_SIZE_SCALE up to VM_KMEM_SIZE_MAX... which
 > in the above case is 800M.   The above settings on a 1G machine would
 > result in you having 512M for the kernel... Please note, however, that
 > this memory is what is allocated for EVERYTHING in the kernel... so if
 > you fill the memory with an MD-disk of 512M, you would probably crash
 > again.
 > 
 > I don't know what will happen if you set VM_KMEM_SIZE_SCALE = 1... I've
 > only ever used about 400M of MD-disk on a 2G machine... in which case
 > VM_KMEM_SIZE_SCALE=2 is still ok.  You'll have to try it out and see...
 >   let me know how it goes.
 > 
 > Please read this:
 > 
 > http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2003-06/
 > 1599.html
 > 
 > to understand how it works a bit more.
 > 
 > And good luck.
 > 
 > Jay
 > 
 > 
 > 
 > On Nov 13, 2003, at 5:30 PM, Brian K. White wrote:
 > 
 > > Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic
 > >
 > > I get the same thing on a 1G machine when trying to use a 768 M
 > > ramdisk when
 > > the ramdisk is only 25% full.
 > > It never fails at exactly the same place, but always within a minute
 > > or two
 > > and it never gets anywhere close to filling up the md-disk.
 > >
 > > In my case I was creating the 768 meg malloc disc and trying to copy
 > > /usr/src into it.
 > > initially I tried a couple times with midnight commander, and after
 > > that
 > > crashed twice, I tried just using tar
 > >
 > > mdconfig -a -t malloc -s 768m
 > > newfs -U /dev/md0
 > > mount /dev/md0 /md0
 > > cd /usr/src ;tar cvf - * |(cd /md0 ;tar xvf - )
 > >
 > > /usr/src is only about 350 megs so it should fit easy, but just in
 > > case some
 > > difference in block sizes cause the same data to take up mor room on
 > > md0
 > > than it does on the hard drive, I watched df in another console while
 > > it was
 > > copying.
 > >
 > > repeatedly looking at df in another consle showed the disk only
 > > reaching
 > > about 25% full at the time of kernel panic.
 > >
 > > I got some different messages each time
 > > one time I just got this line repeated scrolling up the console untill
 > > I
 > > decided to power off:
 > >   initiate_write_filepage: already started
 > >
 > > the last time I got this:
 > >   panic: kmem_malloc(4096) kmem_map too small: 275251200 total
 > > allocated
 > >   syncing disks, buffers remaining... 7095
 > >   panic: initiate_write_inodeblock_ufs2: already started
 > >
 > > This is the 5.1 that is on the 5.1 iso as of about Nov/03/2003
 
 Thank you very much.
 I appreciate the help.
 
 Brian K. White  --  brian@aljex.com  --  http://www.aljex.com/bkw/
 +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
 filePro BBx  Linux SCO  Prosper/FACTS AutoCAD  #callahans Satriani
 

From: Jamie Jones <jamie@bishopston.net>
To: freebsd-gnats-submit@FreeBSD.org, jay@oneway.com
Cc:  
Subject: Re: kern/53350: fill up a malloc md-disk on 5.1-R causes panic
Date: Tue, 2 Dec 2003 02:00:42 +0000

 If this isn't a bug, and we can no longer allocate largeish swap-backed memory 
 disks, can we have MFS back please ?
State-Changed-From-To: open->closed 
State-Changed-By: arved 
State-Changed-When: Tue Aug 24 12:08:50 GMT 2004 
State-Changed-Why:  
Solution (increasing kmem) was mentioned in the PR. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=53350 
>Unformatted:
