From nobody@FreeBSD.org  Wed Dec  3 18:16:55 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 98CBA106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Wed,  3 Dec 2008 18:16:55 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 8830D8FC13
	for <freebsd-gnats-submit@FreeBSD.org>; Wed,  3 Dec 2008 18:16:55 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id mB3IGt7J021095
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 3 Dec 2008 18:16:55 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id mB3IGssd021094;
	Wed, 3 Dec 2008 18:16:54 GMT
	(envelope-from nobody)
Message-Id: <200812031816.mB3IGssd021094@www.freebsd.org>
Date: Wed, 3 Dec 2008 18:16:54 GMT
From: Weldon Godfrey <wgodfrey@ena.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [kern] cannot set vm.kmem_size_max beyond 3GB
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         129396
>Category:       kern
>Synopsis:       [kernel] cannot set vm.kmem_size_max beyond 3GB
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    alc
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Dec 03 18:20:01 UTC 2008
>Closed-Date:    Sat Feb 07 09:17:58 UTC 2009
>Last-Modified:  Wed Feb 11 18:10:05 UTC 2009
>Originator:     Weldon Godfrey
>Release:        FreeBSD 7.1-PRERELEASE amd64
>Organization:
ENA
>Environment:
FreeBSD store1.mail.ena.net 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #2: Thu Nov 20 10:41:36 CST 2008     root@store1.mail.ena.net:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
I noticed during my testing, while trying to copy two 2TB files at the same time (I am using ZFS), when the files get to a certain size, the system panics.  It didn't dump core or log a reason in syslog, however, one time it managed not to reboot (just hang after panic) and I was able to capture that it was complaining that kmem was too small.

I thought that was odd, since I have set the max to 15GB (the system has 16GB of RAM), so I checked sysctl and vm.kmem_max was only at 3GB.  This is in my loader.conf:
vm.kmem_size_max="16106127360"
vm.kmem_size="1073741824"
checking right after boot, the vm.kmem_size was set, it was at 1073741824

here is sysctl right after reboot with the above loader.conf:
store1# sysctl -a | grep kmem
vm.kmem_size_scale: 3
vm.kmem_size_max: 3221225472
vm.kmem_size_min: 0
vm.kmem_size: 1073741824
store1# sysctl -a | grep kern.maxvn
kern.maxvnodes: 100000
store1#



As a work-a-round, I tried to set these options in the kernel config and recompile the kernel, it won't compile, this is the error:

cc -c -O2 -frename-registers -pipe -fno-strict-aliasing  -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc  -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000  -mcmodel=kernel
-mno-red-zone-mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Werror /usr/src/sys/kern/kern_malloc.c
cc1: warnings being treated as errors
/usr/src/sys/kern/kern_malloc.c: In function 'kmeminit':
/usr/src/sys/kern/kern_malloc.c:598: warning: large integer implicitly truncated to unsigned type
*** Error code 1


>How-To-Repeat:
try to set vm.kmem_size_max or OPTIONS VM_KMEM_SIZE_MAX beyond 3221225472
>Fix:
I am guessing these tunables need a larger cast?

u_int vm_kmem_size;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size, CTLFLAG_RD, &vm_kmem_size, 0,
    "Size of kernel memory");
 
u_int vm_kmem_size_min;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_min, CTLFLAG_RD, &vm_kmem_size_min, 0,  
    "Minimum size of kernel memory");
 
u_int vm_kmem_size_max;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_max, CTLFLAG_RD, &vm_kmem_size_max, 0,
    "Maximum size of kernel memory");
   
u_int vm_kmem_size_scale;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_scale, CTLFLAG_RD, &vm_kmem_size_scale, 0,
    "Scale factor for kernel memory size");

>Release-Note:
>Audit-Trail:

From: Bruce Cran <bruce@cran.org.uk>
To: Weldon Godfrey <wgodfrey@ena.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
Date: Wed, 3 Dec 2008 12:06:37 -0800

 It looks like this might be the same problem that was fixed in -CURRENT
 a few months ago - see http://svn.freebsd.org/viewvc/base?view=revision&revision=180308
 
 -- 
 Bruce

From: "Weldon Godfrey" <wgodfrey@ena.com>
To: "Bruce Cran" <bruce@cran.org.uk>
Cc: <freebsd-gnats-submit@FreeBSD.org>
Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
Date: Wed, 3 Dec 2008 14:25:26 -0600

 Ah, sorry, my search for kmem in gnats didn't pull that up.  It does
 sound like the same issue.
 
 Weldon
 
 -----Original Message-----
 From: Bruce Cran [mailto:bruce@cran.org.uk]=20
 Sent: Wednesday, December 03, 2008 2:07 PM
 To: Weldon Godfrey
 Cc: freebsd-gnats-submit@FreeBSD.org
 Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
 
 It looks like this might be the same problem that was fixed in -CURRENT
 a few months ago - see
 http://svn.freebsd.org/viewvc/base?view=3Drevision&revision=3D180308
 
 --=20
 Bruce
Responsible-Changed-From-To: freebsd-bugs->alc 
Responsible-Changed-By: remko 
Responsible-Changed-When: Thu Dec 4 08:01:31 UTC 2008 
Responsible-Changed-Why:  
Potential MFC candidate, if that is not possible, please close this PR 

http://www.freebsd.org/cgi/query-pr.cgi?pr=129396 

From: "Weldon Godfrey" <wgodfrey@ena.com>
To: "Bruce Cran" <bruce@cran.org.uk>,
	<alc@freebsd.org>
Cc: <freebsd-gnats-submit@FreeBSD.org>
Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
Date: Fri, 5 Dec 2008 08:03:11 -0600

 In the two files that was mentioned in the fix for -CURRENT, I applied
 them to my 7.1-PRERELEASE and recompiled/installed the kernel.  I was
 able to set vm.kmem_size_max beyond 4GB.  I ran into a kem shortage
 panic, but this time I paid attention that the size it said it was at
 the time of the panic was just under my setting for vm.kmem_size (1GB).
 I noticed that it had been panicing saying near 1GB size when
 vm.kmem_size_max was shown to be at 3GB or 16GB by sysctl -a.
 
 So, I tried setting kmem_size to 10G, that caused the known too big for
 boot-up panic, so did 2G.  I was able to boot set at 1.5GB.  It stayed
 up a longer, I started my rm -r that starts the panic and it did, but
 this time it said the size was 1.5GB.
 
 From this, I am guessing:
 
 1.  There may be more than just the changes in sys/kern/kern_malloc.c
 and=20
 sys/vm/vm_kern.h
 
 2.  It appears that vm.kmem_size_max isn't being used or there is a
 false alarm panic when vm.kmem_size is exceeded?
 
 
 Weldon
 
 -----Original Message-----
 From: Bruce Cran [mailto:bruce@cran.org.uk]=20
 Sent: Wednesday, December 03, 2008 3:27 PM
 To: alc@freebsd.org
 Cc: Weldon Godfrey
 Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
 
 On Wed, 3 Dec 2008 20:40:05 GMT
 "Weldon Godfrey" <wgodfrey@ena.com> wrote:
 
 > The following reply was made to PR kern/129396; it has been noted by
 > GNATS.
 >=20
 > From: "Weldon Godfrey" <wgodfrey@ena.com>
 > To: "Bruce Cran" <bruce@cran.org.uk>
 > Cc: <freebsd-gnats-submit@FreeBSD.org>
 > Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond
 > 3GB Date: Wed, 3 Dec 2008 14:25:26 -0600
 >=20
 >  Ah, sorry, my search for kmem in gnats didn't pull that up.  It does
 >  sound like the same issue.
 > =20
 >  Weldon
 > =20
 >  -----Original Message-----
 >  From: Bruce Cran [mailto:bruce@cran.org.uk]=3D20
 >  Sent: Wednesday, December 03, 2008 2:07 PM
 >  To: Weldon Godfrey
 >  Cc: freebsd-gnats-submit@FreeBSD.org
 >  Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond
 > 3GB=20
 >  It looks like this might be the same problem that was fixed in
 > -CURRENT a few months ago - see
 >  =
 http://svn.freebsd.org/viewvc/base?view=3D3Drevision&revision=3D3D180308
 > =20
 >  --=3D20
 >  Bruce
 
 Hi alc,
 
 Is there any chance the fix to support a kmem_map greater than 4G could
 be merged back to -STABLE?  It would be nice to have in 7.x at least.
 Or is it one of those changes that can't be made because it would
 change the ABI?
 
 --=20
 Bruce
 brucec@freebsd.org

From: "Weldon Godfrey" <wgodfrey@ena.com>
To: "Weldon Godfrey" <wgodfrey@ena.com>,
	"Bruce Cran" <bruce@cran.org.uk>,
	<alc@freebsd.org>
Cc: <freebsd-gnats-submit@FreeBSD.org>
Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
Date: Thu, 11 Dec 2008 10:41:55 -0600

 I went ahead and upgraded to 8 (HEAD).  I get the panic again (I have
 kmem_size set to 1GB, kmem_size_max set to 15GB, and it panics saying
 about 1GB is allocated.  Is this because kmem_map isn't growing on its
 own or is there a false alarm panic caused by a check someplace?
 
 -----Original Message-----
 From: Weldon Godfrey=20
 Sent: Friday, December 05, 2008 8:03 AM
 To: 'Bruce Cran'; 'alc@freebsd.org'
 Cc: 'freebsd-gnats-submit@FreeBSD.org'
 Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
 
 
 In the two files that was mentioned in the fix for -CURRENT, I applied
 them to my 7.1-PRERELEASE and recompiled/installed the kernel.  I was
 able to set vm.kmem_size_max beyond 4GB.  I ran into a kem shortage
 panic, but this time I paid attention that the size it said it was at
 the time of the panic was just under my setting for vm.kmem_size (1GB).
 I noticed that it had been panicing saying near 1GB size when
 vm.kmem_size_max was shown to be at 3GB or 16GB by sysctl -a.
 
 So, I tried setting kmem_size to 10G, that caused the known too big for
 boot-up panic, so did 2G.  I was able to boot set at 1.5GB.  It stayed
 up a longer, I started my rm -r that starts the panic and it did, but
 this time it said the size was 1.5GB.
 
 From this, I am guessing:
 
 1.  There may be more than just the changes in sys/kern/kern_malloc.c
 and=20
 sys/vm/vm_kern.h
 
 2.  It appears that vm.kmem_size_max isn't being used or there is a
 false alarm panic when vm.kmem_size is exceeded?
 
 
 Weldon
 
 -----Original Message-----
 From: Bruce Cran [mailto:bruce@cran.org.uk]=20
 Sent: Wednesday, December 03, 2008 3:27 PM
 To: alc@freebsd.org
 Cc: Weldon Godfrey
 Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
 
 On Wed, 3 Dec 2008 20:40:05 GMT
 "Weldon Godfrey" <wgodfrey@ena.com> wrote:
 
 > The following reply was made to PR kern/129396; it has been noted by
 > GNATS.
 >=20
 > From: "Weldon Godfrey" <wgodfrey@ena.com>
 > To: "Bruce Cran" <bruce@cran.org.uk>
 > Cc: <freebsd-gnats-submit@FreeBSD.org>
 > Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond
 > 3GB Date: Wed, 3 Dec 2008 14:25:26 -0600
 >=20
 >  Ah, sorry, my search for kmem in gnats didn't pull that up.  It does
 >  sound like the same issue.
 > =20
 >  Weldon
 > =20
 >  -----Original Message-----
 >  From: Bruce Cran [mailto:bruce@cran.org.uk]=3D20
 >  Sent: Wednesday, December 03, 2008 2:07 PM
 >  To: Weldon Godfrey
 >  Cc: freebsd-gnats-submit@FreeBSD.org
 >  Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond
 > 3GB=20
 >  It looks like this might be the same problem that was fixed in
 > -CURRENT a few months ago - see
 >  =
 http://svn.freebsd.org/viewvc/base?view=3D3Drevision&revision=3D3D180308
 > =20
 >  --=3D20
 >  Bruce
 
 Hi alc,
 
 Is there any chance the fix to support a kmem_map greater than 4G could
 be merged back to -STABLE?  It would be nice to have in 7.x at least.
 Or is it one of those changes that can't be made because it would
 change the ABI?
 
 --=20
 Bruce
 brucec@freebsd.org

From: "Weldon Godfrey" <wgodfrey@ena.com>
To: "Weldon Godfrey" <wgodfrey@ena.com>,
	"Bruce Cran" <bruce@cran.org.uk>,
	<alc@freebsd.org>
Cc: <freebsd-gnats-submit@FreeBSD.org>
Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
Date: Thu, 11 Dec 2008 11:01:17 -0600

 I am an ultra-noobie on this, so please forgive me if this is a stupid
 question.  Shouldn't this routine in vm/vm_kern.c have some logic to
 malloc more kmem if under kmem_size_max before panic?
 
         if (vm_map_findspace(map, vm_map_min(map), size, &addr)) {
 
                 vm_map_unlock(map);
                 if ((flags & M_NOWAIT) =3D=3D 0) {
                         for (i =3D 0; i < 8; i++) {
                                 EVENTHANDLER_INVOKE(vm_lowmem, 0);
                                 uma_reclaim();
                                 vm_map_lock(map);
                                 if (vm_map_findspace(map,
 vm_map_min(map),
                                     size, &addr) =3D=3D 0) {
                                         break;
                                 }
                                 vm_map_unlock(map);
                                 tsleep(&i, 0, "nokva", (hz / 4) * (i +
 1));
                         }
                         if (i =3D=3D 8) {
                                 panic("kmem_malloc(%ld): kmem_map too
 small: %ld total allocated",
                                     (long)size, (long)map->size);
                         }
                 } else {
                         return (0);
                 }
         }
 
 Weldon
 
 -----Original Message-----
 From: Weldon Godfrey=20
 Sent: Thursday, December 11, 2008 10:42 AM
 To: Weldon Godfrey; 'Bruce Cran'; 'alc@freebsd.org'
 Cc: 'freebsd-gnats-submit@FreeBSD.org'
 Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
 
 
 I went ahead and upgraded to 8 (HEAD).  I get the panic again (I have
 kmem_size set to 1GB, kmem_size_max set to 15GB, and it panics saying
 about 1GB is allocated.  Is this because kmem_map isn't growing on its
 own or is there a false alarm panic caused by a check someplace?
 
 -----Original Message-----
 From: Weldon Godfrey=20
 Sent: Friday, December 05, 2008 8:03 AM
 To: 'Bruce Cran'; 'alc@freebsd.org'
 Cc: 'freebsd-gnats-submit@FreeBSD.org'
 Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
 
 
 In the two files that was mentioned in the fix for -CURRENT, I applied
 them to my 7.1-PRERELEASE and recompiled/installed the kernel.  I was
 able to set vm.kmem_size_max beyond 4GB.  I ran into a kem shortage
 panic, but this time I paid attention that the size it said it was at
 the time of the panic was just under my setting for vm.kmem_size (1GB).
 I noticed that it had been panicing saying near 1GB size when
 vm.kmem_size_max was shown to be at 3GB or 16GB by sysctl -a.
 
 So, I tried setting kmem_size to 10G, that caused the known too big for
 boot-up panic, so did 2G.  I was able to boot set at 1.5GB.  It stayed
 up a longer, I started my rm -r that starts the panic and it did, but
 this time it said the size was 1.5GB.
 
 From this, I am guessing:
 
 1.  There may be more than just the changes in sys/kern/kern_malloc.c
 and=20
 sys/vm/vm_kern.h
 
 2.  It appears that vm.kmem_size_max isn't being used or there is a
 false alarm panic when vm.kmem_size is exceeded?
 
 
 Weldon
 
 -----Original Message-----
 From: Bruce Cran [mailto:bruce@cran.org.uk]=20
 Sent: Wednesday, December 03, 2008 3:27 PM
 To: alc@freebsd.org
 Cc: Weldon Godfrey
 Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
 
 On Wed, 3 Dec 2008 20:40:05 GMT
 "Weldon Godfrey" <wgodfrey@ena.com> wrote:
 
 > The following reply was made to PR kern/129396; it has been noted by
 > GNATS.
 >=20
 > From: "Weldon Godfrey" <wgodfrey@ena.com>
 > To: "Bruce Cran" <bruce@cran.org.uk>
 > Cc: <freebsd-gnats-submit@FreeBSD.org>
 > Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond
 > 3GB Date: Wed, 3 Dec 2008 14:25:26 -0600
 >=20
 >  Ah, sorry, my search for kmem in gnats didn't pull that up.  It does
 >  sound like the same issue.
 > =20
 >  Weldon
 > =20
 >  -----Original Message-----
 >  From: Bruce Cran [mailto:bruce@cran.org.uk]=3D20
 >  Sent: Wednesday, December 03, 2008 2:07 PM
 >  To: Weldon Godfrey
 >  Cc: freebsd-gnats-submit@FreeBSD.org
 >  Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond
 > 3GB=20
 >  It looks like this might be the same problem that was fixed in
 > -CURRENT a few months ago - see
 >  =
 http://svn.freebsd.org/viewvc/base?view=3D3Drevision&revision=3D3D180308
 > =20
 >  --=3D20
 >  Bruce
 
 Hi alc,
 
 Is there any chance the fix to support a kmem_map greater than 4G could
 be merged back to -STABLE?  It would be nice to have in 7.x at least.
 Or is it one of those changes that can't be made because it would
 change the ABI?
 
 --=20
 Bruce
 brucec@freebsd.org

From: "Weldon Godfrey" <wgodfrey@ena.com>
To: "Bruce Cran" <bruce@cran.org.uk>,
	<alc@freebsd.org>
Cc: <freebsd-gnats-submit@FreeBSD.org>
Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
Date: Mon, 15 Dec 2008 11:24:33 -0600

  As I may have mentioned, I went ahead and installed 8-HEAD.  It did the
 same thing in 1.x, panics on kmem_map too small, and saying it is at the
 size of vm.kmem_size and not near vm.kmem_size_max.
 
 I noticed I can set vm.kmem_size 4GB without kernel boot panic ( 2GB and
 above in 7.x seems to cause panic).  I did try 10GB and that did panic
 at boot.  However, setting it to 4GB allowed me to perform the tests
 fine.  So this is the workaround for me, using 8-HEAD and setting
 vm.kmem_size to 4GB.
 
 Weldon
 
 
State-Changed-From-To: open->closed 
State-Changed-By: alc 
State-Changed-When: Sat Feb 7 09:09:38 UTC 2009 
State-Changed-Why:  
A subset of the kernel and kmem map changes for amd64 have been applied 
to RELENG_7 (7.1-STABLE).  The maximum supported kmem map size on amd64 
is now a little less than 4GB, specifically, I believe it is 4GB-4KB. 
(In contrast, HEAD allows the kmem map to exceed 4GB.) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=129396 

From: "Weldon Godfrey" <wgodfrey@ena.com>
To: <alc@FreeBSD.org>
Cc: <FreeBSD-gnats-submit@FreeBSD.org>
Subject: RE: kern/129396: [kernel] cannot set vm.kmem_size_max beyond 3GB
Date: Mon, 9 Feb 2009 07:36:00 -0600

 When was this fixed  in head?  My copy of 8-CURRENT amd64 doesn't allow
 vm.kmem_size to exceed 4GB or it will panic on boot.  Also, my copy of
 8-CURRENT, vm.kmem_size_max doesn't seem to do anything as the system
 will panic, saying that kmem is at the size vm.kmem_size.
 
 If this was fixed in HEAD since Dec 2008, please let me know and I'll
 plan to upgrade.
 
 Weldon
 
 -----Original Message-----
 From: alc@FreeBSD.org [mailto:alc@FreeBSD.org]=20
 Sent: Saturday, February 07, 2009 3:18 AM
 To: Weldon Godfrey; alc@FreeBSD.org; alc@FreeBSD.org
 Subject: Re: kern/129396: [kernel] cannot set vm.kmem_size_max beyond
 3GB
 
 Synopsis: [kernel] cannot set vm.kmem_size_max beyond 3GB
 
 State-Changed-From-To: open->closed
 State-Changed-By: alc
 State-Changed-When: Sat Feb 7 09:09:38 UTC 2009
 State-Changed-Why:=20
 A subset of the kernel and kmem map changes for amd64 have been applied
 to RELENG_7 (7.1-STABLE).  The maximum supported kmem map size on amd64
 is now a little less than 4GB, specifically, I believe it is 4GB-4KB.
 (In contrast, HEAD allows the kmem map to exceed 4GB.)
 
 http://www.freebsd.org/cgi/query-pr.cgi?pr=3D129396

From: Alan Cox <alc@cs.rice.edu>
To: Weldon Godfrey <wgodfrey@ena.com>
Cc: alc@FreeBSD.org, FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: kern/129396: [kernel] cannot set vm.kmem_size_max beyond 3GB
Date: Tue, 10 Feb 2009 12:45:02 -0600

 Weldon Godfrey wrote:
 > When was this fixed  in head?  My copy of 8-CURRENT amd64 doesn't allow
 > vm.kmem_size to exceed 4GB or it will panic on boot.  Also, my copy of
 > 8-CURRENT, vm.kmem_size_max doesn't seem to do anything as the system
 > will panic, saying that kmem is at the size vm.kmem_size.
 >
 > If this was fixed in HEAD since Dec 2008, please let me know and I'll
 > plan to upgrade.
 >
 >   
 
 Nothing relating to kmem map size has changed in 8-CURRENT since last 
 summer.
 
 I am able to set vm.kmem_size in /boot/loader.conf to greater than 4GB 
 and boot.  Specifically, I set it 4GB+1MB.  After booting, "sysctl vm" 
 reports:
 
 ...
 vm.kmem_size_scale: 3
 vm.kmem_size_max: 4509713203
 vm.kmem_size_min: 0
 vm.kmem_size: 4296015872
 ...
 
 There is still a hard limit on vm.kmem_size.  If you tried setting it to 
 5GB or more, I'm sure it crashed.  The limit is somewhere between 4.5 
 and 5 GB.
 
 Regards,
 Alan
 

From: "Weldon Godfrey" <wgodfrey@ena.com>
To: "Alan Cox" <alc@cs.rice.edu>
Cc: <alc@FreeBSD.org>,
	<FreeBSD-gnats-submit@FreeBSD.org>
Subject: RE: kern/129396: [kernel] cannot set vm.kmem_size_max beyond 3GB
Date: Wed, 11 Feb 2009 11:20:19 -0600

 Ok, then that is the hard limit I am reaching.  (it is amazing now I am
 saying plus or minus a GIG). =20
 
 It isn't a biggie to me currently.  My stable configuration currently:
 
 Zil off (I tried turning it back on and had an episode where the zfs
 filesystem just hangs, I'll try again later on...i don't feel
 comfortable with this setting with NFS)
 Prefetch off
 ARC limited to 2GB
 vm.kmem_size=3D4GB
 vm.kmem_size_max=3D15GB
 
 The only question I have is that I have seen under 7.x and HEAD the
 system panic due to kmem exhaustion and it saying the size of kmem was
 what vm.kmem_size and that was way under what vm.kmem_size_max is, so it
 appears it wasn't growning.  Could it be that it won't grow because my
 vm.kmem_size_max is over the 5GB limit even though sysctl shows it set
 to something like 15GB?  Or is there a bug that doesn't allow kmem to
 grow?
 
 
 Weldon
 
 -----Original Message-----
 From: Alan Cox [mailto:alc@cs.rice.edu]=20
 Sent: Tuesday, February 10, 2009 12:45 PM
 To: Weldon Godfrey
 Cc: alc@FreeBSD.org; FreeBSD-gnats-submit@FreeBSD.org
 Subject: Re: kern/129396: [kernel] cannot set vm.kmem_size_max beyond
 3GB
 
 Weldon Godfrey wrote:
 > When was this fixed  in head?  My copy of 8-CURRENT amd64 doesn't
 allow
 > vm.kmem_size to exceed 4GB or it will panic on boot.  Also, my copy of
 > 8-CURRENT, vm.kmem_size_max doesn't seem to do anything as the system
 > will panic, saying that kmem is at the size vm.kmem_size.
 >
 > If this was fixed in HEAD since Dec 2008, please let me know and I'll
 > plan to upgrade.
 >
 >  =20
 
 Nothing relating to kmem map size has changed in 8-CURRENT since last=20
 summer.
 
 I am able to set vm.kmem_size in /boot/loader.conf to greater than 4GB=20
 and boot.  Specifically, I set it 4GB+1MB.  After booting, "sysctl vm"=20
 reports:
 
 ...
 vm.kmem_size_scale: 3
 vm.kmem_size_max: 4509713203
 vm.kmem_size_min: 0
 vm.kmem_size: 4296015872
 ...
 
 There is still a hard limit on vm.kmem_size.  If you tried setting it to
 
 5GB or more, I'm sure it crashed.  The limit is somewhere between 4.5=20
 and 5 GB.
 
 Regards,
 Alan
 

From: Alan Cox <alc@cs.rice.edu>
To: Weldon Godfrey <wgodfrey@ena.com>
Cc: alc@FreeBSD.org, FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: kern/129396: [kernel] cannot set vm.kmem_size_max beyond 3GB
Date: Wed, 11 Feb 2009 11:42:11 -0600

 Weldon Godfrey wrote:
 > Ok, then that is the hard limit I am reaching.  (it is amazing now I am
 > saying plus or minus a GIG).  
 >
 > It isn't a biggie to me currently.  My stable configuration currently:
 >
 > Zil off (I tried turning it back on and had an episode where the zfs
 > filesystem just hangs, I'll try again later on...i don't feel
 > comfortable with this setting with NFS)
 > Prefetch off
 > ARC limited to 2GB
 > vm.kmem_size=4GB
 > vm.kmem_size_max=15GB
 >
 > The only question I have is that I have seen under 7.x and HEAD the
 > system panic due to kmem exhaustion and it saying the size of kmem was
 > what vm.kmem_size and that was way under what vm.kmem_size_max is, so it
 > appears it wasn't growning.  Could it be that it won't grow because my
 > vm.kmem_size_max is over the 5GB limit even though sysctl shows it set
 > to something like 15GB?  Or is there a bug that doesn't allow kmem to
 > grow?
 >
 >   
 
 The kmem size is static.  It doesn't grow during execution.  It 
 establishes the upper bound on how much physical memory the kernel is 
 allowed to claim for its heap.
 
 vm.kmem_size_max is not what you think it is.  During boot, the kernel 
 computes a size for kmem based on the size of the kernel's overall 
 virtual address space and the amount of physical memory that your 
 machine has.  Ideally, the latter of these factors is the determining 
 factor.  Unfortunately, that was not true until last summer.  Before 
 last summer, the 2GB size of the kernel virtual address space was the 
 limiting factor on any machine with more than a couple gigabytes of 
 memory.  The effect of vm.kmem_size_max is to establish on upper bound 
 on the size computed by this autosizing function or the manual setting 
 of vm.kmem_size.
 
 There is no reason for you to set vm.kmem_size_max any longer.  The 
 default is already larger than your manual setting of vm.kmem_size.
 
 Regards,
 Alan
 

From: "Weldon Godfrey" <wgodfrey@ena.com>
To: "Alan Cox" <alc@cs.rice.edu>
Cc: <alc@FreeBSD.org>,
	<FreeBSD-gnats-submit@FreeBSD.org>
Subject: RE: kern/129396: [kernel] cannot set vm.kmem_size_max beyond 3GB
Date: Wed, 11 Feb 2009 12:04:02 -0600

 Thanks for the learning lesson.
 
 BTW.  We really appreciate the support and all the work for port ZFS
 into FreeBSD.  Thanks!
 
 Weldon
 
 -----Original Message-----
 From: Alan Cox [mailto:alc@cs.rice.edu]=20
 Sent: Wednesday, February 11, 2009 11:42 AM
 To: Weldon Godfrey
 Cc: alc@FreeBSD.org; FreeBSD-gnats-submit@FreeBSD.org
 Subject: Re: kern/129396: [kernel] cannot set vm.kmem_size_max beyond
 3GB
 
 Weldon Godfrey wrote:
 > Ok, then that is the hard limit I am reaching.  (it is amazing now I
 am
 > saying plus or minus a GIG). =20
 >
 > It isn't a biggie to me currently.  My stable configuration currently:
 >
 > Zil off (I tried turning it back on and had an episode where the zfs
 > filesystem just hangs, I'll try again later on...i don't feel
 > comfortable with this setting with NFS)
 > Prefetch off
 > ARC limited to 2GB
 > vm.kmem_size=3D4GB
 > vm.kmem_size_max=3D15GB
 >
 > The only question I have is that I have seen under 7.x and HEAD the
 > system panic due to kmem exhaustion and it saying the size of kmem was
 > what vm.kmem_size and that was way under what vm.kmem_size_max is, so
 it
 > appears it wasn't growning.  Could it be that it won't grow because my
 > vm.kmem_size_max is over the 5GB limit even though sysctl shows it set
 > to something like 15GB?  Or is there a bug that doesn't allow kmem to
 > grow?
 >
 >  =20
 
 The kmem size is static.  It doesn't grow during execution.  It=20
 establishes the upper bound on how much physical memory the kernel is=20
 allowed to claim for its heap.
 
 vm.kmem_size_max is not what you think it is.  During boot, the kernel=20
 computes a size for kmem based on the size of the kernel's overall=20
 virtual address space and the amount of physical memory that your=20
 machine has.  Ideally, the latter of these factors is the determining=20
 factor.  Unfortunately, that was not true until last summer.  Before=20
 last summer, the 2GB size of the kernel virtual address space was the=20
 limiting factor on any machine with more than a couple gigabytes of=20
 memory.  The effect of vm.kmem_size_max is to establish on upper bound=20
 on the size computed by this autosizing function or the manual setting=20
 of vm.kmem_size.
 
 There is no reason for you to set vm.kmem_size_max any longer.  The=20
 default is already larger than your manual setting of vm.kmem_size.
 
 Regards,
 Alan
 
>Unformatted:
