From nobody@FreeBSD.org  Mon Jan 13 01:33:18 2014
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTPS id B86D7A49
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 13 Jan 2014 01:33:18 +0000 (UTC)
Received: from oldred.freebsd.org (oldred.freebsd.org [IPv6:2001:1900:2254:206a::50:4])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id A43E81316
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 13 Jan 2014 01:33:18 +0000 (UTC)
Received: from oldred.freebsd.org ([127.0.1.6])
	by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id s0D1XIel094431
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 13 Jan 2014 01:33:18 GMT
	(envelope-from nobody@oldred.freebsd.org)
Received: (from nobody@localhost)
	by oldred.freebsd.org (8.14.5/8.14.5/Submit) id s0D1XI7G094428;
	Mon, 13 Jan 2014 01:33:18 GMT
	(envelope-from nobody)
Message-Id: <201401130133.s0D1XI7G094428@oldred.freebsd.org>
Date: Mon, 13 Jan 2014 01:33:18 GMT
From: Alfred Perlstein <alfred@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Devices fail to probe on 128GB or larger memory machines
X-Send-Pr-Version: www-3.1
X-GNATS-Notify: alfred, tgs@norse-corp.com

>Number:         185727
>Category:       kern
>Synopsis:       Devices fail to probe on 128GB or larger memory machines
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    alc
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jan 13 01:40:00 UTC 2014
>Closed-Date:    
>Last-Modified:  Sun Apr 20 06:30:00 UTC 2014
>Originator:     Alfred Perlstein <alfred@freebsd.org>
>Release:        -current / 10.x
>Organization:
FreeBSD
>Environment:
N/A
>Description:
FreeBSD machines with large memory, it seems 128GB or higher wind up without enough memory under 4GB for many devices to work.

This results in USB not working (keyboard/mouse doesn't attach), and some HBAs fail as well.  I think other devices may fail as well, but I've not experienced that yet.
>How-To-Repeat:
Boot FreeBSD 10.x or FreeBSD-current with more than 128GB of memory and some hardware configs.
>Fix:
Alan Cox posted the attached patch (
hack2.patch) , however there needs to be some work done still on it according to him:

> The only issue with this patch is that it will pessimize the speed of
> physical memory allocation on amd64 machines with small amounts of
> memory.  I need to augment the attached patch, which just changes some
> #define's, with some changes to vm_phys.c to avoid creating excess free
> page queues on small memory machines.
>
> Alan

Patch attached with submission follows:

Index: amd64/include/vmparam.h
===================================================================
--- amd64/include/vmparam.h	(revision 243366)
+++ amd64/include/vmparam.h	(working copy)
@@ -106,10 +106,13 @@
  * accessible by ISA DMA and VM_FREELIST_ISADMA is for physical pages
  * that are below that address.
  */
-#define	VM_NFREELIST		2
-#define	VM_FREELIST_DEFAULT	0
-#define	VM_FREELIST_ISADMA	1
+#define	VM_NFREELIST		3
+#define	VM_FREELIST_HIGHMEM	0
+#define	VM_FREELIST_DEFAULT	1
+#define	VM_FREELIST_ISADMA	2
 
+#define	VM_HIGHMEM_ADDRESS	((vm_paddr_t)1 << 32)
+
 /*
  * An allocation size of 16MB is supported in order to optimize the
  * use of the direct map by UMA.  Specifically, a cache line contains


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->alc@freebsd.org 
Responsible-Changed-By: alfred 
Responsible-Changed-When: Mon Jan 13 01:49:00 UTC 2014 
Responsible-Changed-Why:  
Assign to alc, fix originator. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=185727 
Responsible-Changed-From-To: alc@freebsd.org->alc 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Jan 13 02:16:30 UTC 2014 
Responsible-Changed-Why:  
Canonicalize assignment. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=185727 

From: Anton Sayetsky <vsjcfm@gmail.com>
To: alfred@freebsd.org, bug-followup@freebsd.org
Cc:  
Subject: Re: kern/185727: Devices fail to probe on 128GB or larger memory machines
Date: Sun, 19 Jan 2014 00:31:09 +0200

 Is this problem exists in 9.x too?
 And about the patch - will it cause performance degradation on systems
 with 128g+ ram?

From: Alfred Perlstein <alfred@freebsd.org>
To: Anton Sayetsky <vsjcfm@gmail.com>, bug-followup@freebsd.org
Cc:  
Subject: Re: kern/185727: Devices fail to probe on 128GB or larger memory
 machines
Date: Sun, 19 Jan 2014 06:28:07 -0800

 On 1/18/14 2:31 PM, Anton Sayetsky wrote:
 > Is this problem exists in 9.x too?
 > And about the patch - will it cause performance degradation on systems
 > with 128g+ ram?
 >
 Probably not, however it may cause problems on machines with less. I'd 
 start to worry about 16GB or even 32GB and less.
 
 -Alfred

From: Anton Sayetsky <vsjcfm@gmail.com>
To: Alfred Perlstein <alfred@freebsd.org>
Cc: bug-followup@freebsd.org
Subject: Re: kern/185727: Devices fail to probe on 128GB or larger memory machines
Date: Sun, 19 Jan 2014 22:52:19 +0200

 2014/1/19 Alfred Perlstein <alfred@freebsd.org>:
 > Probably not, however it may cause problems on machines with less. I'd start
 > to worry about 16GB or even 32GB and less.
 I'm planning to deploy system with 100+ TiB zpool & 128-256 GiB RAM,
 respectively.
 At least now I can try this workaround. It will not be done tomorrow,
 but I'll send followup later.

From: Kurt Lidl <lidl@pix.net>
To: bug-followup@FreeBSD.org, alfred@FreeBSD.org
Cc:  
Subject: Re: kern/185727: Devices fail to probe on 128GB or larger memory
 machines
Date: Fri, 28 Mar 2014 15:45:56 -0400

 Is there any forward progress on resolving this issue?
 
 We just ran into this same issue on a 128GB machine - the USB
 subsystem failed because it could not allocate memory in
 the first 4GB of space.
 
 We're running with the patch noted in this bug report, to
 no avail.
 
 Thanks for any updates.

From: Alfred Perlstein <alfred@freebsd.org>
To: bug-followup@FreeBSD.org, alfred@FreeBSD.org, 
 Kurt Lidl <lidl@pix.net>
Cc:  
Subject: Re: kern/185727: Devices fail to probe on 128GB or larger memory
 machines
Date: Sat, 19 Apr 2014 07:54:39 -0700

 Kurt,
 
 Does the patch work for you or no?

From: Kurt Lidl <lidl@pix.net>
To: Alfred Perlstein <alfred@freebsd.org>, bug-followup@freebsd.org
Cc:  
Subject: Re: kern/185727: Devices fail to probe on 128GB or larger memory
 machines
Date: Sun, 20 Apr 2014 02:28:44 -0400

 On 4/19/14 10:54 AM, Alfred Perlstein wrote:
 > Kurt,
 >
 > Does the patch work for you or no?
 
 More or less.  We have other things that demand
 memory allocation allocations below the 4GB
 mark.  With a little juggling, we were able to
 make this work on our 128GB machines.
 
 Sorry that's sorta a wishy-washy answer, but
 it's the best I can give right now.
 
 -Kurt
 
>Unformatted:
