From nobody@FreeBSD.org  Fri Jul 16 23:07:15 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 3855A106566C
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 16 Jul 2010 23:07:15 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 25B4B8FC0C
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 16 Jul 2010 23:07:15 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o6GN7EOB007555
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 16 Jul 2010 23:07:14 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o6GN7E6B007554;
	Fri, 16 Jul 2010 23:07:14 GMT
	(envelope-from nobody)
Message-Id: <201007162307.o6GN7E6B007554@www.freebsd.org>
Date: Fri, 16 Jul 2010 23:07:14 GMT
From: Joe Landers <jlanders@vmware.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Panic at boot due to integer overflow computing top->cg_mask in smp_topo_none() when mp_ncpus == 32
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         148698
>Category:       kern
>Synopsis:       [panic] Panic at boot due to integer overflow computing top->cg_mask in smp_topo_none() when mp_ncpus == 32
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    attilio
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Jul 16 23:10:07 UTC 2010
>Closed-Date:    Thu Mar 10 17:42:17 UTC 2011
>Last-Modified:  Thu Mar 10 17:42:17 UTC 2011
>Originator:     Joe Landers
>Release:        FreeBSD 8.0 (amd64)
>Organization:
VMware, Inc
>Environment:
Any system with 32 CPUs.
>Description:
Attempting to boot FreeBSD 8.0 (amd64) on machine with 32 CPUs causes the kernel to panic. panic() calls boot() and boot() calls thread_lock(curthread), however curthread->td_lock is NULL so the system starts taking repeated page faults and printing to the screen.

FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 32 package(s) x 1 core(s)
 cpu0 (BSP): APIC ID: 0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
 cpu8 (AP): APIC ID:  8
 cpu9 (AP): APIC ID:  9
 cpu10 (AP): APIC ID: 10
 cpu11 (AP): APIC ID: 11
 cpu12 (AP): APIC ID: 12
 cpu13 (AP): APIC ID: 13
 cpu14 (AP): APIC ID: 14
 cpu15 (AP): APIC ID: 15
 cpu16 (AP): APIC ID: 16
 cpu17 (AP): APIC ID: 17
 cpu18 (AP): APIC ID: 18
 cpu19 (AP): APIC ID: 19
 cpu20 (AP): APIC ID: 20
 cpu21 (AP): APIC ID: 21
 cpu22 (AP): APIC ID: 22
 cpu23 (AP): APIC ID: 23
 cpu24 (AP): APIC ID: 24
 cpu25 (AP): APIC ID: 25
 cpu26 (AP): APIC ID: 26
 cpu27 (AP): APIC ID: 27
 cpu28 (AP): APIC ID: 28
 cpu29 (AP): APIC ID: 29
 cpu30 (AP): APIC ID: 30
 cpu31 (AP): APIC ID: 31
panic: Built bad topology at 0xffffffff8081fc20.  CPU mask 0x0 != 0xFFFFFFFF
cpuid = 0
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x18
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80322e28
stack pointer           = 0x28:0xffffffff809b0b20
frame pointer           = 0x28:0xffffffff809b0b60
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 0 (swapper)
trap number             = 12
panic: page fault
cpuid = 0
kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x18
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80322e28
stack pointer           = 0x28:0xffffffff809b0790
frame pointer           = 0x28:0xffffffff809b07d0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 0 (swapper)
trap number             = 12
panic: page fault
cpuid = 0
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x18
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80322e28
stack pointer           = 0x28:0xffffffff809b0400
frame pointer           = 0x28:0xffffffff809b0440
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 0 (swapper)
trap number             = 12
panic: page fault
cpuid = 0
kernel trap 12 with interrupts disabled

.. <repeats> ...

When the hardware has 32 CPUs, the computation of top->cg_mask in smp_topo_none() overflows and becomes zero. This causes smp_topo() to call panic() with the message about the topology mismatch.

struct cpu_group *
smp_topo_none(void)
{
 struct cpu_group *top;

 top = &group[0];
 top->cg_parent = NULL;
 top->cg_child = NULL;
 top->cg_mask = (1 << mp_ncpus) - 1; <<<<<<<<<<<<<<<<
 top->cg_count = mp_ncpus;
 top->cg_children = 0;
 top->cg_level = CG_SHARE_NONE;
 top->cg_flags = 0;
 
 return (top);
}

0xffffffff805b9743 <smp_topo_none+51>: shl %cl,%eax <<<<<<<<
0xffffffff805b9745 <smp_topo_none+53>: mov %cl,6632169(%rip) # 0xffffffff80c0ca34 <group+20>
0xffffffff805b974b <smp_topo_none+59>: movb $0x0,6632165(%rip) # 0xffffffff80c0ca37 <group+23>
0xffffffff805b9752 <smp_topo_none+66>: sub $0x1,%eax
0xffffffff805b9755 <smp_topo_none+69>: mov %eax,6632149(%rip) # 0xffffffff80c0ca30 <group+16>
0xffffffff805b975b <smp_topo_none+75>: leaveq
0xffffffff805b975c <smp_topo_none+76>: mov $0xffffffff80c0ca20,%rax
0xffffffff805b9763 <smp_topo_none+83>: retq

static int
start_all_aps(void)
{
..
  all_cpus |= (1 << cpu); /* record AP in CPU map */
 }
..
}

struct cpu_group *
smp_topo(void)
{
 /*
  * Verify the returned topology.
  */
 if (top->cg_count != mp_ncpus)
  panic("Built bad topology at %p. CPU count %d != %d",
      top, top->cg_count, mp_ncpus);
 if (top->cg_mask != all_cpus) <<<<<<
  panic("Built bad topology at %p. CPU mask 0x%X != 0x%X",
      top, top->cg_mask, all_cpus);
 return (top);
}

(gdb) x/xw 0xc00000 + (0xffffffff80c0ca20 & 0x1fffff) + 0x10
0xc0ca30: 0x00000000 // top->cg_mask
(gdb) x/xw 0xc00000 + (0xffffffff80c0ca20 & 0x1fffff) + 0x14
0xc0ca34: 0x00000020 // top->cg_count

(gdb) p &all_cpus
$16 = (cpumask_t *) 0xffffffff80c0c9b8
(gdb) x/xw 0xc00000 + (0xffffffff80c0c9b8 & 0x1fffff)
0xc0c9b8: 0xffffffff // all_cpus

(gdb) p &mp_ncpus
$15 = (int *) 0xffffffff80c0c9b0
(gdb) x/xw 0xc00000 + (0xffffffff80c0c9b0 & 0x1fffff)
0xc0c9b0: 0x00000020 // mp_ncpus



0x121db30: 0xffffff00bfed7000 0x0000000000000104
0x121db40: 0xffffffff80951948 0xffffffff80be7600
0x121db50: 0x0000000000000000 0x0000000000000000
0x121db60: 0xffffffff8121dbb0 0xffffffff8057f41f // boot+47


0xffffffff8057f418 <boot+40>: xor %esi,%esi
0xffffffff8057f41a <boot+42>: callq 0xffffffff80571290 <_thread_lock_flags>
0xffffffff8057f41f <boot+47>: mov %gs:0x0,%rdi

0x121db70: 0xffffffff80951948 0xffffffff80be7600
0x121db80: 0x0000000000000000 0x0000000000000104
0x121db90: 0xffffffff80951948 0xffffffff80be7600
0x121dba0: 0x0000000000000000 0x0000000000000000
0x121dbb0: 0xffffffff8121dcb0 0xffffffff8057fcfc // panic+332
0x121dbc0: 0x0000003000000020 0xffffffff8121dcc0
0x121dbd0: 0xffffffff8121dbe0 0x00000000bfed9560
0x121dbe0: 0xffffffff8121dc70 0xffffffff80c0ca20
0x121dbf0: 0x0000000000000000 0x00000000ffffffff
0x121dc00: 0x0000000000000000 0x0000000000000001
0x121dc10: 0xffffff00bfed94e0 0x0000000200000202
0x121dc20: 0xffffff00bfed94e8 0x0000010280b60920
0x121dc30: 0x0000000000000000 0xffffffff80b60920
0x121dc40: 0xffffffff8121dc80 0xffffffff8056db9b
0x121dc50: 0x00000000023a3c80 0xffffff00bfed94c0
0x121dc60: 0x00000000023a3c80 0xffffff00bfed94c0
0x121dc70: 0xffffff00023b0d00 0xffffffff8098dfe8
0x121dc80: 0xffffffff80bf1240 0x0000000000000000
0x121dc90: 0x0000000000000000 0xffffffff8098dfe8
0x121dca0: 0xffffffff80bf1240 0x0000000000000000
0x121dcb0: 0xffffffff8121dcc0 0xffffffff805b9bc7 // smp_topo+263

0xffffffff80951948 <basefix.2920+424>: "Built bad topology at %p. CPU mask 0x%X != 0x%X"

0xffffffff805b9bb9 <smp_topo+249>: mov $0xffffffff80951948,%rdi
0xffffffff805b9bc0 <smp_topo+256>: xor %eax,%eax
0xffffffff805b9bc2 <smp_topo+258>: callq 0xffffffff8057fbb0 <panic>


>How-To-Repeat:
Our engineering team originally came across this issue testing FreeBSD-8.0 (amd64) running on large number of virtual CPUs running as a guest on VMware ESX.

Subsequently, engineering was able to reproduce this issue on physical hardware, booting the FreeBSD-8.0 (amd64) install disk on an IBM x3950 (4 node) Athena M2. The physical system panic()'s identically there as well.
>Fix:


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->suspended 
State-Changed-By: linimon 
State-Changed-When: Fri Jul 16 23:23:00 UTC 2010 
State-Changed-Why:  
I think the answer right now is "unfortunately, you're right". 

IIRC there are people working on code to handle > 32 CPUs, but 
it's not ready to commit yet. 

But take this with a grain of salt -- I'm wearing a bugmeister hat 
as I type this, not a source developer hat. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=148698 

From: Joe Landers <jlanders@vmware.com>
To: "bug-followup@FreeBSD.org" <bug-followup@FreeBSD.org>, Joe Landers
	<jlanders@vmware.com>
Cc:  
Subject: Re: kern/148698: [panic] Panic at boot due to integer overflow
 computing top-&gt;cg_mask in smp_topo_none() when mp_ncpus == 32
Date: Wed, 28 Jul 2010 19:26:19 -0700

 We understand that > 32 CPUs isn't possible with the current code structure=
 .
 However, it is trivial to make the case of mp_ncpus =3D=3D 32 work.
 
 Assuming no error checking, simply change the computation of top->cg_mask f=
 rom:
    top->cg_mask =3D (1 << mp_ncpus) - 1;
 to:
    top->cg_mask =3D ~0U >> (32 - mp_ncpus);
 in subr_smp.c:smp_topo_none().
 
 FreeBSD will then boot fine with 32 or fewer CPUs:
 ...
 FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
 FreeBSD/SMP: 32 package(s) x 1 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
  cpu4 (AP): APIC ID:  4
  cpu5 (AP): APIC ID:  5
  cpu6 (AP): APIC ID:  6
  cpu7 (AP): APIC ID:  7
  cpu8 (AP): APIC ID:  8
  cpu9 (AP): APIC ID:  9
  cpu10 (AP): APIC ID: 10
  cpu11 (AP): APIC ID: 11
  cpu12 (AP): APIC ID: 12
  cpu13 (AP): APIC ID: 13
  cpu14 (AP): APIC ID: 14
  cpu15 (AP): APIC ID: 15
  cpu16 (AP): APIC ID: 16
  cpu17 (AP): APIC ID: 17
  cpu18 (AP): APIC ID: 18
  cpu19 (AP): APIC ID: 19
  cpu20 (AP): APIC ID: 20
  cpu21 (AP): APIC ID: 21
  cpu22 (AP): APIC ID: 22
  cpu23 (AP): APIC ID: 23
  cpu24 (AP): APIC ID: 24
  cpu25 (AP): APIC ID: 25
  cpu26 (AP): APIC ID: 26
  cpu27 (AP): APIC ID: 27
  cpu28 (AP): APIC ID: 28
  cpu29 (AP): APIC ID: 29
  cpu30 (AP): APIC ID: 30
  cpu31 (AP): APIC ID: 31
 ioapic0: Changing APIC ID to 32
 ...
 SMP: AP CPU #28 Launched!
 SMP: AP CPU #27 Launched!
 SMP: AP CPU #1 Launched!
 SMP: AP CPU #2 Launched!
 SMP: AP CPU #31 Launched!
 SMP: AP CPU #3 Launched!
 SMP: AP CPU #13 Launched!
 SMP: AP CPU #23 Launched!
 SMP: AP CPU #4 Launched!
 SMP: AP CPU #9 Launched!
 SMP: AP CPU #20 Launched!
 SMP: AP CPU #21 Launched!
 SMP: AP CPU #5 Launched!
 SMP: AP CPU #12 Launched!
 SMP: AP CPU #15 Launched!
 SMP: AP CPU #16 Launched!
 SMP: AP CPU #7 Launched!
 SMP: AP CPU #26 Launched!
 SMP: AP CPU #29 Launched!
 SMP: AP CPU #17 Launched!
 SMP: AP CPU #25 Launched!
 SMP: AP CPU #6 Launched!
 SMP: AP CPU #11 Launched!
 SMP: AP CPU #19 Launched!
 SMP: AP CPU #14 Launched!
 SMP: AP CPU #18 Launched!
 SMP: AP CPU #22 Launched!
 SMP: AP CPU #10 Launched!
 SMP: AP CPU #24 Launched!
 SMP: AP CPU #8 Launched!
 SMP: AP CPU #30 Launched!
 ...
 
 We verified this fix on both physical and virtual hardware.
 
 Thanks,
 
 Joe Landers
 jlanders@vmware.com
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/148698: commit references a PR
Date: Mon,  9 Aug 2010 00:24:11 +0000 (UTC)

 Author: attilio
 Date: Mon Aug  9 00:23:57 2010
 New Revision: 211087
 URL: http://svn.freebsd.org/changeset/base/211087
 
 Log:
   The r208165 fixed a bug related to unsigned integer overflowing for the
   number of CPUs detection.
   However, that was not mention at all, the problem was not reported, the
   patch has not been MFCed and the fix is mostly improper.
   
   Fix the original overflow (caused when 32 CPUs must be detected) by
   just using a different mathematical computation (it also makes more
   explicit the size of operands involved, which is good in the moment
   waiting for a more complete support for a large number of CPUs).
   
   PR:		kern/148698
   Submitted by:	Joe Landers <jlanders at vmware dot com>
   Tested by:	gianni
   MFC after:	10 days
 
 Modified:
   head/sys/kern/subr_smp.c
 
 Modified: head/sys/kern/subr_smp.c
 ==============================================================================
 --- head/sys/kern/subr_smp.c	Sun Aug  8 23:24:23 2010	(r211086)
 +++ head/sys/kern/subr_smp.c	Mon Aug  9 00:23:57 2010	(r211087)
 @@ -504,10 +504,7 @@ smp_topo_none(void)
  	top = &group[0];
  	top->cg_parent = NULL;
  	top->cg_child = NULL;
 -	if (mp_ncpus == sizeof(top->cg_mask) * 8)
 -		top->cg_mask = -1;
 -	else
 -		top->cg_mask = (1 << mp_ncpus) - 1;
 +	top->cg_mask = ~0U >> (32 - mp_ncpus);
  	top->cg_count = mp_ncpus;
  	top->cg_children = 0;
  	top->cg_level = CG_SHARE_NONE;
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: suspended->patched 
State-Changed-By: linimon 
State-Changed-When: Mon Aug 9 01:16:59 UTC 2010 
State-Changed-Why:  
Patch committed; set as MFC reminder. 


Responsible-Changed-From-To: freebsd-bugs->attilio 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Aug 9 01:16:59 UTC 2010 
Responsible-Changed-Why:  

http://www.freebsd.org/cgi/query-pr.cgi?pr=148698 
State-Changed-From-To: patched->closed 
State-Changed-By: pluknet 
State-Changed-When: Thu Mar 10 17:41:04 UTC 2011 
State-Changed-Why:  
Merged to 8 (as svn r215938). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=148698 
>Unformatted:
