From nobody@FreeBSD.org  Thu Jun 18 10:20:13 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 035301065672
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 18 Jun 2009 10:20:13 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id E3E928FC1B
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 18 Jun 2009 10:20:12 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n5IAKCLQ064280
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 18 Jun 2009 10:20:12 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n5IAKCLC064279;
	Thu, 18 Jun 2009 10:20:12 GMT
	(envelope-from nobody)
Message-Id: <200906181020.n5IAKCLC064279@www.freebsd.org>
Date: Thu, 18 Jun 2009 10:20:12 GMT
From: Adam Kirchhoff <adamk@voicenet.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: kernel panic with VirtualBox on 8.0-CURRENT (Tue Jun 16 13:49:34 EDT 2009)
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         135692
>Category:       kern
>Synopsis:       [mutex] [witness] blockable sleep lock - kernel panic on 8.0-CURRENT
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    jh
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jun 18 10:30:01 UTC 2009
>Closed-Date:    Sat Oct 30 08:09:24 UTC 2010
>Last-Modified:  Sat Oct 30 08:09:24 UTC 2010
>Originator:     Adam Kirchhoff
>Release:        8.0-CURRENT
>Organization:
none
>Environment:
FreeBSD memory.visualtech.com 8.0-CURRENT FreeBSD 8.0-CURRENT #7: Tue Jun 16 13:49:34 EDT 2009     root@memory.visualtech.com:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
I recently updated my workstation at home to -CURRENT, and then noticed (thanks to some discussion on ##freebsd) that VirtualBox had made it into the ports tree.  I installed it via ports, but upon trying to boot a VDI image created under Linux (where it works just fine), I ran into a kernel panic.  When starting the VM, VirtualBox informs me of the key combination to grab/release the mouse and keyboard.  I click "OK" and then the entire machine locks up.
The VM is Windows XP.  256 megs of RAM, 1 processor, VT-x/AMD-V is enabled, with Nesting Paging disabled.  The only option I changed was the memory.  All the others were the defaults.

I reproduced the panic quite easily by rebooting into single user mode, fsck'ing the filesystems manually, booting into X, and then starting up VirtualBox.  Same thing happened when I launched the VM.

The panic:

This GDB was configured as "i386-marcel-freebsd"...                        
Unread portion of the kernel message buffer:
panic: blockable sleep lock (sleep mutex) 32 @ /usr/src/sys/vm/uma_core.c:1990
cpuid = 2                                                                    KDB: enter: panic                                                            Uptime: 1h57m3s                                                              Physical memory: 2029 MB                                                     Dumping 252 MB: 237 221 205 189 173 157 141 125 109 93 77 61 45 29 13       
bReading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done.                                                                                                         
done.                                                                                                                                                                                                            
Loaded symbols for /boot/kernel/linux.ko                                                                                                                                                                         
Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from /boot/kernel/snd_hda.ko.symbols...tdone.                                                                                                     
done.                                                                                                                                                                                                            
Loaded symbols for /boot/kernel/snd_hda.ko                                                                                                                                                                       
Reading symbols from /boot/kernel/sound.ko...Reading symbols from /boot/kernel/sound.ko.symbols...done.                                                                                                          
done.                                                                                                                                                                                                            
Loaded symbols for /boot/kernel/sound.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/aio.ko...Reading symbols from /boot/kernel/aio.ko.symbols...done.
done.
^R
Loaded symbols for /boot/kernel/aio.ko
Reading symbols from /boot/modules/kqemu.ko...done.
Loaded symbols for /boot/modules/kqemu.ko
Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linprocfs.ko
Reading symbols from /boot/kernel/smbfs.ko...Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/smbfs.ko
Reading symbols from /boot/kernel/libiconv.ko...Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/libiconv.ko
Reading symbols from /boot/kernel/libmchain.ko...Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/libmchain.ko
Reading symbols from /boot/kernel/radeon.ko...Reading symbols from /boot/kernel/radeon.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/radeon.ko
Reading symbols from /boot/kernel/drm.ko...Reading symbols from /boot/kernel/drm.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/drm.ko
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
#0  doadump () at pcpu.h:246
246     pcpu.h: No such file or directory.
       in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:246
#1  0xc0870a8e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
#2  0xc0870d62 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:575
#3  0xc08b1eb0 in witness_checkorder (lock=0xc1894788, flags=9, file=0xc0c81d4e "/usr/src/sys/vm/uma_core.c", line=1990, interlock=0x0)
   at /usr/src/sys/kern/subr_witness.c:1063
#4  0xc0861b74 in _mtx_lock_flags (m=0xc1894788, opts=0, file=0xc0c81d4e "/usr/src/sys/vm/uma_core.c", line=1990)
   at /usr/src/sys/kern/kern_mutex.c:200
#5  0xc0ab0ae3 in uma_zalloc_arg (zone=0xc188c700, udata=0x0, flags=1) at /usr/src/sys/vm/uma_core.c:1990
#6  0xc085e1e4 in malloc (size=28, mtp=0xc97c09d0, flags=1) at uma.h:305
#7  0xc97b54b6 in rtMemAlloc () from /boot/modules/vboxdrv.ko
#8  0xc97b3519 in RTMemAlloc () from /boot/modules/vboxdrv.ko
#9  0xc97a6e62 in SUPR0ObjAddRefEx () from /boot/modules/vboxdrv.ko
#10 0xc97a6f00 in SUPR0ObjAddRef () from /boot/modules/vboxdrv.ko
#11 0xc97aee2b in RTHandleTableLookupWithCtx () from /boot/modules/vboxdrv.ko
#12 0xc97a6b26 in SUPSemEventWaitNoResume () from /boot/modules/vboxdrv.ko
#13 0xc97a9305 in supdrvIOCtl () from /boot/modules/vboxdrv.ko
#14 0xc97a4e3a in VBoxDrvFreeBSDIOCtl () from /boot/modules/vboxdrv.ko
#15 0xc07f48d8 in devfs_ioctl_f (fp=0xc6dd4738, com=3364738944, data=0xc88dd780, cred=0xc7adbe00, td=0xc99c9480)
   at /usr/src/sys/fs/devfs/devfs_vnops.c:658
#16 0xc08b4a7d in kern_ioctl (td=0xc99c9480, fd=22, com=3223868955, data=0xc88dd780 "birddrib(") at file.h:262
#17 0xc08b4c04 in ioctl (td=0xc99c9480, uap=0xeab19cf8) at /usr/src/sys/kern/sys_generic.c:677
#18 0xc0b8ed93 in syscall (frame=0xeab19d38) at /usr/src/sys/i386/i386/trap.c:1073
#19 0xc0b71fd0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)

The host:

FreeBSD 8.0-CURRENT #8: Tue Jun 16 14:10:29 EDT 2009
   root@sorrow.ashke.com:/usr/obj/usr/src/sys/GENERIC
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3191.87-MHz 686-class CPU)
 Origin = "GenuineIntel"  Id = 0xf64  Stepping = 4
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 Features2=0xe43d<SSE3,DTES64,MON,DS_CPL,VMX,CNXT-ID,CX16,xTPR,PDCM>
 AMD Features=0x20100000<NX,LM>
 AMD Features2=0x1<LAHF>
 TSC: P-state invariant
real memory  = 2147483648 (2048 MB)
avail memory = 2086457344 (1989 MB)

And I am running virtualbox-2.2.51.r20457_2. 

I have tested this, and experienced the crash, on two systems.  The first is that dualcore xeon (P4 based) with Virtualization enabled in the BIOS.  The second is a core 2 duo, which does not have such an option in the BIOS.  Both are running 8.0-CURRENT as of Tuesday, June 16th, 2009.

>How-To-Repeat:

Install FreeBSD.  Upgrade to 8.0-CURRENT.  Install virtualbox from the ports tree.  Load the vboxdrv kernel module and then try to start a VM.
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-ports-bugs->freebsd-bugs 
Responsible-Changed-By: vwe 
Responsible-Changed-When: Thu Jun 18 12:06:21 UTC 2009 
Responsible-Changed-Why:  
not a ports bug 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135692 
State-Changed-From-To: open->feedback 
State-Changed-By: vwe 
State-Changed-When: Thu Jun 18 20:54:21 UTC 2009 
State-Changed-Why:  
Adam, 
while checking your issue for a moment, it seems like your issue 
should not be seen when using a kernel w/o INVARIANTS support. 
That does not mean, it will work properly and is the suggested way. 
To get the real source of trouble, I would like to see a full backtrace. 
Please do the following (while vboxdrv.ko is loaded): 
``asf -ks `sysctl -n kern.module_path` kernel.asf '' 
kgdb /boot/kernel/kernel /path/to/dump/dir 
and inside kgdb: 
add-symbol-file kernel.asf 
bt full 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135692 

From: Adam K Kirchhoff <adamk@voicenet.com>
To: bug-followup@FreeBSD.org, adamk@voicenet.com
Cc:  
Subject: Re: kern/135692: [mtx] [witness] blockable sleep lock - kernel panic
 on 8.0-CURRENT
Date: Fri, 19 Jun 2009 09:30:46 -0400

 asf -ks `sysctl -n kern.module_path` kernel.asf
 
 worked without any errors.  It created /root/kernel.asf:
 
 [ root@sorrow - ~ ]: ls -l /root/kernel.asf
 -rw-r--r--  1 root  wheel  1067 Jun 18 17:13 /root/kernel.asf
 
 But when I try to add it in kgdb, this is what I get:
 
 (kgdb) add-symbol-file /root/kernel.asf
 add symbol table from file "/root/kernel.asf" at
 (y or n) y
 "/root/kernel.asf": can't read symbols: File format not recognized.
 
 However, I got the gist of things and simply copied and pasted each line 
 of the kernel.asf file into kgdb.  It adds the symbols for each module 
 without any errors.  'bt full' however, only returns a few lines:
 
 Here's the full output from kgdb:
 
 [ root@sorrow - ~ ]: kgdb /boot/kernel/kernel /var/crash/vmcore.7
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you 
 are
 welcome to change it and/or distribute copies of it under certain 
 conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 panic: blockable sleep lock (sleep mutex) 32 @ 
 /usr/src/sys/vm/uma_core.c:1990
 cpuid = 2
 KDB: enter: panic
 Uptime: 1h57m3s
 Physical memory: 2029 MB
 Dumping 252 MB: 237 221 205 189 173 157 141 125 109 93 77 61 45 29 13
 
 Reading symbols from /boot/kernel/linux.ko...Reading symbols from 
 /boot/kernel/linux.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/linux.ko
 Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from 
 /boot/kernel/snd_hda.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/snd_hda.ko
 Reading symbols from /boot/kernel/sound.ko...Reading symbols from 
 /boot/kernel/sound.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/sound.ko
 Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from 
 /boot/kernel/accf_http.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/accf_http.ko
 Reading symbols from /boot/kernel/aio.ko...Reading symbols from 
 /boot/kernel/aio.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/aio.ko
 Reading symbols from /boot/modules/kqemu.ko...done.
 Loaded symbols for /boot/modules/kqemu.ko
 Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from 
 /boot/kernel/linprocfs.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/linprocfs.ko
 Reading symbols from /boot/kernel/smbfs.ko...Reading symbols from 
 /boot/kernel/smbfs.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/smbfs.ko
 Reading symbols from /boot/kernel/libiconv.ko...Reading symbols from 
 /boot/kernel/libiconv.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/libiconv.ko
 Reading symbols from /boot/kernel/libmchain.ko...Reading symbols from 
 /boot/kernel/libmchain.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/libmchain.ko
 Reading symbols from /boot/kernel/radeon.ko...Reading symbols from 
 /boot/kernel/radeon.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/radeon.ko
 Reading symbols from /boot/kernel/drm.ko...Reading symbols from 
 /boot/kernel/drm.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/drm.ko
 Reading symbols from /boot/modules/vboxdrv.ko...done.
 Loaded symbols for /boot/modules/vboxdrv.ko
 #0  doadump () at pcpu.h:246
 246    pcpu.h: No such file or directory.
    in pcpu.h
 (kgdb) add-symbol-file /boot/kernel/linux.ko.symbols 0xc10f2a90 -s .data 
 0xc110c000 -s .bss 0xc110ed54
 add symbol table from file "/boot/kernel/linux.ko.symbols" at
    .text_addr = 0xc10f2a90
    .data_addr = 0xc110c000
    .bss_addr = 0xc110ed54
 (y or n) y
 Reading symbols from /boot/kernel/linux.ko.symbols...done.
 (kgdb) add-symbol-file /boot/kernel/snd_hda.ko.symbols 0xc111c740 -s 
 .data 0xc112fb20 -s .bss 0xc112ff3c
 add symbol table from file "/boot/kernel/snd_hda.ko.symbols" at
    .text_addr = 0xc111c740
    .data_addr = 0xc112fb20
    .bss_addr = 0xc112ff3c
 (y or n) y
 Reading symbols from /boot/kernel/snd_hda.ko.symbols...done.
 (kgdb) add-symbol-file /boot/kernel/sound.ko.symbols 0xc1148fb0 -s .data 
 0xc117f000 -s .bss 0xc1189a20
 add symbol table from file "/boot/kernel/sound.ko.symbols" at
    .text_addr = 0xc1148fb0
    .data_addr = 0xc117f000
    .bss_addr = 0xc1189a20
 (y or n) y
 Reading symbols from /boot/kernel/sound.ko.symbols...done.
 (kgdb) add-symbol-file /boot/kernel/accf_http.ko.symbols 0xc11956b0 -s 
 .data 0xc1196ca0 -s .bss 0xc1196e24
 add symbol table from file "/boot/kernel/accf_http.ko.symbols" at
    .text_addr = 0xc11956b0
    .data_addr = 0xc1196ca0
    .bss_addr = 0xc1196e24
 (y or n) y
 Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
 (kgdb) add-symbol-file /boot/kernel/linprocfs.ko.symbols 0xc5c9d9f0 -s 
 .data 0xc5ca2480 -s .bss 0xc5ca2740
 add symbol table from file "/boot/kernel/linprocfs.ko.symbols" at
    .text_addr = 0xc5c9d9f0
    .data_addr = 0xc5ca2480
    .bss_addr = 0xc5ca2740
 (y or n) y
 Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
 (kgdb) add-symbol-file /boot/kernel/smbfs.ko.symbols 0xc5e9d170 -s .data 
 0xc5eb43c0 -s .bss 0xc5eb4de0
 add symbol table from file "/boot/kernel/smbfs.ko.symbols" at
    .text_addr = 0xc5e9d170
    .data_addr = 0xc5eb43c0
    .bss_addr = 0xc5eb4de0
 (y or n) y
 Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
 (kgdb) add-symbol-file /boot/kernel/libiconv.ko.symbols 0xc5eb6250 -s 
 .data 0xc5eb87e0 -s .bss 0xc5eb8c4c
 add symbol table from file "/boot/kernel/libiconv.ko.symbols" at
    .text_addr = 0xc5eb6250
    .data_addr = 0xc5eb87e0
    .bss_addr = 0xc5eb8c4c
 (y or n) y
 Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
 (kgdb) add-symbol-file /boot/kernel/libmchain.ko.symbols 0xc5f3c640 -s 
 .data 0xc5f3e410 -s .bss 0xc5f3e4a8
 add symbol table from file "/boot/kernel/libmchain.ko.symbols" at
    .text_addr = 0xc5f3c640
    .data_addr = 0xc5f3e410
    .bss_addr = 0xc5f3e4a8
 (y or n) y
 Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
 (kgdb) add-symbol-file /boot/modules/vboxdrv.ko 0xc6072b30 -s .data 
 0xc608d000 -s .bss 0xc608eb00
 add symbol table from file "/boot/modules/vboxdrv.ko" at
    .text_addr = 0xc6072b30
    .data_addr = 0xc608d000
    .bss_addr = 0xc608eb00
 (y or n) y
 Reading symbols from /boot/modules/vboxdrv.ko...done.
 (kgdb) add-symbol-file /boot/kernel/radeon.ko.symbols 0xc6563b60 -s 
 .data 0xc65b6440 -s .bss 0xc65b7d20
 add symbol table from file "/boot/kernel/radeon.ko.symbols" at
    .text_addr = 0xc6563b60
    .data_addr = 0xc65b6440
    .bss_addr = 0xc65b7d20
 (y or n) y
 Reading symbols from /boot/kernel/radeon.ko.symbols...done.
 (kgdb) add-symbol-file /boot/kernel/drm.ko.symbols 0xc65c08b0 -s .data 
 0xc65ce580 -s .bss 0xc65cf6a4
 add symbol table from file "/boot/kernel/drm.ko.symbols" at
    .text_addr = 0xc65c08b0
    .data_addr = 0xc65ce580
    .bss_addr = 0xc65cf6a4
 (y or n) y
 Reading symbols from /boot/kernel/drm.ko.symbols...done.
 (kgdb) bt full
 #0  doadump () at pcpu.h:246
 No locals.
 #1  0xc0870a8e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
    _giantcnt = Variable "_giantcnt" is not available.
 
 
 So 'bt full' only gives me the giantcnt error.  Just a regular 'bt' 
 gives me the same output I posted in my pr.
 
 Am I doing something wrong?
 
 I can possibly try a -CURRENT kernel without invariants in the next day 
 or two, if you think that will make a difference, or just want to 
 confirm your thought.
 

From: Adam K Kirchhoff <adamk@voicenet.com>
To: vwe@freebsd.org, bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/135692: [mtx] [witness] blockable sleep lock - kernel panic
 on 8.0-CURRENT
Date: Sun, 21 Jun 2009 08:51:37 -0400

 You were correct that this problem does not happen with invariants 
 disabled.  However, I am willing to test anything you'd like with 
 invariants enabled again to help fix the source of the problem rather 
 than work around it.
 
 Adam
 
Responsible-Changed-From-To: freebsd-bugs->jh 
Responsible-Changed-By: jh 
Responsible-Changed-When: Sun Sep 19 15:16:39 UTC 2010 
Responsible-Changed-Why:  
Do you still see this on 8.1? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135692 
State-Changed-From-To: feedback->closed 
State-Changed-By: jh 
State-Changed-When: Sat Oct 30 08:09:23 UTC 2010 
State-Changed-Why:  
Feedback timeout. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135692 
>Unformatted:
