From nobody@FreeBSD.org  Thu Aug 20 20:29:14 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id E6D751065672
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 20 Aug 2009 20:29:14 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id CACC28FC51
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 20 Aug 2009 20:29:14 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n7KKTE6m051769
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 20 Aug 2009 20:29:14 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n7KKTEk3051768;
	Thu, 20 Aug 2009 20:29:14 GMT
	(envelope-from nobody)
Message-Id: <200908202029.n7KKTEk3051768@www.freebsd.org>
Date: Thu, 20 Aug 2009 20:29:14 GMT
From: Peter Cornelius <pcc@gmx.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [lor]
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         138002
>Category:       kern
>Synopsis:       [lor] Three lock order reversals: ufs/devfs, bufwait/dirhash, bufwait/ufs
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Aug 20 20:30:01 UTC 2009
>Closed-Date:    
>Last-Modified:  Sun Aug 30 10:00:07 UTC 2009
>Originator:     Peter Cornelius
>Release:        FreeBSD 8.0-BETA2 #4: Mon Aug 17 13:46:34 UTC 2009 (also FreeBSD 8.0-BETA2 #3: Sun Aug 16 10:28:08 UTC 2009)
>Organization:
>Environment:
FreeBSD netserv.ka.cornelius 8.0-BETA2 FreeBSD 8.0-BETA2 #4: Mon Aug 17 13:46:34 UTC 2009     root@netserv.ka.cornelius:/usr/Obj-RELENG_8/usr/Src-RELENG_8/sys/NETSERV  i386

(but also seen on FreeBSD 8.0-BETA2 #3: Sun Aug 16 10:28:08 UTC 2009)
>Description:
*** Observed three lock order reversals almost in a row. All but the second I (naively) think already have been reported elsewhere.

lock order reversal:
 1st 0xc561a270 ufs (ufs) @ /usr/Src-RELENG_8/sys/kern/vfs_mount.c:1054
 2nd 0xc561a058 devfs (devfs) @ /usr/Src-RELENG_8/sys/kern/vfs_subr.c:2083
KDB: stack backtrace:
db_trace_self_wrapper(c0c1fe7c,e7876814,c08be2a5,c08aee9b,c0c22de6,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08aee9b,c0c22de6,c4d1f158,c4d1f088,e7876870,...) at kdb_backtrace+0x29
_witness_debugger(c0c22de6,c561a058,c0c1176c,c4d1f088,c0c2a1f8,...) at _witness_debugger+0x25
witness_checkorder(c561a058,9,c0c2a1f8,823,0,...) at witness_checkorder+0x839
__lockmgr_args(c561a058,80100,c561a074,0,0,...) at __lockmgr_args+0x7a7
vop_stdlock(e7876978,c08be04b,c0c119af,80100,c561a000,...) at vop_stdlock+0x62
VOP_LOCK1_APV(c0cfd660,e7876978,c52059a4,c0d38160,c561a000,...) at VOP_LOCK1_APV+0xb5
_vn_lock(c561a000,80100,c0c2a1f8,823,8,...) at _vn_lock+0x5e
vget(c561a000,80100,c5205900,15e,c0c118c8,...) at vget+0xb9
devfs_allocv(c55e2500,c519aa10,e7876a10,9d,c0edf078,...) at devfs_allocv+0x102
devfs_root(c519aa10,80000,e7876c30,42c,0,...) at devfs_root+0x4a
vfs_donmount(c5205900,0,c5226180,c5226180,bfbfd7f9,...) at vfs_donmount+0x14c2
nmount(c5205900,e7876cf8,c,c,c0d033b8,...) at nmount+0x75
syscall(e7876d38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (378, FreeBSD ELF32, nmount), eip = 0x280e876b, esp = 0xbfbfd7cc, ebp = 0xbfbfdd28 ---
lock order reversal:
 1st 0xd8d299f0 bufwait (bufwait) @ /usr/Src-RELENG_8/sys/kern/vfs_bio.c:2559
 2nd 0xc518e200 dirhash (dirhash) @ /usr/Src-RELENG_8/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
db_trace_self_wrapper(c0c1fe7c,e7894a74,c08be2a5,c08aee9b,c0c22de6,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08aee9b,c0c22de6,c4d1bfc8,c4d1f1c0,e7894ad0,...) at kdb_backtrace+0x29
_witness_debugger(c0c22de6,c518e200,c0c42b70,c4d1f1c0,c0c42800,...) at _witness_debugger+0x25
witness_checkorder(c518e200,9,c0c42800,11d,0,...) at witness_checkorder+0x839
_sx_xlock(c518e200,0,c0c42800,11d,d9c2b36c,...) at _sx_xlock+0x85
ufsdirhash_acquire(0,e,c51b3000,d8d29990,d9c2b36c,...) at ufsdirhash_acquire+0x35
ufsdirhash_remove(c57050e8,d9c2b36c,36c,e7894b60,e7894b5c,...) at ufsdirhash_remove+0x14
ufs_dirremove(c56f410c,c56d8a6c,500800c,0,c56f410c,...) at ufs_dirremove+0xe5
ufs_remove(e7894c34,0,0,0,c571f53c,...) at
ufs_remove+0x6e
VOP_REMOVE_APV(c0d1f760,e7894c34,c571f53c,e7894c0c,80f5200,...) at VOP_REMOVE_APV+0xa5
kern_unlinkat(c570eb40,ffffff9c,80f5200,0,e7894c80,...) at kern_unlinkat+0x181
kern_unlink(c570eb40,80f5200,0,e7894d2c,c0b5b453,...) at kern_unlink+0x27
unlink(c570eb40,e7894cf8,4,c0c3d0bc,c0d00b78,...) at unlink+0x22
syscall(e7894d38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (10, FreeBSD ELF32, unlink), eip = 0x283936cf, esp = 0xbfbfc70c, ebp = 0xbfbfc738 ---
lagg0: link state changed to DOWN
sk1: link state changed to DOWN
vlan0: link state changed to DOWN
vlan2: link state changed to DOWN
vlan3: link state changed to DOWN
vlan1: link state changed to DOWN
sk1: link state changed to UP
CURVNET_SET(0xc4d450e0) in arp_ifscrub() on cpu 0, prev 0xc4d450e0 in soo_ioctl()
lock order reversal:
 1st 0xc5ab0164 ufs (ufs) @ /usr/Src-RELENG_8/sys/kern/vfs_subr.c:2083
 2nd 0xd8e8e8a0 bufwait (bufwait) @ /usr/Src-RELENG_8/sys/ufs/ffs/ffs_softdep.c:6177
 3rd 0xc6e8a6a0 ufs (ufs) @ /usr/Src-RELENG_8/sys/kern/vfs_subr.c:2083
KDB: stack backtrace:
db_trace_self_wrapper(c0c1fe7c,e78fd3cc,c08be2a5,c08aee9b,c0c22dff,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08aee9b,c0c22dff,c4d1bfc8,c4d1f158,e78fd428,...) at kdb_backtrace+0x29
_witness_debugger(c0c22dff,c6e8a6a0,c0c15598,c4d1f158,c0c2a1f8,...) at _witness_debugger+0x25
witness_checkorder(c6e8a6a0,9,c0c2a1f8,823,0,...) at witness_checkorder+0x839
__lockmgr_args(c6e8a6a0,80100,c6e8a6bc,0,0,...) at __lockmgr_args+0x7a7
ffs_lock(e78fd538,c08be04b,c0c296d0,80100,c6e8a648,...) at ffs_lock+0x8a
VOP_LOCK1_APV(c0d1f760,e78fd538,c5a36764,c0d38160,c6e8a648,...) at VOP_LOCK1_APV+0xb5
_vn_lock(c6e8a648,80100,c0c2a1f8,823,4,...) at _vn_lock+0x5e
vget(c6e8a648,80100,c5a366c0,50,0,...) at vget+0xb9
vfs_hash_get(c5459284,9701aa,80000,c5a366c0,e78fd694,...) at vfs_hash_get+0xe6
ffs_vgetf(c5459284,9701aa,80000,e78fd694,1,...) at ffs_vgetf+0x49
softdep_sync_metadata(c5ab010c,0,c0c42470,146,0,...) at softdep_sync_metadata+0x5ba
ffs_syncvnode(c5ab010c,1,c0c1b267,c0c14bcd,3,...) at ffs_syncvnode+0x3e2
ffs_truncate(c5ab010c,200,0,880,c56b3100,...) at ffs_truncate+0x66a
ufs_direnter(c5ab010c,c6e8a648,e78fda1c,e78fdc00,d8f0c300,...) at ufs_direnter+0x8f6
ufs_mkdir(e78fdc28,e78fdc3c,0,0,e78fdb6c,...) at ufs_mkdir+0x8a7
VOP_MKDIR_APV(c0d1f760,e78fdc28,e78fdc00,e78fdb6c,0,...) at VOP_MKDIR_APV+0xa5
kern_mkdirat(c5a366c0,ffffff9c,bfbfe9fd,0,1ff,...) at kern_mkdirat+0x22b
kern_mkdir(c5a366c0,bfbfe9fd,0,1ff,e78fdd2c,...) at kern_mkdir+0x2e
mkdir(c5a366c0,e78fdcf8,8,c0c23750,c0d01940,...) at mkdir+0x29
syscall(e78fdd38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (136, FreeBSD ELF32, mkdir), eip = 0x281698e3, esp = 0xbfbfe78c, ebp = 0xbfbfe858 ---



*** Corresponding kernel dmesg.

Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-BETA2 #4: Mon Aug 17 13:46:34 UTC 2009
    root@netserv.ka.cornelius:/usr/Obj-RELENG_8/usr/Src-RELENG_8/sys/NETSERV
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel Pentium III (1004.52-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x686  Stepping = 6
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 1610612736 (1536 MB)
avail memory = 1559441408 (1487 MB)
ACPI APIC Table: <ASUS   CUV4X-D >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s)
 cpu0 (BSP): APIC ID:  3
 cpu1 (AP): APIC ID:  0
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <ASUS CUV4X-D> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 5ff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0xe408-0xe40b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 82C691 (Apollo Pro) host to PCI bridge> on hostb0
agp0: aperture size is 256M
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xd800-0xd8ff mem 0xfa000000-0xfaffffff,0xf9800000-0xf9800fff irq 16 at device 0.0 on
pci1
isab0: <PCI-ISA bridge> at device 4.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 82C686B UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xb800-0xb80f at device 4.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xb400-0xb41f irq 5 at device 4.2 on pci0
uhci0: [ITHREAD]
usbus0: <VIA 83C572 USB controller> on uhci0
uhci1: <VIA 83C572 USB controller> port 0xb000-0xb01f irq 5 at device 4.3 on pci0
uhci1: [ITHREAD]
usbus1: <VIA 83C572 USB controller> on uhci1
skc0: <D-Link DGE-530T Gigabit Ethernet> port 0xa800-0xa8ff mem 0xf9000000-0xf9003fff at device 10.0 on pci0
skc0: DGE-530T Gigabit Ethernet Adapter rev. (0x1)
sk0: <Marvell Semiconductor, Inc. Yukon> on skc0
sk0: Ethernet address: 00:13:46:6a:2d:f4
miibus0: <MII bus> on sk0
e1000phy0: <Marvell 88E1011 Gigabit PHY> PHY 0 on miibus0
e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
skc0: [ITHREAD]
skc1: <D-Link DGE-530T Gigabit Ethernet> port 0xa400-0xa4ff mem 0xf8800000-0xf8803fff at device 11.0 on pci0
skc1: DGE-530T Gigabit Ethernet Adapter rev. (0x1)
sk1: <Marvell Semiconductor, Inc. Yukon> on skc1
sk1: Ethernet address: 00:13:46:6a:2d:c6
miibus1: <MII bus> on sk1
e1000phy1: <Marvell 88E1011 Gigabit PHY> PHY 0 on miibus1
e1000phy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
skc1: [ITHREAD]
atapci1: <Promise PDC20619 UDMA133 controller> port 0xa000-0xa07f,0x9800-0x980f,0x9400-0x947f mem 0xf8000000-0xf8000fff,0xf7800000-0xf781ffff irq 16 at device 12.0 on pci0
atapci1: [ITHREAD]
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci1
ata5: [ITHREAD]
ahc0: <Adaptec 2940 SCSI adapter> port 0x9000-0x90ff mem 0xf7000000-0xf7000fff irq 19 at device 13.0 on pci0
ahc0: [ITHREAD]
aic7870: Single Channel A, SCSI Id=7, 16/253 SCBs
atrtc0: <AT realtime clock> port 0x70-0x73 irq 8 on acpi0
fdc1: <floppy drive controller> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc1: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77b irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xce7ff,0xd8000-0xd87ff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
fdc0: No FDOUT register!
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
acd0: DVDR <HL-DT-ST DVDRAM GSA-4040B/A301> at ata1-master UDMA33
ad4: 305245MB <Seagate ST3320620A 3.AAC> at ata2-master UDMA100
ad6: 305245MB <Seagate ST3320620A 3.AAC> at ata3-master UDMA100
Waiting 5 seconds for SCSI devices to settle
ugen0.1: <VIA> at usbus0
uhub0: <VIA UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
GEOM: ad4s1: geometry does not match label (255h,63s != 16h,63s).
GEOM: ad6s1: geometry does not match label (255h,63s != 16h,63s).
uhub0: 2 ports with 2 removable, self powered
ugen1.1: <VIA> at usbus1
uhub1: <VIA UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub1: 2 ports with 2 removable, self powered
ar0: 305175MB <Promise Fasttrak RAID1> status: READY
ar0: disk0 READY (master) using ad4 at ata2-master
ar0: disk1 READY (mirror) using ad6 at ata3-master
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/ar0s1a
GEOM_ELI: Device ar0s1e.eli created.
GEOM_ELI: Encryption: AES-CBC 128
GEOM_ELI:     Crypto: software
GEOM_ELI: Device ar0s1f.eli created.
GEOM_ELI: Encryption: AES-CBC 128
GEOM_ELI:     Crypto: software
GEOM_ELI: Device ar0s1d.eli created.
GEOM_ELI: Encryption: AES-CBC 128
GEOM_ELI:     Crypto: software
GEOM_ELI: Device ar0s1b.eli created.
GEOM_ELI: Encryption: AES-CBC 256
GEOM_ELI:     Crypto: software


*** Kernel config file is GENERIC with a few minor changes (please note that build no. 4 was the first one with VIMAGE turned on):

# diff /usr/src/sys/i386/conf/GENERIC /usr/src/sys/i386/conf/NETSERV
21,22c21,22
< cpu           I486_CPU
< cpu           I586_CPU
---
> #cpu          I486_CPU
> #cpu          I586_CPU
24c24,25
< ident         GENERIC
---
> ident         VNETSERV
> #ident                NETSERV
41c42,43
< options       SCTP                    # Stream Control Transmission Protocol
---
> # SCTP is not yet compatible with VIMAGE.
> #options      SCTP                    # Stream Control Transmission Protocol
343a346,354
>
> options               SC_NORM_ATTR=(FG_GREEN|BG_BLACK)
> options               SC_NORM_REV_ATTR=(FG_YELLOW|BG_GREEN)
> options               SC_KERNEL_CONS_ATTR=(FG_RED|BG_BLACK)
> options               SC_KERNEL_CONS_REV_ATTR=(FG_BLACK|BG_RED)
>
> # Network stack virtualisation.
> options               VIMAGE
> options               VNET_DEBUG


*** NB. This LOR, I have also found earlier on the prior kernel build, from the log:

Aug 16 13:51:56 netserv syslogd: kernel boot file is /boot/kernel/kernel
Aug 16 13:51:56 netserv kernel: Copyright (c) 1992-2009 The FreeBSD Project.
Aug 16 13:51:56 netserv kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Aug 16 13:51:56 netserv kernel: The Regents of the University of California. All rights reserved.
Aug 16 13:51:56 netserv kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Aug 16 13:51:56 netserv kernel: FreeBSD 8.0-BETA2 #3: Sun Aug 16 10:28:08 UTC 2009
Aug 16 13:51:56 netserv kernel: root@netserv.ka.cornelius:/usr/Obj-RELENG_8/usr/Src-RELENG_8/sys/NETSERV
Aug 16 13:51:56 netserv kernel: WARNING: WITNESS option enabled, expect reduced performance.
Aug 16 13:51:56 netserv kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Aug 16 13:51:56 netserv kernel: CPU: Intel Pentium III (1004.52-MHz 686-class CPU)
Aug 16 13:51:56 netserv kernel: Origin = "GenuineIntel"  Id = 0x686  Stepping = 6
Aug 16 13:51:56 netserv kernel: Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MM
X,FXSR,SSE>
Aug 16 13:51:56 netserv kernel: real memory  = 1610612736 (1536 MB)
Aug 16 13:51:56 netserv kernel: avail memory = 1559375872 (1487 MB)
Aug 16 13:51:56 netserv kernel: ACPI APIC Table: <ASUS   CUV4X-D >
Aug 16 13:51:56 netserv kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Aug 16 13:51:56 netserv kernel: FreeBSD/SMP: 2 package(s) x 1 core(s)
Aug 16 13:51:56 netserv kernel: cpu0 (BSP): APIC ID:  3
Aug 16 13:51:56 netserv kernel: cpu1 (AP): APIC ID:  0
Aug 16 13:51:56 netserv kernel: ioapic0 <Version 1.1> irqs 0-23 on motherboard
Aug 16 13:51:56 netserv kernel: kbd1 at kbdmux0
Aug 16 13:51:56 netserv kernel: cryptosoft0: <software crypto> on motherboard
Aug 16 13:51:56 netserv kernel: acpi0: <ASUS CUV4X-D> on motherboard
Aug 16 13:51:56 netserv kernel: acpi0: [ITHREAD]
Aug 16 13:51:56 netserv kernel: acpi0: Power Button (fixed)
Aug 16 13:51:56 netserv kernel: acpi0: reservation of 0, a0000 (3) failed
Aug 16 13:51:56 netserv kernel: acpi0: reservation of 100000, 5ff00000 (3) failed
Aug 16 13:51:56 netserv kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
Aug 16 13:51:56 netserv kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0xe408-0xe40b on acpi0
Aug 16 13:51:56 netserv kernel: acpi_button0: <Power Button> on acpi0
Aug 16 13:51:56 netserv kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Aug 16 13:51:56 netserv kernel: pci0: <ACPI PCI bus> on pcib0
Aug 16 13:51:56 netserv kernel: agp0: <VIA 82C691 (Apollo Pro) host to PCI bridge> on hostb0
Aug 16 13:51:56 netserv kernel: agp0: aperture size is 256M
Aug 16 13:51:56 netserv kernel: pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
Aug 16 13:51:56 netserv kernel: pci1: <ACPI PCI bus> on pcib1
Aug 16 13:51:56 netserv kernel: vgapci0: <VGA-compatible display> port 0xd800-0xd8ff mem 0xfa000000-0xfaffffff,0xf9800000-0xf
9800fff irq 16 at device 0.0 on pci1
Aug 16 13:51:56 netserv kernel: isab0: <PCI-ISA bridge> at device 4.0 on pci0
Aug 16 13:51:56 netserv kernel: isa0: <ISA bus> on isab0
Aug 16 13:51:56 netserv kernel: atapci0: <VIA 82C686B UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xb800-0xb
80f at device 4.1 on pci0
Aug 16 13:51:56 netserv kernel: ata0: <ATA channel 0> on atapci0
Aug 16 13:51:56 netserv kernel: ata0: [ITHREAD]
Aug 16 13:51:56 netserv kernel: ata1: <ATA channel 1> on atapci0
Aug 16 13:51:56 netserv kernel: ata1: [ITHREAD]
Aug 16 13:51:56 netserv kernel: uhci0: <VIA 83C572 USB controller> port 0xb400-0xb41f irq 5 at device 4.2 on pci0
Aug 16 13:51:56 netserv kernel: uhci0: [ITHREAD]
Aug 16 13:51:56 netserv kernel: usbus0: <VIA 83C572 USB controller> on uhci0
Aug 16 13:51:56 netserv kernel: uhci1: <VIA 83C572 USB controller> port 0xb000-0xb01f irq 5 at device 4.3 on pci0
Aug 16 13:51:56 netserv kernel: uhci1: [ITHREAD]
Aug 16 13:51:56 netserv kernel: usbus1: <VIA 83C572 USB controller> on uhci1
Aug 16 13:51:56 netserv kernel: skc0: <D-Link DGE-530T Gigabit Ethernet> port 0xa800-0xa8ff mem 0xf9000000-0xf9003fff at devi
ce 10.0 on pci0
Aug 16 13:51:56 netserv kernel: skc0: DGE-530T Gigabit Ethernet Adapter rev. (0x1)
Aug 16 13:51:56 netserv kernel: sk0: <Marvell Semiconductor, Inc. Yukon> on skc0
Aug 16 13:51:56 netserv kernel: sk0: Ethernet address: 00:13:46:6a:2d:f4
Aug 16 13:51:56 netserv kernel: miibus0: <MII bus> on sk0
Aug 16 13:51:56 netserv kernel: e1000phy0: <Marvell 88E1011 Gigabit PHY> PHY 0 on miibus0
Aug 16 13:51:56 netserv kernel: e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
Aug 16 13:51:56 netserv kernel: skc0: [ITHREAD]
Aug 16 13:51:56 netserv kernel: skc1: <D-Link DGE-530T Gigabit Ethernet> port 0xa400-0xa4ff mem 0xf8800000-0xf8803fff at device 11.0 on pci0
Aug 16 13:51:56 netserv kernel: skc1: DGE-530T Gigabit Ethernet Adapter rev. (0x1)
Aug 16 13:51:56 netserv kernel: sk1: <Marvell Semiconductor, Inc. Yukon> on skc1
Aug 16 13:51:56 netserv kernel: sk1: Ethernet address: 00:13:46:6a:2d:c6
Aug 16 13:51:56 netserv kernel: miibus1: <MII bus> on sk1
Aug 16 13:51:56 netserv kernel: e1000phy1: <Marvell 88E1011 Gigabit PHY> PHY 0 on miibus1
Aug 16 13:51:56 netserv kernel: e1000phy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
Aug 16 13:51:56 netserv kernel: skc1: [ITHREAD]
Aug 16 13:51:56 netserv kernel: atapci1: <Promise PDC20619 UDMA133 controller> port 0xa000-0xa07f,0x9800-0x980f,0x9400-0x947f mem 0xf8000000-0xf8000fff,0xf7800000-0xf781ffff irq 16 at device 12.0 on pci0
Aug 16 13:51:56 netserv kernel: atapci1: [ITHREAD]
Aug 16 13:51:56 netserv kernel: atapci1: [ITHREAD]
Aug 16 13:51:56 netserv kernel: ata2: <ATA channel 0> on atapci1
Aug 16 13:51:56 netserv kernel: ata2: [ITHREAD]
Aug 16 13:51:56 netserv kernel: ata3: <ATA channel 1> on atapci1
Aug 16 13:51:56 netserv kernel: ata3: [ITHREAD]
Aug 16 13:51:56 netserv kernel: ata4: <ATA channel 2> on atapci1
Aug 16 13:51:56 netserv kernel: ata4: [ITHREAD]
Aug 16 13:51:56 netserv kernel: ata5: <ATA channel 3> on atapci1
Aug 16 13:51:56 netserv kernel: ata5: [ITHREAD]
Aug 16 13:51:56 netserv kernel: ahc0: <Adaptec 2940 SCSI adapter> port 0x9000-0x90ff mem 0xf7000000-0xf7000fff irq 19 at device 13.0 on pci0
Aug 16 13:51:56 netserv kernel: ahc0: [ITHREAD]
Aug 16 13:51:56 netserv kernel: aic7870: Single Channel A, SCSI Id=7, 16/253 SCBs
Aug 16 13:51:56 netserv kernel: atrtc0: <AT realtime clock> port 0x70-0x73 irq 8 on acpi0
Aug 16 13:51:56 netserv kernel: fdc1: <floppy drive controller> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Aug 16 13:51:56 netserv kernel: fdc1: [FILTER]
Aug 16 13:51:56 netserv kernel: ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77b irq 7 drq 3 on acpi0
Aug 16 13:51:56 netserv kernel: ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
Aug 16 13:51:56 netserv kernel: ppc0: FIFO with 16/16/8 bytes threshold
Aug 16 13:51:56 netserv kernel: ppc0: [ITHREAD]
Aug 16 13:51:56 netserv kernel: ppbus0: <Parallel port bus> on ppc0
Aug 16 13:51:56 netserv kernel: plip0: <PLIP network interface> on ppbus0
Aug 16 13:51:56 netserv kernel: plip0: [ITHREAD]
Aug 16 13:51:56 netserv kernel: lpt0: <Printer> on ppbus0
Aug 16 13:51:56 netserv kernel: lpt0: [ITHREAD]
Aug 16 13:51:56 netserv kernel: lpt0: Interrupt-driven port
Aug 16 13:51:56 netserv kernel: ppi0: <Parallel I/O> on ppbus0
Aug 16 13:51:56 netserv kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Aug 16 13:51:56 netserv kernel: uart0: [FILTER]
Aug 16 13:51:56 netserv kernel: uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
Aug 16 13:51:56 netserv kernel: uart1: [FILTER]
Aug 16 13:51:56 netserv kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Aug 16 13:51:56 netserv kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Aug 16 13:51:56 netserv kernel: kbd0 at atkbd0
Aug 16 13:51:56 netserv kernel: atkbd0: [GIANT-LOCKED]
Aug 16 13:51:56 netserv kernel: atkbd0: [ITHREAD]
Aug 16 13:51:56 netserv kernel: cpu0: <ACPI CPU> on acpi0
Aug 16 13:51:56 netserv kernel: cpu1: <ACPI CPU> on acpi0
Aug 16 13:51:56 netserv kernel: pmtimer0 on isa0
Aug 16 13:51:56 netserv kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xce7ff,0xd8000-0xd87ff pnpid ORM0000 on isa0
Aug 16 13:51:56 netserv kernel: sc0: <System console> at flags 0x100 on isa0
Aug 16 13:51:56 netserv kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Aug 16 13:51:56 netserv kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Aug 16 13:51:56 netserv kernel: fdc0: No FDOUT register!
Aug 16 13:51:56 netserv kernel: Timecounters tick every 1.000 msec
Aug 16 13:51:56 netserv kernel: Waiting 5 seconds for SCSI devices to settle
Aug 16 13:51:56 netserv kernel: usbus0: 12Mbps Full Speed USB v1.0
Aug 16 13:51:56 netserv kernel: usbus1: 12Mbps Full Speed USB v1.0
Aug 16 13:51:56 netserv kernel: acd0: DVDR <HL-DT-ST DVDRAM GSA-4040B/A301> at ata1-master UDMA33
Aug 16 13:51:56 netserv kernel: ad4: 305245MB <Seagate ST3320620A 3.AAC> at ata2-master UDMA100
Aug 16 13:51:56 netserv kernel: ad6: 305245MB <Seagate ST3320620A 3.AAC> at ata3-master UDMA100
Aug 16 13:51:56 netserv kernel: ugen0.1: <VIA> at usbus0
Aug 16 13:51:56 netserv kernel: uhub0: <VIA UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
Aug 16 13:51:56 netserv kernel: GEOM: ad4s1: geometry does not match label (255h,63s != 16h,63s).
Aug 16 13:51:56 netserv kernel: GEOM: ad6s1: geometry does not match label (255h,63s != 16h,63s).
Aug 16 13:51:56 netserv kernel: uhub0: 2 ports with 2 removable, self powered
Aug 16 13:51:56 netserv kernel: ugen1.1: <VIA> at usbus1
Aug 16 13:51:56 netserv kernel: uhub1: <VIA UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
Aug 16 13:51:56 netserv kernel: uhub1: 2 ports with 2 removable, self powered
Aug 16 13:51:56 netserv kernel: ar0: 305175MB <Promise Fasttrak RAID1> status: READY
Aug 16 13:51:56 netserv kernel: ar0: disk0 READY (master) using ad4 at ata2-master
Aug 16 13:51:56 netserv kernel: ar0: disk1 READY (mirror) using ad6 at ata3-master
Aug 16 13:51:56 netserv kernel: SMP: AP CPU #1 Launched!
Aug 16 13:51:56 netserv kernel: WARNING: WITNESS option enabled, expect reduced performance.
Aug 16 13:51:56 netserv kernel: Trying to mount root from ufs:/dev/ar0s1a
Aug 16 13:51:56 netserv kernel: GEOM_ELI: Device ar0s1e.eli created.
Aug 16 13:51:56 netserv kernel: GEOM_ELI: Encryption: AES-CBC 128
Aug 16 13:51:56 netserv kernel: GEOM_ELI:     Crypto: software
Aug 16 13:51:56 netserv kernel: GEOM_ELI: Device ar0s1f.eli created.
Aug 16 13:51:56 netserv kernel: GEOM_ELI: Encryption: AES-CBC 128
Aug 16 13:51:56 netserv kernel: GEOM_ELI:     Crypto: software
Aug 16 13:51:56 netserv kernel: GEOM_ELI: Device ar0s1d.eli created.
Aug 16 13:51:56 netserv kernel: GEOM_ELI: Encryption: AES-CBC 128
Aug 16 13:51:56 netserv kernel: GEOM_ELI:     Crypto: software
Aug 16 13:51:56 netserv kernel: GEOM_ELI: Device ar0s1b.eli created.
Aug 16 13:51:56 netserv kernel: GEOM_ELI: Encryption: AES-CBC 256
Aug 16 13:51:56 netserv kernel: GEOM_ELI:     Crypto: software
Aug 16 13:51:56 netserv root: /etc/rc: WARNING: Dump device does not exist.  Savecore not run.
Aug 16 13:51:58 netserv kernel: sk1: link state changed to UP
Aug 16 13:51:58 netserv kernel: lagg0: link state changed to UP
Aug 16 13:51:58 netserv kernel: vlan0: link state changed to UP
Aug 16 13:51:58 netserv kernel: vlan2: link state changed to UP
Aug 16 13:51:58 netserv kernel: vlan3: link state changed to UP
Aug 16 13:51:58 netserv kernel: vlan1: link state changed to UP
Aug 16 13:51:59 netserv root: /etc/rc: WARNING: Ignoring scratch file /usr/local/etc/rc.d/ezjail.sh.orig
Aug 16 13:51:59 netserv sshd[1417]: error: Bind to port 22 on 192.168.10.10 failed: Can't assign requested address.
Aug 16 13:51:59 netserv sshd[1417]: fatal: Cannot bind any address.
Aug 16 13:52:03 netserv kernel: sk0: link state changed to UP
Aug 16 13:52:13 netserv login: ROOT LOGIN (root) ON ttyv1
Aug 16 13:55:17 netserv login: ROOT LOGIN (root) ON ttyv2
Aug 16 13:57:12 netserv login: ROOT LOGIN (root) ON ttyv3
Aug 16 13:58:56 netserv kernel: lock order reversal:
Aug 16 13:58:56 netserv kernel: 1st 0xd8d413f0 bufwait (bufwait) @ /usr/Src-RELENG_8/sys/kern/vfs_bio.c:2559
Aug 16 13:58:56 netserv kernel: 2nd 0xc5782000 dirhash (dirhash) @ /usr/Src-RELENG_8/sys/ufs/ufs/ufs_dirhash.c:285
Aug 16 13:58:56 netserv kernel: KDB: stack backtrace:
Aug 16 13:58:56 netserv kernel: db_trace_self_wrapper(c0c73804,e782c75c,c08c0d25,c08b1a0b,c0c7675b,...) at db_trace_self_wrapper+0x26
Aug 16 13:58:56 netserv kernel: kdb_backtrace(c08b1a0b,c0c7675b,c4d2be90,c4d2f228,e782c7b8,...) at kdb_backtrace+0x29
Aug 16 13:58:56 netserv kernel: _witness_debugger(c0c7675b,c5782000,c0c97efa,c4d2f228,c0c97b8a,...) at _witness_debugger+0x25
Aug 16 13:58:56 netserv kernel: witness_checkorder(c5782000,9,c0c97b8a,11d,0,...) at witness_checkorder+0x839
Aug 16 13:58:56 netserv kernel: _sx_xlock(c5782000,0,c0c97b8a,11d,c58d9ae0,...) at _sx_xlock+0x85
Aug 16 13:58:56 netserv kernel: ufsdirhash_acquire(d8d41390,da0ab800,200,da0ab818,e782c888,...) at ufsdirhash_acquire+0x35
Aug 16 13:58:56 netserv kernel: ufsdirhash_add(c58d9ae0,e782c8d0,818,e782c874,e782c878,...) at ufsdirhash_add+0x13
Aug 16 13:58:56 netserv kernel: ufs_direnter(c596953c,c596696c,e782c8d0,e782cbd0,0,...) at ufs_direnter+0x729
Aug 16 13:58:56 netserv kernel: ufs_makeinode(e782cbd0,0,e782cabc,e782ca18,c0bbc895,...) at
Aug 16 13:58:56 netserv kernel: ufs_makeinode+0x508
Aug 16 13:58:56 netserv kernel: ufs_create(e782cabc,e782cad4,0,0,e782cba4,...) at ufs_create+0x30
Aug 16 13:58:56 netserv kernel: VOP_CREATE_APV(c0d77740,e782cabc,e782cbd0,e782ca54,0,...) at VOP_CREATE_APV+0xa5
Aug 16 13:58:56 netserv kernel: vn_open_cred(e782cba4,e782cc5c,1a4,0,c51d2b00,...) at vn_open_cred+0x215
Aug 16 13:58:56 netserv kernel: vn_open(e782cba4,e782cc5c,1a4,c51fadc8,c0c98f76,...) at vn_open+0x3b
Aug 16 13:58:56 netserv kernel: kern_openat(c5232000,ffffff9c,28412180,0,a02,...) at kern_openat+0x11f
Aug 16 13:58:56 netserv kernel: kern_open(c5232000,28412180,0,a01,1a4,...) at kern_open+0x35
Aug 16 13:58:56 netserv kernel: open(c5232000,e782ccf8,c,c0c77024,c0d55f6c,...) at open+0x30
Aug 16 13:58:56 netserv kernel: syscall(e782cd38) at syscall+0x2a3
Aug 16 13:58:56 netserv kernel: Xint0x80_syscall() at Xint0x80_syscall+0x20
Aug 16 13:58:56 netserv kernel: --- syscall (5, FreeBSD ELF32, open), eip = 0x28335923, esp = 0xbfbfe7bc, ebp = 0xbfbfe7e8 ---

>How-To-Repeat:
no idea at this time. I just boot and wait...
>Fix:


>Release-Note:
>Audit-Trail:

From: "Peter Cornelius" <pcc@gmx.net>
To: FreeBSD-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/138002: [lor]
Date: Thu, 20 Aug 2009 22:41:38 +0200

 --========GMXBoundary238161250800898746752
 Content-Type: text/plain; charset="iso-8859-1"
 Content-Transfer-Encoding: 8bit
 
 Digging up further LORs from the log (build no 3).
 
 Aug 16 14:16:04 netserv kernel: lock order reversal:
 Aug 16 14:16:04 netserv kernel: 1st 0xc6df37ac ufs (ufs) @ 
 /usr/Src-RELENG_8/sys/kern/vfs_lookup.c:497
 Aug 16 14:16:04 netserv kernel: 2nd 0xd8debf40 bufwait (bufwait) @ 
 /usr/Src-RELENG_8/sys/ufs/ffs/ffs_softdep.c:6177
 Aug 16 14:16:04 netserv kernel: 3rd 0xc6e3f164 ufs (ufs) @ 
 /usr/Src-RELENG_8/sys/kern/vfs_subr.c:2083
 Aug 16 14:16:04 netserv kernel: KDB: stack backtrace:
 Aug 16 14:16:04 netserv kernel: 
 db_trace_self_wrapper(c0c73804,e78b33cc,c08c0d25,c08b1a0b,c0c76774,...) at 
 db_trace_self_wrapper+0x26
 Aug 16 14:16:04 netserv kernel: 
 kdb_backtrace(c08b1a0b,c0c76774,c4d2be90,c4d2f1c0,e78b3428,...) at 
 kdb_backtrace+0x29
 Aug 16 14:16:04 netserv kernel: 
 _witness_debugger(c0c76774,c6e3f164,c0c690c7,c4d2f1c0,c0c7db33,...) at 
 _witness_debugger+0x25
 Aug 16 14:16:04 netserv kernel: 
 witness_checkorder(c6e3f164,9,c0c7db33,823,0,...) at 
 witness_checkorder+0x839
 Aug 16 14:16:04 netserv kernel: 
 __lockmgr_args(c6e3f164,80100,c6e3f180,0,0,...) at __lockmgr_args+0x7a7
 Aug 16 14:16:04 netserv kernel: 
 ffs_lock(e78b3538,c08c0acb,c0c7d00b,80100,c6e3f10c,...) at ffs_lock+0x8a
 Aug 16 14:16:04 netserv kernel: 
 VOP_LOCK1_APV(c0d77740,e78b3538,c549d0a4,c0d90140,c6e3f10c,...) at 
 VOP_LOCK1_APV+0xb5
 Aug 16 14:16:04 netserv kernel: _vn_lock(c6e3f10c,80100,c0c7db33,823,4,...) 
 at _vn_lock+0x5e
 Aug 16 14:16:04 netserv kernel: vget(c6e3f10c,80100,c549d000,50,0,...) at 
 vget+0xb9
 Aug 16 14:16:04 netserv kernel: 
 vfs_hash_get(c5488508,44371d7,80000,c549d000,e78b3694,...) at 
 vfs_hash_get+0xe6
 Aug 16 14:16:04 netserv kernel: 
 ffs_vgetf(c5488508,44371d7,80000,e78b3694,1,...) at ffs_vgetf+0x49
 Aug 16 14:16:04 netserv kernel: 
 softdep_sync_metadata(c6df3754,0,c0c977fa,146,0,...) at 
 softdep_sync_metadata+0x5ba
 Aug 16 14:16:04 netserv kernel: 
 ffs_syncvnode(c6df3754,1,c0c6ec41,c0c686fc,3,...) at ffs_syncvnode+0x3e2
 Aug 16 14:16:04 netserv kernel: 
 ffs_truncate(c6df3754,200,0,880,c5623680,...) at ffs_truncate+0x66a
 Aug 16 14:16:04 netserv kernel: 
 ufs_direnter(c6df3754,c6e3f10c,e78b3a1c,e78b3c00,d8db4e30,...) at 
 ufs_direnter+0x8f6
 Aug 16 14:16:04 netserv kernel: 
 ufs_mkdir(e78b3c28,e78b3c3c,0,0,e78b3b6c,...) at ufs_mkdir+0x8a7
 Aug 16 14:16:04 netserv kernel: 
 VOP_MKDIR_APV(c0d77740,e78b3c28,e78b3c00,e78b3b6c,0,...) at 
 VOP_MKDIR_APV+0xa5
 Aug 16 14:16:04 netserv kernel: 
 kern_mkdirat(c549d000,ffffff9c,bfbfeb16,0,1ff,...) at kern_mkdirat+0x22b
 Aug 16 14:16:04 netserv kernel: 
 kern_mkdir(c549d000,bfbfeb16,0,1ff,e78b3d2c,...) at kern_mkdir+0x2e
 Aug 16 14:16:04 netserv kernel: mkdir(c549d000,e78b3cf8,8,c,c0d56dc0,...) 
 at mkdir+0x29
 Aug 16 14:16:05 netserv kernel: syscall(e78b3d38) at syscall+0x2a3
 Aug 16 14:16:05 netserv kernel: Xint0x80_syscall() at Xint0x80_syscall+0x20
 Aug 16 14:16:05 netserv kernel: --- syscall (136, FreeBSD ELF32, mkdir), 
 eip = 0x281698e3, esp = 0xbfbfe86c, ebp = 0xbfbfe938 ---
 Aug 16 16:02:40 netserv su: pc to root on /dev/ttyv2
 Aug 16 16:05:17 netserv sshd[9432]: error: ssh_msg_send: write
 Aug 17 10:48:25 netserv ntpd[49261]: ntpd 4.2.4p5-a (1)
 Aug 17 10:52:11 netserv ntpd[49262]: kernel time sync status change 2001
 Aug 17 15:44:14 netserv sudo: pc : TTY=ttyv2 ; 
 PWD=/usr/Src-RELENG_8/tools/tools/vimage ; USER=root ; 
 COMMAND=/usr/bin/make all install
 Aug 17 15:47:01 netserv pc: /etc/rc.d/jail: WARNING: devfs_domount(): 
 Unable to mount devfs on /usr/jails/spielwiese/dev
 Aug 17 15:47:01 netserv pc: /etc/rc.d/jail: WARNING: devfs_mount_jail: 
 devfs was not mounted on /usr/jails/spielwiese/dev
 Aug 17 15:47:17 netserv sudo: pc : TTY=ttyv2 ; PWD=/usr/home/pc 
 ; USER=root ; COMMAND=/usr/local/bin/ezjail-admin start spielwiese
 Aug 17 15:47:18 netserv kernel: lock order reversal:
 Aug 17 15:47:18 netserv kernel: 1st 0xc901dce8 ufs (ufs) @ 
 /usr/Src-RELENG_8/sys/kern/vfs_mount.c:1054
 Aug 17 15:47:18 netserv kernel: 2nd 0xc9dfa7ac devfs (devfs) @ 
 /usr/Src-RELENG_8/sys/kern/vfs_subr.c:2083
 Aug 17 15:47:18 netserv kernel: KDB: stack backtrace:
 Aug 17 15:47:18 netserv kernel: 
 db_trace_self_wrapper(c0c73804,e78ad814,c08c0d25,c08b1a0b,c0c7675b,...) at 
 db_trace_self_wrapper+0x26
 Aug 17 15:47:18 netserv kernel: 
 kdb_backtrace(c08b1a0b,c0c7675b,c4d2f1c0,c4d2f0f0,e78ad870,...) at 
 kdb_backtrace+0x29
 Aug 17 15:47:18 netserv kernel: 
 _witness_debugger(c0c7675b,c9dfa7ac,c0c6529f,c4d2f0f0,c0c7db33,...) at 
 _witness_debugger+0x25
 Aug 17 15:47:18 netserv kernel: 
 witness_checkorder(c9dfa7ac,9,c0c7db33,823,0,...) at 
 witness_checkorder+0x839
 Aug 17 15:47:18 netserv kernel: 
 __lockmgr_args(c9dfa7ac,80100,c9dfa7c8,0,0,...) at __lockmgr_args+0x7a7
 Aug 17 15:47:18 netserv kernel: 
 vop_stdlock(e78ad978,c08c0acb,c0c654e2,80100,c9dfa754,...) at 
 vop_stdlock+0x62
 Aug 17 15:47:18 netserv kernel: 
 VOP_LOCK1_APV(c0d52ae0,e78ad978,c549d524,c0d90140,c9dfa754,...) at 
 VOP_LOCK1_APV+0xb5
 Aug 17 15:47:18 netserv kernel: _vn_lock(c9dfa754,80100,c0c7db33,823,8,...) 
 at _vn_lock+0x5e
 Aug 17 15:47:18 netserv kernel: 
 vget(c9dfa754,80100,c549d480,15e,c0c653fb,...) at vget+0xb9
 Aug 17 15:47:18 netserv kernel: 
 devfs_allocv(c5ce4c00,c51ca78c,e78ada10,9d,c0f31578,...) at 
 devfs_allocv+0x102
 Aug 17 15:47:18 netserv kernel: 
 devfs_root(c51ca78c,80000,e78adc30,42c,0,...) at devfs_root+0x4a
 Aug 17 15:47:18 netserv kernel: 
 vfs_donmount(c549d480,0,c5ccea80,c5ccea80,bfbfdac9,...) at 
 vfs_donmount+0x14c2
 Aug 17 15:47:18 netserv kernel: nmount(c549d480,e78adcf8,c,c,c0d58838,...) 
 at nmount+0x75
 Aug 17 15:47:18 netserv kernel: syscall(e78add38) at syscall+0x2a3
 Aug 17 15:47:18 netserv kernel: Xint0x80_syscall() at Xint0x80_syscall+0x20
 Aug 17 15:47:18 netserv kernel: --- syscall (378, FreeBSD ELF32, nmount), 
 eip = 0x280e876b, esp = 0xbfbfda9c, ebp = 0xbfbfdff8 ---
 
         
 -- 
 Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3 -
 sicherer, schneller und einfacher! http://portal.gmx.net/de/go/chbrowser
 
 --========GMXBoundary238161250800898746752
 Content-Type: text/html; charset="utf-8"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
     <head />
         <body style=3D"font-family: sans-serif;">
             Digging up further LORs from the log (build no 3).<br /><br />A=
 ug 16 14:16:04 netserv kernel: lock order reversal:<br />Aug 16 14:16:04 ne=
 tserv kernel: 1st 0xc6df37ac ufs (ufs) @ /usr/Src-RELENG_8/sys/kern/vfs_loo=
 kup.c:497<br />Aug 16 14:16:04 netserv kernel: 2nd 0xd8debf40 bufwait (bufw=
 ait) @ /usr/Src-RELENG_8/sys/ufs/ffs/ffs_softdep.c:6177<br />Aug 16 14:16:0=
 4 netserv kernel: 3rd 0xc6e3f164 ufs (ufs) @ /usr/Src-RELENG_8/sys/kern/vfs=
 _subr.c:2083<br />Aug 16 14:16:04 netserv kernel: KDB: stack backtrace:<br =
 />Aug 16 14:16:04 netserv kernel: db_trace_self_wrapper(c0c73804,e78b33cc,c=
 08c0d25,c08b1a0b,c0c76774,...) at db_trace_self_wrapper+0x26<br />Aug 16 14=
 :16:04 netserv kernel: kdb_backtrace(c08b1a0b,c0c76774,c4d2be90,c4d2f1c0,e7=
 8b3428,...) at kdb_backtrace+0x29<br />Aug 16 14:16:04 netserv kernel: _wit=
 ness_debugger(c0c76774,c6e3f164,c0c690c7,c4d2f1c0,c0c7db33,...) at _witness=
 _debugger+0x25<br />Aug 16 14:16:04 netserv kernel: witness_checkorder(c6e3=
 f164,9,c0c7db33,823,0,...) at witness_checkorder+0x839<br />Aug 16 14:16:04=
  netserv kernel: __lockmgr_args(c6e3f164,80100,c6e3f180,0,0,...) at __lockm=
 gr_args+0x7a7<br />Aug 16 14:16:04 netserv kernel: ffs_lock(e78b3538,c08c0a=
 cb,c0c7d00b,80100,c6e3f10c,...) at ffs_lock+0x8a<br />Aug 16 14:16:04 netse=
 rv kernel: VOP_LOCK1_APV(c0d77740,e78b3538,c549d0a4,c0d90140,c6e3f10c,...) =
 at VOP_LOCK1_APV+0xb5<br />Aug 16 14:16:04 netserv kernel: _vn_lock(c6e3f10=
 c,80100,c0c7db33,823,4,...) at _vn_lock+0x5e<br />Aug 16 14:16:04 netserv k=
 ernel: vget(c6e3f10c,80100,c549d000,50,0,...) at vget+0xb9<br />Aug 16 14:1=
 6:04 netserv kernel: vfs_hash_get(c5488508,44371d7,80000,c549d000,e78b3694,=
 ...) at vfs_hash_get+0xe6<br />Aug 16 14:16:04 netserv kernel: ffs_vgetf(c5=
 488508,44371d7,80000,e78b3694,1,...) at ffs_vgetf+0x49<br />Aug 16 14:16:04=
  netserv kernel: softdep_sync_metadata(c6df3754,0,c0c977fa,146,0,...) at so=
 ftdep_sync_metadata+0x5ba<br />Aug 16 14:16:04 netserv kernel: ffs_syncvnod=
 e(c6df3754,1,c0c6ec41,c0c686fc,3,...) at ffs_syncvnode+0x3e2<br />Aug 16 14=
 :16:04 netserv kernel: ffs_truncate(c6df3754,200,0,880,c5623680,...) at ffs=
 _truncate+0x66a<br />Aug 16 14:16:04 netserv kernel: ufs_direnter(c6df3754,=
 c6e3f10c,e78b3a1c,e78b3c00,d8db4e30,...) at ufs_direnter+0x8f6<br />Aug 16 =
 14:16:04 netserv kernel: ufs_mkdir(e78b3c28,e78b3c3c,0,0,e78b3b6c,...) at u=
 fs_mkdir+0x8a7<br />Aug 16 14:16:04 netserv kernel: VOP_MKDIR_APV(c0d77740,=
 e78b3c28,e78b3c00,e78b3b6c,0,...) at VOP_MKDIR_APV+0xa5<br />Aug 16 14:16:0=
 4 netserv kernel: kern_mkdirat(c549d000,ffffff9c,bfbfeb16,0,1ff,...) at ker=
 n_mkdirat+0x22b<br />Aug 16 14:16:04 netserv kernel: kern_mkdir(c549d000,bf=
 bfeb16,0,1ff,e78b3d2c,...) at kern_mkdir+0x2e<br />Aug 16 14:16:04 netserv =
 kernel: mkdir(c549d000,e78b3cf8,8,c,c0d56dc0,...) at mkdir+0x29<br />Aug 16=
  14:16:05 netserv kernel: syscall(e78b3d38) at syscall+0x2a3<br />Aug 16 14=
 :16:05 netserv kernel: Xint0x80_syscall() at Xint0x80_syscall+0x20<br />Aug=
  16 14:16:05 netserv kernel: --- syscall (136, FreeBSD ELF32, mkdir), eip =
 =3D 0x281698e3, esp =3D 0xbfbfe86c, ebp =3D 0xbfbfe938 ---<br />Aug 16 16:0=
 2:40 netserv su: pc to root on /dev/ttyv2<br />Aug 16 16:05:17 netserv sshd=
 [9432]: error: ssh_msg_send: write<br />Aug 17 10:48:25 netserv ntpd[49261]=
 : ntpd 4.2.4p5-a (1)<br />Aug 17 10:52:11 netserv ntpd[49262]: kernel time =
 sync status change 2001<br />Aug 17 15:44:14 netserv sudo:&nbsp;&nbsp;&nbsp=
 ;&nbsp;&nbsp;&nbsp; pc : TTY=3Dttyv2 ; PWD=3D/usr/Src-RELENG_8/tools/tools/=
 vimage ; USER=3Droot ; COMMAND=3D/usr/bin/make all install<br />Aug 17 15:4=
 7:01 netserv pc: /etc/rc.d/jail: WARNING: devfs_domount(): Unable to mount =
 devfs on /usr/jails/spielwiese/dev<br />Aug 17 15:47:01 netserv pc: /etc/rc=
 .d/jail: WARNING: devfs_mount_jail: devfs was not mounted on /usr/jails/spi=
 elwiese/dev<br />Aug 17 15:47:17 netserv sudo:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
 ;&nbsp; pc : TTY=3Dttyv2 ; PWD=3D/usr/home/pc ; USER=3Droot ; COMMAND=3D/us=
 r/local/bin/ezjail-admin start spielwiese<br />Aug 17 15:47:18 netserv kern=
 el: lock order reversal:<br />Aug 17 15:47:18 netserv kernel: 1st 0xc901dce=
 8 ufs (ufs) @ /usr/Src-RELENG_8/sys/kern/vfs_mount.c:1054<br />Aug 17 15:47=
 :18 netserv kernel: 2nd 0xc9dfa7ac devfs (devfs) @ /usr/Src-RELENG_8/sys/ke=
 rn/vfs_subr.c:2083<br />Aug 17 15:47:18 netserv kernel: KDB: stack backtrac=
 e:<br />Aug 17 15:47:18 netserv kernel: db_trace_self_wrapper(c0c73804,e78a=
 d814,c08c0d25,c08b1a0b,c0c7675b,...) at db_trace_self_wrapper+0x26<br />Aug=
  17 15:47:18 netserv kernel: kdb_backtrace(c08b1a0b,c0c7675b,c4d2f1c0,c4d2f=
 0f0,e78ad870,...) at kdb_backtrace+0x29<br />Aug 17 15:47:18 netserv kernel=
 : _witness_debugger(c0c7675b,c9dfa7ac,c0c6529f,c4d2f0f0,c0c7db33,...) at _w=
 itness_debugger+0x25<br />Aug 17 15:47:18 netserv kernel: witness_checkorde=
 r(c9dfa7ac,9,c0c7db33,823,0,...) at witness_checkorder+0x839<br />Aug 17 15=
 :47:18 netserv kernel: __lockmgr_args(c9dfa7ac,80100,c9dfa7c8,0,0,...) at _=
 _lockmgr_args+0x7a7<br />Aug 17 15:47:18 netserv kernel: vop_stdlock(e78ad9=
 78,c08c0acb,c0c654e2,80100,c9dfa754,...) at vop_stdlock+0x62<br />Aug 17 15=
 :47:18 netserv kernel: VOP_LOCK1_APV(c0d52ae0,e78ad978,c549d524,c0d90140,c9=
 dfa754,...) at VOP_LOCK1_APV+0xb5<br />Aug 17 15:47:18 netserv kernel: _vn_=
 lock(c9dfa754,80100,c0c7db33,823,8,...) at _vn_lock+0x5e<br />Aug 17 15:47:=
 18 netserv kernel: vget(c9dfa754,80100,c549d480,15e,c0c653fb,...) at vget+0=
 xb9<br />Aug 17 15:47:18 netserv kernel: devfs_allocv(c5ce4c00,c51ca78c,e78=
 ada10,9d,c0f31578,...) at devfs_allocv+0x102<br />Aug 17 15:47:18 netserv k=
 ernel: devfs_root(c51ca78c,80000,e78adc30,42c,0,...) at devfs_root+0x4a<br =
 />Aug 17 15:47:18 netserv kernel: vfs_donmount(c549d480,0,c5ccea80,c5ccea80=
 ,bfbfdac9,...) at vfs_donmount+0x14c2<br />Aug 17 15:47:18 netserv kernel: =
 nmount(c549d480,e78adcf8,c,c,c0d58838,...) at nmount+0x75<br />Aug 17 15:47=
 :18 netserv kernel: syscall(e78add38) at syscall+0x2a3<br />Aug 17 15:47:18=
  netserv kernel: Xint0x80_syscall() at Xint0x80_syscall+0x20<br />Aug 17 15=
 :47:18 netserv kernel: --- syscall (378, FreeBSD ELF32, nmount), eip =3D 0x=
 280e876b, esp =3D 0xbfbfda9c, ebp =3D 0xbfbfdff8 ---<br /><br />
         <div class=3D"signature"><br /><br /><br />-- <br />Jetzt kostenlos=
  herunterladen: Internet Explorer 8 und Mozilla Firefox 3 -<br />sicherer, =
 schneller und einfacher! http://portal.gmx.net/de/go/chbrowser</div></body>
     </html>
 
 --========GMXBoundary238161250800898746752--

From: Petr Salinger <Petr.Salinger@seznam.cz>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/138002: [lor] Three lock order reversals: ufs/devfs,
 bufwait/dirhash, bufwait/ufs
Date: Sun, 30 Aug 2009 12:29:01 +0200 (CEST)

 Similarly for me, 8.0 snapshot as of 20090829, in QEMU:
 
 lock order reversal:
   1st 0xc3177250 bufwait (bufwait) @ kern/vfs_bio.c:2559
   2nd 0xc35b2c00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285
 
 lock order reversal:
   1st 0xc35f49c4 ufs (ufs) @ kern/vfs_mount.c:1200
   2nd 0xc35f4df4 devfs (devfs) @ kern/vfs_subr.c:2083
 
 Whole dmesg of boot and reboot:
 
 Copyright (c) 1992-2009 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
          The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 #0 Sat Aug 29 21:10:23 CEST 2009
 WARNING: WITNESS option enabled, expect reduced performance.
 Timecounter "i8254" frequency 1193182 Hz quality 0
 CPU: QEMU Virtual CPU version 0.10.5 (2672.21-MHz 686-class CPU)
    Origin = "AuthenticAMD"  Id = 0x623  Stepping = 3
 
 Features=0x78bf3fd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
    Features2=0x9<SSE3,MON>
    AMD Features=0xe0100800<SYSCALL,NX,LM,3DNow!+,3DNow!>
    AMD Features2=0x4<SVM>
 real memory  = 536805376 (511 MB)
 avail memory = 507310080 (483 MB)
 ACPI APIC Table: <QEMU   QEMUAPIC>
 MADT: Forcing active-low polarity and level trigger for SCI
 ioapic0 <Version 1.1> irqs 0-23 on motherboard
 kbd1 at kbdmux0
 acpi0: <QEMU QEMURSDT> on motherboard
 acpi0: [ITHREAD]
 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
 acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on 
 acpi0
 Timecounter "HPET" frequency 100000000 Hz quality 900
 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 pci_link0: BIOS IRQ 9 does not match initial IRQ 11
 pci0: <ACPI PCI bus> on pcib0
 Correcting Natoma config for non-SMP
 isab0: <PCI-ISA bridge> at device 1.0 on pci0
 isa0: <ISA bus> on isab0
 atapci0: <Intel PIIX3 WDMA2 controller> port 
 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc000-0xc00f at device 1.1 on pci0
 ata0: <ATA channel 0> on atapci0
 ata0: [ITHREAD]
 ata1: <ATA channel 1> on atapci0
 ata1: [ITHREAD]
 pci0: <bridge> at device 1.3 (no driver attached)
 vgapci0: <VGA-compatible display> mem 
 0xc0000000-0xc1ffffff,0xc2000000-0xc2000fff at device 2.0 on pci0
 ed0: <RealTek 8029> port 0xc100-0xc1ff irq 11 at device 3.0 on pci0
 ed0: ed_stop_hw RST never set
 ed0: WARNING: using obsoleted if_watchdog interface
 ed0: Ethernet address: 52:54:00:12:34:56
 ed0: [ITHREAD]
 pci0: <memory, RAM> at device 4.0 (no driver attached)
 atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 atkbd0: [ITHREAD]
 psm0: <PS/2 Mouse> flags 0x100 irq 12 on atkbdc0
 psm0: [GIANT-LOCKED]
 psm0: [ITHREAD]
 psm0: model IntelliMouse Explorer, device ID 4
 ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
 ppc0: [ITHREAD]
 ppbus0: <Parallel port bus> on ppc0
 uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 
 flags 0x10 on acpi0
 uart0: [FILTER]
 cpu0: <ACPI CPU> on acpi0
 pmtimer0 on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=0x300>
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
 Timecounter "TSC" frequency 2672214032 Hz quality 800
 Timecounters tick every 1.000 msec
 driver bug: Unable to set devclass (devname: (null))
 ad0: 4000MB <QEMU HARDDISK 0.10.5> at ata0-master WDMA2
 acd0: CDROM <QEMU DVD-ROM/0.10.5> at ata1-master WDMA2
 WARNING: WITNESS option enabled, expect reduced performance.
 Trying to mount root from ufs:/dev/ad0s1a
 Trying to mount root from ufs:/dev/ad0s1a
 (probe0:ata1:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
 (probe0:ata1:0:0:0): CAM Status: SCSI Status Error
 (probe0:ata1:0:0:0): SCSI Status: Check Condition
 (probe0:ata1:0:0:0): NOT READY asc:3a,0
 (probe0:ata1:0:0:0): Medium not present
 (probe0:ata1:0:0:0): Unretryable error
 cd0 at ata1 bus 0 target 0 lun 0
 cd0: <QEMU QEMU DVD-ROM 0.10> Removable CD-ROM SCSI-0 device
 cd0: 16.000MB/s transfers
 cd0: Attempt to query device size failed: NOT READY, Medium not present
 lock order reversal:
   1st 0xc3177250 bufwait (bufwait) @ kern/vfs_bio.c:2559
   2nd 0xc35b2c00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285
 KDB: stack backtrace:
 db_trace_self_wrapper(d63a1a60,c08dfe38,c0c80821,c0c284c4,d63a1ac8,...) at 
 db_trace_self_wrapper+0x26
 _witness_debugger(c0c19b98,c35b2c00,c0c284c4,c342fce0,c0c80821,...) at 
 _witness_debugger+0x25
 witness_checkorder(c35b2c00,9,c0c80818,11d,0,...) at 
 witness_checkorder+0x7f6
 _sx_xlock(c35b2c00,0,c0c80818,11d,cd637490,...) at _sx_xlock+0x52
 ufsdirhash_acquire(d63a1b2c,0,0,c00,490,...) at ufsdirhash_acquire+0x35
 ufsdirhash_remove(c36e5bc8,cd637490,490,d63a1b60,d63a1b5c,...) at 
 ufsdirhash_remove+0x1d
 ufs_dirremove(c35f4324,c370315c,500800c,0,c35f4324,...) at 
 ufs_dirremove+0x180
 ufs_remove(d63a1c34,d63a1c44,d63a1c0c,d63a1c0c,c3726860,...) at 
 ufs_remove+0x81
 VOP_REMOVE_APV(c0d56ec0,d63a1c34,c3726860,d63a1c0c,bfbfeeec,...) at 
 VOP_REMOVE_APV+0x5d
 kern_unlinkat(c3719900,ffffff9c,bfbfeeec,0,d63a1c80,...) at 
 kern_unlinkat+0x23d
 kern_unlink(c3719900,bfbfeeec,0,d63a1d2c,c0b87b67,...) at kern_unlink+0x27
 unlink(c3719900,d63a1cf8,4,c0c282d2,c0d36218,...) at unlink+0x22
 syscall(d63a1d38) at syscall+0x177
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (10, FreeBSD ELF32, unlink), eip = 0x28135c07, esp = 
 0xbfbfebbc, ebp = 0xbfbfecb8 ---
 ed0: ed_stop_hw RST never set
 ed0: ed_stop_hw RST never set
 Waiting (max 60 seconds) for system process `vnlru' to stop...done
 Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
 Waiting (max 60 seconds) for system process `syncer' to stop...
 Syncing disks, vnodes remaining...0 done
 All buffers synced.
 lock order reversal:
   1st 0xc35f49c4 ufs (ufs) @ kern/vfs_mount.c:1200
   2nd 0xc35f4df4 devfs (devfs) @ kern/vfs_subr.c:2083
 KDB: stack backtrace:
 db_trace_self_wrapper(d636e9e4,c08dfe38,c0c1e124,c0c11fcc,d636ea4c,...) at 
 db_trace_self_wrapper+0x26
 _witness_debugger(c0c19b98,c35f4df4,c0c11fcc,c342fba8,c0c1e124,...) at 
 _witness_debugger+0x25
 witness_checkorder(c35f4df4,9,c0c1e11b,823,0,...) at 
 witness_checkorder+0x7f6
 __lockmgr_args(c35f4df4,80100,c35f4e10,0,0,...) at __lockmgr_args+0xf00
 vop_stdlock(d636eb20,3,c35b0994,80000,c35f4d9c,...) at vop_stdlock+0x54
 VOP_LOCK1_APV(c0d32b00,d636eb20,c342ba40,c0d6f720,c35f4d9c,...) at 
 VOP_LOCK1_APV+0x7f
 _vn_lock(c35f4d9c,80100,c0c1e11b,823,15f,...) at _vn_lock+0x49
 vget(c35f4d9c,80100,c36f5b40,15e,9d,...) at vget+0x66
 devfs_allocv(c36dd080,c36df000,d636ebb4,9d,c35f496c,...) at 
 devfs_allocv+0x110
 devfs_root(c36df000,80000,d636ebf0,4f5,c31b0df0,...) at devfs_root+0x4a
 dounmount(c36df000,80000,c36f5b40,c36f5b40,c3176230,...) at 
 dounmount+0x53e
 vfs_unmountall(c31763e0,0,c0c174e4,129,0,...) at vfs_unmountall+0x50
 boot(c0da3a30,0,c0c174e4,ac,c0d36100,...) at boot+0x757
 reboot(c36f5b40,d636ecf8,4,c0c1add0,c0d36704,...) at reboot+0x77
 syscall(d636ed38) at syscall+0x177
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (55, FreeBSD ELF32, reboot), eip = 0x28132687, esp = 
 0xbfbfed3c, ebp = 0x1 ---
 Uptime: 25m7s
 Rebooting...
 
>Unformatted:
