From nobody@FreeBSD.org  Sun Mar 21 17:14:53 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 6099F106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 21 Mar 2010 17:14:53 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 4F7F88FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 21 Mar 2010 17:14:53 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o2LHEqxu031059
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 21 Mar 2010 17:14:52 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o2LHEquK031058;
	Sun, 21 Mar 2010 17:14:52 GMT
	(envelope-from nobody)
Message-Id: <201003211714.o2LHEquK031058@www.freebsd.org>
Date: Sun, 21 Mar 2010 17:14:52 GMT
From: Ryan Steinmetz <rpsfa@rit.edu>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [lor] vfs_bio.c + ufs_dirhash.c
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         144929
>Category:       kern
>Synopsis:       [ufs] [lor] vfs_bio.c + ufs_dirhash.c
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Mar 21 17:20:03 UTC 2010
>Closed-Date:    
>Last-Modified:  Thu Jun  2 01:10:09 UTC 2011
>Originator:     Ryan Steinmetz
>Release:        9.0-CURRENT-201002
>Organization:
Rochester Institute of Technology
>Environment:
FreeBSD box 9.0-CURRENT-201002 FreeBSD 9.0-CURRENT-201002 #0: Fri Feb 12 22:57:16 UTC 2010     root@obrian.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

>Description:

lock order reversal:
 1st 0xcd2e2b88 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559
 2nd 0xc3554400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
db_trace_self_wrapper(c0ca3e8b,d616074c,c08d7395,c08c7fab,c0ca6d96,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08c7fab,c0ca6d96,c312efc8,c3132228,d61607a8,...) at kdb_backtrace+0x29
_witness_debugger(c0ca6d96,c3554400,c0cc88f8,c3132228,c0cc857d,...) at _witness_debugger+0x25
witness_checkorder(c3554400,9,c0cc857d,11d,0,...) at witness_checkorder+0x839
_sx_xlock(c3554400,0,c0cc857d,11d,c399cb54,...) at _sx_xlock+0x85
ufsdirhash_acquire(cd2e2b28,ce0e4800,200,ce0e484c,d6160878,...) at ufsdirhash_acquire+0x35
ufsdirhash_add(c399cb54,d61608d0,84c,d6160864,d6160868,...) at ufsdirhash_add+0x13
ufs_direnter(c398a330,c3a23cc0,d61608d0,d6160bd0,0,...) at ufs_direnter+0x729
ufs_makeinode(d6160bd0,0,d6160abc,d6160a18,c0be1d15,...) at ufs_makeinode+0x50d
ufs_create(d6160abc,d6160ad4,0,0,d6160ba4,...) at ufs_create+0x30
VOP_CREATE_APV(c0dad960,d6160abc,d6160bd0,d6160a54,0,...) at VOP_CREATE_APV+0xa5
vn_open_cred(d6160ba4,d6160c5c,1a4,0,c3575a00,...) at vn_open_cred+0x215
vn_open(d6160ba4,d6160c5c,1a4,c34e61f8,c0cc99d2,...) at vn_open+0x3b
kern_openat(c398db40,ffffff9c,284661f0,0,a02,...) at kern_openat+0x11f
kern_open(c398db40,284661f0,0,a01,1a4,...) at kern_open+0x35
open(c398db40,d6160cf8,c0cdac1c,c0ca75e6,c394c000,...) at open+0x30
syscall(d6160d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip = 0x28342a93, esp = 0xbfbfe89c, ebp = 0xbfbfe8c8 ---
>How-To-Repeat:
Freshly installed the 9.0-CURRENT snapshot 201002 (under ESXi) and ran 'portsnap fetch extract' and received the LOR message.
>Fix:


Patch attached with submission follows:

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-CURRENT-201002 #0: Fri Feb 12 22:57:16 UTC 2010
    root@obrian.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.00GHz (3141.72-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf41  Stepping = 1
  Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
  Features2=0x80000001<SSE3,<b31>>
  AMD Features=0x100000<NX>
  TSC: P-state invariant
real memory  = 536870912 (512 MB)
avail memory = 506249216 (482 MB)
ACPI APIC Table: <PTLTD          APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL 440BX> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
pci0: <base peripheral> at device 7.7 (no driver attached)
vgapci0: <VGA-compatible display> port 0x10d0-0x10df mem 0xd4000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection 6.9.25> port 0x2000-0x203f mem 0xd8920000-0xd893ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on pci2
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:0c:29:c0:ba:dd
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pci13: <ACPI PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pci14: <ACPI PCI bus> on pcib14
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pci15: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pci16: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pci17: <ACPI PCI bus> on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pci18: <ACPI PCI bus> on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci19: <ACPI PCI bus> on pcib19
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pci20: <ACPI PCI bus> on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pci21: <ACPI PCI bus> on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pci22: <ACPI PCI bus> on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pci23: <ACPI PCI bus> on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pci24: <ACPI PCI bus> on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pci25: <ACPI PCI bus> on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pci26: <ACPI PCI bus> on pcib26
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pci27: <ACPI PCI bus> on pcib27
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pci28: <ACPI PCI bus> on pcib28
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pci29: <ACPI PCI bus> on pcib29
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pci30: <ACPI PCI bus> on pcib30
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pci31: <ACPI PCI bus> on pcib31
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pci32: <ACPI PCI bus> on pcib32
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pci33: <ACPI PCI bus> on pcib33
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pci34: <ACPI PCI bus> on pcib34
acpi_acad0: <AC Adapter> on acpi0
acpi_button0: <Sleep Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
Timecounter "TSC" frequency 3141717412 Hz quality 800
Timecounters tick every 10.000 msec
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata1-master UDMA33 
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 10240MB (20971520 512 byte sectors: 255H 63S/T 1305C)
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
lock order reversal:
 1st 0xcd2e2b88 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559
 2nd 0xc3554400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
db_trace_self_wrapper(c0ca3e8b,d616074c,c08d7395,c08c7fab,c0ca6d96,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08c7fab,c0ca6d96,c312efc8,c3132228,d61607a8,...) at kdb_backtrace+0x29
_witness_debugger(c0ca6d96,c3554400,c0cc88f8,c3132228,c0cc857d,...) at _witness_debugger+0x25
witness_checkorder(c3554400,9,c0cc857d,11d,0,...) at witness_checkorder+0x839
_sx_xlock(c3554400,0,c0cc857d,11d,c399cb54,...) at _sx_xlock+0x85
ufsdirhash_acquire(cd2e2b28,ce0e4800,200,ce0e484c,d6160878,...) at ufsdirhash_acquire+0x35
ufsdirhash_add(c399cb54,d61608d0,84c,d6160864,d6160868,...) at ufsdirhash_add+0x13
ufs_direnter(c398a330,c3a23cc0,d61608d0,d6160bd0,0,...) at ufs_direnter+0x729
ufs_makeinode(d6160bd0,0,d6160abc,d6160a18,c0be1d15,...) at ufs_makeinode+0x50d
ufs_create(d6160abc,d6160ad4,0,0,d6160ba4,...) at ufs_create+0x30
VOP_CREATE_APV(c0dad960,d6160abc,d6160bd0,d6160a54,0,...) at VOP_CREATE_APV+0xa5
vn_open_cred(d6160ba4,d6160c5c,1a4,0,c3575a00,...) at vn_open_cred+0x215
vn_open(d6160ba4,d6160c5c,1a4,c34e61f8,c0cc99d2,...) at vn_open+0x3b
kern_openat(c398db40,ffffff9c,284661f0,0,a02,...) at kern_openat+0x11f
kern_open(c398db40,284661f0,0,a01,1a4,...) at kern_open+0x35
open(c398db40,d6160cf8,c0cdac1c,c0ca75e6,c394c000,...) at open+0x30
syscall(d6160d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip = 0x28342a93, esp = 0xbfbfe89c, ebp = 0xbfbfe8c8 ---

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sun Mar 21 20:08:19 UTC 2010 
Responsible-Changed-Why:  
Take a guess and turn this over to freebsd-fs@. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144929 

From: Ryan Steinmetz <rpsfa@rit.edu>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/144929: [lor] vfs_bio.c + ufs_dirhash.c
Date: Wed, 1 Jun 2011 21:02:26 -0400

 This is from FreeBSD-9.0-CURRENT-201105, under ESX4i:
 
 freebsd9# uname -a
 FreeBSD freebsd9 9.0-CURRENT FreeBSD 9.0-CURRENT #0: Thu May 12 11:28:09 UTC 2011     root@obrian.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
 
 Jun  2 00:52:39 freebsd9 kernel: lock order reversal:
 Jun  2 00:52:39 freebsd9 kernel: 1st 0xc2a3aaf8 isofs (isofs) @ /usr/src/sys/kern/vfs_mount.c:1193
 Jun  2 00:52:39 freebsd9 kernel: 2nd 0xc2a3c168 devfs (devfs) @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1267
 Jun  2 00:52:39 freebsd9 kernel: KDB: stack backtrace:
 Jun  2 00:52:39 freebsd9 kernel: db_trace_self_wrapper(c0e8047a,7366662f,7366765f,2e73706f,32313a63,...) at db_trace_self_wrapper+0x26
 Jun  2 00:52:39 freebsd9 kernel: kdb_backtrace(c09ddcdb,c0e83cff,c2568a48,c25689e0,c739e9ac,...) at kdb_backtrace+0x2a
 Jun  2 00:52:39 freebsd9 kernel: _witness_debugger(c0e83cff,c2a3c168,c0e6b68b,c25689e0,c0eb124c,...) at _witness_debugger+0x25
 Jun  2 00:52:39 freebsd9 kernel: witness_checkorder(c2a3c168,9,c0eb124c,4f3,c2a3c188,...) at witness_checkorder+0x839
 Jun  2 00:52:39 freebsd9 kernel: __lockmgr_args(c2a3c168,80400,c2a3c188,0,0,...) at __lockmgr_args+0x804
 Jun  2 00:52:39 freebsd9 kernel: vop_stdlock(c739eacc,c2a3c1bc,c739eac4,80400,c2a3c110,...) at vop_stdlock+0x62
 Jun  2 00:52:39 freebsd9 kernel: VOP_LOCK1_APV(c0f85ec0,c739eacc,c299f000,c0fca6a0,c2a3c110,...) at VOP_LOCK1_APV+0xb5
 Jun  2 00:52:39 freebsd9 kernel: _vn_lock(c2a3c110,80400,c0eb124c,4f3,c27c5400,...) at _vn_lock+0x5e
 Jun  2 00:52:39 freebsd9 kernel: ffs_flushfiles(c28e5510,2,c28ce5c0,59f,3,...) at ffs_flushfiles+0xa7
 Jun  2 00:52:39 freebsd9 kernel: softdep_flushfiles(c28e5510,2,c28ce5c0,0,c0fca6c0,...) at softdep_flushfiles+0x2e
 Jun  2 00:52:39 freebsd9 kernel: ffs_unmount(c28e5510,80000,c739ebc0,4ee,c739ebcc,...) at ffs_unmount+0x149
 Jun  2 00:52:39 freebsd9 kernel: dounmount(c28e5510,80000,c28ce5c0,c221762c,0,...) at dounmount+0x466
 Jun  2 00:52:39 freebsd9 kernel: vfs_unmountall(c0e7c3b9,0,c0e7c307,12f,c28ce5c0,...) at vfs_unmountall+0x4e
 Jun  2 00:52:39 freebsd9 kernel: kern_reboot(0,0,c0e7c307,b2,c28c8000,...) at kern_reboot+0x450
 Jun  2 00:52:39 freebsd9 kernel: reboot(c28ce5c0,c739ecec,c739ed80,c739ec88,0,...) at reboot+0x6c
 Jun  2 00:52:39 freebsd9 kernel: syscallenter(c28ce5c0,c739ece4,c0ccb2f6,3,c10148c0,...) at syscallenter+0x263
 Jun  2 00:52:39 freebsd9 kernel: syscall(c739ed28) at syscall+0x34
 Jun  2 00:52:39 freebsd9 kernel: Xint0x80_syscall() at Xint0x80_syscall+0x21
 Jun  2 00:52:39 freebsd9 kernel: --- syscall (55, FreeBSD ELF32, reboot), eip = 0x280b8ccb, esp = 0xbfbfec8c, ebp = 0xbfbfedd8 ---
 
 Jun  2 00:58:38 freebsd9 kernel: lock order reversal:
 Jun  2 00:58:38 freebsd9 kernel: 1st 0xc225db78 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2658
 Jun  2 00:58:38 freebsd9 kernel: 2nd 0xc2721800 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
 Jun  2 00:58:38 freebsd9 kernel: KDB: stack backtrace:
 Jun  2 00:58:38 freebsd9 kernel: db_trace_self_wrapper(c0e8047a,7366752f,7366752f,7269645f,68736168,...) at db_tra
 Jun  2 00:58:38 freebsd9 kernel: ce_self_wrapper+0x26
 Jun  2 00:58:38 freebsd9 kernel: kdb_backtrace(c09ddcdb,c0e83cff,c25652a0,c2568b18,ccffd7e8,...) at 
 Jun  2 00:58:38 freebsd9 kernel: kdb_backtrace+0x2a
 Jun  2 00:58:38 freebsd9 kernel: _witness_debugger(c0e83cff,c2721800,c0eb20ae,c2568b18,c0eb1d33,...) at 
 Jun  2 00:58:38 freebsd9 kernel: _witness_debugger+0x25
 Jun  2 00:58:38 freebsd9 kernel: witness_checkorder(c2721800,9,c0eb1d33,11c,0,...) at witness_checkorder+0x839
 Jun  2 00:58:38 freebsd9 kernel: _
 Jun  2 00:58:38 freebsd9 kernel: sx_xlock(c2721800,0,c0eb1d33,11c,c2bf015c,...) at _s
 Jun  2 00:58:38 freebsd9 kernel: x_xlock+0x85
 Jun  2 00:58:38 freebsd9 kernel: ufsdirhash_acquire(c225db18,c8138800,200,c8138814,ccffd8b8,...) at ufsdirhash_acq
 Jun  2 00:58:38 freebsd9 kernel: uire+0x35
 Jun  2 00:58:38 freebsd9 kernel: ufsdirhash_add(c2bf015c,ccffd914,814,ccffd8a4,ccffd8a8,...) at ufsdirhash_add+0x13
 Jun  2 00:58:38 freebsd9 kernel: ufs_direnter(c2bf9990,c2c27440,ccffd914,ccffdba4,0,...) at ufs_direnter+0x6f9
 Jun  2 00:58:38 freebsd9 kernel: ufs_makeinode(ccffdba4,0,ccffdb00,ccffda5c,c0cecb55,...) at ufs_makeinode+0x557
 Jun  2 00:58:38 freebsd9 kernel: ufs_create(ccffdb00,ccffdb18,0,0,ccffdb78,...) at ufs_create+0x30
 Jun  2 00:58:38 freebsd9 kernel: VOP_CREATE_APV(c0faf7c0,ccffdb00,ccffdba4,ccffda98,0,...) at VOP_CREATE_APV+0xa5
 Jun  2 00:58:38 freebsd9 kernel: vn_open_cred(ccffdb78,ccffdc2c,1a4,0,c2bcd500,...) at vn_open_cred+0x215
 Jun  2 00:58:38 freebsd9 kernel: vn_open(ccffdb78,ccffdc2c,1a4,c2909070,0,...) at vn_open+0x3b
 Jun  2 00:58:38 freebsd9 kernel: kern_openat(c2a9ab80,ffffff9c,28414180,0,a02,...) at kern_openat+0x12f
 Jun  2 00:58:38 freebsd9 kernel: kern_open(c2a9ab80,28414180,0,a01,1a4,...) at kern_open+0x35
 Jun  2 00:58:38 freebsd9 kernel: open(c2a9ab80,ccffdcec,ccffdd28,c0e825f6,0,...) at open+0x30
 Jun  2 00:58:38 freebsd9 kernel: syscallenter(c2a9ab80,ccffdce4,ccffdce4,2,c10148c0,...) at syscallenter+0x263
 Jun  2 00:58:38 freebsd9 kernel: syscall(ccffdd28) at syscall+0x34
 Jun  2 00:58:38 freebsd9 kernel: Xint0x80_syscall() at Xint0x80_syscall+0x21
 Jun  2 00:58:38 freebsd9 kernel: --- syscall (5, FreeBSD ELF32, open), eip = 0x283706a3, esp = 0xbfbfe6dc, ebp = 0xbfbfe708 ---
>Unformatted:
