From mexas@bristol.ac.uk  Mon Mar  3 11:04:46 2014
Return-Path: <mexas@bristol.ac.uk>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTPS id 4297CBBB
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  3 Mar 2014 11:04:46 +0000 (UTC)
Received: from eu1sys200aog103.obsmtp.com (eu1sys200aog103.obsmtp.com [207.126.144.115])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id B9CA18B6
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  3 Mar 2014 11:03:56 +0000 (UTC)
Received: from mail-wg0-f47.google.com ([74.125.82.47]) (using TLSv1) by eu1sys200aob103.postini.com ([207.126.147.11]) with SMTP
	ID DSNKUxRhhNiZi+SlV3h5/+w+nvIAIckECBoz@postini.com; Mon, 03 Mar 2014 11:03:56 UTC
Received: by mail-wg0-f47.google.com with SMTP id x12so497460wgg.30
        for <FreeBSD-gnats-submit@freebsd.org>; Mon, 03 Mar 2014 03:03:31 -0800 (PST)
Received: from mech-as28.men.bris.ac.uk (mech-as28.men.bris.ac.uk. [137.222.187.28])
        by mx.google.com with ESMTPSA id z1sm33489927wjq.19.2014.03.03.03.03.29
        for <FreeBSD-gnats-submit@freebsd.org>
        (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Mon, 03 Mar 2014 03:03:30 -0800 (PST)
Received: from mech-as28.men.bris.ac.uk (localhost [127.0.0.1])
	by mech-as28.men.bris.ac.uk (8.14.7/8.14.7) with ESMTP id s23B3T9k001132
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 3 Mar 2014 11:03:29 GMT
	(envelope-from mexas@mech-as28.men.bris.ac.uk)
Received: (from mexas@localhost)
	by mech-as28.men.bris.ac.uk (8.14.7/8.14.7/Submit) id s23B3TBV001131;
	Mon, 3 Mar 2014 11:03:29 GMT
	(envelope-from mexas)
Message-Id: <201403031103.s23B3TBV001131@mech-as28.men.bris.ac.uk>
Date: Mon, 03 Mar 2014 03:03:30 -0800 (PST)
From: Anton Shterenlikht <mexas@bris.ac.uk>
Sender: Anton Shterenlikht <mexas@bristol.ac.uk>
Reply-To: Anton Shterenlikht <mexas@bris.ac.uk>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: panic: 11.0-CURRENT #20 r260914: Panic String: deadlkres: possible deadlock detected for 0xe000000012079b00, blocked for 1801960 ticks
X-Send-Pr-Version: 3.114
X-GNATS-Notify:

>Number:         187219
>Category:       ia64
>Synopsis:       [panic] 11.0-CURRENT #20 r260914: Panic String: deadlkres: possible deadlock detected for 0xe000000012079b00, blocked for 1801960 ticks
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-ia64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 03 11:10:00 UTC 2014
>Closed-Date:    
>Last-Modified:  Sun Apr 20 02:28:55 UTC 2014
>Originator:     Anton Shterenlikht
>Release:        FreeBSD 11.0-CURRENT ia64
>Organization:
University of Bristol	
>Environment:
System: FreeBSD mech-as28.men.bris.ac.uk 11.0-CURRENT FreeBSD 11.0-CURRENT #20 r260914: Fri Feb 28 14:36:02 GMT 2014 root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV ia64


	
>Description:

With textdump set up, I collected these files:

# cat info.last 
Dump header from device /dev/da2p1
  Architecture: ia64
  Architecture Version: 1
  Dump Length: 67584B (0 MB)
  Blocksize: 512
  Dumptime: Mon Mar  3 10:35:01 2014
  Hostname: mech-as28.men.bris.ac.uk
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 11.0-CURRENT #20 r260914: Fri Feb 28 14:36:02 GMT 2014
    root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV
  Panic String: deadlkres: possible deadlock detected for 0xe000000012079b00, blocked for 1801960 ticks

  Dump Parity: 2806468409
  Bounds: 8
  Dump Status: good
# 

# cat panic.txt
deadlkres: possible deadlock detected for 0xe000000012079b00, blocked for 1801960 ticks
#

# cat version.txt
FreeBSD 11.0-CURRENT #20 r260914: Fri Feb 28 14:36:02 GMT 2014
    root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV
# 

# cat config.txt

options	CONFIG_AUTOGENERATED
ident	ZEEV
machine	ia64
cpu	ITANIUM2
makeoptions	MODULES_OVERRIDE=geom/geom_part opensolaris zfs acl_nfs4 fdescfs
makeoptions	DEBUG=-g
options	USB_DEBUG
options	DEBUG_VFS_LOCKS
options	DEBUG_LOCKS
options	TEXTDUMP_VERBOSE
options	TEXTDUMP_PREFERRED
options	MALLOC_DEBUG_MAXZONES=8
options	WITNESS_SKIPSPIN
options	WITNESS
options	INVARIANT_SUPPORT
options	INVARIANTS
options	DEADLKRES
options	GDB
options	DDB
options	KDB_TRACE
options	KDB
options	_KPOSIX_PRIORITY_SCHEDULING
options	QUOTA
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SYSVSHM
options	SYSVSEM
options	SYSVMSG
options	STACK
options	SOFTUPDATES
options	SMP
options	SCTP
options	SCSI_DELAY=5000
options	SCHED_ULE
options	PSEUDOFS
options	PROCFS
options	PRINTF_BUFR_SIZE=128
options	PREEMPTION
options	P1003_1B_SEMAPHORES
options	NFS_ROOT
options	NFSD
options	NFSLOCKD
options	NFSCL
options	MSDOSFS
options	MD_ROOT
options	MAC
options	KTRACE
options	INET6
options	INET
options	INCLUDE_CONFIG_FILE
options	GEOM_LABEL
options	FFS
options	TMPFS
options	NULLFS
options	FDESCFS
options	COMPAT_FREEBSD7
options	CD9660
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_GPT
options	GEOM_PART_BSD
device	acpi
device	io
device	mem
device	uart_ns8250
device	miibus
device	pci
device	scbus
device	usb
device	isp
device	ispfw
device	mpt
device	cd
device	da
device	pass
device	sa
device	ehci
device	ohci
device	uhci
device	umass
device	em
device	ether
device	faith
device	firmware
device	gif
device	loop
device	md
device	puc
device	random
device	tun
device	uart
device	vlan

# cat msgbuf.txt
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #20 r260914: Fri Feb 28 14:36:02 GMT 2014
    root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV ia64
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
CPU: Madison II (1600 MHz Itanium 2)
  Origin = "GenuineIntel"  Revision = 2
  Features = 0x1<LB>
real memory  = 10737418240 (10240 MB)
avail memory = 10494246912 (10008 MB)
FPSWA Revision = 0x10012, Entry = 0xe0000040ffcc4050
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP)
cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0
random: <Software, Yarrow> initialized
Event timer "ITC" frequency 1600000000 Hz quality 1000
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603)
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603)
acpi0: <HP> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0
acpi_tz0: <Thermal Zone> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pcib0: could not evaluate _ADR - AE_NOT_FOUND
pci0: <ACPI PCI bus> on pcib0
ohci0: <NEC uPD 9210 USB controller> mem 0x80002000-0x80002fff irq 16 at device 1.0 on pci0
usbus0 on ohci0
ohci1: <NEC uPD 9210 USB controller> mem 0x80001000-0x80001fff irq 17 at device 1.1 on pci0
usbus1 on ohci1
ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x80000000-0x800000ff irq 18 at device 1.2 on pci0
usbus2: EHCI version 0.95
usbus2 on ehci0
pci0: <mass storage, ATA> at device 2.0 (no driver attached)
pcib1: <ACPI Host-PCI bridge> on acpi0
pcib1: could not evaluate _ADR - AE_NOT_FOUND
pci32: <ACPI PCI bus> on pcib1
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2100-0x21ff mem 0x903a0000-0x903bffff,0x90380000-0x9039ffff irq 27 at device 1.0 on pci32
mpt0: MPI Version=1.2.12.0
mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0x90360000-0x9037ffff,0x90340000-0x9035ffff irq 28 at device 1.1 on pci32
mpt1: MPI Version=1.2.12.0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2240-0x227f mem 0x90320000-0x9033ffff,0x90280000-0x902fffff irq 29 at device 2.0 on pci32
em0: Ethernet address: 00:13:21:5b:05:1c
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2200-0x223f mem 0x90300000-0x9031ffff irq 30 at device 2.1 on pci32
em1: Ethernet address: 00:13:21:5b:05:1d
pcib2: <ACPI Host-PCI bridge> on acpi0
pcib2: could not evaluate _ADR - AE_NOT_FOUND
pci64: <ACPI PCI bus> on pcib2
pcib3: <ACPI Host-PCI bridge> on acpi0
pcib3: could not evaluate _ADR - AE_NOT_FOUND
pci96: <ACPI PCI bus> on pcib3
pcib4: <ACPI Host-PCI bridge> on acpi0
pcib4: could not evaluate _ADR - AE_NOT_FOUND
pci128: <ACPI PCI bus> on pcib4
pcib5: <ACPI Host-PCI bridge> on acpi0
pcib5: could not evaluate _ADR - AE_NOT_FOUND
pci192: <ACPI PCI bus> on pcib5
isp0: <Qlogic ISP 2422 PCI FC-AL Adapter> port 0xc000-0xc0ff mem 0xe0040000-0xe0040fff irq 71 at device 1.0 on pci192
pcib6: <ACPI Host-PCI bridge> on acpi0
pcib6: could not evaluate _ADR - AE_NOT_FOUND
pci224: <ACPI PCI bus> on pcib6
uart0: <HP Auxiliary Diva Serial Port> mem 0xf8051000-0xf805100f irq 82 at device 1.0 on pci224
puc0: <HP Diva Serial [GSP] Multiport UART - Everest SP2> mem 0xf8050000-0xf8050fff,0xf8020000-0xf803ffff irq 82 at device 1.1 on pci224
uart1: <Non-standard ns8250 class UART with FIFOs> at port 1 on puc0
uart1: console (9600,n,8,1)
uart2: <Non-standard ns8250 class UART with FIFOs> at port 2 on puc0
uart3: <Non-standard ns8250 class UART with FIFOs> at port 3 on puc0
vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf0000000-0xf7ffffff,0xf8040000-0xf804ffff at device 2.0 on pci224
vgapci0: Boot video device
uart4: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0
uart5: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0
uart5: debug port (9600,n,8,1)
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
ugen0.1: <NEC> at usbus0
ugen1.1: <NEC> at usbus1
uhub0: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub1: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <NEC> at usbus2
uhub2: <NEC EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
random: unblocking device.
uhub1: 2 ports with 2 removable, self powered
uhub0: 3 ports with 3 removable, self powered
uhub2: 5 ports with 5 removable, self powered
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <HP 73.4G ST373454LC HPC2> Fixed Direct Access SCSI-3 device 
da0: Serial Number 3KP05D0W00007532H5SA
da0: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
da0: Command Queueing enabled
da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
da1 at isp0 bus 0 scbus2 target 0 lun 1
da1: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device 
da1: Serial Number P56350GX3RX04Z
da1: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da1: Command Queueing enabled
da1: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
da2 at isp0 bus 0 scbus2 target 0 lun 2
da2: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device 
da2: Serial Number P56350GX3RX04Z
da2: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da2: Command Queueing enabled
da2: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
da3 at isp0 bus 0 scbus2 target 0 lun 5
da3: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device 
da3: Serial Number P56350GX3RX04Z
da3: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da3: Command Queueing enabled
da3: 140011MB (286744185 512 byte sectors: 255H 63S/T 17849C)
pass1 at isp0 bus 0 scbus2 target 0 lun 0
pass1: <COMPAQ MSA1000 4.32> Fixed Storage Array SCSI-4 device 
pass1: Serial Number P56350GX3RX04Z
pass1: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
pass1: Command Queueing enabled
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0p2 [rw]...
WARNING: / was not properly dismounted
<118>Setting hostuuid: ee652761-3dcc-11df-9074-0013215b051c.
<118>Setting hostid: 0x8968724b.
<118>Starting ddb.
<118>Entropy harvesting: interrupts ethernet point_to_point swi.
<118>Starting file system checks:
<118>/dev/da0p2: 560961 files, 3745925 used, 30870499 free (304931 frags, 3820696 blocks, 0.9% fragmentation)
<118>/dev/da1p1: 2133 files, 78964 used, 17144397 free (461 frags, 2142992 blocks, 0.0% fragmentation)
<118>/dev/da3p1: 6067 files, 1327969 used, 68099534 free (6214 frags, 8511665 blocks, 0.0% fragmentation)
<118>Mounting local file systems:.
<118>Writing entropy file:.
<118>Setting hostname: mech-as28.men.bris.ac.uk.
<118>Starting Network: lo0 em0 em1.
<118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
<118>	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
<118>	inet6 ::1 prefixlen 128 
<118>	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
<118>	inet 127.0.0.1 netmask 0xff000000 
<118>	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
<118>em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
<118>	ether 00:13:21:5b:05:1c
<118>	inet 137.222.187.28 netmask 0xffffff00 broadcast 137.222.187.255 
<118>	inet6 fe80::213:21ff:fe5b:51c%em0 prefixlen 64 scopeid 0x1 
<118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118>	media: Ethernet autoselect
<118>	status: no carrier
<118>em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
<118>	ether 00:13:21:5b:05:1d
<118>	inet 10.10.10.14 netmask 0xffffff00 broadcast 10.10.10.255 
<118>	inet6 fe80::213:21ff:fe5b:51d%em1 prefixlen 64 scopeid 0x2 
<118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118>	media: Ethernet autoselect (100baseTX <full-duplex>)
<118>	status: active
<118>Starting devd.
<118>Starting pflogd: 
<118>add net default: gateway 137.222.187.250
<118>add net fe80::: gateway ::1
<118>add net ff02::: gateway ::1
<118>add net ::ffff:0.0.0.0: gateway ::1
<118>add net ::0.0.0.0: gateway ::1
<118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
<118>Creating and/or trimming log files.
<118>Starting syslogd.
<118>savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe000000012424480, blocked for 1802103 ticks
<118>Mar  3 09:50:45 mech-as28 savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe000000012424480, blocked for 1802103 ticks
<118>savecore: writing core to /var/crash/textdump.tar.7
<118>/var/crash/vmcore.7 not found
<118>Starting casperd.
lock order reversal:
 1st 0xa00000009e060078 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3081
 2nd 0xe000000010c38000 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
KDB: stack backtrace:
db_trace_self(0x9ffc0000000d89c0) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc0000005842c0) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000e4eba8, 0x9ffc0000005ba3e0) at kdb_backtrace+0xc0
_witness_debugger(0x1, 0x9ffc000000d4bdf8, 0x9ffc0000005bddc0, 0xb9d, 0x9ffc000000d799e8) at _witness_debugger+0x60
witness_checkorder(0xe000000010c38000, 0x9ffc000000d4b740, 0x9ffc000000d799e8, 0x11c, 0x0) at witness_checkorder+0x15c0
_sx_xlock(0xe000000010c38000, 0x0, 0x9ffc000000d799e8, 0x11c) at _sx_xlock+0x120
ufsdirhash_acquire(0xe0000000123c8f18, 0xe000000010c38000, 0x9ffc000000989250) at ufsdirhash_acquire+0x70
ufsdirhash_add(0xe0000000123c8f18, 0xa00000000036d098, 0x90c, 0xa00000000036d040, 0x9ffc000000991140, 0xda1) at ufsdirhash_add+0x20
ufs_direnter(0xe000000012689380, 0xe000000012694750, 0xa00000000036d098, 0xa00000000036d318, 0xa00000009e0609b0, 0x0) at ufs_direnter+0x1260
ufs_mkdir(0xa00000000036d370, 0xa00000000036d098, 0xe000000012689380) at ufs_mkdir+0x14f0
VOP_MKDIR_APV(0x9ffc000000e13a50, 0xa00000000036d370, 0x0, 0x9ffc000000d8a708, 0x9ffc000000677b70, 0x818, 0x9ffc000001022ba0) at VOP_MKDIR_APV+0x270
kern_mkdirat(0xe000000012636480, 0xffffffffffffff9c, 0x140804040, 0x0, 0x1c0) at kern_mkdirat+0x3a0
kern_mkdir(0xe000000012636480, 0x140804040, 0x0, 0x1c0) at kern_mkdir+0x40
sys_mkdir(0xe000000012636480, 0xa00000000036d4e8, 0x9ffc000000a40c80, 0x48d) at sys_mkdir+0x30
syscall(0xe0000000124464a0, 0x1c0, 0x9ffc000000a3ccb0, 0xe000000012636480, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
<118>Clearing /tmp (X related).
<118>Recovering vi editor sessions:.
<118>Updating motd:.
<118>Mounting late file systems:.
<118>Starting ntpd.
<118>Starting bsdstats.
<118>Posting monthly OS statistics to rpt.bsdstats.org
<118>Performing sanity check on sshd configuration.
<118>Starting sshd.
<118>Starting sendmail_submit.
<118>Starting sendmail_msp_queue.
<118>Starting cron.
<118>Local package initialization:cd: /var/portbuild/: No such file or directory
<118>.
<118>Starting inetd.
<118>Starting background file system checks in 60 seconds.
<118>
<118>Mon Mar  3 09:50:54 GMT 2014
<118>Mar  3 09:56:32 mech-as28 login: ROOT LOGIN (root) ON ttyu1
panic: deadlkres: possible deadlock detected for 0xe000000012079b00, blocked for 1801960 ticks

cpuid = 0
KDB: stack backtrace:
db_trace_self(0x9ffc0000000d89c0) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc0000005842c0) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000e4eba8, 0x9ffc00000050b420, 0x40c, 0x9ffc000001022ba0) at kdb_backtrace+0xc0
vpanic(0x9ffc000000d36760, 0xa000000000275518) at vpanic+0x260
panic(0x9ffc000000d36760, 0x9ffc000000d369b8, 0xe000000012079b00, 0x1b7ee8) at panic+0x80
deadlkres(0x9ffc000000d366f0, 0xe000000012079b00, 0x9ffc000000de9210, 0x9ffc000000d366f0) at deadlkres+0x700
fork_exit(0x9ffc000000d970a0, 0x0, 0xa000000000275550) at fork_exit+0x120
enter_userland() at enter_userland
KDB: enter: panic

# cat ddb.txt

db:0:kdb.enter.panic>  run lockinfo
db:1:lockinfo> show locks
db:1:locks>  show alllocks
Process 1125 (sh) thread 0xe0000000122e3200 (100077)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1123 (sh) thread 0xe000000012716d80 (100093)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1121 (sh) thread 0xe000000012719680 (100104)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1119 (sh) thread 0xe000000012716480 (100095)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1117 (sh) thread 0xe0000000122e2d80 (100078)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1113 (sh) thread 0xe00000001282e480 (100101)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1111 (sh) thread 0xe00000001282e000 (100102)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1109 (sh) thread 0xe0000000123e8480 (100073)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1107 (sshd) thread 0xe0000000123e8d80 (100071)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1106 (sh) thread 0xe0000000123e8900 (100072)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1104 (sh) thread 0xe000000012079b00 (100055)
shared lockmgr ufs (ufs) r = 0 (0xe0000000124387b8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 1102 (who) thread 0xe000000012719b00 (100103)
exclusive lockmgr ufs (ufs) r = 0 (0xe00000001261e2d8) locked @ /usr/src/sys/kern/vfs_vnops.c:373
Process 1100 (vi) thread 0xe0000000122e3680 (100076)
exclusive lockmgr bufwait (bufwait) r = 0 (0xa00000009e062028) locked @ /usr/src/sys/kern/vfs_bio.c:3081
exclusive lockmgr ufs (ufs) r = 0 (0xe0000000128dc2d8) locked @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1711
exclusive lockmgr ufs (ufs) r = 0 (0xe0000000124693e8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
Process 22 (syncer) thread 0xe0000000122d8d80 (100064)
exclusive lockmgr ufs (ufs) r = 0 (0xe000000012633b38) locked @ /usr/src/sys/kern/vfs_subr.c:1720
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.panic>  show pcpu
cpuid        = 0
dynamic pcpu = 0x40040040ff1be380
curthread    = 0xe000000011651200: pid 0 "deadlkres"
curpcb       = 0
fpcurthread  = none
idlethread   = 0xe00000001117bb00: tid 100003 "idle: cpu0"
MD: vhpt       = 0xe0000040ff000000
MD: lid        = 0
MD: clock      = 0x41c83545197
MD: clock_mode = 2
MD: clock_load = 0
MD: stats      = 0x9ffc0000011c8330
MD: pmap       = 0x9ffc000000e77240
spin locks held:
db:0:kdb.enter.panic>  bt
Tracing pid 0 tid 100053 td 0xe000000011651200
kdb_enter(0x9ffc000000d40e50, 0x9ffc000000d40e50, 0x9ffc00000050b470, 0x40c) at kdb_enter+0x92
vpanic(0x9ffc000000d36760, 0xa000000000275518) at vpanic+0x2b0
panic(0x9ffc000000d36760, 0x9ffc000000d369b8, 0xe000000012079b00, 0x1b7ee8) at panic+0x80
deadlkres(0x9ffc000000d366f0, 0xe000000012079b00, 0x9ffc000000de9210, 0x9ffc000000d366f0) at deadlkres+0x700
fork_exit(0x9ffc000000d970a0, 0x0, 0xa000000000275550) at fork_exit+0x120
enter_userland() at enter_userland
db:0:kdb.enter.panic>  ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
 1125  1124  1125     0  Ds      ufs      0xe00000001261e2d8 sh
 1124   935   935     0  S       piperd   0xe00000001237ce70 cron
 1123  1122  1123     2  Ds      ufs      0xe00000001261e2d8 sh
 1122   935   935     0  S       piperd   0xe0000000127904d0 cron
 1121  1120  1121     0  Ds      ufs      0xe00000001261e2d8 sh
 1120   935   935     0  S       piperd   0xe000000012790000 cron
 1119  1118  1119     0  Ds      ufs      0xe00000001261e2d8 sh
 1118   935   935     0  S       piperd   0xe000000012820268 cron
 1117  1116  1117     2  Ds      ufs      0xe00000001261e2d8 sh
 1116   935   935     0  S       piperd   0xe000000012820738 cron
 1113  1112  1113     0  Ds      ufs      0xe00000001261e2d8 sh
 1112   935   935     0  S       piperd   0xe000000012820c08 cron
 1111  1110  1111     0  Ds      ufs      0xe00000001261e2d8 sh
 1110   935   935     0  S       piperd   0xe00000001237d340 cron
 1109  1108  1109     2  Ds      ufs      0xe00000001261e2d8 sh
 1108   935   935     0  S       piperd   0xe0000000128210d8 cron
 1107   924  1107     0  Ds      ufs      0xe00000001261e2d8 sshd
 1106  1105  1106     0  Ds      ufs      0xe00000001261e2d8 sh
 1105   935   935     0  S       piperd   0xe00000001237dce0 cron
 1104  1103  1104     0  Ds      ufs      0xe00000001261e2d8 sh
 1103   935   935     0  S       piperd   0xe000000012791810 cron
 1102  1030  1102     0  D+      getblk   0xa00000009e0433c8 who
 1100  1051  1051  1001  D+      biowr    0xa00000009e061fa0 vi
 1051  1046  1051  1001  S+      wait     0xe000000012449720 sh
 1046     1  1046  1001  Ss+     pause    0xe0000000124469e8 tcsh
 1043   924  1043     0  Ds      ufs      0xe0000000124387b8 sshd
 1030   995  1030     0  S+      wait     0xe0000000122d7280 sh
  995     1   995     0  Ss+     wait     0xe000000012796000 login
  994     1     1     0  S       ttydcd   0xe0000000116b90e8 getty
  974     1   974     0  Ss      select   0xe000000012301c40 inetd
  935     1   935     0  Ss      nanslp   0x9ffc000000e83eb8 cron
  930     1   930    25  Ss      pause    0xe0000000111797c8 sendmail
  927     1   927     0  Ss      select   0xe000000012300dc0 sendmail
  924     1   924     0  Ss      select   0xe0000000120aaa40 sshd
  857     1   857     0  Ss      select   0xe000000012300a40 ntpd
  793     1   793     0  Ss      select   0xe000000012300e40 casperd
  792     1    24     0  S+      select   0xe000000012300940 casperd
  763     1   763     0  Ds      ufs      0xe000000012633b38 syslogd
  638     1   638     0  Ss      select   0xe0000000120aa2c0 devd
  129     1   129     0  Ss      pause    0xe0000000124480a8 adjkerntz
   23     0     0     0  DL      sdflush  0x9ffc000000e4fd50 [softdepflush]
   22     0     0     0  DL      getblk   0xa00000009e0422b8 [syncer]
   21     0     0     0  DL      vlruwt   0xe0000000122d1280 [vnlru]
   20     0     0     0  DL      psleep   0x9ffc000000e4f130 [bufdaemon]
   19     0     0     0  DL      pgzero   0x9ffc000000e500bc [pagezero]
   18     0     0     0  DL      psleep   0x9ffc000000e50024 [vmdaemon]
   17     0     0     0  DL      psleep   0x9ffc000000e50054 [pagedaemon]
    7     0     0     0  DL      waiting_ 0x9ffc0000011bc670 [sctp_iterator]
    6     0     0     0  DL      (threaded)                  [zfskern]
100052                   D       l2arc_fe 0x9ffc00000161b640 [l2arc_feed_thread]
100051                   D       arc_recl 0x9ffc00000160b820 [arc_reclaim_thread]
    5     0     0     0  DL      ispf     0xe0000000116a0000 [isp0: fc_thrd0]
    4     0     0     0  DL      idle     0xa000000000a26000 [mpt_recovery1]
    3     0     0     0  DL      idle     0xa000000000a0e000 [mpt_recovery0]
   16     0     0     0  DL      (threaded)                  [usb]
100039                   D       -        0xa000000000a0ae18 [usbus2]
100038                   D       -        0xa000000000a0adc0 [usbus2]
100037                   D       -        0xa000000000a0ad68 [usbus2]
100036                   D       -        0xa000000000a0ad10 [usbus2]
100034                   D       -        0xa000000000a01460 [usbus1]
100033                   D       -        0xa000000000a01408 [usbus1]
100032                   D       -        0xa000000000a013b0 [usbus1]
100031                   D       -        0xa000000000a01358 [usbus1]
100029                   D       -        0xa0000000009fd460 [usbus0]
100028                   D       -        0xa0000000009fd408 [usbus0]
100027                   D       -        0xa0000000009fd3b0 [usbus0]
100026                   D       -        0xa0000000009fd358 [usbus0]
   15     0     0     0  DL      tzpoll   0x9ffc000000e4dac0 [acpi_thermal]
    2     0     0     0  DL      (threaded)                  [cam]
100058                   D       ccb_scan 0x9ffc000000e503b8 [scanner]
100023                   D       -        0x9ffc000000e50500 [doneq0]
   14     0     0     0  DL      -        0x9ffc000000e4ddb0 [rand_harvestq]
   13     0     0     0  DL      (threaded)                  [geom]
100011                   D       -        0x9ffc000000e4e2f0 [g_down]
100010                   D       -        0x9ffc000000e4e2e8 [g_up]
100009                   D       -        0x9ffc000000e4e2d8 [g_event]
   12     0     0     0  WL      (threaded)                  [intr]
100048                   I                                   [swi0: uart uart++++]
100046                   I                                   [irq71: isp0]
100042                   I                                   [irq28: mpt1]
100040                   I                                   [irq27: mpt0]
100035                   I                                   [irq18: ehci0]
100030                   I                                   [irq17: ohci1]
100025                   I                                   [irq16: ohci0]
100019                   I                                   [swi6: task queue]
100018                   I                                   [swi6: Giant taskq]
100016                   I                                   [swi5: fast taskq]
100008                   I                                   [swi4: clock]
100007                   I                                   [swi4: clock]
100006                   I                                   [swi1: netisr 0]
100005                   I                                   [swi3: vm]
   11     0     0     0  RL      (threaded)                  [idle]
100004                   Run     CPU 1                       [idle: cpu1]
100003                   CanRun                              [idle: cpu0]
    1     0     1     0  SLs     wait     0xe0000000111784a0 [init]
   10     0     0     0  DL      audit_wo 0x9ffc0000011c0d08 [audit]
    0     0     0     0  RLs     (threaded)                  [kernel]
100057                   D       -        0xe00000001118c000 [CAM taskq]
100053                   Run     CPU 0                       [deadlkres]
100050                   D       -        0xe000000011255300 [system_taskq_1]
100049                   D       -        0xe000000011255300 [system_taskq_0]
100045                   D       -        0xe000000011256900 [em1 taskq]
100044                   D       -        0xe000000011256b00 [em0 taskq]
100022                   D       -        0xe00000001118c100 [acpi_task_2]
100021                   D       -        0xe00000001118c100 [acpi_task_1]
100020                   D       -        0xe00000001118c100 [acpi_task_0]
100017                   D       -        0xe00000001118c600 [thread taskq]
100015                   D       -        0xe00000001118c900 [kqueue taskq]
100014                   D       -        0xe00000001118ca00 [ffs_trim taskq]
100012                   D       -        0xe00000001118d600 [firmware taskq]
100000                   D       swapin   0x9ffc000000e76808 [swapper]
 1045  1043  1043  1001  Z                                   sshd
db:0:kdb.enter.panic>  alltrace

Tracing command sh pid 1125 tid 100077 td 0xe0000000122e3200
cpu_switch(0xe0000000122e3200, 0xe00000001117bb00, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000122e3200, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000122e3200, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe0000000122e3578, 0xe0000000122e3200) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe0000000122e3200, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa000000000335070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa000000000335050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe0000000122e3200, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa0000000003352e8, 0xa000000000335310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa000000000335310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa0000000003350f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa000000000335288, 0x200000, 0xe000000012438750, 0x0, 0xa000000000335318) at lookup+0xe30
namei(0xa000000000335288, 0xe000000012448940, 0x9ffc0000009b89f0, 0x0, 0xa0000000003352f8) at namei+0xb40
vn_open_cred(0xa000000000335288, 0xa000000000335390, 0x1c8, 0x0, 0xe000000011163600, 0xe00000001234b590) at vn_open_cred+0x500
vn_open(0xa000000000335288, 0xa000000000335390, 0x1c8, 0xe00000001234b590, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe0000000122e3200, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe0000000122e3200, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe0000000122e3200, 0xa0000000003354e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe000000012448940, 0x100000, 0x9ffc000000e791d8, 0xe0000000122e3200, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1124 tid 100106 td 0xe000000012718d80
cpu_switch(0xe000000012718d80, 0xe00000001117b680, 0x9ffc000000ed5bd0, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe000000012718d80, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012718d80, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001237ce70, 0x5c, 0xe0000000127190f8, 0xe000000012718d80) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe00000001237ce70, 0x5c, 0x9ffc000000d48810, 0x9ffc000000d48bd0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe00000001237ce70, 0x5c, 0x9ffc000000d4c260, 0x100, 0x9ffc00000051d880) at sleepq_wait_sig+0x30
_sleep(0xe00000001237ce70, 0xe00000001237d0b0, 0x5c, 0x9ffc000000d4c260, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000000012348484, 0xa00000000041d360, 0xa00000000041d378, 0x0, 0xe00000001237cf78) at pipe_read+0x740
dofileread(0xe000000012718d80, 0x6, 0xe000000012348460, 0xa00000000041d360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000012718d80, 0x6, 0xa00000000041d360) at kern_readv+0xa0
sys_read(0xe000000012718d80, 0xa00000000041d4e8, 0x9ffc000000a40c80, 0x48d) at sys_read+0x100
syscall(0xe00000001293b720, 0x140c2c000, 0x2000, 0xe000000012718d80, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 1123 tid 100093 td 0xe000000012716d80
cpu_switch(0xe000000012716d80, 0xe00000001117bb00, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe000000012716d80, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012716d80, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe0000000127170f8, 0xe000000012716d80) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe000000012716d80, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa0000000003b5070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa0000000003b5050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe000000012716d80, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa0000000003b52e8, 0xa0000000003b5310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa0000000003b5310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa0000000003b50f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa0000000003b5288, 0x200000, 0xe000000012438750, 0x0, 0xa0000000003b5318) at lookup+0xe30
namei(0xa0000000003b5288, 0xe000000012449280, 0x9ffc0000009b89f0, 0x0, 0xa0000000003b52f8) at namei+0xb40
vn_open_cred(0xa0000000003b5288, 0xa0000000003b5390, 0x1c8, 0x0, 0xe000000011163700, 0xe00000001234b680) at vn_open_cred+0x500
vn_open(0xa0000000003b5288, 0xa0000000003b5390, 0x1c8, 0xe00000001234b680, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe000000012716d80, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe000000012716d80, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe000000012716d80, 0xa0000000003b54e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe000000012449280, 0x100000, 0x9ffc000000e791d8, 0xe000000012716d80, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1122 tid 100070 td 0xe0000000116ab680
cpu_switch(0xe0000000116ab680, 0xe00000001117b680, 0x9ffc000000ed5f40, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000116ab680, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000116ab680, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe0000000127904d0, 0x5c, 0xe0000000116ab9f8, 0xe0000000116ab680) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000127904d0, 0x5c, 0x9ffc000000d48810, 0x9ffc000000d48bd0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000127904d0, 0x5c, 0x9ffc000000d4c260, 0x100, 0x9ffc00000051d880) at sleepq_wait_sig+0x30
_sleep(0xe0000000127904d0, 0xe000000012790710, 0x5c, 0x9ffc000000d4c260, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000000012348574, 0xa0000000002fd360, 0xa0000000002fd378, 0x0, 0xe0000000127905d8) at pipe_read+0x740
dofileread(0xe0000000116ab680, 0x6, 0xe000000012348550, 0xa0000000002fd360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe0000000116ab680, 0x6, 0xa0000000002fd360) at kern_readv+0xa0
sys_read(0xe0000000116ab680, 0xa0000000002fd4e8, 0x9ffc000000a40c80, 0x48d) at sys_read+0x100
syscall(0xe0000000123cade0, 0x140c2c000, 0x2000, 0xe0000000116ab680, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 1121 tid 100104 td 0xe000000012719680
cpu_switch(0xe000000012719680, 0xe00000001117b680, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe000000012719680, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012719680, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe0000000127199f8, 0xe000000012719680) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe000000012719680, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa00000000040d070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa00000000040d050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe000000012719680, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa00000000040d2e8, 0xa00000000040d310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa00000000040d310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa00000000040d0f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa00000000040d288, 0x200000, 0xe000000012438750, 0x0, 0xa00000000040d318) at lookup+0xe30
namei(0xa00000000040d288, 0xe0000000127964a0, 0x9ffc0000009b89f0, 0x0, 0xa00000000040d2f8) at namei+0xb40
vn_open_cred(0xa00000000040d288, 0xa00000000040d390, 0x1c8, 0x0, 0xe000000011165200, 0xe0000000123767d0) at vn_open_cred+0x500
vn_open(0xa00000000040d288, 0xa00000000040d390, 0x1c8, 0xe0000000123767d0, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe000000012719680, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe000000012719680, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe000000012719680, 0xa00000000040d4e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe0000000127964a0, 0x100000, 0x9ffc000000e791d8, 0xe000000012719680, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1120 tid 100094 td 0xe000000012716900
cpu_switch(0xe000000012716900, 0xe00000001117bb00, 0x9ffc000000ed3e20, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe000000012716900, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012716900, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe000000012790000, 0x5c, 0xe000000012716c78, 0xe000000012716900) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012790000, 0x5c, 0x9ffc000000d48810, 0x9ffc000000d48bd0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012790000, 0x5c, 0x9ffc000000d4c260, 0x100, 0x9ffc00000051d880) at sleepq_wait_sig+0x30
_sleep(0xe000000012790000, 0xe000000012790240, 0x5c, 0x9ffc000000d4c260, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe00000001234b654, 0xa0000000003bd360, 0xa0000000003bd378, 0x0, 0xe000000012790108) at pipe_read+0x740
dofileread(0xe000000012716900, 0x6, 0xe00000001234b630, 0xa0000000003bd360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000012716900, 0x6, 0xa0000000003bd360) at kern_readv+0xa0
sys_read(0xe000000012716900, 0xa0000000003bd4e8, 0x9ffc000000a40c80, 0x48d) at sys_read+0x100
syscall(0xe000000012448de0, 0x140c2c000, 0x2000, 0xe000000012716900, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 1119 tid 100095 td 0xe000000012716480
cpu_switch(0xe000000012716480, 0xe00000001117bb00, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe000000012716480, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012716480, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe0000000127167f8, 0xe000000012716480) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe000000012716480, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa0000000003c5070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa0000000003c5050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe000000012716480, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa0000000003c52e8, 0xa0000000003c5310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa0000000003c5310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa0000000003c50f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa0000000003c5288, 0x200000, 0xe000000012438750, 0x0, 0xa0000000003c5318) at lookup+0xe30
namei(0xa0000000003c5288, 0xe0000000127b0940, 0x9ffc0000009b89f0, 0x0, 0xa0000000003c52f8) at namei+0xb40
vn_open_cred(0xa0000000003c5288, 0xa0000000003c5390, 0x1c8, 0x0, 0xe000000011163900, 0xe000000012376960) at vn_open_cred+0x500
vn_open(0xa0000000003c5288, 0xa0000000003c5390, 0x1c8, 0xe000000012376960, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe000000012716480, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe000000012716480, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe000000012716480, 0xa0000000003c54e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe0000000127b0940, 0x100000, 0x9ffc000000e791d8, 0xe000000012716480, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1118 tid 100097 td 0xe0000000122d8000
cpu_switch(0xe0000000122d8000, 0xe00000001117b680, 0x9ffc000000ed4eb0, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000122d8000, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000122d8000, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe000000012820268, 0x5c, 0xe0000000122d8378, 0xe0000000122d8000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012820268, 0x5c, 0x9ffc000000d48810, 0x9ffc000000d48bd0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012820268, 0x5c, 0x9ffc000000d4c260, 0x100, 0x9ffc00000051d880) at sleepq_wait_sig+0x30
_sleep(0xe000000012820268, 0xe0000000128204a8, 0x5c, 0x9ffc000000d4c260, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000000012348894, 0xa0000000003d5360, 0xa0000000003d5378, 0x0, 0xe000000012820370) at pipe_read+0x740
dofileread(0xe0000000122d8000, 0x6, 0xe000000012348870, 0xa0000000003d5360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe0000000122d8000, 0x6, 0xa0000000003d5360) at kern_readv+0xa0
sys_read(0xe0000000122d8000, 0xa0000000003d54e8, 0x9ffc000000a40c80, 0x48d) at sys_read+0x100
syscall(0xe0000000122d0000, 0x140c2c000, 0x2000, 0xe0000000122d8000, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 1117 tid 100078 td 0xe0000000122e2d80
cpu_switch(0xe0000000122e2d80, 0xe00000001117bb00, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000122e2d80, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000122e2d80, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe0000000122e30f8, 0xe0000000122e2d80) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe0000000122e2d80, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa00000000033d070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa00000000033d050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe0000000122e2d80, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa00000000033d2e8, 0xa00000000033d310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa00000000033d310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa00000000033d0f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa00000000033d288, 0x200000, 0xe000000012438750, 0x0, 0xa00000000033d318) at lookup+0xe30
namei(0xa00000000033d288, 0xe0000000124484a0, 0x9ffc0000009b89f0, 0x0, 0xa00000000033d2f8) at namei+0xb40
vn_open_cred(0xa00000000033d288, 0xa00000000033d390, 0x1c8, 0x0, 0xe000000011163a00, 0xe000000012376c30) at vn_open_cred+0x500
vn_open(0xa00000000033d288, 0xa00000000033d390, 0x1c8, 0xe000000012376c30, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe0000000122e2d80, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe0000000122e2d80, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe0000000122e2d80, 0xa00000000033d4e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe0000000124484a0, 0x100000, 0x9ffc000000e791d8, 0xe0000000122e2d80, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1116 tid 100090 td 0xe000000012717b00
cpu_switch(0xe000000012717b00, 0xe00000001117b680, 0x9ffc000000ed47f8, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe000000012717b00, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012717b00, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe000000012820738, 0x5c, 0xe000000012717e78, 0xe000000012717b00) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012820738, 0x5c, 0x9ffc000000d48810, 0x9ffc000000d48bd0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012820738, 0x5c, 0x9ffc000000d4c260, 0x100, 0x9ffc00000051d880) at sleepq_wait_sig+0x30
_sleep(0xe000000012820738, 0xe000000012820978, 0x5c, 0x9ffc000000d4c260, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe0000000123499c4, 0xa00000000039d360, 0xa00000000039d378, 0x0, 0xe000000012820840) at pipe_read+0x740
dofileread(0xe000000012717b00, 0x6, 0xe0000000123499a0, 0xa00000000039d360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000012717b00, 0x6, 0xa00000000039d360) at kern_readv+0xa0
sys_read(0xe000000012717b00, 0xa00000000039d4e8, 0x9ffc000000a40c80, 0x48d) at sys_read+0x100
syscall(0xe0000000126c84a0, 0x140c2c000, 0x2000, 0xe000000012717b00, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 1113 tid 100101 td 0xe00000001282e480
cpu_switch(0xe00000001282e480, 0xe00000001117bb00, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe00000001282e480, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001282e480, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe00000001282e7f8, 0xe00000001282e480) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe00000001282e480, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa0000000003f5070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa0000000003f5050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe00000001282e480, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa0000000003f52e8, 0xa0000000003f5310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa0000000003f5310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa0000000003f50f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa0000000003f5288, 0x200000, 0xe000000012438750, 0x0, 0xa0000000003f5318) at lookup+0xe30
namei(0xa0000000003f5288, 0xe000000012797280, 0x9ffc0000009b89f0, 0x0, 0xa0000000003f52f8) at namei+0xb40
vn_open_cred(0xa0000000003f5288, 0xa0000000003f5390, 0x1c8, 0x0, 0xe000000011163b00, 0xe000000012348780) at vn_open_cred+0x500
vn_open(0xa0000000003f5288, 0xa0000000003f5390, 0x1c8, 0xe000000012348780, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe00000001282e480, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe00000001282e480, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe00000001282e480, 0xa0000000003f54e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe000000012797280, 0x100000, 0x9ffc000000e791d8, 0xe00000001282e480, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1112 tid 100084 td 0xe000000012636480
cpu_switch(0xe000000012636480, 0xe00000001117b680, 0x9ffc000000ed3ec0, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe000000012636480, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012636480, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe000000012820c08, 0x5c, 0xe0000000126367f8, 0xe000000012636480) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012820c08, 0x5c, 0x9ffc000000d48810, 0x9ffc000000d48bd0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012820c08, 0x5c, 0x9ffc000000d4c260, 0x100, 0x9ffc00000051d880) at sleepq_wait_sig+0x30
_sleep(0xe000000012820c08, 0xe000000012820e48, 0x5c, 0x9ffc000000d4c260, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe0000000123489d4, 0xa00000000036d360, 0xa00000000036d378, 0x0, 0xe000000012820d10) at pipe_read+0x740
dofileread(0xe000000012636480, 0x6, 0xe0000000123489b0, 0xa00000000036d360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe000000012636480, 0x6, 0xa00000000036d360) at kern_readv+0xa0
sys_read(0xe000000012636480, 0xa00000000036d4e8, 0x9ffc000000a40c80, 0x48d) at sys_read+0x100
syscall(0xe0000000124464a0, 0x140c2c000, 0x2000, 0xe000000012636480, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 1111 tid 100102 td 0xe00000001282e000
cpu_switch(0xe00000001282e000, 0xe00000001117b680, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe00000001282e000, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001282e000, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe00000001282e378, 0xe00000001282e000) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe00000001282e000, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa0000000003fd070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa0000000003fd050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe00000001282e000, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa0000000003fd2e8, 0xa0000000003fd310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa0000000003fd310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa0000000003fd0f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa0000000003fd288, 0x200000, 0xe000000012438750, 0x0, 0xa0000000003fd318) at lookup+0xe30
namei(0xa0000000003fd288, 0xe000000012796de0, 0x9ffc0000009b89f0, 0x0, 0xa0000000003fd2f8) at namei+0xb40
vn_open_cred(0xa0000000003fd288, 0xa0000000003fd390, 0x1c8, 0x0, 0xe000000011164200, 0xe000000012348e10) at vn_open_cred+0x500
vn_open(0xa0000000003fd288, 0xa0000000003fd390, 0x1c8, 0xe000000012348e10, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe00000001282e000, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe00000001282e000, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe00000001282e000, 0xa0000000003fd4e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe000000012796de0, 0x100000, 0x9ffc000000e791d8, 0xe00000001282e000, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1110 tid 100096 td 0xe0000000122d8480
cpu_switch(0xe0000000122d8480, 0xe00000001117bb00, 0x9ffc000000ed5518, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000122d8480, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000122d8480, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001237d340, 0x5c, 0xe0000000122d87f8, 0xe0000000122d8480) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe00000001237d340, 0x5c, 0x9ffc000000d48810, 0x9ffc000000d48bd0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe00000001237d340, 0x5c, 0x9ffc000000d4c260, 0x100, 0x9ffc00000051d880) at sleepq_wait_sig+0x30
_sleep(0xe00000001237d340, 0xe00000001237d580, 0x5c, 0x9ffc000000d4c260, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe00000001234b794, 0xa0000000003cd360, 0xa0000000003cd378, 0x0, 0xe00000001237d448) at pipe_read+0x740
dofileread(0xe0000000122d8480, 0x6, 0xe00000001234b770, 0xa0000000003cd360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe0000000122d8480, 0x6, 0xa0000000003cd360) at kern_readv+0xa0
sys_read(0xe0000000122d8480, 0xa0000000003cd4e8, 0x9ffc000000a40c80, 0x48d) at sys_read+0x100
syscall(0xe0000000122d04a0, 0x140c2c000, 0x2000, 0xe0000000122d8480, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 1109 tid 100073 td 0xe0000000123e8480
cpu_switch(0xe0000000123e8480, 0xe00000001117bb00, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000123e8480, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000123e8480, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe0000000123e87f8, 0xe0000000123e8480) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe0000000123e8480, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa000000000315070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa000000000315050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe0000000123e8480, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa0000000003152e8, 0xa000000000315310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa000000000315310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa0000000003150f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa000000000315288, 0x200000, 0xe000000012438750, 0x0, 0xa000000000315318) at lookup+0xe30
namei(0xa000000000315288, 0xe0000000123ca000, 0x9ffc0000009b89f0, 0x0, 0xa0000000003152f8) at namei+0xb40
vn_open_cred(0xa000000000315288, 0xa000000000315390, 0x1c8, 0x0, 0xe000000011163d00, 0xe000000012349d60) at vn_open_cred+0x500
vn_open(0xa000000000315288, 0xa000000000315390, 0x1c8, 0xe000000012349d60, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe0000000123e8480, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe0000000123e8480, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe0000000123e8480, 0xa0000000003154e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe0000000123ca000, 0x100000, 0x9ffc000000e791d8, 0xe0000000123e8480, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1108 tid 100088 td 0xe0000000123e9200
cpu_switch(0xe0000000123e9200, 0xe00000001117b680, 0x9ffc000000ed5d60, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000123e9200, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000123e9200, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe0000000128210d8, 0x5c, 0xe0000000123e9578, 0xe0000000123e9200) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000128210d8, 0x5c, 0x9ffc000000d48810, 0x9ffc000000d48bd0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000128210d8, 0x5c, 0x9ffc000000d4c260, 0x100, 0x9ffc00000051d880) at sleepq_wait_sig+0x30
_sleep(0xe0000000128210d8, 0xe000000012821318, 0x5c, 0x9ffc000000d4c260, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000000012348b14, 0xa00000000038d360, 0xa00000000038d378, 0x0, 0xe0000000128211e0) at pipe_read+0x740
dofileread(0xe0000000123e9200, 0x6, 0xe000000012348af0, 0xa00000000038d360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe0000000123e9200, 0x6, 0xa00000000038d360) at kern_readv+0xa0
sys_read(0xe0000000123e9200, 0xa00000000038d4e8, 0x9ffc000000a40c80, 0x48d) at sys_read+0x100
syscall(0xe0000000126c8de0, 0x140c2c000, 0x2000, 0xe0000000123e9200, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sshd pid 1107 tid 100071 td 0xe0000000123e8d80
cpu_switch(0xe0000000123e8d80, 0xe00000001117bb00, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000123e8d80, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000123e8d80, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe0000000123e90f8, 0xe0000000123e8d80) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe0000000123e8d80, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa000000000305070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa000000000305050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe0000000123e8d80, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa0000000003052e8, 0xa000000000305310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa000000000305310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa0000000003050f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa000000000305288, 0x200000, 0xe000000012438750, 0x0, 0xa000000000305318) at lookup+0xe30
namei(0xa000000000305288, 0xe0000000123ca940, 0x9ffc0000009b89f0, 0x0, 0xa0000000003052f8) at namei+0xb40
vn_open_cred(0xa000000000305288, 0xa000000000305390, 0x0, 0x0, 0xe000000011165000, 0xe0000000123768c0) at vn_open_cred+0x500
vn_open(0xa000000000305288, 0xa000000000305390, 0x0, 0xe0000000123768c0, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe0000000123e8d80, 0xffffffffffffff9c, 0x1400f99a0, 0x0, 0x0, 0x4010d000) at kern_openat+0x490
kern_open(0xe0000000123e8d80, 0x1400f99a0, 0x0, 0x100000, 0x4010d000) at kern_open+0x40
sys_open(0xe0000000123e8d80, 0xa0000000003054e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe0000000123ca940, 0x100000, 0x14010d000, 0xe0000000123e8d80, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 1106 tid 100072 td 0xe0000000123e8900
cpu_switch(0xe0000000123e8900, 0xe00000001117bb00, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000123e8900, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000123e8900, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe0000000123e8c78, 0xe0000000123e8900) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe0000000123e8900, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa00000000030d070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa00000000030d050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe0000000123e8900, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa00000000030d2e8, 0xa00000000030d310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa00000000030d310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa00000000030d0f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa00000000030d288, 0x200000, 0xe000000012438750, 0x0, 0xa00000000030d318) at lookup+0xe30
namei(0xa00000000030d288, 0xe0000000123ca4a0, 0x9ffc0000009b89f0, 0x0, 0xa00000000030d2f8) at namei+0xb40
vn_open_cred(0xa00000000030d288, 0xa00000000030d390, 0x1c8, 0x0, 0xe000000011163e00, 0xe000000012348a50) at vn_open_cred+0x500
vn_open(0xa00000000030d288, 0xa00000000030d390, 0x1c8, 0xe000000012348a50, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe0000000123e8900, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe0000000123e8900, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe0000000123e8900, 0xa00000000030d4e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe0000000123ca4a0, 0x100000, 0x9ffc000000e791d8, 0xe0000000123e8900, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1105 tid 100100 td 0xe00000001282e900
cpu_switch(0xe00000001282e900, 0xe00000001117b680, 0x9ffc000000ed4780, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe00000001282e900, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001282e900, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001237dce0, 0x5c, 0xe00000001282ec78, 0xe00000001282e900) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe00000001237dce0, 0x5c, 0x9ffc000000d48810, 0x9ffc000000d48bd0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe00000001237dce0, 0x5c, 0x9ffc000000d4c260, 0x100, 0x9ffc00000051d880) at sleepq_wait_sig+0x30
_sleep(0xe00000001237dce0, 0xe00000001237df20, 0x5c, 0x9ffc000000d4c260, 0x0, 0x0, 0x100) at _sleep+0x7b0
pipe_read(0xe000000012348d44, 0xa0000000003ed360, 0xa0000000003ed378, 0x0, 0xe00000001237dde8) at pipe_read+0x740
dofileread(0xe00000001282e900, 0x6, 0xe000000012348d20, 0xa0000000003ed360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe00000001282e900, 0x6, 0xa0000000003ed360) at kern_readv+0xa0
sys_read(0xe00000001282e900, 0xa0000000003ed4e8, 0x9ffc000000a40c80, 0x48d) at sys_read+0x100
syscall(0xe000000012797720, 0x140c2c000, 0x2000, 0xe00000001282e900, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 1104 tid 100055 td 0xe000000012079b00
cpu_switch(0xe000000012079b00, 0xe00000001117bb00, 0x9ffc000000ed4730, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe000000012079b00, 0xe00000001117bb00, 0x9ffc000000e9e118, 0x9ffc000000e9e100) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012079b00, 0x9ffc00000059de10) at mi_switch+0x3f0
sleepq_switch(0xe00000001261e2d8, 0x60, 0xe000000012079e78, 0xe000000012079b00) at sleepq_switch+0x2e0
sleepq_wait(0xe00000001261e2d8, 0x60, 0x9ffc000000d48810, 0xe000000012079b00, 0x9ffc0000004d4e20, 0x12ab) at sleepq_wait+0xb0
__lockmgr_args(0xe00000001261e2d8, 0x200100, 0xe00000001261e3a0, 0x9ffc000000d34f08, 0x60, 0x0, 0x9ffc000000d560c8, 0x835) at __lockmgr_args+0xae0
ffs_lock(0xa000000000285070, 0xe00000001261e2d8, 0x200100) at ffs_lock+0x160
VOP_LOCK1_APV(0x9ffc000000e13110, 0xa000000000285050, 0x9ffc000000df12f0) at VOP_LOCK1_APV+0x230
_vn_lock(0xe00000001261e270, 0x200100, 0x9ffc000000d560c8, 0x835) at _vn_lock+0x130
vget(0xe00000001261e270, 0x200100, 0xe000000012079b00, 0x9ffc0000011b9d48) at vget+0x100
cache_lookup(0xf, 0xa0000000002852e8, 0xa000000000285310, 0x0, 0x0) at cache_lookup+0xf70
vfs_cache_lookup(0xe000000012438750, 0x0, 0xa000000000285310) at vfs_cache_lookup+0x1a0
VOP_LOOKUP_APV(0x9ffc000000e13a50, 0xa0000000002850f8, 0x0, 0x9ffc000000d8a240) at VOP_LOOKUP_APV+0x270
lookup(0xa000000000285288, 0x200000, 0xe000000012438750, 0x0, 0xa000000000285318) at lookup+0xe30
namei(0xa000000000285288, 0xe000000011660000, 0x9ffc0000009b89f0, 0x0, 0xa0000000002852f8) at namei+0xb40
vn_open_cred(0xa000000000285288, 0xa000000000285390, 0x1c8, 0x0, 0xe000000011164000, 0xe000000012348960) at vn_open_cred+0x500
vn_open(0xa000000000285288, 0xa000000000285390, 0x1c8, 0xe000000012348960, 0x9ffc00000067ea60, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe000000012079b00, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe791d8) at kern_openat+0x490
kern_open(0xe000000012079b00, 0x1400c39a0, 0x0, 0x100000, 0xe791d8) at kern_open+0x40
sys_open(0xe000000012079b00, 0xa0000000002854e8, 0x9ffc000000a40c80, 0x48d) at sys_open+0x40
syscall(0xe000000011660000, 0x100000, 0x9ffc000000e791d8, 0xe000000012079b00, 0x0, 0x0, 0x9ffc000000a3cca0, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 1103 tid 100087 td 0xe0000000123e9680
cpu_switch(0xe0000000123e9680, 0xe00000001117b680, 0x9ffc000000ed3f60, 0x9ffc000000564d40) at cpu_switch+0xd0
sched_switch(0xe0000000123e9680, 0xe00000001117b680, 0x9ffc000000e9ee18, 0x9ffc000000e9ee00) at sched_switch+0x890
mi_switch(0x104, 0x

	
>How-To-Repeat:
	
>Fix:

	


>Release-Note:
>Audit-Trail:

From: Anton Shterenlikht <mexas@bris.ac.uk>
To: bug-followup@freebsd.org
Cc:  
Subject: ia64/187219: panic: 11.0-CURRENT #20 r260914: Panic String: deadlkres: possible deadlock detected for 0xe000000012079b00, blocked for 1801960 ticks
Date: Thu, 06 Mar 2014 02:28:38 -0800 (PST)

 Another deadlock panic with many DEBUG
 options removed from the kernel.
 
 # cat info.last 
 Dump header from device /dev/da2p1
   Architecture: ia64
   Architecture Version: 1
   Dump Length: 67584B (0 MB)
   Blocksize: 512
   Dumptime: Thu Mar  6 09:58:45 2014
   Hostname: mech-as28.men.bris.ac.uk
   Magic: FreeBSD Text Dump
   Version String: FreeBSD 11.0-CURRENT #22 r260914: Wed Mar  5 10:12:04 GMT 2014
     root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV
   Panic String: deadlkres: possible deadlock detected for 0xe000000012651200, blocked for 1801802 ticks
 
   Dump Parity: 3209697066
   Bounds: 3
   Dump Status: good
 # 
 
 # cat panic.txt 
 deadlkres: possible deadlock detected for 0xe000000012651200, blocked for 1801802 ticks
 #
 
 # cat version.txt 
 FreeBSD 11.0-CURRENT #22 r260914: Wed Mar  5 10:12:04 GMT 2014
     root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV
 #
 
 # cat config.txt 
 options CONFIG_AUTOGENERATED
 ident   ZEEV
 machine ia64
 cpu     ITANIUM2
 makeoptions     MODULES_OVERRIDE=geom/geom_part opensolaris zfs acl_nfs4 fdescfs
 makeoptions     DEBUG=-g
 options TEXTDUMP_VERBOSE
 options TEXTDUMP_PREFERRED
 options WITNESS_SKIPSPIN
 options WITNESS
 options INVARIANT_SUPPORT
 options INVARIANTS
 options DEADLKRES
 options GDB
 options DDB
 options KDB_TRACE
 options KDB
 options _KPOSIX_PRIORITY_SCHEDULING
 options QUOTA
 options UFS_GJOURNAL
 options UFS_DIRHASH
 options UFS_ACL
 options SYSVSHM
 options SYSVSEM
 options SYSVMSG
 options STACK
 options SOFTUPDATES
 options SMP
 options SCTP
 options SCSI_DELAY=5000
 options SCHED_ULE
 options PSEUDOFS
 options PROCFS
 options PRINTF_BUFR_SIZE=128
 options PREEMPTION
 options P1003_1B_SEMAPHORES
 options NFS_ROOT
 options NFSD
 options NFSLOCKD
 options NFSCL
 options MSDOSFS
 options MD_ROOT
 options MAC
 options KTRACE
 options INET6
 options INET
 options INCLUDE_CONFIG_FILE
 options GEOM_LABEL
 options FFS
 options TMPFS
 options NULLFS
 options FDESCFS
 options COMPAT_FREEBSD7
 options CD9660
 options CAPABILITIES
 options CAPABILITY_MODE
 options AUDIT
 options NEW_PCIB
 options GEOM_PART_MBR
 options GEOM_PART_GPT
 options GEOM_PART_BSD
 device  acpi
 device  io
 device  mem
 device  uart_ns8250
 device  miibus
 device  pci
 device  scbus
 device  usb
 device  isp
 device  ispfw
 device  mpt
 device  cd
 device  da
 device  pass
 device  sa
 device  ehci
 device  ohci
 device  uhci
 device  umass
 device  em
 device  ether
 device  faith
 device  firmware
 device  gif
 device  loop
 device  md
 device  puc
 device  random
 device  tun
 device  uart
 device  vlan
 # 
 
 *****************************
 msgbuf.txt
 *****************************
 
 GDB: debug ports: uart
 GDB: current port: uart
 KDB: debugger backends: ddb gdb
 KDB: current backend: ddb
 Copyright (c) 1992-2014 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 11.0-CURRENT #22 r260914: Wed Mar  5 10:12:04 GMT 2014
     root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV ia64
 gcc version 4.2.1 20070831 patched [FreeBSD]
 WARNING: WITNESS option enabled, expect reduced performance.
 CPU: Madison II (1600 MHz Itanium 2)
   Origin = "GenuineIntel"  Revision = 2
   Features = 0x1<LB>
 real memory  = 10737418240 (10240 MB)
 avail memory = 10508304384 (10021 MB)
 FPSWA Revision = 0x10012, Entry = 0xe0000040ffcc4050
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP)
 cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0
 random: <Software, Yarrow> initialized
 Event timer "ITC" frequency 1600000000 Hz quality 1000
 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603)
 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603)
 acpi0: <HP> on motherboard
 acpi0: Power Button (fixed)
 acpi0: Sleep Button (fixed)
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
 acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0
 acpi_tz0: <Thermal Zone> on acpi0
 pcib0: <ACPI Host-PCI bridge> on acpi0
 pcib0: could not evaluate _ADR - AE_NOT_FOUND
 pci0: <ACPI PCI bus> on pcib0
 ohci0: <NEC uPD 9210 USB controller> mem 0x80002000-0x80002fff irq 16 at device 1.0 on pci0
 usbus0 on ohci0
 ohci1: <NEC uPD 9210 USB controller> mem 0x80001000-0x80001fff irq 17 at device 1.1 on pci0
 usbus1 on ohci1
 ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x80000000-0x800000ff irq 18 at device 1.2 on pci0
 usbus2: EHCI version 0.95
 usbus2 on ehci0
 pci0: <mass storage, ATA> at device 2.0 (no driver attached)
 pcib1: <ACPI Host-PCI bridge> on acpi0
 pcib1: could not evaluate _ADR - AE_NOT_FOUND
 pci32: <ACPI PCI bus> on pcib1
 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2100-0x21ff mem 0x903a0000-0x903bffff,0x90380000-0x9039ffff irq 27 at device 1.0 on pci32
 mpt0: MPI Version=1.2.12.0
 mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0x90360000-0x9037ffff,0x90340000-0x9035ffff irq 28 at device 1.1 on pci32
 mpt1: MPI Version=1.2.12.0
 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2240-0x227f mem 0x90320000-0x9033ffff,0x90280000-0x902fffff irq 29 at device 2.0 on pci32
 em0: Ethernet address: 00:13:21:5b:05:1c
 em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2200-0x223f mem 0x90300000-0x9031ffff irq 30 at device 2.1 on pci32
 em1: Ethernet address: 00:13:21:5b:05:1d
 pcib2: <ACPI Host-PCI bridge> on acpi0
 pcib2: could not evaluate _ADR - AE_NOT_FOUND
 pci64: <ACPI PCI bus> on pcib2
 pcib3: <ACPI Host-PCI bridge> on acpi0
 pcib3: could not evaluate _ADR - AE_NOT_FOUND
 pci96: <ACPI PCI bus> on pcib3
 pcib4: <ACPI Host-PCI bridge> on acpi0
 pcib4: could not evaluate _ADR - AE_NOT_FOUND
 pci128: <ACPI PCI bus> on pcib4
 pcib5: <ACPI Host-PCI bridge> on acpi0
 pcib5: could not evaluate _ADR - AE_NOT_FOUND
 pci192: <ACPI PCI bus> on pcib5
 isp0: <Qlogic ISP 2422 PCI FC-AL Adapter> port 0xc000-0xc0ff mem 0xe0040000-0xe0040fff irq 71 at device 1.0 on pci192
 pcib6: <ACPI Host-PCI bridge> on acpi0
 pcib6: could not evaluate _ADR - AE_NOT_FOUND
 pci224: <ACPI PCI bus> on pcib6
 uart0: <HP Auxiliary Diva Serial Port> mem 0xf8051000-0xf805100f irq 82 at device 1.0 on pci224
 puc0: <HP Diva Serial [GSP] Multiport UART - Everest SP2> mem 0xf8050000-0xf8050fff,0xf8020000-0xf803ffff irq 82 at device 1.1 on pci224
 uart1: <Non-standard ns8250 class UART with FIFOs> at port 1 on puc0
 uart1: console (9600,n,8,1)
 uart2: <Non-standard ns8250 class UART with FIFOs> at port 3 on puc0
 vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf0000000-0xf7ffffff,0xf8040000-0xf804ffff at device 2.0 on pci224
 vgapci0: Boot video device
 uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0
 uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0
 uart4: debug port (9600,n,8,1)
 Timecounters tick every 1.000 msec
 usbus0: 12Mbps Full Speed USB v1.0
 usbus1: 12Mbps Full Speed USB v1.0
 usbus2: 480Mbps High Speed USB v2.0
 ugen1.1: <NEC> at usbus1
 ugen0.1: <NEC> at usbus0
 uhub0: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
 uhub1: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
 ugen2.1: <NEC> at usbus2
 uhub2: <NEC EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
 random: unblocking device.
 uhub0: 2 ports with 2 removable, self powered
 uhub1: 3 ports with 3 removable, self powered
 uhub2: 5 ports with 5 removable, self powered
 pass1 at isp0 bus 0 scbus2 target 0 lun 0
 da1 at isp0 bus 0 scbus2 target 0 lun 1
 pass1: <COMPAQ MSA1000 4.32> Fixed Storage Array SCSI-4 device 
 da1: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device 
 pass1: Serial Number P56350GX3RX04Z
 da1: Serial Number P56350GX3RX04Z
 pass1: 200.000MB/s transfersda1: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 WWPN 0x500805f3000ec221 PortID 0x10000
  PortID 0x10000
 pass1: Command Queueing enabled
 da1: Command Queueing enabled
 da1: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
 da2 at isp0 bus 0 scbus2 target 0 lun 2
 da2: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device 
 da2: Serial Number P56350GX3RX04Z
 da2: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
 da2: Command Queueing enabled
 da2: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
 da3 at isp0 bus 0 scbus2 target 0 lun 5
 da3: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device 
 da3: Serial Number P56350GX3RX04Z
 da3: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
 da3: Command Queueing enabled
 da3: 140011MB (286744185 512 byte sectors: 255H 63S/T 17849C)
 da0 at mpt0 bus 0 scbus0 target 0 lun 0
 da0: <HP 73.4G ST373454LC HPC2> Fixed Direct Access SCSI-3 device 
 da0: Serial Number 3KP05D0W00007532H5SA
 da0: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
 da0: Command Queueing enabled
 da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
 WARNING: WITNESS option enabled, expect reduced performance.
 Trying to mount root from ufs:/dev/da0p2 [rw]...
 WARNING: / was not properly dismounted
 <118>Setting hostuuid: ee652761-3dcc-11df-9074-0013215b051c.
 <118>Setting hostid: 0x8968724b.
 <118>Starting ddb.
 <118>Entropy harvesting: interrupts ethernet point_to_point swi.
 <118>Starting file system checks:
 <118>/dev/da0p2: 562514 files, 3749278 used, 30867146 free (305122 frags, 3820253 blocks, 0.9% fragmentation)
 <118>/dev/da1p1: 2152 files, 81832 used, 17141529 free (465 frags, 2142633 blocks, 0.0% fragmentation)
 <118>/dev/da3p1: 6077 files, 1328197 used, 68099306 free (6234 frags, 8511634 blocks, 0.0% fragmentation)
 <118>Mounting local file systems:.
 <118>Writing entropy file:.
 <118>Setting hostname: mech-as28.men.bris.ac.uk.
 <118>Starting Network: lo0 em0 em1.
 <118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 <118>	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
 <118>	inet6 ::1 prefixlen 128 
 <118>	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
 <118>	inet 127.0.0.1 netmask 0xff000000 
 <118>	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 <118>em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 <118>	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
 <118>	ether 00:13:21:5b:05:1c
 <118>	inet 137.222.187.28 netmask 0xffffff00 broadcast 137.222.187.255 
 <118>	inet6 fe80::213:21ff:fe5b:51c%em0 prefixlen 64 scopeid 0x1 
 <118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 <118>	media: Ethernet autoselect
 <118>	status: no carrier
 <118>em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 <118>	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
 <118>	ether 00:13:21:5b:05:1d
 <118>	inet 10.10.10.14 netmask 0xffffff00 broadcast 10.10.10.255 
 <118>	inet6 fe80::213:21ff:fe5b:51d%em1 prefixlen 64 scopeid 0x2 
 <118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 <118>	media: Ethernet autoselect (100baseTX <full-duplex>)
 <118>	status: active
 <118>Starting devd.
 <118>Starting pflogd: 
 <118>add net default: gateway 137.222.187.250
 <118>add net fe80::: gateway ::1
 <118>add net ff02::: gateway ::1
 <118>add net ::ffff:0.0.0.0: gateway ::1
 <118>add net ::0.0.0.0: gateway ::1
 <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
 <118>Creating and/or trimming log files.
 <118>Starting syslogd.
 <118>savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe000000011781680, blocked for 1800556 ticks
 <118>Mar  6 09:26:13 mech-as28 savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe000000011781680, blocked for 1800556 ticks
 <118>savecore: writing core to /var/crash/textdump.tar.2
 <118>/var/crash/vmcore.2 not found
 <118>Starting casperd.
 lock order reversal:
  1st 0xa00000009e049220 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3081
  2nd 0xe000000012136c00 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000d89c0) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc000000572870) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000e2b2d8, 0x9ffc0000005a8990) at kdb_backtrace+0xc0
 _witness_debugger(0x1, 0x9ffc000000d2c138, 0x9ffc0000005ac370, 0xb9d, 0x9ffc000000d59270) at _witness_debugger+0x60
 witness_checkorder(0xe000000012136c00, 0x9ffc000000d2ba80, 0x9ffc000000d59270, 0x11c, 0x0) at witness_checkorder+0x15c0
 _sx_xlock(0xe000000012136c00, 0x0, 0x9ffc000000d59270, 0x11c) at _sx_xlock+0x120
 ufsdirhash_acquire(0xe0000000123f5500, 0xe000000012136c00, 0x9ffc000000975d60) at ufsdirhash_acquire+0x50
 ufsdirhash_add(0xe0000000123f5500, 0xa0000000002f1098, 0x934, 0xa0000000002f1040, 0x9ffc00000097dbf0, 0xda1) at ufsdirhash_add+0x20
 ufs_direnter(0xe000000012664760, 0xe0000000126641d8, 0xa0000000002f1098, 0xa0000000002f1318, 0xa00000009e049720, 0x0) at ufs_direnter+0x1260
 ufs_mkdir(0xa0000000002f1370, 0xa0000000002f1098, 0xe000000012664760) at ufs_mkdir+0x14f0
 VOP_MKDIR_APV(0x9ffc000000df2660, 0xa0000000002f1370, 0xa0000000002f1318, 0x0, 0x9ffc0000006650c0, 0x818, 0x9ffc0000010015a0) at VOP_MKDIR_APV+0x220
 kern_mkdirat(0xe000000012384900, 0xffffffffffffff9c, 0x140804040, 0x0, 0x1c0) at kern_mkdirat+0x3a0
 kern_mkdir(0xe000000012384900, 0x140804040, 0x0, 0x1c0) at kern_mkdir+0x40
 sys_mkdir(0xe000000012384900, 0xa0000000002f14e8, 0x9ffc000000a2cc80, 0x48d) at sys_mkdir+0x30
 syscall(0xe00000001244ade0, 0x1c0, 0x9ffc000000a29630, 0xe000000012384900, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 <118>Clearing /tmp (X related).
 <118>Recovering vi editor sessions:.
 <118>Updating motd:.
 <118>Mounting late file systems:.
 <118>Starting ntpd.
 <118>Starting bsdstats.
 <118>Posting monthly OS statistics to rpt.bsdstats.org
 <118>Performing sanity check on sshd configuration.
 <118>Starting sshd.
 <118>Starting sendmail_submit.
 <118>Starting sendmail_msp_queue.
 <118>Starting cron.
 <118>Local package initialization:cd: /var/portbuild/: No such file or directory
 <118>.
 <118>Starting inetd.
 <118>Starting background file system checks in 60 seconds.
 <118>
 <118>Thu Mar  6 09:26:22 GMT 2014
 <118>Mar  6 09:26:58 mech-as28 su: mexas to root on /dev/pts/0
 panic: deadlkres: possible deadlock detected for 0xe000000012651200, blocked for 1801802 ticks
 
 cpuid = 0
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000d89c0) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc000000572870) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000e2b2d8, 0x9ffc0000004f99e0, 0x40c, 0x9ffc0000010015a0) at kdb_backtrace+0xc0
 vpanic(0x9ffc000000d16c10, 0xa00000009dfff518) at vpanic+0x260
 panic(0x9ffc000000d16c10, 0x9ffc000000d16e68, 0xe000000012651200, 0x1b7e4a) at panic+0x80
 deadlkres(0x9ffc000000d16ba0, 0xe000000012651200, 0x9ffc000000dc8358, 0x9ffc000000d16ba0) at deadlkres+0x700
 fork_exit(0x9ffc000000d6b130, 0x0, 0xa00000009dfff550) at fork_exit+0x120
 enter_userland() at enter_userland
 KDB: enter: panic
 
 *************************************
 ddb.txt
 *************************************
 db:0:kdb.enter.panic>  run lockinfo
 db:1:lockinfo> show locks
 db:1:locks>  show alllocks
 Process 1062 (atrun) thread 0xe000000012769680 (100100)
 shared lockmgr ufs (ufs) r = 0 (0xe000000012425c10) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1061 (sh) thread 0xe000000012766d80 (100099)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000120f1c10) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1058 (atrun) thread 0xe0000000128cc480 (100096)
 shared lockmgr ufs (ufs) r = 0 (0xe000000012425c10) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1056 (atrun) thread 0xe0000000123fe480 (100073)
 shared lockmgr ufs (ufs) r = 0 (0xe000000012425c10) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1054 (sh) thread 0xe000000012650900 (100083)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000120f1c10) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1052 (atrun) thread 0xe00000001237e000 (100070)
 shared lockmgr ufs (ufs) r = 0 (0xe000000012425c10) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1050 (atrun) thread 0xe0000000123fed80 (100071)
 shared lockmgr ufs (ufs) r = 0 (0xe000000012425c10) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1048 (dd) thread 0xe00000001237f680 (100065)
 exclusive lockmgr bufwait (bufwait) r = 0 (0xa00000009e03d9c0) locked @ /usr/src/sys/kern/vfs_bio.c:3081
 exclusive lockmgr ufs (ufs) r = 0 (0xe000000012838068) locked @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1711
 exclusive lockmgr ufs (ufs) r = 0 (0xe0000000124245f0) locked @ /usr/src/sys/kern/vfs_lookup.c:518
 Process 1045 (atrun) thread 0xe000000012768000 (100091)
 exclusive lockmgr bufwait (bufwait) r = 0 (0xa00000009e06a348) locked @ /usr/src/sys/kern/vfs_bio.c:2345
 exclusive lockmgr ufs (ufs) r = 0 (0xe0000000128914b0) locked @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1711
 shared lockmgr ufs (ufs) r = 0 (0xe000000012425c10) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1043 (fsck_ufs) thread 0xe000000012385b00 (100075)
 exclusive lockmgr ufs (ufs) r = 0 (0xe0000000124425f0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1042 (grep) thread 0xe000000012384480 (100062)
 exclusive lockmgr bufwait (bufwait) r = 0 (0xa00000009e06a8d0) locked @ /usr/src/sys/kern/vfs_bio.c:2345
 shared lockmgr ufs (ufs) r = 0 (0xe0000000128392d8) locked @ /usr/src/sys/kern/vfs_vnops.c:704
 Process 1004 (login) thread 0xe000000011691b00 (100051)
 exclusive lockmgr ufs (ufs) r = 0 (0xe000000012424d50) locked @ /usr/src/sys/kern/vfs_vnops.c:373
 Process 20 (syncer) thread 0xe00000001231c900 (100059)
 exclusive lockmgr ufs (ufs) r = 0 (0xe000000012645de8) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 exclusive lockmgr syncer (syncer) r = 0 (0xe000000012425a38) locked @ /usr/src/sys/kern/vfs_subr.c:1720
 db:1:alllocks>  show lockedvnods
 Locked vnodes
 db:0:kdb.enter.panic>  show pcpu
 cpuid        = 0
 dynamic pcpu = 0x40040040ff1df980
 curthread    = 0xe000000011780480: pid 0 "deadlkres"
 curpcb       = 0
 fpcurthread  = none
 idlethread   = 0xe000000011151b00: tid 100003 "idle: cpu0"
 MD: vhpt       = 0xe0000040ff000000
 MD: lid        = 0
 MD: clock      = 0x31b2d05669a
 MD: clock_mode = 2
 MD: clock_load = 0
 MD: stats      = 0x9ffc0000011a4a30
 MD: pmap       = 0x9ffc000000e53920
 spin locks held:
 db:0:kdb.enter.panic>  bt
 Tracing pid 0 tid 100049 td 0xe000000011780480
 kdb_enter(0x9ffc000000d21190, 0x9ffc000000d21190, 0x9ffc0000004f9a30, 0x40c) at kdb_enter+0x92
 vpanic(0x9ffc000000d16c10, 0xa00000009dfff518) at vpanic+0x2b0
 panic(0x9ffc000000d16c10, 0x9ffc000000d16e68, 0xe000000012651200, 0x1b7e4a) at panic+0x80
 deadlkres(0x9ffc000000d16ba0, 0xe000000012651200, 0x9ffc000000dc8358, 0x9ffc000000d16ba0) at deadlkres+0x700
 fork_exit(0x9ffc000000d6b130, 0x0, 0xa00000009dfff550) at fork_exit+0x120
 enter_userland() at enter_userland
 db:0:kdb.enter.panic>  ps
   pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
  1062  1060  1062     0  Ds      ufs      0xe0000000128914b0 atrun
  1061  1059  1061     2  Ds      ufs      0xe0000000124245f0 sh
  1060   944   944     0  S       piperd   0xe0000000123aba78 cron
  1059   944   944     0  S       piperd   0xe00000001289f0d8 cron
  1058  1057  1058     0  Ds      ufs      0xe0000000128914b0 atrun
  1057   944   944     0  S       piperd   0xe0000000123ab5a8 cron
  1056  1055  1056     0  Ds      ufs      0xe0000000128914b0 atrun
  1055   944   944     0  S       piperd   0xe00000001289f5a8 cron
  1054  1053  1054     2  Ds      ufs      0xe0000000124245f0 sh
  1053   944   944     0  S       piperd   0xe0000000126a55a8 cron
  1052  1051  1052     0  Ds      ufs      0xe0000000128914b0 atrun
  1051   944   944     0  S       piperd   0xe0000000123aae70 cron
  1050  1049  1050     0  Ds      ufs      0xe0000000128914b0 atrun
  1049   944   944     0  S       piperd   0xe0000000126a50d8 cron
  1048  1047  1047     2  D       biowr    0xa00000009e03d938 dd
  1047  1046  1047     2  Ss      wait     0xe00000001244a000 sh
  1046   944   944     0  S       piperd   0xe0000000126a4c08 cron
  1045  1044  1045     0  Ds      biord    0xa00000009e06a2c0 atrun
  1044   944   944     0  S       piperd   0xe00000001289fa78 cron
  1043   999    23     0  D       getblk   0xa00000009e032160 fsck_ufs
  1042  1039  1042     0  D+      biord    0xa00000009e06a848 grep
  1039  1038  1039     0  S+      wait     0xe00000001244cde0 sh
  1038  1034  1038  1001  S+      wait     0xe0000000126ae4a0 su
  1034  1033  1034  1001  Ss+     pause    0xe00000001244d328 tcsh
  1033  1031  1031  1001  S       select   0xe000000012241640 sshd
  1031   933  1031     0  Ss      select   0xe000000012241a40 sshd
  1004     1  1004     0  Ds+     getblk   0xa00000009e032160 login
  1003     1     1     0  S       ttydcd   0xe00000001173b0e8 getty
  1000     1    23     0  S+      piperd   0xe0000000123aa9a0 logger
   999     1    23     0  S+      wait     0xe0000000123f84a0 fsck
   983     1   983     0  Ss      select   0xe000000012242040 inetd
   944     1   944     0  Ss      nanslp   0x9ffc000000e60598 cron
   939     1   939    25  Ss      pause    0xe00000001244ae88 sendmail
   936     1   936     0  Ss      select   0xe000000012241440 sendmail
   933     1   933     0  Ss      select   0xe00000001178a0c0 sshd
   866     1   866     0  Ss      select   0xe00000001178b8c0 ntpd
   802     1   802     0  Ss      select   0xe00000001178b1c0 casperd
   801     1    23     0  S+      select   0xe000000011699e40 casperd
   772     1   772     0  Ds      ufs      0xe0000000124425f0 syslogd
   647     1   647     0  Ss      select   0xe000000011736bc0 devd
   138     1   138     0  Ss      pause    0xe00000001244b328 adjkerntz
    22     0     0     0  DL      sdflush  0x9ffc000000e2c480 [softdepflush]
    21     0     0     0  DL      vlruwt   0xe000000012318de0 [vnlru]
    20     0     0     0  DL      getblk   0xa00000009e032160 [syncer]
    19     0     0     0  DL      psleep   0x9ffc000000e2b860 [bufdaemon]
    18     0     0     0  DL      pgzero   0x9ffc000000e2c7ec [pagezero]
    17     0     0     0  DL      psleep   0x9ffc000000e2c754 [vmdaemon]
     9     0     0     0  DL      psleep   0x9ffc000000e2c784 [pagedaemon]
     6     0     0     0  DL      waiting_ 0x9ffc000001198d70 [sctp_iterator]
     5     0     0     0  DL      ispf     0xe00000001173c800 [isp0: fc_thrd0]
     4     0     0     0  DL      idle     0xa000000000a22000 [mpt_recovery1]
     3     0     0     0  DL      idle     0xa000000000a0a000 [mpt_recovery0]
    16     0     0     0  DL      (threaded)                  [usb]
 100039                   D       -        0xa000000000a06e18 [usbus2]
 100038                   D       -        0xa000000000a06dc0 [usbus2]
 100037                   D       -        0xa000000000a06d68 [usbus2]
 100036                   D       -        0xa000000000a06d10 [usbus2]
 100034                   D       -        0xa0000000009fd460 [usbus1]
 100033                   D       -        0xa0000000009fd408 [usbus1]
 100032                   D       -        0xa0000000009fd3b0 [usbus1]
 100031                   D       -        0xa0000000009fd358 [usbus1]
 100029                   D       -        0xa0000000009f9460 [usbus0]
 100028                   D       -        0xa0000000009f9408 [usbus0]
 100027                   D       -        0xa0000000009f93b0 [usbus0]
 100026                   D       -        0xa0000000009f9358 [usbus0]
    15     0     0     0  DL      tzpoll   0x9ffc000000e2a288 [acpi_thermal]
     2     0     0     0  DL      (threaded)                  [cam]
 100054                   D       ccb_scan 0x9ffc000000e2cb38 [scanner]
 100015                   D       -        0x9ffc000000e2cc80 [doneq0]
    14     0     0     0  DL      -        0x9ffc000000e2a578 [rand_harvestq]
    13     0     0     0  DL      (threaded)                  [geom]
 100011                   D       -        0x9ffc000000e2aa28 [g_down]
 100010                   D       -        0x9ffc000000e2aa20 [g_up]
 100009                   D       -        0x9ffc000000e2aa10 [g_event]
    12     0     0     0  WL      (threaded)                  [intr]
 100048                   I                                   [swi0: uart uart+++]
 100046                   I                                   [irq71: isp0]
 100042                   I                                   [irq28: mpt1]
 100040                   I                                   [irq27: mpt0]
 100035                   I                                   [irq18: ehci0]
 100030                   I                                   [irq17: ohci1]
 100025                   I                                   [irq16: ohci0]
 100023                   I                                   [swi6: Giant taskq]
 100021                   I                                   [swi5: fast taskq]
 100014                   I                                   [swi6: task queue]
 100008                   I                                   [swi1: netisr 0]
 100007                   I                                   [swi3: vm]
 100006                   I                                   [swi4: clock]
 100005                   I                                   [swi4: clock]
    11     0     0     0  RL      (threaded)                  [idle]
 100004                   Run     CPU 1                       [idle: cpu1]
 100003                   CanRun                              [idle: cpu0]
     1     0     1     0  SLs     wait     0xe00000001114e4a0 [init]
    10     0     0     0  DL      audit_wo 0x9ffc00000119d408 [audit]
     0     0     0     0  RLs     (threaded)                  [kernel]
 100053                   D       -        0xe000000011142300 [CAM taskq]
 100049                   Run     CPU 0                       [deadlkres]
 100045                   D       -        0xe000000011207e00 [em1 taskq]
 100044                   D       -        0xe000000011208200 [em0 taskq]
 100022                   D       -        0xe000000011141a00 [thread taskq]
 100020                   D       -        0xe000000011141d00 [kqueue taskq]
 100019                   D       -        0xe000000011141e00 [ffs_trim taskq]
 100018                   D       -        0xe000000011142000 [acpi_task_2]
 100017                   D       -        0xe000000011142000 [acpi_task_1]
 100016                   D       -        0xe000000011142000 [acpi_task_0]
 100012                   D       -        0xe000000011143100 [firmware taskq]
 100000                   D       swapin   0x9ffc000000e52ee8 [swapper]
 db:0:kdb.enter.panic>  alltrace
 
 Tracing command atrun pid 1062 tid 100100 td 0xe000000012769680
 cpu_switch(0xe000000012769680, 0xe000000012769b00, 0x9ffc000000eb1ec0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012769680, 0xe000000012769b00, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012769680, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000128914b0, 0x60, 0xe0000000127699f8, 0xe000000012769680) at sleepq_switch+0x2e0
 sleepq_wait(0xe0000000128914b0, 0x60, 0x9ffc000000d28b50, 0xe000000012769680, 0x9ffc0000004c43a0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe0000000128914b0, 0x200100, 0xe0000000128914e0, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0x16b0
 ffs_lock(0xa000000000399140, 0xe0000000128914b0, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000399120, 0x8) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012891448, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012891448, 0x200100, 0xe000000012769680, 0x57) at vget+0x100
 vfs_hash_get(0xe000000012346330, 0x1c2a80, 0xe000000012891448, 0xe000000012769680, 0xa000000000399198, 0x0, 0x0, 0x9ffc000000d353d8) at vfs_hash_get+0x180
 ffs_vgetf(0xe000000012346330, 0x1c2a80, 0x200000, 0xa000000000399198, 0x0) at ffs_vgetf+0x50
 ffs_vget(0xe000000012346330, 0x1c2a80, 0x200000, 0xa000000000399198, 0x9ffc00000097c2b0, 0x1836) at ffs_vget+0x40
 ufs_lookup_ino(0xe000000012425ba8, 0xa000000000399310, 0xa000000000399338, 0x0) at ufs_lookup_ino+0x1cd0
 ufs_lookup(0xa0000000003991e8, 0x9ffc000000a42240, 0x38a) at ufs_lookup+0x40
 VOP_CACHEDLOOKUP_APV(0x9ffc000000df2660, 0xa0000000003991d0, 0xa000000000399338, 0x0, 0x9ffc000000630460) at VOP_CACHEDLOOKUP_APV+0x220
 vfs_cache_lookup(0xe000000012425ba8, 0x0, 0xa000000000399338) at vfs_cache_lookup+0x1f0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa000000000399208, 0x4004144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000003992b0, 0x200000, 0xe000000012425ba8, 0x0, 0xa000000000399340) at lookup+0xe10
 namei(0xa0000000003992b0, 0xe0000000128c3280, 0x4000144, 0x0, 0xa000000000399320, 0xa000000000399368, 0xa000000000399360, 0xa000000000399310) at namei+0xb40
 kern_chdir(0xe000000012769680, 0x100004d10, 0x0) at kern_chdir+0x80
 sys_chdir(0xe000000012769680, 0xa0000000003994e8, 0x9ffc000000a2cc80, 0x48d) at sys_chdir+0x30
 syscall(0xe0000000128c3280, 0x7fffffffffffed40, 0x100004e38, 0xe000000012769680, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1061 tid 100099 td 0xe000000012766d80
 cpu_switch(0xe000000012766d80, 0xe000000011151b00, 0x9ffc000000eb2168, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012766d80, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012766d80, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000124245f0, 0x60, 0xe0000000127670f8, 0xe000000012766d80) at sleepq_switch+0x2e0
 sleepq_wait(0xe0000000124245f0, 0x60, 0x9ffc000000d28b50, 0xe000000012766d80, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe0000000124245f0, 0x200100, 0xe000000012424620, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa000000000391090, 0xe0000000124245f0, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000391070, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012424588, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012424588, 0x200100, 0xe000000012766d80, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0x7, 0xa000000000391220, 0xa000000000391248, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe0000000120f1ba8, 0x0, 0xa000000000391248) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa000000000391118, 0x400c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000003911c0, 0x200000, 0xe0000000120f1ba8, 0x0, 0xa000000000391250) at lookup+0xe10
 namei(0xa0000000003911c0, 0xe0000000128e1280, 0x4000144, 0x0, 0xa000000000391230, 0xa000000000391278, 0xa000000000391270, 0xa000000000391220) at namei+0xb40
 kern_statat_vnhook(0xe000000012766d80, 0x0, 0xffffffffffffff9c, 0x140c0ab10, 0x0, 0xa000000000391320, 0x0) at kern_statat_vnhook+0xf0
 kern_statat(0xe000000012766d80, 0x0, 0xffffffffffffff9c, 0x140c0ab10, 0x0, 0xa000000000391320) at kern_statat+0x40
 kern_stat(0xe000000012766d80, 0x140c0ab10, 0x0, 0xa000000000391320) at kern_stat+0x40
 sys_stat(0xe000000012766d80, 0xa0000000003914e8, 0x9ffc000000a2cc80, 0x48d) at sys_stat+0x30
 syscall(0xe0000000128e1280, 0x7fffffffffffe708, 0x14031d640, 0xe000000012766d80, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1060 tid 100098 td 0xe000000012769b00
 cpu_switch(0xe000000012769b00, 0xe000000011151680, 0x9ffc000000eb2370, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012769b00, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012769b00, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000123aba78, 0x5c, 0xe000000012769e78, 0xe000000012769b00) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000123aba78, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000123aba78, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000123aba78, 0xe0000000123abcb8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000126f7014, 0xa000000000389360, 0xa000000000389378, 0x0, 0xe0000000123abb80) at pipe_read+0x740
 dofileread(0xe000000012769b00, 0x6, 0xe0000000126f6ff0, 0xa000000000389360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012769b00, 0x6, 0xa000000000389360) at kern_readv+0xa0
 sys_read(0xe000000012769b00, 0xa0000000003894e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe0000000128c3720, 0x140c26000, 0x2000, 0xe000000012769b00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1059 tid 100097 td 0xe0000000128cc000
 cpu_switch(0xe0000000128cc000, 0xe000000011151680, 0x9ffc000000eb0b60, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000128cc000, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000128cc000, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe00000001289f0d8, 0x5c, 0xe0000000128cc378, 0xe0000000128cc000) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe00000001289f0d8, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe00000001289f0d8, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe00000001289f0d8, 0xe00000001289f318, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe00000001238de24, 0xa000000000381360, 0xa000000000381378, 0x0, 0xe00000001289f1e0) at pipe_read+0x740
 dofileread(0xe0000000128cc000, 0x6, 0xe00000001238de00, 0xa000000000381360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe0000000128cc000, 0x6, 0xa000000000381360) at kern_readv+0xa0
 sys_read(0xe0000000128cc000, 0xa0000000003814e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe0000000128c8000, 0x140c26000, 0x2000, 0xe0000000128cc000, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command atrun pid 1058 tid 100096 td 0xe0000000128cc480
 cpu_switch(0xe0000000128cc480, 0xe000000011151b00, 0x9ffc000000eb1ec0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000128cc480, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000128cc480, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000128914b0, 0x60, 0xe0000000128cc7f8, 0xe0000000128cc480) at sleepq_switch+0x2e0
 sleepq_wait(0xe0000000128914b0, 0x60, 0x9ffc000000d28b50, 0xe0000000128cc480, 0x9ffc0000004c43a0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe0000000128914b0, 0x200100, 0xe0000000128914e0, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0x16b0
 ffs_lock(0xa000000000379140, 0xe0000000128914b0, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000379120, 0x8) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012891448, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012891448, 0x200100, 0xe0000000128cc480, 0x57) at vget+0x100
 vfs_hash_get(0xe000000012346330, 0x1c2a80, 0xe000000012891448, 0xe0000000128cc480, 0xa000000000379198, 0x0, 0x0, 0x9ffc000000d353d8) at vfs_hash_get+0x180
 ffs_vgetf(0xe000000012346330, 0x1c2a80, 0x200000, 0xa000000000379198, 0x0) at ffs_vgetf+0x50
 ffs_vget(0xe000000012346330, 0x1c2a80, 0x200000, 0xa000000000379198, 0x9ffc00000097c2b0, 0x1836) at ffs_vget+0x40
 ufs_lookup_ino(0xe000000012425ba8, 0xa000000000379310, 0xa000000000379338, 0x0) at ufs_lookup_ino+0x1cd0
 ufs_lookup(0xa0000000003791e8, 0x9ffc000000a42240, 0x38a) at ufs_lookup+0x40
 VOP_CACHEDLOOKUP_APV(0x9ffc000000df2660, 0xa0000000003791d0, 0xa000000000379338, 0x0, 0x9ffc000000630460) at VOP_CACHEDLOOKUP_APV+0x220
 vfs_cache_lookup(0xe000000012425ba8, 0x0, 0xa000000000379338) at vfs_cache_lookup+0x1f0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa000000000379208, 0x4004144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000003792b0, 0x200000, 0xe000000012425ba8, 0x0, 0xa000000000379340) at lookup+0xe10
 namei(0xa0000000003792b0, 0xe0000000128c84a0, 0x4000144, 0x0, 0xa000000000379320, 0xa000000000379368, 0xa000000000379360, 0xa000000000379310) at namei+0xb40
 kern_chdir(0xe0000000128cc480, 0x100004d10, 0x0) at kern_chdir+0x80
 sys_chdir(0xe0000000128cc480, 0xa0000000003794e8, 0x9ffc000000a2cc80, 0x48d) at sys_chdir+0x30
 syscall(0xe0000000128c84a0, 0x7fffffffffffed40, 0x100004e38, 0xe0000000128cc480, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1057 tid 100095 td 0xe000000012767200
 cpu_switch(0xe000000012767200, 0xe000000011151680, 0x9ffc000000eb09a8, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012767200, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012767200, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000123ab5a8, 0x5c, 0xe000000012767578, 0xe000000012767200) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000123ab5a8, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000123ab5a8, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000123ab5a8, 0xe0000000123ab7e8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000126f7104, 0xa000000000371360, 0xa000000000371378, 0x0, 0xe0000000123ab6b0) at pipe_read+0x740
 dofileread(0xe000000012767200, 0x6, 0xe0000000126f70e0, 0xa000000000371360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012767200, 0x6, 0xa000000000371360) at kern_readv+0xa0
 sys_read(0xe000000012767200, 0xa0000000003714e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe0000000128e1720, 0x140c26000, 0x2000, 0xe000000012767200, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command atrun pid 1056 tid 100073 td 0xe0000000123fe480
 cpu_switch(0xe0000000123fe480, 0xe000000011151680, 0x9ffc000000eb1ec0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000123fe480, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000123fe480, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000128914b0, 0x60, 0xe0000000123fe7f8, 0xe0000000123fe480) at sleepq_switch+0x2e0
 sleepq_wait(0xe0000000128914b0, 0x60, 0x9ffc000000d28b50, 0xe0000000123fe480, 0x9ffc0000004c43a0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe0000000128914b0, 0x200100, 0xe0000000128914e0, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0x16b0
 ffs_lock(0xa0000000002c1140, 0xe0000000128914b0, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa0000000002c1120, 0x8) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012891448, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012891448, 0x200100, 0xe0000000123fe480, 0x57) at vget+0x100
 vfs_hash_get(0xe000000012346330, 0x1c2a80, 0xe000000012891448, 0xe0000000123fe480, 0xa0000000002c1198, 0x0, 0x0, 0x9ffc000000d353d8) at vfs_hash_get+0x180
 ffs_vgetf(0xe000000012346330, 0x1c2a80, 0x200000, 0xa0000000002c1198, 0x0) at ffs_vgetf+0x50
 ffs_vget(0xe000000012346330, 0x1c2a80, 0x200000, 0xa0000000002c1198, 0x9ffc00000097c2b0, 0x1836) at ffs_vget+0x40
 ufs_lookup_ino(0xe000000012425ba8, 0xa0000000002c1310, 0xa0000000002c1338, 0x0) at ufs_lookup_ino+0x1cd0
 ufs_lookup(0xa0000000002c11e8, 0x9ffc000000a42240, 0x38a) at ufs_lookup+0x40
 VOP_CACHEDLOOKUP_APV(0x9ffc000000df2660, 0xa0000000002c11d0, 0xa0000000002c1338, 0x0, 0x9ffc000000630460) at VOP_CACHEDLOOKUP_APV+0x220
 vfs_cache_lookup(0xe000000012425ba8, 0x0, 0xa0000000002c1338) at vfs_cache_lookup+0x1f0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000002c1208, 0x4004144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000002c12b0, 0x200000, 0xe000000012425ba8, 0x0, 0xa0000000002c1340) at lookup+0xe10
 namei(0xa0000000002c12b0, 0xe00000001231ade0, 0x4000144, 0x0, 0xa0000000002c1320, 0xa0000000002c1368, 0xa0000000002c1360, 0xa0000000002c1310) at namei+0xb40
 kern_chdir(0xe0000000123fe480, 0x100004d10, 0x0) at kern_chdir+0x80
 sys_chdir(0xe0000000123fe480, 0xa0000000002c14e8, 0x9ffc000000a2cc80, 0x48d) at sys_chdir+0x30
 syscall(0xe00000001231ade0, 0x7fffffffffffed40, 0x100004e38, 0xe0000000123fe480, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1055 tid 100094 td 0xe0000000128cc900
 cpu_switch(0xe0000000128cc900, 0xe000000011151b00, 0x9ffc000000eb13a8, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000128cc900, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000128cc900, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe00000001289f5a8, 0x5c, 0xe0000000128ccc78, 0xe0000000128cc900) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe00000001289f5a8, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe00000001289f5a8, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe00000001289f5a8, 0xe00000001289f7e8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe00000001238df14, 0xa000000000369360, 0xa000000000369378, 0x0, 0xe00000001289f6b0) at pipe_read+0x740
 dofileread(0xe0000000128cc900, 0x6, 0xe00000001238def0, 0xa000000000369360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe0000000128cc900, 0x6, 0xa000000000369360) at kern_readv+0xa0
 sys_read(0xe0000000128cc900, 0xa0000000003694e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe0000000128c8940, 0x140c26000, 0x2000, 0xe0000000128cc900, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1054 tid 100083 td 0xe000000012650900
 cpu_switch(0xe000000012650900, 0xe000000011151b00, 0x9ffc000000eb2168, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012650900, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012650900, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000124245f0, 0x60, 0xe000000012650c78, 0xe000000012650900) at sleepq_switch+0x2e0
 sleepq_wait(0xe0000000124245f0, 0x60, 0x9ffc000000d28b50, 0xe000000012650900, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe0000000124245f0, 0x200100, 0xe000000012424620, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa000000000311090, 0xe0000000124245f0, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000311070, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012424588, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012424588, 0x200100, 0xe000000012650900, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0x7, 0xa000000000311220, 0xa000000000311248, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe0000000120f1ba8, 0x0, 0xa000000000311248) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa000000000311118, 0x400c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000003111c0, 0x200000, 0xe0000000120f1ba8, 0x0, 0xa000000000311250) at lookup+0xe10
 namei(0xa0000000003111c0, 0xe0000000126af720, 0x4000144, 0x0, 0xa000000000311230, 0xa000000000311278, 0xa000000000311270, 0xa000000000311220) at namei+0xb40
 kern_statat_vnhook(0xe000000012650900, 0x0, 0xffffffffffffff9c, 0x140c0ab10, 0x0, 0xa000000000311320, 0x0) at kern_statat_vnhook+0xf0
 kern_statat(0xe000000012650900, 0x0, 0xffffffffffffff9c, 0x140c0ab10, 0x0, 0xa000000000311320) at kern_statat+0x40
 kern_stat(0xe000000012650900, 0x140c0ab10, 0x0, 0xa000000000311320) at kern_stat+0x40
 sys_stat(0xe000000012650900, 0xa0000000003114e8, 0x9ffc000000a2cc80, 0x48d) at sys_stat+0x30
 syscall(0xe0000000126af720, 0x7fffffffffffe708, 0x14031d640, 0xe000000012650900, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1053 tid 100068 td 0xe00000001237e900
 cpu_switch(0xe00000001237e900, 0xe000000011151680, 0x9ffc000000eb2ca8, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe00000001237e900, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe00000001237e900, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000126a55a8, 0x5c, 0xe00000001237ec78, 0xe00000001237e900) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000126a55a8, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000126a55a8, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000126a55a8, 0xe0000000126a57e8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000126f71f4, 0xa000000000299360, 0xa000000000299378, 0x0, 0xe0000000126a56b0) at pipe_read+0x740
 dofileread(0xe00000001237e900, 0x6, 0xe0000000126f71d0, 0xa000000000299360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe00000001237e900, 0x6, 0xa000000000299360) at kern_readv+0xa0
 sys_read(0xe00000001237e900, 0xa0000000002994e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe0000000123f8940, 0x140c26000, 0x2000, 0xe00000001237e900, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command atrun pid 1052 tid 100070 td 0xe00000001237e000
 cpu_switch(0xe00000001237e000, 0xe000000011151b00, 0x9ffc000000eb1ec0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe00000001237e000, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe00000001237e000, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000128914b0, 0x60, 0xe00000001237e378, 0xe00000001237e000) at sleepq_switch+0x2e0
 sleepq_wait(0xe0000000128914b0, 0x60, 0x9ffc000000d28b50, 0xe00000001237e000, 0x9ffc0000004c43a0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe0000000128914b0, 0x200100, 0xe0000000128914e0, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0x16b0
 ffs_lock(0xa0000000002a9140, 0xe0000000128914b0, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa0000000002a9120, 0x8) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012891448, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012891448, 0x200100, 0xe00000001237e000, 0x57) at vget+0x100
 vfs_hash_get(0xe000000012346330, 0x1c2a80, 0xe000000012891448, 0xe00000001237e000, 0xa0000000002a9198, 0x0, 0x0, 0x9ffc000000d353d8) at vfs_hash_get+0x180
 ffs_vgetf(0xe000000012346330, 0x1c2a80, 0x200000, 0xa0000000002a9198, 0x0) at ffs_vgetf+0x50
 ffs_vget(0xe000000012346330, 0x1c2a80, 0x200000, 0xa0000000002a9198, 0x9ffc00000097c2b0, 0x1836) at ffs_vget+0x40
 ufs_lookup_ino(0xe000000012425ba8, 0xa0000000002a9310, 0xa0000000002a9338, 0x0) at ufs_lookup_ino+0x1cd0
 ufs_lookup(0xa0000000002a91e8, 0x9ffc000000a42240, 0x38a) at ufs_lookup+0x40
 VOP_CACHEDLOOKUP_APV(0x9ffc000000df2660, 0xa0000000002a91d0, 0xa0000000002a9338, 0x0, 0x9ffc000000630460) at VOP_CACHEDLOOKUP_APV+0x220
 vfs_cache_lookup(0xe000000012425ba8, 0x0, 0xa0000000002a9338) at vfs_cache_lookup+0x1f0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000002a9208, 0x4004144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000002a92b0, 0x200000, 0xe000000012425ba8, 0x0, 0xa0000000002a9340) at lookup+0xe10
 namei(0xa0000000002a92b0, 0xe0000000123f8000, 0x4000144, 0x0, 0xa0000000002a9320, 0xa0000000002a9368, 0xa0000000002a9360, 0xa0000000002a9310) at namei+0xb40
 kern_chdir(0xe00000001237e000, 0x100004d10, 0x0) at kern_chdir+0x80
 sys_chdir(0xe00000001237e000, 0xa0000000002a94e8, 0x9ffc000000a2cc80, 0x48d) at sys_chdir+0x30
 syscall(0xe0000000123f8000, 0x7fffffffffffed40, 0x100004e38, 0xe00000001237e000, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1051 tid 100064 td 0xe00000001237fb00
 cpu_switch(0xe00000001237fb00, 0xe000000011151680, 0x9ffc000000eb27d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe00000001237fb00, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe00000001237fb00, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000123aae70, 0x5c, 0xe00000001237fe78, 0xe00000001237fb00) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000123aae70, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000123aae70, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000123aae70, 0xe0000000123ab0b0, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000126f72e4, 0xa000000000279360, 0xa000000000279378, 0x0, 0xe0000000123aaf78) at pipe_read+0x740
 dofileread(0xe00000001237fb00, 0x6, 0xe0000000126f72c0, 0xa000000000279360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe00000001237fb00, 0x6, 0xa000000000279360) at kern_readv+0xa0
 sys_read(0xe00000001237fb00, 0xa0000000002794e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe00000001114ede0, 0x140c26000, 0x2000, 0xe00000001237fb00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command atrun pid 1050 tid 100071 td 0xe0000000123fed80
 cpu_switch(0xe0000000123fed80, 0xe000000011151b00, 0x9ffc000000eb1ec0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000123fed80, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000123fed80, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000128914b0, 0x60, 0xe0000000123ff0f8, 0xe0000000123fed80) at sleepq_switch+0x2e0
 sleepq_wait(0xe0000000128914b0, 0x60, 0x9ffc000000d28b50, 0xe0000000123fed80, 0x9ffc0000004c43a0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe0000000128914b0, 0x200100, 0xe0000000128914e0, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0x16b0
 ffs_lock(0xa0000000002b1140, 0xe0000000128914b0, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa0000000002b1120, 0x8) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012891448, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012891448, 0x200100, 0xe0000000123fed80, 0x57) at vget+0x100
 vfs_hash_get(0xe000000012346330, 0x1c2a80, 0xe000000012891448, 0xe0000000123fed80, 0xa0000000002b1198, 0x0, 0x0, 0x9ffc000000d353d8) at vfs_hash_get+0x180
 ffs_vgetf(0xe000000012346330, 0x1c2a80, 0x200000, 0xa0000000002b1198, 0x0) at ffs_vgetf+0x50
 ffs_vget(0xe000000012346330, 0x1c2a80, 0x200000, 0xa0000000002b1198, 0x9ffc00000097c2b0, 0x1836) at ffs_vget+0x40
 ufs_lookup_ino(0xe000000012425ba8, 0xa0000000002b1310, 0xa0000000002b1338, 0x0) at ufs_lookup_ino+0x1cd0
 ufs_lookup(0xa0000000002b11e8, 0x9ffc000000a42240, 0x38a) at ufs_lookup+0x40
 VOP_CACHEDLOOKUP_APV(0x9ffc000000df2660, 0xa0000000002b11d0, 0xa0000000002b1338, 0x0, 0x9ffc000000630460) at VOP_CACHEDLOOKUP_APV+0x220
 vfs_cache_lookup(0xe000000012425ba8, 0x0, 0xa0000000002b1338) at vfs_cache_lookup+0x1f0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000002b1208, 0x4004144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000002b12b0, 0x200000, 0xe000000012425ba8, 0x0, 0xa0000000002b1340) at lookup+0xe10
 namei(0xa0000000002b12b0, 0xe00000001231b720, 0x4000144, 0x0, 0xa0000000002b1320, 0xa0000000002b1368, 0xa0000000002b1360, 0xa0000000002b1310) at namei+0xb40
 kern_chdir(0xe0000000123fed80, 0x100004d10, 0x0) at kern_chdir+0x80
 sys_chdir(0xe0000000123fed80, 0xa0000000002b14e8, 0x9ffc000000a2cc80, 0x48d) at sys_chdir+0x30
 syscall(0xe00000001231b720, 0x7fffffffffffed40, 0x100004e38, 0xe0000000123fed80, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1049 tid 100080 td 0xe000000012651680
 cpu_switch(0xe000000012651680, 0xe000000011151680, 0x9ffc000000eb1a60, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012651680, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012651680, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000126a50d8, 0x5c, 0xe0000000126519f8, 0xe000000012651680) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000126a50d8, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000126a50d8, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000126a50d8, 0xe0000000126a5318, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000126f73d4, 0xa0000000002f9360, 0xa0000000002f9378, 0x0, 0xe0000000126a51e0) at pipe_read+0x740
 dofileread(0xe000000012651680, 0x6, 0xe0000000126f73b0, 0xa0000000002f9360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012651680, 0x6, 0xa0000000002f9360) at kern_readv+0xa0
 sys_read(0xe000000012651680, 0xa0000000002f94e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe00000001244a940, 0x140c26000, 0x2000, 0xe000000012651680, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command dd pid 1048 tid 100065 td 0xe00000001237f680
 cpu_switch(0xe00000001237f680, 0xe000000011151680, 0x9ffc000000eb2848, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe00000001237f680, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe00000001237f680, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xa00000009e03d938, 0x5c, 0xe00000001237f9f8, 0xe00000001237f680) at sleepq_switch+0x2e0
 sleepq_wait(0xa00000009e03d938, 0x5c, 0x9ffc000000d28b50, 0xe00000001237f680, 0x9ffc00000050be60) at sleepq_wait+0xb0
 _sleep(0xa00000009e03d938, 0xe000000010c20ad0, 0x5c, 0x9ffc000000d331a8, 0x0, 0x0, 0x100) at _sleep+0x7d0
 bwait(0xa00000009e03d938, 0x5c, 0x9ffc000000d331a8) at bwait+0xc0
 bufwait(0xa00000009e03d938, 0x9ffc000000628270, 0x38a) at bufwait+0x80
 bufwrite(0xa00000009e03d938, 0xa000, 0x0, 0x0, 0x9ffc000000960d10, 0x30d, 0x9ffc0000010015a0) at bufwrite+0x3a0
 ffs_bufwrite(0xa00000009e03d938, 0x9ffc000000d58388, 0x0, 0x9ffc0000009165a0, 0x12ad, 0x0) at ffs_bufwrite+0x520
 ffs_update(0xe000000012838000, 0x1, 0xa00000009e03d938, 0xe0000000123ec588, 0xe00000001213f0b8, 0xe0000000127b4ab0) at ffs_update+0x8b0
 ufs_makeinode(0x8100, 0xe000000012424588, 0xa0000000002812e8, 0xa000000000281310) at ufs_makeinode+0x820
 ufs_create(0xa000000000281258, 0x9ffc000000a45e80, 0x38a) at ufs_create+0x80
 VOP_CREATE_APV(0x9ffc000000df2660, 0xa000000000281240, 0xa000000000281310, 0x0, 0x9ffc0000006762f0) at VOP_CREATE_APV+0x220
 vn_open_cred(0xa000000000281288, 0xa000000000281390, 0x100, 0x0, 0xe000000012355600, 0xe0000000126f7450) at vn_open_cred+0x330
 vn_open(0xa000000000281288, 0xa000000000281390, 0x100, 0xe0000000126f7450, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe00000001237f680, 0xffffffffffffff9c, 0x140806043, 0x0, 0x0, 0x1b6) at kern_openat+0x490
 kern_open(0xe00000001237f680, 0x140806043, 0x0, 0x602, 0x1b6) at kern_open+0x40
 sys_open(0xe00000001237f680, 0xa0000000002814e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe0000000123f9720, 0x602, 0x1b6, 0xe00000001237f680, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1047 tid 100082 td 0xe000000012650d80
 cpu_switch(0xe000000012650d80, 0xe000000011151b00, 0x9ffc000000eb1e20, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012650d80, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012650d80, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe00000001244a000, 0x6c, 0xe0000000126510f8, 0xe000000012650d80) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe00000001244a000, 0x6c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe00000001244a000, 0x6c, 0x9ffc000000d57ae8, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe00000001244a000, 0xe00000001244a100, 0x6c, 0x9ffc000000d57ae8, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
 kern_wait6(0xe000000012650d80, 0x7, 0x0, 0xa000000000309390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
 kern_wait(0xe000000012650d80, 0xffffffff, 0xa000000000309390, 0x0, 0x0) at kern_wait+0xe0
 sys_wait4(0xe000000012650d80, 0xa0000000003094e8, 0xa000000000309390, 0x9ffc000000a2cc80) at sys_wait4+0x70
 syscall(0xe00000001244a000, 0x7fffffffffffe8e0, 0x0, 0xe000000012650d80, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1046 tid 100087 td 0xe0000000123ffb00
 cpu_switch(0xe0000000123ffb00, 0xe000000011151680, 0x9ffc000000eb0fc0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000123ffb00, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000123ffb00, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000126a4c08, 0x5c, 0xe0000000123ffe78, 0xe0000000123ffb00) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000126a4c08, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000126a4c08, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000126a4c08, 0xe0000000126a4e48, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe00000001238c2f4, 0xa000000000331360, 0xa000000000331378, 0x0, 0xe0000000126a4d10) at pipe_read+0x740
 dofileread(0xe0000000123ffb00, 0x6, 0xe00000001238c2d0, 0xa000000000331360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe0000000123ffb00, 0x6, 0xa000000000331360) at kern_readv+0xa0
 sys_read(0xe0000000123ffb00, 0xa0000000003314e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe0000000126ae940, 0x140c26000, 0x2000, 0xe0000000123ffb00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command atrun pid 1045 tid 100091 td 0xe000000012768000
 cpu_switch(0xe000000012768000, 0xe000000011151b00, 0x9ffc000000eb1470, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012768000, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012768000, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xa00000009e06a2c0, 0x5c, 0xe000000012768378, 0xe000000012768000) at sleepq_switch+0x2e0
 sleepq_wait(0xa00000009e06a2c0, 0x5c, 0x9ffc000000d28b50, 0xe000000012768000, 0x9ffc00000050be60) at sleepq_wait+0xb0
 _sleep(0xa00000009e06a2c0, 0xe000000010c203b0, 0x5c, 0x9ffc000000d331a0, 0x0, 0x0, 0x100) at _sleep+0x7d0
 bwait(0xa00000009e06a2c0, 0x5c, 0x9ffc000000d331a0) at bwait+0xc0
 bufwait(0xa00000009e06a2c0, 0x9ffc00000062b7a0, 0x591, 0x0) at bufwait+0x50
 breadn_flags(0xe0000000123ec588, 0xa00000009e06a2c0, 0xa00000009e06a3b8, 0x0, 0x0, 0x0, 0x0, 0x0) at breadn_flags+0x300
 ffs_vgetf(0xe000000012346330, 0x1c2a80, 0x80000, 0xa000000000351198, 0x0) at ffs_vgetf+0x6d0
 ffs_vget(0xe000000012346330, 0x1c2a80, 0x200000, 0xa000000000351198, 0x9ffc00000097c2b0, 0x1836) at ffs_vget+0x40
 ufs_lookup_ino(0xe000000012425ba8, 0xa000000000351310, 0xa000000000351338, 0x0) at ufs_lookup_ino+0x1cd0
 ufs_lookup(0xa0000000003511e8, 0x9ffc000000a42240, 0x38a) at ufs_lookup+0x40
 VOP_CACHEDLOOKUP_APV(0x9ffc000000df2660, 0xa0000000003511d0, 0xa000000000351338, 0x0, 0x9ffc000000630460) at VOP_CACHEDLOOKUP_APV+0x220
 vfs_cache_lookup(0xe000000012425ba8, 0x0, 0xa000000000351338) at vfs_cache_lookup+0x1f0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa000000000351208, 0x4004144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000003512b0, 0x200000, 0xe000000012425ba8, 0x0, 0xa000000000351340) at lookup+0xe10
 namei(0xa0000000003512b0, 0xe00000001244d720, 0x4000144, 0x0, 0xa000000000351320, 0xa000000000351368, 0xa000000000351360, 0xa000000000351310) at namei+0xb40
 kern_chdir(0xe000000012768000, 0x100004d10, 0x0) at kern_chdir+0x80
 sys_chdir(0xe000000012768000, 0xa0000000003514e8, 0x9ffc000000a2cc80, 0x48d) at sys_chdir+0x30
 syscall(0xe00000001244d720, 0x7fffffffffffed40, 0x100004e38, 0xe000000012768000, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1044 tid 100074 td 0xe0000000123fe000
 cpu_switch(0xe0000000123fe000, 0xe000000011151b00, 0x9ffc000000eb1970, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000123fe000, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000123fe000, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe00000001289fa78, 0x5c, 0xe0000000123fe378, 0xe0000000123fe000) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe00000001289fa78, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe00000001289fa78, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe00000001289fa78, 0xe00000001289fcb8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123a4114, 0xa0000000002c9360, 0xa0000000002c9378, 0x0, 0xe00000001289fb80) at pipe_read+0x740
 dofileread(0xe0000000123fe000, 0x6, 0xe0000000123a40f0, 0xa0000000002c9360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe0000000123fe000, 0x6, 0xa0000000002c9360) at kern_readv+0xa0
 sys_read(0xe0000000123fe000, 0xa0000000002c94e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe00000001244c940, 0x140c26000, 0x2000, 0xe0000000123fe000, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command fsck_ufs pid 1043 tid 100075 td 0xe000000012385b00
 cpu_switch(0xe000000012385b00, 0xe000000011151680, 0x9ffc000000eb0f48, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012385b00, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012385b00, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xa00000009e032160, 0x60, 0xe000000012385e78, 0xe000000012385b00) at sleepq_switch+0x2e0
 sleepq_wait(0xa00000009e032160, 0x60, 0x9ffc000000d28b50, 0xe000000012385b00, 0x9ffc0000004c43a0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xa00000009e032160, 0x81900, 0xe0000000123ec658, 0x9ffc000000d16b98, 0x60, 0x0, 0x9ffc000000d32db0, 0xc09) at __lockmgr_args+0x16b0
 getblk(0xe0000000123ec588, 0x7b81d00, 0x8000, 0x0, 0x0, 0x0) at getblk+0x2b0
 breadn_flags(0xe0000000123ec588, 0x7b81d00, 0x8000, 0x0, 0x0, 0x0, 0x0, 0x0) at breadn_flags+0x40
 ffs_update(0xe000000012442588, 0x0, 0xf70378, 0xe0000000123ec588, 0xe00000001213f0b8, 0xe00000001264eff0, 0xe00000001264efc0, 0xe00000001213f000) at ffs_update+0x3e0
 ffs_syncvnode(0xe000000012442588, 0x0, 0x0, 0x0, 0x9ffc000000d58b68, 0xe000000012442670) at ffs_syncvnode+0x810
 ffs_sync(0xe000000012346330, 0x2, 0xe000000012442588, 0x9ffc000000664710) at ffs_sync+0x670
 sys_sync(0x0, 0x9ffc000000d37be0, 0x9ffc000000e278d8, 0xe000000012385be4) at sys_sync+0x350
 syscall(0xe00000001244c4a0, 0x7fffffffffffed58, 0x0, 0xe000000012385b00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command grep pid 1042 tid 100062 td 0xe000000012384480
 cpu_switch(0xe000000012384480, 0xe000000011151b00, 0x9ffc000000eb2820, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012384480, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012384480, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xa00000009e06a848, 0x5c, 0xe0000000123847f8, 0xe000000012384480) at sleepq_switch+0x2e0
 sleepq_wait(0xa00000009e06a848, 0x5c, 0x9ffc000000d28b50, 0xe000000012384480, 0x9ffc00000050be60) at sleepq_wait+0xb0
 _sleep(0xa00000009e06a848, 0xe000000010c205f0, 0x5c, 0x9ffc000000d331a0, 0x0, 0x0, 0x100) at _sleep+0x7d0
 bwait(0xa00000009e06a848, 0x5c, 0x9ffc000000d331a0) at bwait+0xc0
 bufwait(0xa00000009e06a848, 0x9ffc00000062b7a0, 0x591, 0x0) at bufwait+0x50
 breadn_flags(0xe000000012839270, 0xa00000009e06a848, 0xa00000009e06a940, 0x0, 0x0, 0x0, 0x0, 0x8) at breadn_flags+0x300
 ffs_read(0xa000000000269240, 0x1000, 0x8000) at ffs_read+0x540
 VOP_READ_APV(0x9ffc000000df1d20, 0xa000000000269240, 0xe000000012839270, 0x0) at VOP_READ_APV+0x220
 vn_read(0xe0000000123a4dc0, 0xa000000000269360

From: Anton Shterenlikht <mexas@bris.ac.uk>
To: bug-followup@freebsd.org
Cc:  
Subject: ia64/187219: panic: 11.0-CURRENT #20 r260914: Panic String: deadlkres: possible deadlock detected for 0xe000000012079b00, blocked for 1801960 ticks
Date: Mon, 10 Mar 2014 05:29:53 -0700 (PDT)

 Another one:
 
 # cat info.last 
 Dump header from device /dev/da2p1
   Architecture: ia64
   Architecture Version: 1
   Dump Length: 67584B (0 MB)
   Blocksize: 512
   Dumptime: Mon Mar 10 11:40:02 2014
   Hostname: mech-as28.men.bris.ac.uk
   Magic: FreeBSD Text Dump
   Version String: FreeBSD 11.0-CURRENT #22 r260914: Wed Mar  5 10:12:04 GMT 2014
     root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV
   Panic String: deadlkres: possible deadlock detected for 0xe0000000123a6900, blocked for 1802278 ticks
 
   Dump Parity: 2881296935
   Bounds: 6
   Dump Status: good
 # 
 
 # cat panic.txt 
 deadlkres: possible deadlock detected for 0xe0000000123a6900, blocked for 1802278 ticks
 #
 
 # cat version.txt 
 FreeBSD 11.0-CURRENT #22 r260914: Wed Mar  5 10:12:04 GMT 2014
     root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV
 # 
 
 config.txt:
 
 
 options	CONFIG_AUTOGENERATED
 ident	ZEEV
 machine	ia64
 cpu	ITANIUM2
 makeoptions	MODULES_OVERRIDE=geom/geom_part opensolaris zfs acl_nfs4 fdescfs
 makeoptions	DEBUG=-g
 options	TEXTDUMP_VERBOSE
 options	TEXTDUMP_PREFERRED
 options	WITNESS_SKIPSPIN
 options	WITNESS
 options	INVARIANT_SUPPORT
 options	INVARIANTS
 options	DEADLKRES
 options	GDB
 options	DDB
 options	KDB_TRACE
 options	KDB
 options	_KPOSIX_PRIORITY_SCHEDULING
 options	QUOTA
 options	UFS_GJOURNAL
 options	UFS_DIRHASH
 options	UFS_ACL
 options	SYSVSHM
 options	SYSVSEM
 options	SYSVMSG
 options	STACK
 options	SOFTUPDATES
 options	SMP
 options	SCTP
 options	SCSI_DELAY=5000
 options	SCHED_ULE
 options	PSEUDOFS
 options	PROCFS
 options	PRINTF_BUFR_SIZE=128
 options	PREEMPTION
 options	P1003_1B_SEMAPHORES
 options	NFS_ROOT
 options	NFSD
 options	NFSLOCKD
 options	NFSCL
 options	MSDOSFS
 options	MD_ROOT
 options	MAC
 options	KTRACE
 options	INET6
 options	INET
 options	INCLUDE_CONFIG_FILE
 options	GEOM_LABEL
 options	FFS
 options	TMPFS
 options	NULLFS
 options	FDESCFS
 options	COMPAT_FREEBSD7
 options	CD9660
 options	CAPABILITIES
 options	CAPABILITY_MODE
 options	AUDIT
 options	NEW_PCIB
 options	GEOM_PART_MBR
 options	GEOM_PART_GPT
 options	GEOM_PART_BSD
 device	acpi
 device	io
 device	mem
 device	uart_ns8250
 device	miibus
 device	pci
 device	scbus
 device	usb
 device	isp
 device	ispfw
 device	mpt
 device	cd
 device	da
 device	pass
 device	sa
 device	ehci
 device	ohci
 device	uhci
 device	umass
 device	em
 device	ether
 device	faith
 device	firmware
 device	gif
 device	loop
 device	md
 device	puc
 device	random
 device	tun
 device	uart
 device	vlan
 
 msgbuf.txt
 
 
 GDB: debug ports: uart
 GDB: current port: uart
 KDB: debugger backends: ddb gdb
 KDB: current backend: ddb
 Copyright (c) 1992-2014 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 11.0-CURRENT #22 r260914: Wed Mar  5 10:12:04 GMT 2014
     root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV ia64
 gcc version 4.2.1 20070831 patched [FreeBSD]
 WARNING: WITNESS option enabled, expect reduced performance.
 CPU: Madison II (1600 MHz Itanium 2)
   Origin = "GenuineIntel"  Revision = 2
   Features = 0x1<LB>
 real memory  = 10737418240 (10240 MB)
 avail memory = 10508304384 (10021 MB)
 FPSWA Revision = 0x10012, Entry = 0xe0000040ffcc4050
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP)
 cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0
 random: <Software, Yarrow> initialized
 Event timer "ITC" frequency 1600000000 Hz quality 1000
 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603)
 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603)
 acpi0: <HP> on motherboard
 acpi0: Power Button (fixed)
 acpi0: Sleep Button (fixed)
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
 acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0
 acpi_tz0: <Thermal Zone> on acpi0
 pcib0: <ACPI Host-PCI bridge> on acpi0
 pcib0: could not evaluate _ADR - AE_NOT_FOUND
 pci0: <ACPI PCI bus> on pcib0
 ohci0: <NEC uPD 9210 USB controller> mem 0x80002000-0x80002fff irq 16 at device 1.0 on pci0
 usbus0 on ohci0
 ohci1: <NEC uPD 9210 USB controller> mem 0x80001000-0x80001fff irq 17 at device 1.1 on pci0
 usbus1 on ohci1
 ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x80000000-0x800000ff irq 18 at device 1.2 on pci0
 usbus2: EHCI version 0.95
 usbus2 on ehci0
 pci0: <mass storage, ATA> at device 2.0 (no driver attached)
 pcib1: <ACPI Host-PCI bridge> on acpi0
 pcib1: could not evaluate _ADR - AE_NOT_FOUND
 pci32: <ACPI PCI bus> on pcib1
 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2100-0x21ff mem 0x903a0000-0x903bffff,0x90380000-0x9039ffff irq 27 at device 1.0 on pci32
 mpt0: MPI Version=1.2.12.0
 mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0x90360000-0x9037ffff,0x90340000-0x9035ffff irq 28 at device 1.1 on pci32
 mpt1: MPI Version=1.2.12.0
 em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2240-0x227f mem 0x90320000-0x9033ffff,0x90280000-0x902fffff irq 29 at device 2.0 on pci32
 em0: Ethernet address: 00:13:21:5b:05:1c
 em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2200-0x223f mem 0x90300000-0x9031ffff irq 30 at device 2.1 on pci32
 em1: Ethernet address: 00:13:21:5b:05:1d
 pcib2: <ACPI Host-PCI bridge> on acpi0
 pcib2: could not evaluate _ADR - AE_NOT_FOUND
 pci64: <ACPI PCI bus> on pcib2
 pcib3: <ACPI Host-PCI bridge> on acpi0
 pcib3: could not evaluate _ADR - AE_NOT_FOUND
 pci96: <ACPI PCI bus> on pcib3
 pcib4: <ACPI Host-PCI bridge> on acpi0
 pcib4: could not evaluate _ADR - AE_NOT_FOUND
 pci128: <ACPI PCI bus> on pcib4
 pcib5: <ACPI Host-PCI bridge> on acpi0
 pcib5: could not evaluate _ADR - AE_NOT_FOUND
 pci192: <ACPI PCI bus> on pcib5
 isp0: <Qlogic ISP 2422 PCI FC-AL Adapter> port 0xc000-0xc0ff mem 0xe0040000-0xe0040fff irq 71 at device 1.0 on pci192
 pcib6: <ACPI Host-PCI bridge> on acpi0
 pcib6: could not evaluate _ADR - AE_NOT_FOUND
 pci224: <ACPI PCI bus> on pcib6
 uart0: <HP Auxiliary Diva Serial Port> mem 0xf8051000-0xf805100f irq 82 at device 1.0 on pci224
 puc0: <HP Diva Serial [GSP] Multiport UART - Everest SP2> mem 0xf8050000-0xf8050fff,0xf8020000-0xf803ffff irq 82 at device 1.1 on pci224
 uart1: <Non-standard ns8250 class UART with FIFOs> at port 1 on puc0
 uart1: console (9600,n,8,1)
 uart2: <Non-standard ns8250 class UART with FIFOs> at port 2 on puc0
 uart3: <Non-standard ns8250 class UART with FIFOs> at port 3 on puc0
 vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf0000000-0xf7ffffff,0xf8040000-0xf804ffff at device 2.0 on pci224
 vgapci0: Boot video device
 uart4: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0
 uart5: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0
 uart5: debug port (9600,n,8,1)
 Timecounters tick every 1.000 msec
 usbus0: 12Mbps Full Speed USB v1.0
 usbus1: 12Mbps Full Speed USB v1.0
 usbus2: 480Mbps High Speed USB v2.0
 ugen1.1: <NEC> at usbus1
 uhub0: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
 ugen0.1: <NEC> at usbus0
 ugen2.1: <NEC> at usbus2
 uhub1: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
 uhub2: <NEC EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
 random: unblocking device.
 uhub0: 2 ports with 2 removable, self powered
 uhub1: 3 ports with 3 removable, self powered
 uhub2: 5 ports with 5 removable, self powered
 pass1 at isp0 bus 0 scbus2 target 0 lun 0
 da1 at isp0 bus 0 scbus2 target 0 lun 1
 pass1: <COMPAQ MSA1000 4.32> Fixed Storage Array SCSI-4 device 
 da1: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device 
 pass1: Serial Number P56350GX3RX04Z
 da1: Serial Number P56350GX3RX04Z
 pass1: 200.000MB/s transfersda1: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 WWPN 0x500805f3000ec221 PortID 0x10000
  PortID 0x10000
 pass1: Command Queueing enabled
 da1: Command Queueing enabled
 da1: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
 da2 at isp0 bus 0 scbus2 target 0 lun 2
 da2: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device 
 da2: Serial Number P56350GX3RX04Z
 da2: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
 da2: Command Queueing enabled
 da2: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
 da3 at isp0 bus 0 scbus2 target 0 lun 5
 da3: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device 
 da3: Serial Number P56350GX3RX04Z
 da3: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
 da3: Command Queueing enabled
 da3: 140011MB (286744185 512 byte sectors: 255H 63S/T 17849C)
 da0 at mpt0 bus 0 scbus0 target 0 lun 0
 da0: <HP 73.4G ST373454LC HPC2> Fixed Direct Access SCSI-3 device 
 da0: Serial Number 3KP05D0W00007532H5SA
 da0: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
 da0: Command Queueing enabled
 da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
 WARNING: WITNESS option enabled, expect reduced performance.
 Trying to mount root from ufs:/dev/da0p2 [rw]...
 WARNING: / was not properly dismounted
 <118>Setting hostuuid: ee652761-3dcc-11df-9074-0013215b051c.
 <118>Setting hostid: 0x8968724b.
 <118>Starting ddb.
 <118>Entropy harvesting: interrupts ethernet point_to_point swi.
 <118>Starting file system checks:
 <118>/dev/da0p2: 563434 files, 3754911 used, 30861513 free (305153 frags, 3819545 blocks, 0.9% fragmentation)
 <118>/dev/da1p1: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
 <118>/dev/da1p1: SUMMARY INFORMATION BAD (SALVAGED)
 <118>/dev/da1p1: BLK(S) MISSING IN BIT MAPS (SALVAGED)
 <118>/dev/da1p1: 2158 files, 81894 used, 17141467 free (443 frags, 2142628 blocks, 0.0% fragmentation)
 <118>/dev/da3p1: 6085 files, 1328251 used, 68099252 free (6244 frags, 8511626 blocks, 0.0% fragmentation)
 <118>Mounting local file systems:.
 <118>Writing entropy file:.
 <118>Setting hostname: mech-as28.men.bris.ac.uk.
 <118>Starting Network: lo0 em0 em1.
 <118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 <118>	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
 <118>	inet6 ::1 prefixlen 128 
 <118>	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
 <118>	inet 127.0.0.1 netmask 0xff000000 
 <118>	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 <118>em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 <118>	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
 <118>	ether 00:13:21:5b:05:1c
 <118>	inet 137.222.187.28 netmask 0xffffff00 broadcast 137.222.187.255 
 <118>	inet6 fe80::213:21ff:fe5b:51c%em0 prefixlen 64 scopeid 0x1 
 <118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 <118>	media: Ethernet autoselect
 <118>	status: no carrier
 <118>em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 <118>	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
 <118>	ether 00:13:21:5b:05:1d
 <118>	inet 10.10.10.14 netmask 0xffffff00 broadcast 10.10.10.255 
 <118>	inet6 fe80::213:21ff:fe5b:51d%em1 prefixlen 64 scopeid 0x2 
 <118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 <118>	media: Ethernet autoselect (100baseTX <full-duplex>)
 <118>	status: active
 <118>Starting devd.
 <118>Starting pflogd: 
 <118>add net default: gateway 137.222.187.250
 <118>add net fe80::: gateway ::1
 <118>add net ff02::: gateway ::1
 <118>add net ::ffff:0.0.0.0: gateway ::1
 <118>add net ::0.0.0.0: gateway ::1
 <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
 <118>Creating and/or trimming log files.
 <118>Starting syslogd.
 <118>savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe00000001271e900, blocked for 1800893 ticks
 <118>Mar 10 09:03:39 mech-as28 savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe00000001271e900, blocked for 1800893 ticks
 <118>savecore: writing core to /var/crash/textdump.tar.5
 <118>/var/crash/vmcore.5 not found
 <118>Starting casperd.
 lock order reversal:
  1st 0xa00000009e049220 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3081
  2nd 0xe000000012130a00 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000d89c0) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc000000572870) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000e2b2d8, 0x9ffc0000005a8990) at kdb_backtrace+0xc0
 _witness_debugger(0x1, 0x9ffc000000d2c138, 0x9ffc0000005ac370, 0xb9d, 0x9ffc000000d59270) at _witness_debugger+0x60
 witness_checkorder(0xe000000012130a00, 0x9ffc000000d2ba80, 0x9ffc000000d59270, 0x11c, 0x0) at witness_checkorder+0x15c0
 _sx_xlock(0xe000000012130a00, 0x0, 0x9ffc000000d59270, 0x11c) at _sx_xlock+0x120
 ufsdirhash_acquire(0xe000000012423308, 0xe000000012130a00, 0x9ffc000000975d60) at ufsdirhash_acquire+0x50
 ufsdirhash_add(0xe000000012423308, 0xa000000000279098, 0x934, 0xa000000000279040, 0x9ffc00000097dbf0, 0xda1) at ufsdirhash_add+0x20
 ufs_direnter(0xe00000001267a1d8, 0xe000000012681ba8, 0xa000000000279098, 0xa000000000279318, 0xa00000009e049720, 0x0) at ufs_direnter+0x1260
 ufs_mkdir(0xa000000000279370, 0xa000000000279098, 0xe00000001267a1d8) at ufs_mkdir+0x14f0
 VOP_MKDIR_APV(0x9ffc000000df2660, 0xa000000000279370, 0xa000000000279318, 0x0, 0x9ffc0000006650c0, 0x818, 0x9ffc0000010015a0) at VOP_MKDIR_APV+0x220
 kern_mkdirat(0xe0000000123a7b00, 0xffffffffffffff9c, 0x140804040, 0x0, 0x1c0) at kern_mkdirat+0x3a0
 kern_mkdir(0xe0000000123a7b00, 0x140804040, 0x0, 0x1c0) at kern_mkdir+0x40
 sys_mkdir(0xe0000000123a7b00, 0xa0000000002794e8, 0x9ffc000000a2cc80, 0x48d) at sys_mkdir+0x30
 syscall(0xe00000001114ede0, 0x1c0, 0x9ffc000000a29630, 0xe0000000123a7b00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 <118>Clearing /tmp (X related).
 <118>Recovering vi editor sessions:.
 <118>Updating motd:.
 <118>Mounting late file systems:.
 <118>Starting ntpd.
 <118>Starting bsdstats.
 <118>Posting monthly OS statistics to rpt.bsdstats.org
 <118>Performing sanity check on sshd configuration.
 <118>Starting sshd.
 <118>Starting sendmail_submit.
 <118>Starting sendmail_msp_queue.
 <118>Starting cron.
 <118>Local package initialization:cd: /var/portbuild/: No such file or directory
 <118>.
 <118>Starting inetd.
 <118>Starting background file system checks in 60 seconds.
 <118>
 <118>Mon Mar 10 09:03:49 GMT 2014
 <118>Mar 10 11:08:30 mech-as28 su: mexas to root on /dev/pts/0
 panic: deadlkres: possible deadlock detected for 0xe0000000123a6900, blocked for 1802278 ticks
 
 cpuid = 0
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000d89c0) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc000000572870) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000e2b2d8, 0x9ffc0000004f99e0, 0x40c, 0x9ffc0000010015a0) at kdb_backtrace+0xc0
 vpanic(0x9ffc000000d16c10, 0xa00000009dfff518) at vpanic+0x260
 panic(0x9ffc000000d16c10, 0x9ffc000000d16e68, 0xe0000000123a6900, 0x1b8026) at panic+0x80
 deadlkres(0x9ffc000000d16ba0, 0xe0000000123a6900, 0x9ffc000000dc8358, 0x9ffc000000d16ba0) at deadlkres+0x700
 fork_exit(0x9ffc000000d6b130, 0x0, 0xa00000009dfff550) at fork_exit+0x120
 enter_userland() at enter_userland
 KDB: enter: panic
 
 
 ddb.txt
 
 
 db:0:kdb.enter.panic>  run lockinfo
 db:1:lockinfo> show locks
 db:1:locks>  show alllocks
 Process 1281 (sh) thread 0xe0000000123a7680 (100065)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1279 (sshd) thread 0xe000000012a58900 (100100)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1278 (sh) thread 0xe000000012a58480 (100101)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1274 (sh) thread 0xe0000000123adb00 (100075)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1272 (sh) thread 0xe000000011781b00 (100089)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1270 (sshd) thread 0xe0000000123a6000 (100070)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1269 (sh) thread 0xe000000012799680 (100093)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1267 (sh) thread 0xe000000012799b00 (100092)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1265 (sh) thread 0xe0000000123a7200 (100066)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1263 (sh) thread 0xe000000012663680 (100080)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1261 (sh) thread 0xe000000012662d80 (100082)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1259 (sh) thread 0xe0000000123a6900 (100068)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 1257 (sh) thread 0xe000000012433680 (100087)
 exclusive lockmgr ufs (ufs) r = 0 (0xe000000012899688) locked @ /usr/src/sys/kern/vfs_vnops.c:373
 Process 1243 (pkg) thread 0xe0000000123a6d80 (100067)
 exclusive lockmgr ufs (ufs) r = 0 (0xe00000001286f100) locked @ /usr/src/sys/kern/vfs_vnops.c:373
 Process 1002 (login) thread 0xe000000012662900 (100083)
 shared lockmgr ufs (ufs) r = 0 (0xe0000000124474b0) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 20 (syncer) thread 0xe000000012344900 (100059)
 exclusive lockmgr ufs (ufs) r = 0 (0xe000000012680068) locked @ /usr/src/sys/kern/vfs_subr.c:1720
 db:1:alllocks>  show lockedvnods
 Locked vnodes
 db:0:kdb.enter.panic>  show pcpu
 cpuid        = 0
 dynamic pcpu = 0x40040040ff1df980
 curthread    = 0xe000000011780480: pid 0 "deadlkres"
 curpcb       = 0
 fpcurthread  = none
 idlethread   = 0xe000000011151b00: tid 100003 "idle: cpu0"
 MD: vhpt       = 0xe0000040ff000000
 MD: lid        = 0
 MD: clock      = 0xde8220a6b52
 MD: clock_mode = 2
 MD: clock_load = 0
 MD: stats      = 0x9ffc0000011a4a30
 MD: pmap       = 0x9ffc000000e53920
 spin locks held:
 db:0:kdb.enter.panic>  bt
 Tracing pid 0 tid 100049 td 0xe000000011780480
 kdb_enter(0x9ffc000000d21190, 0x9ffc000000d21190, 0x9ffc0000004f9a30, 0x40c) at kdb_enter+0x92
 vpanic(0x9ffc000000d16c10, 0xa00000009dfff518) at vpanic+0x2b0
 panic(0x9ffc000000d16c10, 0x9ffc000000d16e68, 0xe0000000123a6900, 0x1b8026) at panic+0x80
 deadlkres(0x9ffc000000d16ba0, 0xe0000000123a6900, 0x9ffc000000dc8358, 0x9ffc000000d16ba0) at deadlkres+0x700
 fork_exit(0x9ffc000000d6b130, 0x0, 0xa00000009dfff550) at fork_exit+0x120
 enter_userland() at enter_userland
 db:0:kdb.enter.panic>  ps
   pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
  1281  1280  1281     0  Ds      ufs      0xe000000012899688 sh
  1280   942   942     0  S       piperd   0xe000000012a7ba78 cron
  1279   931  1279     0  Ds      ufs      0xe000000012899688 sshd
  1278  1277  1278     0  Ds      ufs      0xe000000012899688 sh
  1277   942   942     0  S       piperd   0xe000000012794e70 cron
  1274  1273  1274     2  Ds      ufs      0xe000000012899688 sh
  1273   942   942     0  S       piperd   0xe0000000123dd0d8 cron
  1272  1271  1272     0  Ds      ufs      0xe000000012899688 sh
  1271   942   942     0  S       piperd   0xe000000012795a78 cron
  1270   931  1270     0  Ds      ufs      0xe000000012899688 sshd
  1269  1268  1269     0  Ds      ufs      0xe000000012899688 sh
  1268   942   942     0  S       piperd   0xe0000000123dc268 cron
  1267  1266  1267     2  Ds      ufs      0xe000000012899688 sh
  1266   942   942     0  S       piperd   0xe0000000123ddce0 cron
  1265  1264  1265     0  Ds      ufs      0xe000000012899688 sh
  1264   942   942     0  S       piperd   0xe0000000123dce70 cron
  1263  1262  1263     0  Ds      ufs      0xe000000012899688 sh
  1262   942   942     0  S       piperd   0xe0000000127955a8 cron
  1261  1260  1261     2  Ds      ufs      0xe000000012899688 sh
  1260   942   942     0  S       piperd   0xe0000000123dda78 cron
  1259  1258  1259     0  Ds      ufs      0xe000000012899688 sh
  1258   942   942     0  S       piperd   0xe000000012794738 cron
  1257  1243  1243     0  D+      getblk   0xa00000009e032e48 sh
  1243  1241  1243     0  DE+     getblk   0xa00000009e032510 pkg
  1241  1240  1241     0  S+      wait     0xe000000012340000 sh
  1240  1236  1240  1001  S+      wait     0xe000000012429720 su
  1236     1  1236  1001  Ss+     pause    0xe0000000123437c8 tcsh
  1233   931  1233     0  Ds      ufs      0xe0000000124474b0 sshd
  1002     1  1002     0  Ds+     ufs      0xe000000012899688 login
  1001     1     1     0  S       ttydcd   0xe00000001173b0e8 getty
   981     1   981     0  Ss      select   0xe00000001214c7c0 inetd
   942     1   942     0  Ss      nanslp   0x9ffc000000e60598 cron
   937     1   937    25  Ss      pause    0xe00000001114f328 sendmail
   934     1   934     0  Ss      select   0xe00000001178b6c0 sendmail
   931     1   931     0  Ss      select   0xe00000001178ba40 sshd
   864     1   864     0  Ss      select   0xe0000000122f3cc0 ntpd
   800     1   800     0  Ss      select   0xe00000001178bb40 casperd
   799     1    23     0  S+      select   0xe0000000117372c0 casperd
   770     1   770     0  Ds      ufs      0xe000000012680068 syslogd
   645     1   645     0  Ss      select   0xe0000000122159c0 devd
   136     1   136     0  Ss      pause    0xe000000012437328 adjkerntz
    22     0     0     0  DL      sdflush  0x9ffc000000e2c480 [softdepflush]
    21     0     0     0  DL      vlruwt   0xe000000012340de0 [vnlru]
    20     0     0     0  DL      getblk   0xa00000009e032160 [syncer]
    19     0     0     0  DL      psleep   0x9ffc000000e2b860 [bufdaemon]
    18     0     0     0  DL      pgzero   0x9ffc000000e2c7ec [pagezero]
    17     0     0     0  DL      psleep   0x9ffc000000e2c754 [vmdaemon]
     9     0     0     0  DL      psleep   0x9ffc000000e2c784 [pagedaemon]
     6     0     0     0  DL      waiting_ 0x9ffc000001198d70 [sctp_iterator]
     5     0     0     0  DL      ispf     0xe00000001173c800 [isp0: fc_thrd0]
     4     0     0     0  DL      idle     0xa000000000a22000 [mpt_recovery1]
     3     0     0     0  DL      idle     0xa000000000a0a000 [mpt_recovery0]
    16     0     0     0  DL      (threaded)                  [usb]
 100039                   D       -        0xa000000000a06e18 [usbus2]
 100038                   D       -        0xa000000000a06dc0 [usbus2]
 100037                   D       -        0xa000000000a06d68 [usbus2]
 100036                   D       -        0xa000000000a06d10 [usbus2]
 100034                   D       -        0xa0000000009fd460 [usbus1]
 100033                   D       -        0xa0000000009fd408 [usbus1]
 100032                   D       -        0xa0000000009fd3b0 [usbus1]
 100031                   D       -        0xa0000000009fd358 [usbus1]
 100029                   D       -        0xa0000000009f9460 [usbus0]
 100028                   D       -        0xa0000000009f9408 [usbus0]
 100027                   D       -        0xa0000000009f93b0 [usbus0]
 100026                   D       -        0xa0000000009f9358 [usbus0]
    15     0     0     0  DL      tzpoll   0x9ffc000000e2a288 [acpi_thermal]
     2     0     0     0  DL      (threaded)                  [cam]
 100054                   D       ccb_scan 0x9ffc000000e2cb38 [scanner]
 100015                   D       -        0x9ffc000000e2cc80 [doneq0]
    14     0     0     0  DL      -        0x9ffc000000e2a578 [rand_harvestq]
    13     0     0     0  DL      (threaded)                  [geom]
 100011                   D       -        0x9ffc000000e2aa28 [g_down]
 100010                   D       -        0x9ffc000000e2aa20 [g_up]
 100009                   D       -        0x9ffc000000e2aa10 [g_event]
    12     0     0     0  WL      (threaded)                  [intr]
 100048                   I                                   [swi0: uart uart++++]
 100046                   I                                   [irq71: isp0]
 100042                   I                                   [irq28: mpt1]
 100040                   I                                   [irq27: mpt0]
 100035                   I                                   [irq18: ehci0]
 100030                   I                                   [irq17: ohci1]
 100025                   I                                   [irq16: ohci0]
 100023                   I                                   [swi6: Giant taskq]
 100021                   I                                   [swi5: fast taskq]
 100014                   I                                   [swi6: task queue]
 100008                   I                                   [swi1: netisr 0]
 100007                   I                                   [swi3: vm]
 100006                   I                                   [swi4: clock]
 100005                   I                                   [swi4: clock]
    11     0     0     0  RL      (threaded)                  [idle]
 100004                   Run     CPU 1                       [idle: cpu1]
 100003                   CanRun                              [idle: cpu0]
     1     0     1     0  SLs     wait     0xe00000001114e4a0 [init]
    10     0     0     0  DL      audit_wo 0x9ffc00000119d408 [audit]
     0     0     0     0  RLs     (threaded)                  [kernel]
 100053                   D       -        0xe000000011142300 [CAM taskq]
 100049                   Run     CPU 0                       [deadlkres]
 100045                   D       -        0xe000000011207e00 [em1 taskq]
 100044                   D       -        0xe000000011208200 [em0 taskq]
 100022                   D       -        0xe000000011141a00 [thread taskq]
 100020                   D       -        0xe000000011141d00 [kqueue taskq]
 100019                   D       -        0xe000000011141e00 [ffs_trim taskq]
 100018                   D       -        0xe000000011142000 [acpi_task_2]
 100017                   D       -        0xe000000011142000 [acpi_task_1]
 100016                   D       -        0xe000000011142000 [acpi_task_0]
 100012                   D       -        0xe000000011143100 [firmware taskq]
 100000                   D       swapin   0x9ffc000000e52ee8 [swapper]
  1235  1233  1233  1001  Z                                   sshd
 db:0:kdb.enter.panic>  alltrace
 
 Tracing command sh pid 1281 tid 100065 td 0xe0000000123a7680
 cpu_switch(0xe0000000123a7680, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000123a7680, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000123a7680, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe0000000123a79f8, 0xe0000000123a7680) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe0000000123a7680, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa000000000281070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000281050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe0000000123a7680, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000002812e8, 0xa000000000281310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa000000000281310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000002810f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa000000000281288, 0x200000, 0xe000000012447448, 0x0, 0xa000000000281318) at lookup+0xe10
 namei(0xa000000000281288, 0xe0000000123e5720, 0x9ffc0000009a5410, 0x0, 0xa0000000002812f8) at namei+0xb40
 vn_open_cred(0xa000000000281288, 0xa000000000281390, 0x8a8, 0x0, 0xe00000001229d200, 0xe0000000123b3720) at vn_open_cred+0x500
 vn_open(0xa000000000281288, 0xa000000000281390, 0x8a8, 0xe0000000123b3720, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe0000000123a7680, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe558b8) at kern_openat+0x490
 kern_open(0xe0000000123a7680, 0x1400c39a0, 0x0, 0x100000, 0xe558b8) at kern_open+0x40
 sys_open(0xe0000000123a7680, 0xa0000000002814e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe0000000123e5720, 0x100000, 0x9ffc000000e558b8, 0xe0000000123a7680, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1280 tid 100103 td 0xe00000001279bb00
 cpu_switch(0xe00000001279bb00, 0xe000000011151680, 0x9ffc000000eb2370, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe00000001279bb00, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe00000001279bb00, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012a7ba78, 0x5c, 0xe00000001279be78, 0xe00000001279bb00) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe000000012a7ba78, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe000000012a7ba78, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe000000012a7ba78, 0xe000000012a7bcb8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123b50b4, 0xa0000000003b1360, 0xa0000000003b1378, 0x0, 0xe000000012a7bb80) at pipe_read+0x740
 dofileread(0xe00000001279bb00, 0x6, 0xe0000000123b5090, 0xa0000000003b1360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe00000001279bb00, 0x6, 0xa0000000003b1360) at kern_readv+0xa0
 sys_read(0xe00000001279bb00, 0xa0000000003b14e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe0000000123b1720, 0x140c26000, 0x2000, 0xe00000001279bb00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sshd pid 1279 tid 100100 td 0xe000000012a58900
 cpu_switch(0xe000000012a58900, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012a58900, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012a58900, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe000000012a58c78, 0xe000000012a58900) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe000000012a58900, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa000000000399070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000399050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe000000012a58900, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000003992e8, 0xa000000000399310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa000000000399310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000003990f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa000000000399288, 0x200000, 0xe000000012447448, 0x0, 0xa000000000399318) at lookup+0xe10
 namei(0xa000000000399288, 0xe000000012a06940, 0x9ffc0000009a5410, 0x0, 0xa0000000003992f8) at namei+0xb40
 vn_open_cred(0xa000000000399288, 0xa000000000399390, 0x0, 0x0, 0xe000000011143200, 0xe0000000123b5130) at vn_open_cred+0x500
 vn_open(0xa000000000399288, 0xa000000000399390, 0x0, 0xe0000000123b5130, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe000000012a58900, 0xffffffffffffff9c, 0x1400f99a0, 0x0, 0x0, 0x4010d000) at kern_openat+0x490
 kern_open(0xe000000012a58900, 0x1400f99a0, 0x0, 0x100000, 0x4010d000) at kern_open+0x40
 sys_open(0xe000000012a58900, 0xa0000000003994e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe000000012a06940, 0x100000, 0x14010d000, 0xe000000012a58900, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1278 tid 100101 td 0xe000000012a58480
 cpu_switch(0xe000000012a58480, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012a58480, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012a58480, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe000000012a587f8, 0xe000000012a58480) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe000000012a58480, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa0000000003a1070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa0000000003a1050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe000000012a58480, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000003a12e8, 0xa0000000003a1310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa0000000003a1310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000003a10f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000003a1288, 0x200000, 0xe000000012447448, 0x0, 0xa0000000003a1318) at lookup+0xe10
 namei(0xa0000000003a1288, 0xe000000012a064a0, 0x9ffc0000009a5410, 0x0, 0xa0000000003a12f8) at namei+0xb40
 vn_open_cred(0xa0000000003a1288, 0xa0000000003a1390, 0x8a8, 0x0, 0xe00000001230c000, 0xe0000000123b2a50) at vn_open_cred+0x500
 vn_open(0xa0000000003a1288, 0xa0000000003a1390, 0x8a8, 0xe0000000123b2a50, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe000000012a58480, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe558b8) at kern_openat+0x490
 kern_open(0xe000000012a58480, 0x1400c39a0, 0x0, 0x100000, 0xe558b8) at kern_open+0x40
 sys_open(0xe000000012a58480, 0xa0000000003a14e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe000000012a064a0, 0x100000, 0x9ffc000000e558b8, 0xe000000012a58480, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1277 tid 100102 td 0xe000000012a58000
 cpu_switch(0xe000000012a58000, 0xe000000011151680, 0x9ffc000000eb0ed0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012a58000, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012a58000, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012794e70, 0x5c, 0xe000000012a58378, 0xe000000012a58000) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe000000012794e70, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe000000012794e70, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe000000012794e70, 0xe0000000127950b0, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123b4e34, 0xa0000000003a9360, 0xa0000000003a9378, 0x0, 0xe000000012794f78) at pipe_read+0x740
 dofileread(0xe000000012a58000, 0x6, 0xe0000000123b4e10, 0xa0000000003a9360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012a58000, 0x6, 0xa0000000003a9360) at kern_readv+0xa0
 sys_read(0xe000000012a58000, 0xa0000000003a94e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe000000012a06000, 0x140c26000, 0x2000, 0xe000000012a58000, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1274 tid 100075 td 0xe0000000123adb00
 cpu_switch(0xe0000000123adb00, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000123adb00, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000123adb00, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe0000000123ade78, 0xe0000000123adb00) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe0000000123adb00, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa0000000002d1070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa0000000002d1050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe0000000123adb00, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000002d12e8, 0xa0000000002d1310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa0000000002d1310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000002d10f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000002d1288, 0x200000, 0xe000000012447448, 0x0, 0xa0000000002d1318) at lookup+0xe10
 namei(0xa0000000002d1288, 0xe0000000124284a0, 0x9ffc0000009a5410, 0x0, 0xa0000000002d12f8) at namei+0xb40
 vn_open_cred(0xa0000000002d1288, 0xa0000000002d1390, 0x8a8, 0x0, 0xe00000001215db00, 0xe0000000123d6dc0) at vn_open_cred+0x500
 vn_open(0xa0000000002d1288, 0xa0000000002d1390, 0x8a8, 0xe0000000123d6dc0, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe0000000123adb00, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe558b8) at kern_openat+0x490
 kern_open(0xe0000000123adb00, 0x1400c39a0, 0x0, 0x100000, 0xe558b8) at kern_open+0x40
 sys_open(0xe0000000123adb00, 0xa0000000002d14e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe0000000124284a0, 0x100000, 0x9ffc000000e558b8, 0xe0000000123adb00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1273 tid 100099 td 0xe000000012663b00
 cpu_switch(0xe000000012663b00, 0xe000000011151680, 0x9ffc000000eb0660, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012663b00, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012663b00, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000123dd0d8, 0x5c, 0xe000000012663e78, 0xe000000012663b00) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000123dd0d8, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000123dd0d8, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000123dd0d8, 0xe0000000123dd318, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123b2074, 0xa000000000391360, 0xa000000000391378, 0x0, 0xe0000000123dd1e0) at pipe_read+0x740
 dofileread(0xe000000012663b00, 0x6, 0xe0000000123b2050, 0xa000000000391360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012663b00, 0x6, 0xa000000000391360) at kern_readv+0xa0
 sys_read(0xe000000012663b00, 0xa0000000003914e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe000000012a06de0, 0x140c26000, 0x2000, 0xe000000012663b00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1272 tid 100089 td 0xe000000011781b00
 cpu_switch(0xe000000011781b00, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000011781b00, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000011781b00, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe000000011781e78, 0xe000000011781b00) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe000000011781b00, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa000000000341070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000341050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe000000011781b00, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000003412e8, 0xa000000000341310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa000000000341310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000003410f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa000000000341288, 0x200000, 0xe000000012447448, 0x0, 0xa000000000341318) at lookup+0xe10
 namei(0xa000000000341288, 0xe0000000123404a0, 0x9ffc0000009a5410, 0x0, 0xa0000000003412f8) at namei+0xb40
 vn_open_cred(0xa000000000341288, 0xa000000000341390, 0x8a8, 0x0, 0xe0000000121c8500, 0xe0000000123b2aa0) at vn_open_cred+0x500
 vn_open(0xa000000000341288, 0xa000000000341390, 0x8a8, 0xe0000000123b2aa0, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe000000011781b00, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe558b8) at kern_openat+0x490
 kern_open(0xe000000011781b00, 0x1400c39a0, 0x0, 0x100000, 0xe558b8) at kern_open+0x40
 sys_open(0xe000000011781b00, 0xa0000000003414e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe0000000123404a0, 0x100000, 0x9ffc000000e558b8, 0xe000000011781b00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1271 tid 100098 td 0xe000000012798000
 cpu_switch(0xe000000012798000, 0xe000000011151680, 0x9ffc000000eb0a70, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012798000, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012798000, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012795a78, 0x5c, 0xe000000012798378, 0xe000000012798000) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe000000012795a78, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe000000012795a78, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe000000012795a78, 0xe000000012795cb8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123b2164, 0xa000000000389360, 0xa000000000389378, 0x0, 0xe000000012795b80) at pipe_read+0x740
 dofileread(0xe000000012798000, 0x6, 0xe0000000123b2140, 0xa000000000389360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012798000, 0x6, 0xa000000000389360) at kern_readv+0xa0
 sys_read(0xe000000012798000, 0xa0000000003894e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe000000012a07280, 0x140c26000, 0x2000, 0xe000000012798000, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sshd pid 1270 tid 100070 td 0xe0000000123a6000
 cpu_switch(0xe0000000123a6000, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000123a6000, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000123a6000, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe0000000123a6378, 0xe0000000123a6000) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe0000000123a6000, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa0000000002a9070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa0000000002a9050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe0000000123a6000, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000002a92e8, 0xa0000000002a9310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa0000000002a9310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000002a90f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000002a9288, 0x200000, 0xe000000012447448, 0x0, 0xa0000000002a9318) at lookup+0xe10
 namei(0xa0000000002a9288, 0xe0000000123e4000, 0x9ffc0000009a5410, 0x0, 0xa0000000002a92f8) at namei+0xb40
 vn_open_cred(0xa0000000002a9288, 0xa0000000002a9390, 0x0, 0x0, 0xe000000011143200, 0xe0000000123b2cd0) at vn_open_cred+0x500
 vn_open(0xa0000000002a9288, 0xa0000000002a9390, 0x0, 0xe0000000123b2cd0, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe0000000123a6000, 0xffffffffffffff9c, 0x1400f99a0, 0x0, 0x0, 0x4010d000) at kern_openat+0x490
 kern_open(0xe0000000123a6000, 0x1400f99a0, 0x0, 0x100000, 0x4010d000) at kern_open+0x40
 sys_open(0xe0000000123a6000, 0xa0000000002a94e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe0000000123e4000, 0x100000, 0x14010d000, 0xe0000000123a6000, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1269 tid 100093 td 0xe000000012799680
 cpu_switch(0xe000000012799680, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012799680, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012799680, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe0000000127999f8, 0xe000000012799680) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe000000012799680, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa000000000361070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000361050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe000000012799680, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000003612e8, 0xa000000000361310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa000000000361310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000003610f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa000000000361288, 0x200000, 0xe000000012447448, 0x0, 0xa000000000361318) at lookup+0xe10
 namei(0xa000000000361288, 0xe000000012428de0, 0x9ffc0000009a5410, 0x0, 0xa0000000003612f8) at namei+0xb40
 vn_open_cred(0xa000000000361288, 0xa000000000361390, 0x8a8, 0x0, 0xe00000001230d400, 0xe0000000123b2fa0) at vn_open_cred+0x500
 vn_open(0xa000000000361288, 0xa000000000361390, 0x8a8, 0xe0000000123b2fa0, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe000000012799680, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe558b8) at kern_openat+0x490
 kern_open(0xe000000012799680, 0x1400c39a0, 0x0, 0x100000, 0xe558b8) at kern_open+0x40
 sys_open(0xe000000012799680, 0xa0000000003614e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe000000012428de0, 0x100000, 0x9ffc000000e558b8, 0xe000000012799680, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1268 tid 100097 td 0xe000000012798480
 cpu_switch(0xe000000012798480, 0xe000000011151680, 0x9ffc000000eb1fb0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012798480, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012798480, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000123dc268, 0x5c, 0xe0000000127987f8, 0xe000000012798480) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000123dc268, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000123dc268, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000123dc268, 0xe0000000123dc4a8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123b22f4, 0xa000000000381360, 0xa000000000381378, 0x0, 0xe0000000123dc370) at pipe_read+0x740
 dofileread(0xe000000012798480, 0x6, 0xe0000000123b22d0, 0xa000000000381360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012798480, 0x6, 0xa000000000381360) at kern_readv+0xa0
 sys_read(0xe000000012798480, 0xa0000000003814e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe000000012a07720, 0x140c26000, 0x2000, 0xe000000012798480, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1267 tid 100092 td 0xe000000012799b00
 cpu_switch(0xe000000012799b00, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012799b00, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012799b00, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe000000012799e78, 0xe000000012799b00) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe000000012799b00, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa000000000359070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000359050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe000000012799b00, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000003592e8, 0xa000000000359310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa000000000359310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000003590f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa000000000359288, 0x200000, 0xe000000012447448, 0x0, 0xa000000000359318) at lookup+0xe10
 namei(0xa000000000359288, 0xe000000012429280, 0x9ffc0000009a5410, 0x0, 0xa0000000003592f8) at namei+0xb40
 vn_open_cred(0xa000000000359288, 0xa000000000359390, 0x8a8, 0x0, 0xe00000001229e600, 0xe0000000123b2910) at vn_open_cred+0x500
 vn_open(0xa000000000359288, 0xa000000000359390, 0x8a8, 0xe0000000123b2910, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe000000012799b00, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe558b8) at kern_openat+0x490
 kern_open(0xe000000012799b00, 0x1400c39a0, 0x0, 0x100000, 0xe558b8) at kern_open+0x40
 sys_open(0xe000000012799b00, 0xa0000000003594e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe000000012429280, 0x100000, 0x9ffc000000e558b8, 0xe000000012799b00, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1266 tid 100096 td 0xe000000012798900
 cpu_switch(0xe000000012798900, 0xe000000011151680, 0x9ffc000000eb0e80, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012798900, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012798900, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000123ddce0, 0x5c, 0xe000000012798c78, 0xe000000012798900) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000123ddce0, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000123ddce0, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000123ddce0, 0xe0000000123ddf20, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123b2344, 0xa000000000379360, 0xa000000000379378, 0x0, 0xe0000000123ddde8) at pipe_read+0x740
 dofileread(0xe000000012798900, 0x6, 0xe0000000123b2320, 0xa000000000379360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012798900, 0x6, 0xa000000000379360) at kern_readv+0xa0
 sys_read(0xe000000012798900, 0xa0000000003794e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe000000012a0c000, 0x140c26000, 0x2000, 0xe000000012798900, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1265 tid 100066 td 0xe0000000123a7200
 cpu_switch(0xe0000000123a7200, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000123a7200, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000123a7200, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe0000000123a7578, 0xe0000000123a7200) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe0000000123a7200, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa000000000289070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000289050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe0000000123a7200, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000002892e8, 0xa000000000289310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa000000000289310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000002890f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa000000000289288, 0x200000, 0xe000000012447448, 0x0, 0xa000000000289318) at lookup+0xe10
 namei(0xa000000000289288, 0xe0000000123e5280, 0x9ffc0000009a5410, 0x0, 0xa0000000002892f8) at namei+0xb40
 vn_open_cred(0xa000000000289288, 0xa000000000289390, 0x8a8, 0x0, 0xe00000001229e300, 0xe0000000123b2b90) at vn_open_cred+0x500
 vn_open(0xa000000000289288, 0xa000000000289390, 0x8a8, 0xe0000000123b2b90, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe0000000123a7200, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe558b8) at kern_openat+0x490
 kern_open(0xe0000000123a7200, 0x1400c39a0, 0x0, 0x100000, 0xe558b8) at kern_open+0x40
 sys_open(0xe0000000123a7200, 0xa0000000002894e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe0000000123e5280, 0x100000, 0x9ffc000000e558b8, 0xe0000000123a7200, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1264 tid 100095 td 0xe000000012798d80
 cpu_switch(0xe000000012798d80, 0xe000000011151680, 0x9ffc000000eb22d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012798d80, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012798d80, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000123dce70, 0x5c, 0xe0000000127990f8, 0xe000000012798d80) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000123dce70, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000123dce70, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000123dce70, 0xe0000000123dd0b0, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123d67f4, 0xa000000000371360, 0xa000000000371378, 0x0, 0xe0000000123dcf78) at pipe_read+0x740
 dofileread(0xe000000012798d80, 0x6, 0xe0000000123d67d0, 0xa000000000371360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012798d80, 0x6, 0xa000000000371360) at kern_readv+0xa0
 sys_read(0xe000000012798d80, 0xa0000000003714e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe000000012a0c4a0, 0x140c26000, 0x2000, 0xe000000012798d80, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1263 tid 100080 td 0xe000000012663680
 cpu_switch(0xe000000012663680, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012663680, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012663680, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe0000000126639f8, 0xe000000012663680) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe000000012663680, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa0000000002f9070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa0000000002f9050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe000000012663680, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000002f92e8, 0xa0000000002f9310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa0000000002f9310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000002f90f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa0000000002f9288, 0x200000, 0xe000000012447448, 0x0, 0xa0000000002f9318) at lookup+0xe10
 namei(0xa0000000002f9288, 0xe000000012436940, 0x9ffc0000009a5410, 0x0, 0xa0000000002f92f8) at namei+0xb40
 vn_open_cred(0xa0000000002f9288, 0xa0000000002f9390, 0x8a8, 0x0, 0xe00000001215c300, 0xe0000000123b4190) at vn_open_cred+0x500
 vn_open(0xa0000000002f9288, 0xa0000000002f9390, 0x8a8, 0xe0000000123b4190, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe000000012663680, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe558b8) at kern_openat+0x490
 kern_open(0xe000000012663680, 0x1400c39a0, 0x0, 0x100000, 0xe558b8) at kern_open+0x40
 sys_open(0xe000000012663680, 0xa0000000002f94e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe000000012436940, 0x100000, 0x9ffc000000e558b8, 0xe000000012663680, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1262 tid 100094 td 0xe000000012799200
 cpu_switch(0xe000000012799200, 0xe000000011151680, 0x9ffc000000eb2ca8, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012799200, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012799200, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000127955a8, 0x5c, 0xe000000012799578, 0xe000000012799200) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000127955a8, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000127955a8, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000127955a8, 0xe0000000127957e8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123b3794, 0xa000000000369360, 0xa000000000369378, 0x0, 0xe0000000127956b0) at pipe_read+0x740
 dofileread(0xe000000012799200, 0x6, 0xe0000000123b3770, 0xa000000000369360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012799200, 0x6, 0xa000000000369360) at kern_readv+0xa0
 sys_read(0xe000000012799200, 0xa0000000003694e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe000000012a0c940, 0x140c26000, 0x2000, 0xe000000012799200, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1261 tid 100082 td 0xe000000012662d80
 cpu_switch(0xe000000012662d80, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012662d80, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012662d80, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe0000000126630f8, 0xe000000012662d80) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe000000012662d80, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0xe0000000128996b8, 0x9ffc000000d153b8, 0x60, 0x0, 0x9ffc000000d363a0, 0x835) at __lockmgr_args+0xae0
 ffs_lock(0xa000000000309070, 0xe000000012899688, 0x200100) at ffs_lock+0x130
 VOP_LOCK1_APV(0x9ffc000000df1d20, 0xa000000000309050, 0x9ffc0000004f6150) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000012899620, 0x200100, 0x9ffc000000d363a0, 0x835) at _vn_lock+0x100
 vget(0xe000000012899620, 0x200100, 0xe000000012662d80, 0x9ffc000001196448) at vget+0x100
 cache_lookup(0xf, 0xa0000000003092e8, 0xa000000000309310, 0x0, 0x0) at cache_lookup+0xf70
 vfs_cache_lookup(0xe000000012447448, 0x0, 0xa000000000309310) at vfs_cache_lookup+0x1a0
 VOP_LOOKUP_APV(0x9ffc000000df2660, 0xa0000000003090f8, 0x420c144, 0x0) at VOP_LOOKUP_APV+0x220
 lookup(0xa000000000309288, 0x200000, 0xe000000012447448, 0x0, 0xa000000000309318) at lookup+0xe10
 namei(0xa000000000309288, 0xe000000012436000, 0x9ffc0000009a5410, 0x0, 0xa0000000003092f8) at namei+0xb40
 vn_open_cred(0xa000000000309288, 0xa000000000309390, 0x8a8, 0x0, 0xe00000001229dd00, 0xe0000000123b28c0) at vn_open_cred+0x500
 vn_open(0xa000000000309288, 0xa000000000309390, 0x8a8, 0xe0000000123b28c0, 0x9ffc00000066bfb0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe000000012662d80, 0xffffffffffffff9c, 0x1400c39a0, 0x0, 0x0, 0xe558b8) at kern_openat+0x490
 kern_open(0xe000000012662d80, 0x1400c39a0, 0x0, 0x100000, 0xe558b8) at kern_open+0x40
 sys_open(0xe000000012662d80, 0xa0000000003094e8, 0x9ffc000000a2cc80, 0x48d) at sys_open+0x40
 syscall(0xe000000012436000, 0x100000, 0x9ffc000000e558b8, 0xe000000012662d80, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cron pid 1260 tid 100088 td 0xe000000012433200
 cpu_switch(0xe000000012433200, 0xe000000011151680, 0x9ffc000000eb1e70, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe000000012433200, 0xe000000011151680, 0x9ffc000000e7b518, 0x9ffc000000e7b500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012433200, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe0000000123dda78, 0x5c, 0xe000000012433578, 0xe000000012433200) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000000123dda78, 0x5c, 0x9ffc000000d28b50, 0x9ffc000000d28f10) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000000123dda78, 0x5c, 0x9ffc000000d2c5a0, 0x100, 0x9ffc00000050be40) at sleepq_wait_sig+0x30
 _sleep(0xe0000000123dda78, 0xe0000000123ddcb8, 0x5c, 0x9ffc000000d2c5a0, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000123d69d4, 0xa000000000339360, 0xa000000000339378, 0x0, 0xe0000000123ddb80) at pipe_read+0x740
 dofileread(0xe000000012433200, 0x6, 0xe0000000123d69b0, 0xa000000000339360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012433200, 0x6, 0xa000000000339360) at kern_readv+0xa0
 sys_read(0xe000000012433200, 0xa0000000003394e8, 0x9ffc000000a2cc80, 0x48d) at sys_read+0x100
 syscall(0xe0000000126e2000, 0x140c26000, 0x2000, 0xe000000012433200, 0x0, 0x0, 0x9ffc000000a29620, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 1259 tid 100068 td 0xe0000000123a6900
 cpu_switch(0xe0000000123a6900, 0xe000000011151b00, 0x9ffc000000eb09d0, 0x9ffc0000005532f0) at cpu_switch+0xd0
 sched_switch(0xe0000000123a6900, 0xe000000011151b00, 0x9ffc000000e7a818, 0x9ffc000000e7a800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000000123a6900, 0x9ffc00000058c3c0) at mi_switch+0x3f0
 sleepq_switch(0xe000000012899688, 0x60, 0xe0000000123a6c78, 0xe0000000123a6900) at sleepq_switch+0x2e0
 sleepq_wait(0xe000000012899688, 0x60, 0x9ffc000000d28b50, 0xe0000000123a6900, 0x9ffc0000004c37d0, 0x12ab) at sleepq_wait+0xb0
 __lockmgr_args(0xe000000012899688, 0x200100, 0
 
>Unformatted:
