From mexas@bristol.ac.uk  Tue Oct 15 21:34:20 2013
Return-Path: <mexas@bristol.ac.uk>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTP id 21BBA651
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 15 Oct 2013 21:34:20 +0000 (UTC)
	(envelope-from mexas@bristol.ac.uk)
Received: from eu1sys200aog121.obsmtp.com (eu1sys200aog121.obsmtp.com [207.126.144.151])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id 078CC2A42
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 15 Oct 2013 21:34:18 +0000 (UTC)
Received: from mail-ee0-f54.google.com ([74.125.83.54]) (using TLSv1) by eu1sys200aob121.postini.com ([207.126.147.11]) with SMTP
	ID DSNKUl202YsBv4UPZVG7+c53AXo2wQHcTqas@postini.com; Tue, 15 Oct 2013 21:34:19 UTC
Received: by mail-ee0-f54.google.com with SMTP id e53so4298204eek.13
        for <FreeBSD-gnats-submit@freebsd.org>; Tue, 15 Oct 2013 14:34:17 -0700 (PDT)
Received: from mech-aslap33.men.bris.ac.uk (cpc2-aztw9-0-0-cust169.18-1.cable.virginmedia.com. [77.97.103.170])
        by mx.google.com with ESMTPSA id bn13sm171304257eeb.11.1969.12.31.16.00.00
        (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
        Tue, 15 Oct 2013 14:34:16 -0700 (PDT)
Received: from mech-aslap33.men.bris.ac.uk (localhost [127.0.0.1])
	by mech-aslap33.men.bris.ac.uk (8.14.7/8.14.7) with ESMTP id r9FLYEXE010288
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 15 Oct 2013 22:34:14 +0100 (BST)
	(envelope-from mexas@mech-aslap33.men.bris.ac.uk)
Received: (from mexas@localhost)
	by mech-aslap33.men.bris.ac.uk (8.14.7/8.14.7/Submit) id r9FLYE06010287;
	Tue, 15 Oct 2013 22:34:14 +0100 (BST)
	(envelope-from mexas)
Message-Id: <201310152134.r9FLYE06010287@mech-aslap33.men.bris.ac.uk>
Date: Tue, 15 Oct 2013 22:34:14 +0100 (BST)
From: Anton Shterenlikht <mexas@bris.ac.uk>
Sender: Anton Shterenlikht <mexas@bristol.ac.uk>
Reply-To: Anton Shterenlikht <mexas@bris.ac.uk>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: ia64: panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks
X-Send-Pr-Version: 3.114
X-GNATS-Notify:

>Number:         183007
>Category:       kern
>Synopsis:       ia64: panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-ia64
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Oct 15 21:40:01 UTC 2013
>Closed-Date:    Mon Jan 20 18:28:23 UTC 2014
>Last-Modified:  Mon Jan 20 18:28:23 UTC 2014
>Originator:     Anton Shterenlikht
>Release:        FreeBSD 10.0-CURRENT amd64
>Organization:
University of Bristol	
>Environment:
System: FreeBSD mech-aslap33.men.bris.ac.uk 10.0-CURRENT FreeBSD 10.0-CURRENT #6 r255093: Sat Aug 31 13:49:24 BST 2013 root@mech-aslap33.men.bris.ac.uk:/usr/obj/usr/src/sys/MINKY amd64


	
>Description:

below are the outputs of ddb commands:

show msgbuf
show proc
run lockinfo
show pcpu
bt
show threads
ps
alltrace
show pginfo


db> show msgbuf
msgbufp = 0xe0000040fffdffb8
magic = 63062, size = 98232, r= 11031, w = 11766, ptr = 0xe0000040fffc8000, cksum= 898967
panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks

cpuid = 1
KDB: stack backtrace:
db_trace_self(0x9ffc000000158380) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc00000060b900) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000ee1b48, 0x9ffc000000592a40, 0x40c, 0x9ffc0000010b48a0) at kdb_backtrace+0xc0
vpanic(0x9ffc000000dc54d8, 0xa00000009de3b518) at vpanic+0x260
panic(0x9ffc000000dc54d8, 0x9ffc000000dc5768, 0xe0000000128ecd80, 0xdbbdc) at panic+0x80
deadlkres(0xdbbdc, 0xe0000000128ecd80, 0x9ffc000000dc8140, 0x9ffc000000dc5468) at deadlkres+0x420
fork_exit(0x9ffc000000e23230, 0x0, 0xa00000009de3b550) at fork_exit+0x120
enter_userland() at enter_userland
KDB: enter: panic
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #7 r255488: Tue Oct 15 10:09:07 BST 2013
    root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV ia64
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
CPU: Madison II (1600 MHz Itanium 2)
  Origin = "GenuineIntel"  Revision = 2
  Features = 0x1<LB>
real memory  = 10737418240 (10240 MB)
avail memory = 10484695040 (9998 MB)
FPSWA Revision = 0x10012, Entry = 0xe0000040ffcc4050
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP)
cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0
random: <Software, Yarrow> initialized
Event timer "ITC" frequency 1600000000 Hz quality 1000
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603)
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603)
acpi0: <HP> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0
acpi_tz0: <Thermal Zone> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pcib0: could not evaluate _ADR - AE_NOT_FOUND
pci0: <ACPI PCI bus> on pcib0
ohci0: <NEC uPD 9210 USB controller> mem 0x80002000-0x80002fff irq 16 at device 1.0 on pci0
usbus0 on ohci0
ohci1: <NEC uPD 9210 USB controller> mem 0x80001000-0x80001fff irq 17 at device 1.1 on pci0
usbus1 on ohci1
ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x80000000-0x800000ff irq 18 at device 1.2 on pci0
usbus2: EHCI version 0.95
usbus2 on ehci0
pci0: <mass storage, ATA> at device 2.0 (no driver attached)
pcib1: <ACPI Host-PCI bridge> on acpi0
pcib1: could not evaluate _ADR - AE_NOT_FOUND
pci32: <ACPI PCI bus> on pcib1
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2100-0x21ff mem 0x903a0000-0x903bffff,0x90380000-0x9039ffff irq 27 at device 1.0 on pci32
mpt0: MPI Version=1.2.12.0
mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0x90360000-0x9037ffff,0x90340000-0x9035ffff irq 28 at device 1.1 on pci32
mpt1: MPI Version=1.2.12.0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2240-0x227f mem 0x90320000-0x9033ffff,0x90280000-0x902fffff irq 29 at device 2.0 on pci32
em0: Ethernet address: 00:13:21:5b:05:1c
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2200-0x223f mem 0x90300000-0x9031ffff irq 30 at device 2.1 on pci32
em1: Ethernet address: 00:13:21:5b:05:1d
pcib2: <ACPI Host-PCI bridge> on acpi0
pcib2: could not evaluate _ADR - AE_NOT_FOUND
pci64: <ACPI PCI bus> on pcib2
pcib3: <ACPI Host-PCI bridge> on acpi0
pcib3: could not evaluate _ADR - AE_NOT_FOUND
pci96: <ACPI PCI bus> on pcib3
pcib4: <ACPI Host-PCI bridge> on acpi0
pcib4: could not evaluate _ADR - AE_NOT_FOUND
pci128: <ACPI PCI bus> on pcib4
pcib5: <ACPI Host-PCI bridge> on acpi0
pcib5: could not evaluate _ADR - AE_NOT_FOUND
pci192: <ACPI PCI bus> on pcib5
isp0: <Qlogic ISP 2422 PCI FC-AL Adapter> port 0xc000-0xc0ff mem 0xe0040000-0xe0040fff irq 71 at device 1.0 on pci192
pcib6: <ACPI Host-PCI bridge> on acpi0
pcib6: could not evaluate _ADR - AE_NOT_FOUND
pci224: <ACPI PCI bus> on pcib6
uart0: <HP Auxiliary Diva Serial Port> mem 0xf8051000-0xf805100f irq 82 at device 1.0 on pci224
puc0: <HP Diva Serial [GSP] Multiport UART - Everest SP2> mem 0xf8050000-0xf8050fff,0xf8020000-0xf803ffff irq 82 at device 1.1 on pci224
uart1: <Non-standard ns8250 class UART with FIFOs> at port 1 on puc0
uart1: console (9600,n,8,1)
uart2: <Non-standard ns8250 class UART with FIFOs> at port 3 on puc0
vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf0000000-0xf7ffffff,0xf8040000-0xf804ffff at device 2.0 on pci224
uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0
uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0
uart4: debug port (9600,n,8,1)
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
IP Filter: v5.1.2 initialized.  Default = block all, Logging = enabled
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
ugen2.1: <NEC> at usbus2
uhub0: <NEC EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen1.1: <NEC> at usbus1
uhub1: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen0.1: <NEC> at usbus0
uhub2: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub1: 2 ports with 2 removable, self powered
uhub2: 3 ports with 3 removable, self powered
uhub0: 5 ports with 5 removable, self powered
Expensive timeout(9) function: 0x9ffc000000e1e1e0(0x9ffc000000ee3c00) 0.020669666 s
Expensive timeout(9) function: 0x9ffc000000e222e0(0xa0000000009ed320) 0.282418016 s
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <HP 73.4G ST373454LC HPC2> Fixed Direct Access SCSI-3 device
da0: 320.000MB/s transfersWARNING: WITNESS option enabled, expect reduced performance.
 (160.000MHz, offset 63WARNING: DIAGNOSTIC option enabled, expect reduced performance.
, 16bit)
da0: Command Queueing enabled
da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <SEAGATE ST318452LC 2213> Fixed Direct Access SCSI-3 device
da1: 160.000MB/s transfers (80.000MHz, offset 63, 16bit)
da1: Command Queueing enabled
da1: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C)
da2 at mpt1 bus 0 scbus1 target 2 lun 0
da2: <SEAGATE ST318452LC 2213> Fixed Direct Access SCSI-3 device
da2: 160.000MB/s transfers (80.000MHz, offset 63, 16bit)
da2: Command Queueing enabled
da2: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C)
Trying to mount root from ufs:/dev/da0p2 [rw]...
WARNING: / was not properly dismounted
WITNESS: unable to allocate a new witness object
<118>Setting hostuuid: ee652761-3dcc-11df-9074-0013215b051c.
<118>Setting hostid: 0x8968724b.
<118>dumpon: /dev/da4p1: No such file or directory
<118>/etc/rc: WARNING: unable to specify /dev/da4p1 as a dump device
<118>Starting ddb.
<118>Entropy harvesting: interrupts ethernet point_to_point kickstart.
<118>swapon: /dev/da4p1: No such file or directory
<118>Starting file system checks:
da3 at isp0 bus 0 scbus2 target 0 lun 1
da3: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device
da3: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da3: Command Queueing enabled
da3: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
da4 at isp0 bus 0 scbus2 target 0 lun 2
da4: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device
da4: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da4: Command Queueing enabled
da4: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
da5 at isp0 bus 0 scbus2 target 0 lun 3
da5: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device
da5: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da5: Command Queueing enabled
da5: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
da6 at isp0 bus 0 scbus2 target 0 lun 5
da6: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device
da6: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da6: Command Queueing enabled
da6: 140011MB (286744185 512 byte sectors: 255H 63S/T 17849C)
<118>/dev/da0p2: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
<118>/dev/da0p2: 582356 files, 3927896 used, 30688528 free (183320 frags, 3813151 blocks, 0.5% fragmentation)
<118>/dev/da3p1: INCORRECT BLOCK COUNT I=642145 (8 should be 0) (CORRECTED)
<118>/dev/da3p1: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
<118>/dev/da3p1: SUMMARY INFORMATION BAD (SALVAGED)
<118>/dev/da3p1: BLK(S) MISSING IN BIT MAPS (SALVAGED)
<118>/dev/da3p1: 1967 files, 7223674 used, 9999687 free (263 frags, 1249928 blocks, 0.0% fragmentation)
<118>/dev/da5p1: 436860 files, 1247548 used, 15975813 free (36405 frags, 1992426 blocks, 0.2% fragmentation)
<118>/dev/da6p1: 5481 files, 1323234 used, 68104269 free (5773 frags, 8512312 blocks, 0.0% fragmentation)
<118>Mounting local file systems:.
<118>Writing entropy file:.
<118>Setting hostname: mech-as28.men.bris.ac.uk.
<118>Enabling ipfilter.
<118>Starting Network: lo0 em0 em1.
<118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
<118>   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
<118>   inet6 ::1 prefixlen 128
<118>   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
<118>   inet 127.0.0.1 netmask 0xff000000
<118>   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
<118>em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>   options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
<118>   ether 00:13:21:5b:05:1c
<118>   inet 137.222.187.28 netmask 0xffffff00 broadcast 137.222.187.255
<118>   inet6 fe80::213:21ff:fe5b:51c%em0 prefixlen 64 scopeid 0x1
<118>   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118>   media: Ethernet autoselect
<118>   status: no carrier
<118>em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>   options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
<118>   ether 00:13:21:5b:05:1d
<118>   inet 10.10.10.14 netmask 0xffffff00 broadcast 10.10.10.255
<118>   inet6 fe80::213:21ff:fe5b:51d%em1 prefixlen 64 scopeid 0x2
<118>   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118>   media: Ethernet autoselect (100baseTX <full-duplex>)
<118>   status: active
<118>filter sync'd
<118>Starting devd.
<118>add net default: gateway 137.222.187.250
<118>add net fe80::: gateway ::1
<118>add net ff02::: gateway ::1
<118>add net ::ffff:0.0.0.0: gateway ::1
<118>add net ::0.0.0.0: gateway ::1
<118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
<118>Starting ipmon.
<118>Creating and/or trimming log files.
<118>Starting syslogd.
<118>No core dumps found.
<118>Clearing /tmp (X related).
<118>Recovering vi editor sessions:.
<118>Updating motd:.
<118>Mounting late file systems:.
<118>Starting ntpd.
<118>Starting bsdstats.
<118>Posting monthly OS statistics to rpt.bsdstats.org
<118>Performing sanity check on sshd configuration.
<118>Starting sshd.
<118>Starting cron.
<118>Local package initialization:cd: /var/portbuild/: No such file or directory
<118>.  
<118>Starting inetd.
<118>Starting background file system checks in 60 seconds.
<118>   
<118>Tue Oct 15 16:15:03 BST 2013
<118>Oct 15 16:15:43 mech-as28 su: mexas to root on /dev/pts/0
<118>Oct 15 16:26:25 mech-as28 su: mexas to root on /dev/pts/1
<118>Oct 15 20:42:07 mech-as28 su: mexas to root on /dev/pts/2
        
db>     

db> show proc
Process 0 (kernel) at 0x9ffc000000efeac8:
 state: NORMAL
 uid: 0  gids: 0
 ABI: null
 threads: 13
100053                   Run     CPU 1                       [deadlkres]
100050                   D       -        0xe000000011795300 [system_taskq_1]
100049                   D       -        0xe000000011795300 [system_taskq_0]
100045                   D       -        0xe0000000111a2500 [em1 taskq]
100044                   D       -        0xe0000000111a2600 [em0 taskq]
100023                   D       -        0xe0000000111a5300 [ffs_trim taskq]
100020                   RunQ                                [thread taskq]
100017                   D       -        0xe0000000111a5900 [acpi_task_2]
100016                   D       -        0xe0000000111a5900 [acpi_task_1]
100015                   D       -        0xe0000000111a5900 [acpi_task_0]
100014                   D       -        0xe0000000111a5a00 [kqueue taskq]
100009                   D       -        0xe0000000111a5d00 [firmware taskq]
100000                   D       swapin   0x9ffc000000efeac8 [swapper]
db> 

db> run lockinfo
db:0:lockinfo> show locks
db:0:locks>  show alllocks
db:0:alllocks>  show lockedvnods
Locked vnodes

0xe0000000120e1ad0: tag devfs, type VDIR
    usecount 3, writecount 0, refcount 4 mountedhere 0
    flags (VV_ROOT|VI_ACTIVE)
    lock type devfs: EXCL by thread 0xe000000015800900 (pid 20716, sh, tid 100129)
 with exclusive waiters pending

0xe0000000120e3860: tag devfs, type VCHR
    usecount 29, writecount 16, refcount 30 mountedhere 0xe0000000111a7800
    flags (VV_ETERNALDEV|VI_ACTIVE)
    lock type devfs: EXCL by thread 0xe0000000157a4d80 (pid 69993, sh, tid 100125)
 with exclusive waiters pending
        dev null
db> 

db> show pcpu
cpuid        = 1
dynamic pcpu = 0x400009f8d2680
curthread    = 0xe00000001178cd80: pid 0 "deadlkres"
curpcb       = 0
fpcurthread  = none
idlethread   = 0xe000000011182480: tid 100004 "idle: cpu1"
MD: vhpt       = 0xe000000011200000
MD: lid        = 0x1000000
MD: clock      = 0x1caaca1b73b6
MD: clock_mode = 2
MD: clock_load = 0
MD: stats      = 0xe000000011158a30
MD: pmap       = 0x9ffc000000eff500
spin locks held:
db> 

db> bt
Tracing pid 0 tid 100053 td 0xe00000001178cd80
db>

db> show threads
  100138 (0xe0000000157e5b00) (stack 0xa0000000002ca000)  ***
  100139 (0xe0000000157e5680) (stack 0xa0000000002d2000)  ***
  100147 (0xe0000001d2c36000) (stack 0xa000000000302000)  ***
  100093 (0xe00000001258b680) (stack 0xa00000009df5e000)  ***
  100120 (0xe00000001574bb00) (stack 0xa00000000023a000)  ***
  100078 (0xe00000001258a000) (stack 0xa00000009defe000)  ***
  100112 (0xe00000001574a000) (stack 0xa00000009dff6000)  ***
  100104 (0xe0000000128ecd80) (stack 0xa00000009dfb6000)  ***
  100151 (0xe0000001d2c37b00) (stack 0xa000000000322000)  ***
  100116 (0xe000000015770000) (stack 0xa00000000021a000)  ***
  100119 (0xe000000015771680) (stack 0xa000000000232000)  ***
  100107 (0xe000000012919b00) (stack 0xa00000009dfce000)  ***
  100117 (0xe0000000157a4000) (stack 0xa000000000222000)  ***
  100126 (0xe0000000157a4900) (stack 0xa00000000026a000)  ***
  100129 (0xe000000015800900) (stack 0xa000000000282000)  ***
  100100 (0xe00000001279bb00) (stack 0xa00000009df96000)  ***
  100125 (0xe0000000157a4d80) (stack 0xa000000000262000)  ***
  100131 (0xe0000000157e4480) (stack 0xa000000000292000)  ***
  100118 (0xe000000015771b00) (stack 0xa00000000022a000)  ***
  100103 (0xe0000000128ed200) (stack 0xa00000009dfae000)  ***
  100056 (0xe000000012168d80) (stack 0xa00000009de4e000)  ***
  100070 (0xe0000000124f8900) (stack 0xa00000009debe000)  ***
  100073 (0xe00000001258a900) (stack 0xa00000009ded6000)  ***
  100098 (0xe0000000128ec480) (stack 0xa00000009df86000)  ***
  100096 (0xe000000012918480) (stack 0xa00000009df76000)  ***
  100099 (0xe0000000128ec000) (stack 0xa00000009df8e000)  ***
  100091 (0xe00000001258b200) (stack 0xa00000009df4e000)  ***
  100065 (0xe000000012168480) (stack 0xa00000009de96000)  ***
  100095 (0xe000000012918900) (stack 0xa00000009df6e000)  ***
  100081 (0xe00000001279a000) (stack 0xa00000009df16000)  ***
  100088 (0xe00000001279ad80) (stack 0xa00000009df36000)  ***
  100068 (0xe0000000124f9200) (stack 0xa00000009deae000)  ***
  100079 (0xe0000000124f9b00) (stack 0xa00000009df06000)  ***
  100076 (0xe0000000123fa900) (stack 0xa00000009deee000)  ***
  100074 (0xe0000000123fb200) (stack 0xa00000009dede000)  ***
  100066 (0xe000000012168000) (stack 0xa00000009de9e000)  ***
  100064 (0xe0000000123fb680) (stack 0xa00000009de8e000)  ***
  100063 (0xe0000000123fbb00) (stack 0xa00000009de86000)  ***
  100062 (0xe000000012169200) (stack 0xa00000009de7e000)  ***
  100061 (0xe000000012169680) (stack 0xa00000009de76000)  ***
  100060 (0xe000000012169b00) (stack 0xa00000009de6e000)  ***
  100059 (0xe0000000123fa000) (stack 0xa00000009de66000)  ***
  100058 (0xe0000000123fa480) (stack 0xa00000009de5e000)  ***
  100057 (0xe000000012168900) (stack 0xa00000009de56000)  ***
  100054 (0xe00000001178c900) (stack 0xa00000009de3e000)  ***
  100052 (0xe00000001178d200) (stack 0xa00000009de2c000)  ***
  100051 (0xe00000001178d680) (stack 0xa00000009de24000)  ***
  100047 (0xe00000001174db00) (stack 0xa00000009de04000)  ***
  100043 (0xe00000001174c000) (stack 0xa00000009ddc6000)  ***
  100041 (0xe00000001174c900) (stack 0xa00000009dd68000)  ***
  100039 (0xe000000011744480) (stack 0xa00000009dd0a000)  ***
  100038 (0xe000000011744900) (stack 0xa00000009dd02000)  ***
  100037 (0xe000000011744d80) (stack 0xa00000009dcfa000)  ***
  100036 (0xe000000011745200) (stack 0xa00000009dcf2000)  ***
  100034 (0xe000000011480d80) (stack 0xa0000000e2fe6000)  ***
  100033 (0xe000000011481200) (stack 0xa0000000e2fde000)  ***
  100032 (0xe000000011481680) (stack 0xa0000000e2fd6000)  ***
  100031 (0xe000000011481b00) (stack 0xa0000000e2fce000)  ***
  100029 (0xe000000011425680) (stack 0xa0000000e2f36000)  ***
  100028 (0xe000000011425b00) (stack 0xa0000000e2f2e000)  ***
  100027 (0xe000000011480000) (stack 0xa0000000e2f26000)  ***
  100026 (0xe000000011480480) (stack 0xa0000000e2f1e000)  ***
  100024 (0xe000000011424000) (stack 0xa0000000e2e8a000)  ***
  100013 (0xe0000000111a1680) (stack 0xa0000000a07f6000)  ***
  100012 (0xe0000000111a1b00) (stack 0xa0000000a07ee000)  ***
  100011 (0xe0000000111b0000) (stack 0xa0000000a07e6000)  ***
  100010 (0xe000000011183680) (stack 0xa0000000a07de000)  ***
  100048 (0xe00000001174d680) (stack 0xa00000009de0c000)  ***
  100046 (0xe00000001178c000) (stack 0xa00000009ddde000)  ***
  100042 (0xe00000001174c480) (stack 0xa00000009dd70000)  ***
  100040 (0xe000000011744000) (stack 0xa00000009dd12000)  ***
  100035 (0xe000000011480900) (stack 0xa00000009dcea000)  ***
  100030 (0xe000000011425200) (stack 0xa0000000e2fc6000)  ***
  100025 (0xe0000000111b1b00) (stack 0xa0000000e2f16000)  ***
  100022 (0xe000000011424900) (stack 0xa0000000e2e7a000)  ***
  100021 (0xe000000011424d80) (stack 0xa0000000e2e72000)  ***
  100019 (0xe0000000111b0900) (stack 0xa0000000e2e62000)  ***
  100018 (0xe0000000111b0d80) (stack 0xa0000000e2e5a000)  ***
  100008 (0xe0000000111a0000) (stack 0xa0000000a07ce000)  ***
  100007 (0xe0000000111a0480) (stack 0xa0000000a07c6000)  ***
  100006 (0xe0000000111a0900) (stack 0xa0000000a07be000)  ***
  100005 (0xe000000011182000) (stack 0xa0000000a07b6000)  ***
  100004 (0xe000000011182480) (stack 0xa0000000a07ae000)  ***
  100003 (0xe000000011182900) (stack 0xa0000000a07a6000)  ***
  100002 (0xe000000011182d80) (stack 0xa0000000a079e000)  ***
  100001 (0xe000000011183200) (stack 0xa0000000a0796000)  ***
  100053 (0xe00000001178cd80) (stack 0xa00000009de34000)  ***
  100050 (0xe00000001174cd80) (stack 0xa00000009de1c000)  ***
  100049 (0xe00000001174d200) (stack 0xa00000009de14000)  ***
  100045 (0xe000000011745680) (stack 0xa00000009ddd6000)  ***
  100044 (0xe000000011745b00) (stack 0xa00000009ddce000)  ***
  100023 (0xe000000011424480) (stack 0xa0000000e2e82000)  ***
  100020 (0xe0000000111b0480) (stack 0xa0000000e2e6a000)  ***
  100017 (0xe0000000111b1200) (stack 0xa0000000e2e52000)  ***
  100016 (0xe0000000111b1680) (stack 0xa0000000e2e4a000)  ***
  100015 (0xe0000000111a0d80) (stack 0xa0000000e2e42000)  ***
  100014 (0xe0000000111a1200) (stack 0xa0000000e2e3a000)  ***
  100009 (0xe000000011183b00) (stack 0xa0000000a07d6000)  ***
  100000 (0x9ffc000000efef70) (stack 0xe0000040fffc0000)  ***
db>


db> ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
37126  1147  1147     0  D+      allproc  0x9ffc000000f0a350 sh
37125 42826  1147     0  DE+     allproc  0x9ffc000000f0a350 sed
37124 42826  1147     0  DE+     allproc  0x9ffc000000f0a350 awk
37123 42826  1147     0  DE+     proctree 0x9ffc000000f0a370 awk
37115 37110  1147     0  S+J     piperd   0xe0000000129509a0 as
37111 37110  1147     0  LL+J   *pmap pv  0xe00000001216b8c0 cc1plus
37110 28849  1147     0  S+J     wait     0xe000000015790940 g++
36890 36889  1147     0  LL+J   *pmap pv  0xe00000001216b8c0 ld
36889 11278  1147     0  SW+J    wait     0xe0000000157664a0 c++
36591 36590  1147     0  LL+J   *pmap pv  0xe00000001216b8c0 cc1plus
36590 14781  1147     0  SW+J    wait     0xe00000001290d280 xg++
28849 61132  1147     0  S+J     wait     0xe000000015767720 gmake
18342 18341 18342  1001  Ds+     allproc  0x9ffc000000f0a350 tcsh
18341 18317 18317  1001  S       select   0xe000000010c286c0 sshd
18317   924 18317     0  Ss      select   0xe00000001273f640 sshd
14781  2824  1147     0  SW+J    wait     0xe00000001572e000 gmake
11278 59577  1147     0  SW+J    wait     0xe000000012762de0 gmake
61132 61124  1147     0  SW+J    wait     0xe0000000157ec4a0 gmake
61124 61123  1147     0  SW+J    wait     0xe000000015791720 gmake
61123 60934  1147     0  SW+J    wait     0xe0000001d4970000 sh
60934 60932  1147     0  SW+J    wait     0xe000000011797280 make
60932 20716  1147     0  SW+     wait     0xe0000001d300cde0 sh
59577 59537  1147     0  SW+J    wait     0xe0000000124a6940 gmake
59537 59516  1147     0  SW+J    wait     0xe000000015791280 gmake
59516 59515  1147     0  SW+J    wait     0xe000000015a1b720 gmake
59515 59505  1147     0  SW+J    wait     0xe00000001572f280 sh
59505 59479  1147     0  SW+J    wait     0xe000000015a1b280 make
59479 59477  1147     0  SW+J    wait     0xe0000000157b9280 make
59477 69993  1147     0  SW+     wait     0xe00000001572e4a0 sh
 2824 77085  1147     0  SW+J    wait     0xe000000015766940 gmake
28012 20716  1147     0  S+      piperd   0xe00000001586a738 tee
20716  1147  1147     0  D+      devfs    0xe0000000120e38c8 sh
77085 77056  1147     0  SW+J    wait     0xe0000000157ecde0 gmake
77056 77055  1147     0  SW+J    wait     0xe000000012762000 gmake
77055 77043  1147     0  SW+J    wait     0xe0000000127ae940 sh
77043 77023  1147     0  SW+J    wait     0xe0000000157b8000 make
77023 77021  1147     0  SW+J    wait     0xe0000000157904a0 make
77021 62311  1147     0  SW+     wait     0xe0000000123f4940 sh
70398 69993  1147     0  S+      piperd   0xe000000012863ce0 tee
69993  1147  1147     0  D+      proctree 0x9ffc000000f0a370 sh
67686 62311  1147     0  S+      piperd   0xe0000001d4ed1340 tee
62311  1147  1147     0  D+      devfs    0xe0000000120e1b38 sh
42826  1147  1147     0  S+      wait     0xe00000001290c940 sh
 1147  1030  1147     0  S+      piperd   0xe00000001586ae70 sh
 1122  1121  1122     0  S+      ttyin    0xe00000001254e4a8 sh
 1121  1047  1121  1001  SW+     wait     0xe00000001289c4a0 su
 1047  1046  1047  1001  SWs+    pause    0xe0000000127ae548 tcsh
 1046  1042  1042  1001  S       select   0xe00000001273fd40 sshd
 1042   924  1042     0  Ss      select   0xe000000010c287c0 sshd
 1030  1029  1030     0  SW+     wait     0xe00000001289d720 sh
 1029  1025  1029  1001  SW+     wait     0xe00000001289c940 su
 1025  1024  1025  1001  SWs+    pause    0xe00000001290c0a8 tcsh
 1024  1022  1022  1001  S       select   0xe000000010c289c0 sshd
 1022   924  1022     0  Ss      select   0xe00000001273f7c0 sshd
  997     1   997     0  Ss+     ttyin    0xe0000000114288a8 getty
  996     1     1     0  S       ttydcd   0xe000000011428ce8 getty
  974     1   974     0  Ss      select   0xe000000010c288c0 inetd
  935     1   935     0  LLs    *pmap pv  0xe00000001216b8c0 cron
  930     1   930    25  SWs     pause    0xe0000000124a7328 sendmail
  927     1   927     0  Rs                                  sendmail
  924     1   924     0  Ss      select   0xe000000010c28e40 sshd
  864     1   864     0  LLs    *pmap pv  0xe00000001216b8c0 ntpd
  784     1   784     0  Rs                                  syslogd
  775     1   775     0  Ss      select   0xe000000010c29040 ipmon
  653     1   653     0  Ss      select   0xe00000001273fe40 devd
  138     1   138     0  SWs     pause    0xe0000000111d77c8 adjkerntz
   23     0     0     0  DL      sdflush  0x9ffc000000ee2d30 [softdepflush]
   22     0     0     0  RL                                  [vnlru]
   21     0     0     0  RL                                  [syncer]
   20     0     0     0  RL                                  [bufdaemon]
   19     0     0     0  DL      pgzero   0x9ffc000000ee30d4 [pagezero]
   18     0     0     0  RL      CPU 0                       [vmdaemon]
   17     0     0     0  LL     *pmap pv  0xe00000001216b8c0 [pagedaemon]
    9     0     0     0  DL      ccb_scan 0x9ffc000000ee33a0 [xpt_thrd]
    6     0     0     0  DL      waiting_ 0x9ffc000001104130 [sctp_iterator]
    5     0     0     0  RL      (threaded)                  [zfskern]
100052                   RunQ                                [l2arc_feed_thread]
100051                   RunQ                                [arc_reclaim_thread]
    4     0     0     0  DL      ispf     0xe000000011768000 [isp0: fc_thrd0]
    3     0     0     0  DL      idle     0xa000000000a16000 [mpt_recovery1]
    2     0     0     0  DL      idle     0xa0000000009fe000 [mpt_recovery0]
   16     0     0     0  DL      (threaded)                  [usb]
100039                   D       -        0xa0000000009fae18 [usbus2]
100038                   D       -        0xa0000000009fadc0 [usbus2]
100037                   D       -        0xa0000000009fad68 [usbus2]
100036                   D       -        0xa0000000009fad10 [usbus2]
100034                   D       -        0xa0000000009f1460 [usbus1]
100033                   D       -        0xa0000000009f1408 [usbus1]
100032                   D       -        0xa0000000009f13b0 [usbus1]
100031                   D       -        0xa0000000009f1358 [usbus1]
100029                   D       -        0xa0000000009ed460 [usbus0]
100028                   D       -        0xa0000000009ed408 [usbus0]
100027                   D       -        0xa0000000009ed3b0 [usbus0]
100026                   D       -        0xa0000000009ed358 [usbus0]
   15     0     0     0  DL      tzpoll   0x9ffc000000ee0a50 [acpi_thermal]
   14     0     0     0  RL                                  [rand_harvestq]
   13     0     0     0  DL      (threaded)                  [geom]
100012                   D       -        0x9ffc000000ee1270 [g_down]
100011                   D       -        0x9ffc000000ee1268 [g_up]
100010                   D       -        0x9ffc000000ee1258 [g_event]
   12     0     0     0  WL      (threaded)                  [intr]
100048                   I                                   [swi0: uart uart+++]
100046                   I                                   [irq71: isp0]
100042                   I                                   [irq28: mpt1]
100040                   I                                   [irq27: mpt0]
100035                   I                                   [irq18: ehci0]
100030                   I                                   [irq17: ohci1]
100025                   I                                   [irq16: ohci0]
100022                   I                                   [swi6: task queue]
100021                   I                                   [swi6: Giant taskq]
100019                   I                                   [swi5: fast taskq]
100018                   I                                   [swi2: cambio]
100008                   I                                   [swi4: clock]
100007                   I                                   [swi4: clock]
100006                   I                                   [swi3: vm]
100005                   I                                   [swi1: netisr 0]
   11     0     0     0  RL      (threaded)                  [idle]
100004                   CanRun                              [idle: cpu1]
100003                   CanRun                              [idle: cpu0]
    1     0     1     0  SLs     wait     0xe000000011180de0 [init]
   10     0     0     0  DL      audit_wo 0x9ffc0000011087c8 [audit]
    0     0     0     0  RLs     (threaded)                  [kernel]
100053                   Run     CPU 1                       [deadlkres]
100050                   D       -        0xe000000011795300 [system_taskq_1]
100049                   D       -        0xe000000011795300 [system_taskq_0]
100045                   D       -        0xe0000000111a2500 [em1 taskq]
100044                   D       -        0xe0000000111a2600 [em0 taskq]
100023                   D       -        0xe0000000111a5300 [ffs_trim taskq]
100020                   RunQ                                [thread taskq]
100017                   D       -        0xe0000000111a5900 [acpi_task_2]
100016                   D       -        0xe0000000111a5900 [acpi_task_1]
100015                   D       -        0xe0000000111a5900 [acpi_task_0]
100014                   D       -        0xe0000000111a5a00 [kqueue taskq]
100009                   D       -        0xe0000000111a5d00 [firmware taskq]
100000                   D       swapin   0x9ffc000000efeac8 [swapper]
db> 

db> alltrace

Tracing command sh pid 37126 tid 100138 td 0xe0000000157e5b00

Tracing command sed pid 37125 tid 100139 td 0xe0000000157e5680

Tracing command awk pid 37124 tid 100147 td 0xe0000001d2c36000

Tracing command awk pid 37123 tid 100093 td 0xe00000001258b680

Tracing command as pid 37115 tid 100120 td 0xe00000001574bb00

Tracing command cc1plus pid 37111 tid 100078 td 0xe00000001258a000

Tracing command g++ pid 37110 tid 100112 td 0xe00000001574a000

Tracing command ld pid 36890 tid 100104 td 0xe0000000128ecd80

Tracing command c++ pid 36889 tid 100115 td 0xe000000012919200

Tracing command cc1plus pid 36591 tid 100151 td 0xe0000001d2c37b00
        
Tracing command xg++ pid 36590 tid 100101 td 0xe0000000128edb00

Tracing command gmake pid 28849 tid 100116 td 0xe000000015770000

Tracing command tcsh pid 18342 tid 100119 td 0xe000000015771680

Tracing command sshd pid 18341 tid 100107 td 0xe000000012919b00

Tracing command sshd pid 18317 tid 100117 td 0xe0000000157a4000

Tracing command gmake pid 14781 tid 100111 td 0xe00000001574a480

Tracing command gmake pid 11278 tid 100077 td 0xe00000001258a480

Tracing command gmake pid 61132 tid 100134 td 0xe000000015801680

Tracing command gmake pid 61124 tid 100122 td 0xe000000015770d80

Tracing command sh pid 61123 tid 100149 td 0xe0000001d492c480
        
Tracing command make pid 60934 tid 100067 td 0xe00000001178db00

Tracing command sh pid 60932 tid 100141 td 0xe000000015801200

Tracing command gmake pid 59577 tid 100071 td 0xe0000000124f8480

Tracing command gmake pid 59537 tid 100123 td 0xe0000000157a5680

Tracing command gmake pid 59516 tid 100145 td 0xe0000001d2c36480

Tracing command sh pid 59515 tid 100105 td 0xe0000000128ec900

Tracing command make pid 59505 tid 100135 td 0xe000000015800480

Tracing command make pid 59479 tid 100127 td 0xe0000000157a4480

Tracing command sh pid 59477 tid 100110 td 0xe00000001574a900

Tracing command gmake pid 2824 tid 100109 td 0xe000000012919680
        
Tracing command tee pid 28012 tid 100126 td 0xe0000000157a4900

Tracing command sh pid 20716 tid 100129 td 0xe000000015800900

Tracing command gmake pid 77085 tid 100132 td 0xe0000000157e4000

Tracing command gmake pid 77056 tid 100080 td 0xe0000000124f9680

Tracing command sh pid 77055 tid 100085 td 0xe00000001258ad80

Tracing command make pid 77043 tid 100121 td 0xe000000015771200

Tracing command make pid 77023 tid 100113 td 0xe000000015770900

Tracing command sh pid 77021 tid 100075 td 0xe0000000123fad80

Tracing command tee pid 70398 tid 100100 td 0xe00000001279bb00

Tracing command sh pid 69993 tid 100125 td 0xe0000000157a4d80
        
Tracing command tee pid 67686 tid 100131 td 0xe0000000157e4480

Tracing command sh pid 62311 tid 100118 td 0xe000000015771b00

Tracing command sh pid 42826 tid 100103 td 0xe0000000128ed200

Tracing command sh pid 1147 tid 100056 td 0xe000000012168d80

Tracing command sh pid 1122 tid 100070 td 0xe0000000124f8900

Tracing command su pid 1121 tid 100090 td 0xe00000001258bb00

Tracing command tcsh pid 1047 tid 100086 td 0xe00000001279b680

Tracing command sshd pid 1046 tid 100073 td 0xe00000001258a900

Tracing command sshd pid 1042 tid 100098 td 0xe0000000128ec480

Tracing command sh pid 1030 tid 100094 td 0xe000000012918d80
        
Tracing command su pid 1029 tid 100097 td 0xe000000012918000
        
Tracing command tcsh pid 1025 tid 100092 td 0xe00000001279a480
        
Tracing command sshd pid 1024 tid 100096 td 0xe000000012918480

Tracing command sshd pid 1022 tid 100099 td 0xe0000000128ec000

Tracing command getty pid 997 tid 100091 td 0xe00000001258b200

Tracing command getty pid 996 tid 100065 td 0xe000000012168480

Tracing command inetd pid 974 tid 100095 td 0xe000000012918900

Tracing command cron pid 935 tid 100081 td 0xe00000001279a000

Tracing command sendmail pid 930 tid 100069 td 0xe0000000124f8d80

Tracing command sendmail pid 927 tid 100088 td 0xe00000001279ad80

Tracing command sshd pid 924 tid 100068 td 0xe0000000124f9200

Tracing command ntpd pid 864 tid 100079 td 0xe0000000124f9b00

Tracing command syslogd pid 784 tid 100076 td 0xe0000000123fa900

Tracing command ipmon pid 775 tid 100074 td 0xe0000000123fb200

Tracing command devd pid 653 tid 100066 td 0xe000000012168000

Tracing command adjkerntz pid 138 tid 100055 td 0xe00000001178c480

Tracing command softdepflush pid 23 tid 100064 td 0xe0000000123fb680

Tracing command vnlru pid 22 tid 100063 td 0xe0000000123fbb00

Tracing command syncer pid 21 tid 100062 td 0xe000000012169200

Tracing command bufdaemon pid 20 tid 100061 td 0xe000000012169680

Tracing command pagezero pid 19 tid 100060 td 0xe000000012169b00

Tracing command vmdaemon pid 18 tid 100059 td 0xe0000000123fa000

Tracing command pagedaemon pid 17 tid 100058 td 0xe0000000123fa480

Tracing command xpt_thrd pid 9 tid 100057 td 0xe000000012168900

Tracing command sctp_iterator pid 6 tid 100054 td 0xe00000001178c900

Tracing command zfskern pid 5 tid 100052 td 0xe00000001178d200

Tracing command zfskern pid 5 tid 100051 td 0xe00000001178d680

Tracing command isp0: fc_thrd0 pid 4 tid 100047 td 0xe00000001174db00

Tracing command mpt_recovery1 pid 3 tid 100043 td 0xe00000001174c000

Tracing command mpt_recovery0 pid 2 tid 100041 td 0xe00000001174c900

Tracing command usb pid 16 tid 100039 td 0xe000000011744480

Tracing command usb pid 16 tid 100038 td 0xe000000011744900

Tracing command usb pid 16 tid 100037 td 0xe000000011744d80

Tracing command usb pid 16 tid 100036 td 0xe000000011745200

Tracing command usb pid 16 tid 100034 td 0xe000000011480d80

Tracing command usb pid 16 tid 100033 td 0xe000000011481200

Tracing command usb pid 16 tid 100032 td 0xe000000011481680

Tracing command usb pid 16 tid 100031 td 0xe000000011481b00

Tracing command usb pid 16 tid 100029 td 0xe000000011425680

Tracing command usb pid 16 tid 100028 td 0xe000000011425b00

Tracing command usb pid 16 tid 100027 td 0xe000000011480000

Tracing command usb pid 16 tid 100026 td 0xe000000011480480

Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000011424000

Tracing command rand_harvestq pid 14 tid 100013 td 0xe0000000111a1680

Tracing command geom pid 13 tid 100012 td 0xe0000000111a1b00

Tracing command geom pid 13 tid 100011 td 0xe0000000111b0000

Tracing command geom pid 13 tid 100010 td 0xe000000011183680

Tracing command intr pid 12 tid 100048 td 0xe00000001174d680

Tracing command intr pid 12 tid 100046 td 0xe00000001178c000

Tracing command intr pid 12 tid 100042 td 0xe00000001174c480

Tracing command intr pid 12 tid 100040 td 0xe000000011744000

Tracing command intr pid 12 tid 100035 td 0xe000000011480900


Tracing command intr pid 12 tid 100030 td 0xe000000011425200

Tracing command intr pid 12 tid 100025 td 0xe0000000111b1b00

Tracing command intr pid 12 tid 100022 td 0xe000000011424900

Tracing command intr pid 12 tid 100021 td 0xe000000011424d80

Tracing command intr pid 12 tid 100019 td 0xe0000000111b0900

Tracing command intr pid 12 tid 100018 td 0xe0000000111b0d80

Tracing command intr pid 12 tid 100008 td 0xe0000000111a0000

Tracing command intr pid 12 tid 100007 td 0xe0000000111a0480

Tracing command intr pid 12 tid 100006 td 0xe0000000111a0900

Tracing command intr pid 12 tid 100005 td 0xe000000011182000

Tracing command idle pid 11 tid 100004 td 0xe000000011182480

Tracing command idle pid 11 tid 100003 td 0xe000000011182900

Tracing command init pid 1 tid 100002 td 0xe000000011182d80

Tracing command audit pid 10 tid 100001 td 0xe000000011183200

Tracing command kernel pid 0 tid 100053 td 0xe00000001178cd80

Tracing command kernel pid 0 tid 100050 td 0xe00000001174cd80

Tracing command kernel pid 0 tid 100049 td 0xe00000001174d200

Tracing command kernel pid 0 tid 100045 td 0xe000000011745680

Tracing command kernel pid 0 tid 100044 td 0xe000000011745b00

Tracing command kernel pid 0 tid 100023 td 0xe000000011424480

Tracing command kernel pid 0 tid 100020 td 0xe0000000111b0480

Tracing command kernel pid 0 tid 100017 td 0xe0000000111b1200

Tracing command kernel pid 0 tid 100016 td 0xe0000000111b1680

Tracing command kernel pid 0 tid 100015 td 0xe0000000111a0d80

Tracing command kernel pid 0 tid 100014 td 0xe0000000111a1200

Tracing command kernel pid 0 tid 100009 td 0xe000000011183b00

Tracing command kernel pid 0 tid 100000 td 0x9ffc000000efef70
db>  

db> show pginfo 0xe0000000128ecd80  
page 0xe0000000128ecd80 obj 0 pidx 0x9ffc000000f27760 phys 0x0 q 0 hold 0 wire 10074176
  af 0x0 of 0x0 f 0xa000 act 24 busy 1273e880 valid 0x1240 dirty 0x0
db> 

db> textdump dump
db> call doadump
= 0x6
db> 

	
>How-To-Repeat:
	
>Fix:

	


>Release-Note:
>Audit-Trail:

From: Anton Shterenlikht <mexas@bris.ac.uk>
To: bug-followup@freebsd.org
Cc:  
Subject: RE: kern/183007: ia64: panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks
Date: Wed, 23 Oct 2013 10:12:33 +0100 (BST)

 Another one.
 
 deadlkres: possible deadlock detected for 0xe000000012aed200, blocked for 900014 ticks
 
 FreeBSD 10.0-CURRENT #12 r255488M: Tue Oct 22 15:42:23 BST 2013
     root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI
 
 db:0:kdb.enter.panic>  run lockinfo
 db:1:lockinfo> show locks
 db:1:locks>  show alllocks
 Process 56111 (cc1) thread 0xe000000012aed200 (100381)
 shared sx vm map (user) (vm map (user)) r = 0 (0xe0000001149eee48) locked @ /usr/src/sys/vm/vm_map.c:3834
 Process 56101 (cc1) thread 0xe000000114cdfb00 (100412)
 shared sx vm map (user) (vm map (user)) r = 0 (0xe0000001149efa88) locked @ /usr/src/sys/vm/vm_map.c:3834
 Process 55822 (find) thread 0xe000000114cded80 (100407)
 shared sx vm map (user) (vm map (user)) r = 0 (0xe00000001268da88) locked @ /usr/src/sys/vm/vm_map.c:3834
 Process 53016 (rsync) thread 0xe000000114c47b00 (100416)
 exclusive rw vm object (vm object) r = 0 (0xe00000011ddc5a00) locked @ /usr/src/sys/kern/vfs_bio.c:1850
 exclusive lockmgr bufwait (bufwait) r = 0 (0xa000000060432c08) locked @ /usr/src/sys/kern/vfs_bio.c:2305
 exclusive lockmgr bufwait (bufwait) r = 0 (0xa0000000604050c8) locked @ /usr/src/sys/kern/vfs_bio.c:3059
 shared lockmgr ufs (ufs) r = 0 (0xe00000011e0e4a28) locked @ /usr/src/sys/kern/vfs_vnops.c:699
 Process 23692 (top) thread 0xe000000013a33680 (100409)
 shared sx vm map (user) (vm map (user)) r = 0 (0xe0000000126b1a88) locked @ /usr/src/sys/vm/vm_map.c:2053
 Process 851 (ntpd) thread 0xe000000012335680 (100065)
 shared sx vm map (user) (vm map (user)) r = 0 (0xe000000010b9f900) locked @ /usr/src/sys/vm/vm_map.c:3834
 Process 19 (syncer) thread 0xe0000000122acd80 (100056)
 exclusive lockmgr bufwait (bufwait) r = 0 (0xa000000060432e78) locked @ /usr/src/sys/kern/vfs_bio.c:2305
 exclusive lockmgr bufwait (bufwait) r = 0 (0xa00000005f673c38) locked @ /usr/src/sys/kern/vfs_default.c:660
 exclusive lockmgr devfs (devfs) r = 0 (0xe0000000121edb38) locked @ /usr/src/sys/kern/vfs_subr.c:1718
 Process 9 (vmdaemon) thread 0xe000000011c48d80 (100053)
 exclusive sleep mutex pmap (pmap) r = 0 (0xe000000114f4cf00) locked @ /usr/src/sys/ia64/ia64/pmap.c:1545
 exclusive rw pmap pv global (pmap pv global) r = 0 (0x9ffc000000e27d80) locked @ /usr/src/sys/ia64/ia64/pmap.c:1544
 exclusive sx vm map (user) (vm map (user)) r = 0 (0xe000000114f4ce48) locked @ /usr/src/sys/vm/vm_pageout.c:822
 shared sx allproc (allproc) r = 0 (0x9ffc000000c21760) locked @ /usr/src/sys/vm/vm_pageout.c:1763
 Process 8 (pagedaemon) thread 0xe000000011c49200 (100052)
 exclusive rw vm object (vm object) r = 0 (0xe00000001e414300) locked @ /usr/src/sys/vm/vm_pageout.c:997
 exclusive sleep mutex vm page (vm page) r = 0 (0x9ffc000000e21b80) locked @ /usr/src/sys/vm/vm_pageout.c:305
 Process 0 (kernel) thread 0xe000000012414900 (100221)
 exclusive rw kmem vm object (kmem vm object) r = 0 (0x9ffc000000e21440) locked @ /usr/src/sys/vm/vm_kern.c:344
 Process 0 (kernel) thread 0xe000000010bbc900 (100019)
 exclusive sleep mutex kmem arena (kmem arena) r = 0 (0x9ffc000000c78180) locked @ /usr/src/sys/kern/subr_vmem.c:728
 exclusive sleep mutex vmem list lock (vmem list lock) r = 0 (0x9ffc000000c77a80) locked @ /usr/src/sys/kern/subr_vmem.c:723
 Process 0 (kernel) thread 0x9ffc000000c16380 (100000)
 exclusive rw vm object (vm object) r = 0 (0xe000000012352500) locked @ /usr/src/sys/vm/vm_glue.c:528
 db:1:alllocks>  show lockedvnods
 Locked vnodes
 db:0:kdb.enter.panic>  show pcpu
 cpuid        = 1
 dynamic pcpu = 0x400005fc2ad00
 curthread    = 0xe000000011973b00: pid 0 "deadlkres"
 curpcb       = 0
 fpcurthread  = none
 idlethread   = 0xe000000010b92480: tid 100004 "idle: cpu1"
 MD: vhpt       = 0xe000004040600000
 MD: lid        = 0x1000000
 MD: clock      = 0x361e86823ab1
 MD: clock_mode = 2
 MD: clock_load = 0
 MD: stats      = 0xe000000010b6ea30
 MD: pmap       = 0x9ffc000000c16910
 spin locks held:
 db:0:kdb.enter.panic>  bt
 Tracing pid 0 tid 100047 td 0xe000000011973b00
 kdb_enter(0x9ffc000000af8d60, 0x9ffc000000af8d60, 0x9ffc0000004573f0, 0x40c) at kdb_enter+0x92
 vpanic(0x9ffc000000aee760, 0xa00000008fdd1518) at vpanic+0x2b0
 panic(0x9ffc000000aee760, 0x9ffc000000aee9f8, 0xe000000012aed200, 0xdbbae) at panic+0x80
 deadlkres(0xdbbae, 0xe000000012aed200, 0x9ffc000000af13d0, 0x9ffc000000aee6f0) at deadlkres+0x420
 fork_exit(0x9ffc000000b456a0, 0x0, 0xa00000008fdd1550) at fork_exit+0x120
 enter_userland() at enter_userland
 db:0:kdb.enter.panic>  ps
   pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
 56112 56110 56109     0  S+      piperd   0xe000000012a74268 as
 56111 56110 56109     0  LL+    *pmap pv  0xe0000000122ae900 cc1
 56110 56109 56109     0  S+      wait     0xe00000011c5b3720 cc
 56109 56108 56109     0  S+      wait     0xe00000011e1e1280 sh
 56108 50069 50069     0  R+                                  make
 56102 56100 56099     0  S+      piperd   0xe000000114f670d8 as
 56101 56100 56099     0  LL+    *pmap pv  0xe0000000122ae900 cc1
 56100 56099 56099     0  SW+     wait     0xe00000011e276de0 cc
 56099 56082 56099     0  SW+     wait     0xe000000012875720 sh
 56082 56081 56081     0  R+                                  make
 56081 56080 56081     0  SW+     wait     0xe000000012aea000 sh
 56080 55313 55313     0  R+                                  make
 55825 55823 55597     0  S       piperd   0xe000000012374000 cat
 55823 55818 55597     0  SW      wait     0xe00000011e1e0de0 sh
 55822 55818 55597     0  LL     *pmap pv  0xe0000000122ae900 find
 55818 55816 55597     0  SW      wait     0xe000000012b0a4a0 sh
 55817 55809 55597     0  S       piperd   0xe000000012a75810 mail
 55816 55809 55597     0  SW      wait     0xe000000013aca000 sh
 55809 55808 55597     0  SW      wait     0xe00000011e276000 sh
 55808 55806 55597     0  SW      wait     0xe00000011e2764a0 lockf
 55806 55805 55597     0  SW      wait     0xe000000012b0ade0 sh
 55805 55607 55597     0  SW      wait     0xe00000011e1e0940 sh
 55608 55600 55597     0  S       piperd   0xe000000114f66738 mail
 55607 55600 55597     0  SW      wait     0xe000000114c6c4a0 sh
 55600 55599 55597     0  SW      wait     0xe0000001149fd280 sh
 55599 55597 55597     0  SW      wait     0xe000000114c6c000 lockf
 55597 55595 55597     0  SWs     wait     0xe000000013acb720 sh
 55595   920   920     0  S       piperd   0xe000000114f66c08 cron
 55313 55312 55313     0  SW+     wait     0xe000000114df9720 sh
 55312 50963 50963     0  R+                                  make
 53021 53016 53013  1001  S       select   0xe00000002e6544c0 rsync
 53016 53013 53013  1001  L      *vm page  0xe0000000122af200 rsync
 53013 53012 53013  1001  SWs     pause    0xe000000114df89e8 tcsh
 53012 53002 53002  1001  S       select   0xe00000002ffa19c0 sshd
 53002   910 53002     0  Ss      select   0xe00000002ffa0a40 sshd
 50963 50962 50963     0  SW+     wait     0xe000000114c6c940 sh
 50962 50961 50961     0  R+                                  make
 50961  1006 50961     0  SW+     wait     0xe00000011c5b3280 sh
 50069 50068 50069     0  S+      wait     0xe0000001149fd720 sh
 50068 50066 50066     0  R+                                  make
 50066  1006 50066     0  SW+     wait     0xe000000012aea4a0 sh
  1006  1005  1005     0  R+                                  make
  1005  6794  1005     0  SW+     wait     0xe000000114cbf720 sh
 23692  4715 23692     0  L+     *vm objec 0xe000000012338480 top
  6794  6793  6793     0  R+                                  make
  6793  6788  6793     0  SW+     wait     0xe000000013acade0 sh
  6788  4634  4634     0  R+                                  make
  4715  4714  4715     0  SW+     wait     0xe000000012b0a000 sh
  4714  4506  4714  1001  SW+     wait     0xe000000012b0a940 su
  4634  4404  4634     0  SW+     wait     0xe0000000123d04a0 sh
  4506  4505  4506  1001  SWs+    pause    0xe000000012aeb328 tcsh
  4505  4503  4503  1001  S       select   0xe0000000122ecf40 sshd
  4503   910  4503     0  Ss      select   0xe0000000108317c0 sshd
  4432  4431  4431    80  SW      kqread   0xe00000001227db00 nginx
  4431     1  4431     0  SWs     pause    0xe000000012aea9e8 nginx
  4404  4403  4404     0  SW+     wait     0xe0000000122b7280 sh
  4403  4398  4403  1001  SW+     wait     0xe000000012a4f280 su
  4398  4397  4398  1001  SWs+    pause    0xe000000012aeb7c8 tcsh
  4397  4395  4395  1001  S       select   0xe000000010831340 sshd
  4395   910  4395     0  Ss      select   0xe0000000108310c0 sshd
  2622   979  2622     0  S+      ttyin    0xe000000010c348a8 sh
   988   987   988     0  S+      ttyin    0xe0000000126564a8 sh
   987   983   987  1001  SW+     wait     0xe0000000123d1720 su
   983   982   983  1001  SWs+    pause    0xe0000000122b77c8 tcsh
   982   980   980  1001  S       select   0xe0000000122ed3c0 sshd
   980   910   980     0  Ss      select   0xe0000000108315c0 sshd
   979     1   979     0  SWs+    wait     0xe0000000122284a0 login
   978     1     1     0  S       ttydcd   0xe000000010c34ce8 getty
   958     1   958     0  Ss      select   0xe0000000108319c0 inetd
   920     1   920     0  ?WLs                                cron
   916     1   916    25  SWs     pause    0xe0000000122a37c8 sendmail
   913     1   913     0  Rs                                  sendmail
   910     1   910     0  Ss      select   0xe0000000122ed340 sshd
   851     1   851     0  LLs    *pmap pv  0xe0000000122ae900 ntpd
   773     1   773     0  Rs                                  syslogd
   635     1   635     0  Ss      select   0xe000000010831bc0 devd
   134     1   134     0  SWs     pause    0xe0000000123d09e8 adjkerntz
    40     0     0     0  RL      (threaded)                  [zfskern]
 100349                   D       zio->io_ 0xe00000001245c6e8 [txg_thread_enter]
 100348                   D       tx->tx_q 0xe0000000123c8238 [txg_thread_enter]
 100346                   RunQ                                [trim tank]
 100068                   RunQ                                [l2arc_feed_thread]
 100064                   D       arc_recl 0xa0000000016c20a0 [arc_reclaim_thread]
    21     0     0     0  DL      sdflush  0x9ffc000000bfc808 [softdepflush]
    20     0     0     0  DL      vlruwt   0xe000000012228de0 [vnlru]
    19     0     0     0  LL     *vm objec 0xe000000114c48cc0 [syncer]
    18     0     0     0  DL      psleep   0x9ffc000000bfbba0 [bufdaemon]
    17     0     0     0  DL      pgzero   0x9ffc000000bfcbac [pagezero]
     9     0     0     0  RL      CPU 0                       [vmdaemon]
     8     0     0     0  LL     *pmap pv  0xe0000000122ae900 [pagedaemon]
     7     0     0     0  DL      ccb_scan 0x9ffc000000bfcea0 [xpt_thrd]
     4     0     0     0  DL      waiting_ 0x9ffc000000e1b530 [sctp_iterator]
     3     0     0     0  DL      idle     0xa000000000a12000 [mpt_recovery1]
     2     0     0     0  DL      idle     0xa0000000009fa000 [mpt_recovery0]
    16     0     0     0  DL      (threaded)                  [usb]
 100039                   D       -        0xa0000000009f6e18 [usbus2]
 100038                   D       -        0xa0000000009f6dc0 [usbus2]
 100037                   D       -        0xa0000000009f6d68 [usbus2]
 100036                   D       -        0xa0000000009f6d10 [usbus2]
 100034                   D       -        0xa0000000009ed460 [usbus1]
 100033                   D       -        0xa0000000009ed408 [usbus1]
 100032                   D       -        0xa0000000009ed3b0 [usbus1]
 100031                   D       -        0xa0000000009ed358 [usbus1]
 100029                   D       -        0xa0000000009e9460 [usbus0]
 100028                   D       -        0xa0000000009e9408 [usbus0]
 100027                   D       -        0xa0000000009e93b0 [usbus0]
 100026                   D       -        0xa0000000009e9358 [usbus0]
    15     0     0     0  DL      tzpoll   0x9ffc000000bfa5c8 [acpi_thermal]
    14     0     0     0  RL                                  [rand_harvestq]
    13     0     0     0  DL      (threaded)                  [geom]
 100011                   D       -        0x9ffc000000bfad68 [g_down]
 100010                   D       -        0x9ffc000000bfad60 [g_up]
 100009                   D       -        0x9ffc000000bfad50 [g_event]
    12     0     0     0  WL      (threaded)                  [intr]
 100046                   I                                   [swi0: uart uart++++]
 100045                   I                                   [irq29: bge0]
 100043                   I                                   [irq28: mpt1]
 100041                   I                                   [irq27: mpt0]
 100040                   I                                   [irq20: fxp0]
 100035                   I                                   [irq18: ehci0]
 100030                   I                                   [irq17: ohci1]
 100025                   I                                   [irq16: ohci0]
 100021                   I                                   [swi6: task queue]
 100020                   I                                   [swi6: Giant taskq]
 100018                   I                                   [swi5: fast taskq]
 100017                   I                                   [swi2: cambio]
 100008                   I                                   [swi4: clock]
 100007                   I                                   [swi4: clock]
 100006                   I                                   [swi3: vm]
 100005                   I                                   [swi1: netisr 0]
    11     0     0     0  RL      (threaded)                  [idle]
 100004                   CanRun                              [idle: cpu1]
 100003                   CanRun                              [idle: cpu0]
     1     0     1     0  SLs     wait     0xe000000010b90de0 [init]
    10     0     0     0  DL      audit_wo 0x9ffc000000e1fbc8 [audit]
     0     0     0     0  RLs     (threaded)                  [kernel]
 100390                   D       -        0xe00000001261b200 [zil_clean]
 100389                   D       -        0xe00000001261d600 [zil_clean]
 100386                   D       -        0xe00000001261c200 [zil_clean]
 100385                   D       -        0xe00000001261b500 [zil_clean]
 100354                   D       -        0xe00000001261d100 [zil_clean]
 100347                   D       -        0xe0000000122fa200 [zfs_vn_rele_taskq]
 100345                   D       -        0xe0000000122f9d00 [zio_ioctl_intr]
 100344                   D       -        0xe0000000122fb000 [zio_ioctl_issue]
 100343                   D       -        0xe0000000122faf00 [zio_claim_intr]
 100342                   D       -        0xe0000000122fae00 [zio_claim_issue]
 100341                   D       -        0xe0000000122fad00 [zio_free_intr]
 100340                   D       -        0xe0000000122fac00 [zio_free_issue_99]
 100339                   D       -        0xe0000000122fac00 [zio_free_issue_98]
 100338                   D       -        0xe0000000122fac00 [zio_free_issue_97]
 100337                   D       -        0xe0000000122fac00 [zio_free_issue_96]
 100336                   D       -        0xe0000000122fac00 [zio_free_issue_95]
 100335                   D       -        0xe0000000122fac00 [zio_free_issue_94]
 100334                   D       -        0xe0000000122fac00 [zio_free_issue_93]
 100333                   D       -        0xe0000000122fac00 [zio_free_issue_92]
 100332                   D       -        0xe0000000122fac00 [zio_free_issue_91]
 100331                   D       -        0xe0000000122fac00 [zio_free_issue_90]
 100330                   D       -        0xe0000000122fac00 [zio_free_issue_89]
 100329                   D       -        0xe0000000122fac00 [zio_free_issue_88]
 100328                   D       -        0xe0000000122fac00 [zio_free_issue_87]
 100327                   D       -        0xe0000000122fac00 [zio_free_issue_86]
 100326                   D       -        0xe0000000122fac00 [zio_free_issue_85]
 100325                   D       -        0xe0000000122fac00 [zio_free_issue_84]
 100324                   D       -        0xe0000000122fac00 [zio_free_issue_83]
 100323                   D       -        0xe0000000122fac00 [zio_free_issue_82]
 100322                   D       -        0xe0000000122fac00 [zio_free_issue_81]
 100321                   D       -        0xe0000000122fac00 [zio_free_issue_80]
 100320                   D       -        0xe0000000122fac00 [zio_free_issue_79]
 100319                   D       -        0xe0000000122fac00 [zio_free_issue_78]
 100318                   D       -        0xe0000000122fac00 [zio_free_issue_77]
 100317                   D       -        0xe0000000122fac00 [zio_free_issue_76]
 100316                   D       -        0xe0000000122fac00 [zio_free_issue_75]
 100315                   D       -        0xe0000000122fac00 [zio_free_issue_74]
 100314                   D       -        0xe0000000122fac00 [zio_free_issue_73]
 100313                   D       -        0xe0000000122fac00 [zio_free_issue_72]
 100312                   D       -        0xe0000000122fac00 [zio_free_issue_71]
 100311                   D       -        0xe0000000122fac00 [zio_free_issue_70]
 100310                   D       -        0xe0000000122fac00 [zio_free_issue_69]
 100309                   D       -        0xe0000000122fac00 [zio_free_issue_68]
 100308                   D       -        0xe0000000122fac00 [zio_free_issue_67]
 100307                   D       -        0xe0000000122fac00 [zio_free_issue_66]
 100306                   D       -        0xe0000000122fac00 [zio_free_issue_65]
 100305                   D       -        0xe0000000122fac00 [zio_free_issue_64]
 100304                   D       -        0xe0000000122fac00 [zio_free_issue_63]
 100303                   D       -        0xe0000000122fac00 [zio_free_issue_62]
 100302                   D       -        0xe0000000122fac00 [zio_free_issue_61]
 100301                   D       -        0xe0000000122fac00 [zio_free_issue_60]
 100300                   D       -        0xe0000000122fac00 [zio_free_issue_59]
 100299                   D       -        0xe0000000122fac00 [zio_free_issue_58]
 100298                   D       -        0xe0000000122fac00 [zio_free_issue_57]
 100297                   D       -        0xe0000000122fac00 [zio_free_issue_56]
 100296                   D       -        0xe0000000122fac00 [zio_free_issue_55]
 100295                   D       -        0xe0000000122fac00 [zio_free_issue_54]
 100294                   D       -        0xe0000000122fac00 [zio_free_issue_53]
 100293                   D       -        0xe0000000122fac00 [zio_free_issue_52]
 100292                   D       -        0xe0000000122fac00 [zio_free_issue_51]
 100291                   D       -        0xe0000000122fac00 [zio_free_issue_50]
 100290                   D       -        0xe0000000122fac00 [zio_free_issue_49]
 100289                   D       -        0xe0000000122fac00 [zio_free_issue_48]
 100288                   D       -        0xe0000000122fac00 [zio_free_issue_47]
 100287                   D       -        0xe0000000122fac00 [zio_free_issue_46]
 100286                   D       -        0xe0000000122fac00 [zio_free_issue_45]
 100285                   D       -        0xe0000000122fac00 [zio_free_issue_44]
 100284                   D       -        0xe0000000122fac00 [zio_free_issue_43]
 100283                   D       -        0xe0000000122fac00 [zio_free_issue_42]
 100282                   D       -        0xe0000000122fac00 [zio_free_issue_41]
 100281                   D       -        0xe0000000122fac00 [zio_free_issue_40]
 100280                   D       -        0xe0000000122fac00 [zio_free_issue_39]
 100279                   D       -        0xe0000000122fac00 [zio_free_issue_38]
 100278                   D       -        0xe0000000122fac00 [zio_free_issue_37]
 100277                   D       -        0xe0000000122fac00 [zio_free_issue_36]
 100276                   D       -        0xe0000000122fac00 [zio_free_issue_35]
 100275                   D       -        0xe0000000122fac00 [zio_free_issue_34]
 100274                   D       -        0xe0000000122fac00 [zio_free_issue_33]
 100273                   D       -        0xe0000000122fac00 [zio_free_issue_32]
 100272                   D       -        0xe0000000122fac00 [zio_free_issue_31]
 100271                   D       -        0xe0000000122fac00 [zio_free_issue_30]
 100270                   D       -        0xe0000000122fac00 [zio_free_issue_29]
 100269                   D       -        0xe0000000122fac00 [zio_free_issue_28]
 100268                   D       -        0xe0000000122fac00 [zio_free_issue_27]
 100267                   D       -        0xe0000000122fac00 [zio_free_issue_26]
 100266                   D       -        0xe0000000122fac00 [zio_free_issue_25]
 100265                   D       -        0xe0000000122fac00 [zio_free_issue_24]
 100264                   D       -        0xe0000000122fac00 [zio_free_issue_23]
 100263                   D       -        0xe0000000122fac00 [zio_free_issue_22]
 100262                   D       -        0xe0000000122fac00 [zio_free_issue_21]
 100261                   D       -        0xe0000000122fac00 [zio_free_issue_20]
 100260                   D       -        0xe0000000122fac00 [zio_free_issue_19]
 100259                   D       -        0xe0000000122fac00 [zio_free_issue_18]
 100258                   D       -        0xe0000000122fac00 [zio_free_issue_17]
 100257                   D       -        0xe0000000122fac00 [zio_free_issue_16]
 100256                   D       -        0xe0000000122fac00 [zio_free_issue_15]
 100255                   D       -        0xe0000000122fac00 [zio_free_issue_14]
 100254                   D       -        0xe0000000122fac00 [zio_free_issue_13]
 100253                   D       -        0xe0000000122fac00 [zio_free_issue_12]
 100252                   D       -        0xe0000000122fac00 [zio_free_issue_11]
 100251                   D       -        0xe0000000122fac00 [zio_free_issue_10]
 100250                   D       -        0xe0000000122fac00 [zio_free_issue_9]
 100249                   D       -        0xe0000000122fac00 [zio_free_issue_8]
 100248                   D       -        0xe0000000122fac00 [zio_free_issue_7]
 100247                   D       -        0xe0000000122fac00 [zio_free_issue_6]
 100246                   D       -        0xe0000000122fac00 [zio_free_issue_5]
 100245                   D       -        0xe0000000122fac00 [zio_free_issue_4]
 100244                   D       -        0xe0000000122fac00 [zio_free_issue_3]
 100243                   D       -        0xe0000000122fac00 [zio_free_issue_2]
 100242                   D       -        0xe0000000122fac00 [zio_free_issue_1]
 100241                   D       -        0xe0000000122fac00 [zio_free_issue_0]
 100240                   D       -        0xe0000000122fab00 [zio_write_intr_high]
 100239                   D       -        0xe0000000122fab00 [zio_write_intr_high]
 100238                   D       -        0xe0000000122fab00 [zio_write_intr_high]
 100237                   D       -        0xe0000000122fab00 [zio_write_intr_high]
 100236                   D       -        0xe0000000122fab00 [zio_write_intr_high]
 100235                   D       -        0xe0000000122faa00 [zio_write_intr_7]
 100234                   D       -        0xe0000000122faa00 [zio_write_intr_6]
 100233                   D       -        0xe0000000122faa00 [zio_write_intr_5]
 100232                   D       -        0xe0000000122faa00 [zio_write_intr_4]
 100231                   D       -        0xe0000000122faa00 [zio_write_intr_3]
 100230                   D       -        0xe0000000122faa00 [zio_write_intr_2]
 100229                   D       -        0xe0000000122faa00 [zio_write_intr_1]
 100228                   D       -        0xe0000000122faa00 [zio_write_intr_0]
 100227                   D       -        0xe0000000122fa900 [zio_write_issue_hig]
 100226                   D       -        0xe0000000122fa900 [zio_write_issue_hig]
 100225                   D       -        0xe0000000122fa900 [zio_write_issue_hig]
 100224                   D       -        0xe0000000122fa900 [zio_write_issue_hig]
 100223                   D       -        0xe0000000122fa900 [zio_write_issue_hig]
 100222                   L      *kmem vm  0xe000000012444e40 [zio_write_issue_1]
 100221                   L      *pmap pv  0xe0000000122ae900 [zio_write_issue_0]
 100220                   D       -        0xe0000000122fa700 [zio_read_intr_1]
 100219                   D       -        0xe0000000122fa700 [zio_read_intr_0]
 100218                   D       -        0xe0000000122fa600 [zio_read_issue_7]
 100217                   D       -        0xe0000000122fa600 [zio_read_issue_6]
 100216                   D       -        0xe0000000122fa600 [zio_read_issue_5]
 100215                   D       -        0xe0000000122fa600 [zio_read_issue_4]
 100214                   D       -        0xe0000000122fa600 [zio_read_issue_3]
 100213                   D       -        0xe0000000122fa600 [zio_read_issue_2]
 100212                   D       -        0xe0000000122fa600 [zio_read_issue_1]
 100211                   D       -        0xe0000000122fa600 [zio_read_issue_0]
 100210                   D       -        0xe0000000122fa500 [zio_null_intr]
 100209                   D       -        0xe0000000122fa400 [zio_null_issue]
 100067                   D       -        0xe0000000122fb300 [system_taskq_1]
 100066                   D       -        0xe0000000122fb300 [system_taskq_0]
 100047                   Run     CPU 1                       [deadlkres]
 100023                   D       -        0xe000000010bc7300 [kqueue taskq]
 100022                   D       -        0xe000000010bc7400 [ffs_trim taskq]
 100019                   RunQ                                [thread taskq]
 100016                   D       -        0xe000000010bc7a00 [acpi_task_2]
 100015                   D       -        0xe000000010bc7a00 [acpi_task_1]
 100014                   D       -        0xe000000010bc7a00 [acpi_task_0]
 100012                   D       -        0xe000000010bc7d00 [firmware taskq]
 100000                   L      *vm page  0xe0000000122af200 [swapper]
 db:0:kdb.enter.panic>  alltrace
 
 Tracing command as pid 56112 tid 100434 td 0xe00000011e201680
 cpu_switch(0xe00000011e201680, 0xe000000012aed200, 0x9ffc000000c73bb0, 0x9ffc0000004b0d70) at cpu_switch+0xd0
 sched_switch(0xe00000011e201680, 0xe000000012aed200, 0x9ffc000000c3d818, 0x9ffc000000c3d800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe00000011e201680, 0x9ffc0000004e9a60) at mi_switch+0x3f0
 sleepq_switch(0xe000000012a74268, 0x5c, 0xe00000011e2019f8, 0xe00000011e201680) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe000000012a74268, 0x5c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe000000012a74268, 0x5c, 0x9ffc000000b04218, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
 _sleep(0xe000000012a74268, 0xe000000012a744a8, 0x5c, 0x9ffc000000b04218, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe000000114f58c04, 0xa00000009057d360, 0xa00000009057d378, 0x0, 0xe000000012a74370) at pipe_read+0x740
 dofileread(0xe00000011e201680, 0x0, 0xe000000114f58be0, 0xa00000009057d360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe00000011e201680, 0x0, 0xa00000009057d360) at kern_readv+0xa0
 sys_read(0xe00000011e201680, 0xa00000009057d4e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
 syscall(0xe00000011e4164a0, 0x1208e0000, 0x2000, 0xe00000011e201680, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cc1 pid 56111 tid 100381 td 0xe000000012aed200
 cpu_switch(0xe000000012aed200, 0xe000000114c47b00, 0xe0000000122ae900, 0x9ffc0000004b0d70) at cpu_switch+0xd0
 sched_switch(0xe000000012aed200, 0xe000000114c47b00, 0x9ffc000000c3d818, 0x9ffc000000c3d800) at sched_switch+0x890
 mi_switch(0x103, 0x0, 0xe000000012aed200, 0x9ffc0000004f60e0) at mi_switch+0x3f0
 turnstile_wait(0xe0000000122ae900, 0xe000000011c48d80, 0x0, 0x9ffc000000b014b8, 0xe000000012aed200) at turnstile_wait+0x960
 __rw_wlock_hard(0x9ffc000000e27d98, 0xe000000012aed200, 0x0, 0x9ffc000000bf691c, 0x9ffc000000e27d80) at __rw_wlock_hard+0x4c0
 _rw_wlock_cookie(0x9ffc000000e27d80, 0x9ffc000000b3f248, 0x68a, 0x9ffc00000097ebe0, 0x815) at _rw_wlock_cookie+0x150
 pmap_enter(0xe0000001149eef00, 0x121642000, 0x2, 0xe0000040fb4fb270, 0x3, 0x0, 0x9ffc000000b3f248, 0xe00000011dd13518) at pmap_enter+0x40
 vm_fault_hold(0xe0000001149eedc8, 0x121642000, 0x2, 0x0, 0x0) at vm_fault_hold+0x2c70
 vm_fault(0xe0000001149eedc8, 0x121642000, 0x2, 0x0, 0xe000000012aed200, 0x9ffc000000987ce0, 0x716, 0xe000000012b0b380) at vm_fault+0xf0
 trap(0x14, 0xa0000000903d5400) at trap+0x9e0
 ivt_Data_TLB() at ivt_Data_TLB+0x1d0
 
 Tracing command cc pid 56110 tid 100428 td 0xe00000011e200480
 cpu_switch(0xe00000011e200480, 0xe00000011e201680, 0x9ffc000000c738b8, 0x9ffc0000004b0d70) at cpu_switch+0xd0
 sched_switch(0xe00000011e200480, 0xe00000011e201680, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe00000011e200480, 0x9ffc0000004e9a60) at mi_switch+0x3f0
 sleepq_switch(0xe00000011c5b3720, 0x6c, 0xe00000011e2007f8, 0xe00000011e200480) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe00000011c5b3720, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe00000011c5b3720, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
 _sleep(0xe00000011c5b3720, 0xe00000011c5b3820, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
 kern_wait6(0xe00000011e200480, 0x0, 0xdb2f, 0xa00000009054d390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
 kern_wait(0xe00000011e200480, 0xdb2f, 0xa00000009054d390, 0x0, 0x0) at kern_wait+0xe0
 sys_wait4(0xe00000011e200480, 0xa00000009054d4e8, 0xa00000009054d390, 0x9ffc000000986c80) at sys_wait4+0x70
 syscall(0xe00000011c5b3720, 0x1208760e8, 0x0, 0xe00000011e200480, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 56109 tid 100433 td 0xe00000011e201b00
 cpu_switch(0xe00000011e201b00, 0xe000000114cdfb00, 0x9ffc000000c74bf0, 0x9ffc0000004b0d70) at cpu_switch+0xd0
 sched_switch(0xe00000011e201b00, 0xe000000114cdfb00, 0x9ffc000000c3d818, 0x9ffc000000c3d800) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe00000011e201b00, 0x9ffc0000004e9a60) at mi_switch+0x3f0
 sleepq_switch(0xe00000011e1e1280, 0x6c, 0xe00000011e201e78, 0xe00000011e201b00) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe00000011e1e1280, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe00000011e1e1280, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
 _sleep(0xe00000011e1e1280, 0xe00000011e1e1380, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
 kern_wait6(0xe00000011e201b00, 0x7, 0x0, 0xa000000090575390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
 kern_wait(0xe00000011e201b00, 0xffffffff, 0xa000000090575390, 0x0, 0x0) at kern_wait+0xe0
 sys_wait4(0xe00000011e201b00, 0xa0000000905754e8, 0xa000000090575390, 0x9ffc000000986c80) at sys_wait4+0x70
 syscall(0xe00000011e1e1280, 0x7fffffffffffe2c0, 0x0, 0xe00000011e201b00, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command make pid 56108 tid 100415 td 0xe000000114cde000
 cpu_switch(0xe000000114cde000, 0xe000000010bbc900, 0x9ffc000000c73ca0, 0x9ffc0000004b0d70) at cpu_switch+0xd0
 sched_switch(0xe000000114cde000, 0xe000000010bbc900, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000114cde000, 0x9ffc0000004e9a60) at mi_switch+0x3f0
 sleepq_switch(0xe00000001082f0c0, 0x0, 0xe000000114cde378, 0xe000000114cde000) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe00000001082f0c0, 0x0, 0x9ffc000000b00730, 0xe00000001082f0a0) at sleepq_catch_signals+0x190
 sleepq_timedwait_sig(0xe00000001082f0c0, 0x0, 0x9ffc0000003bf0b0, 0x9ffc0000003bf0f0, 0x713) at sleepq_timedwait_sig+0x30
 _cv_timedwait_sig_sbt(0xe00000001082f0c0, 0xe00000001082f0a0, 0x9a79df2e176f, 0x4fffffa3, 0x200) at _cv_timedwait_sig_sbt+0x4f0
 seltdwait(0xe00000001082f080, 0x9a79df2e176f, 0x4fffffa3, 0xe00000001082f0b8) at seltdwait+0xf0
 sys_poll(0xe000000114cde000, 0xa0000000904e54e8, 0xe000000113560200, 0xa0000000904e52a0) at sys_poll+0x620
 syscall(0xe000000114df9280, 0x3, 0x1388, 0xe000000114cde000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command as pid 56102 tid 100435 td 0xe00000011e201200
 cpu_switch(0xe00000011e201200, 0xe000000114cdfb00, 0x9ffc000000c74f60, 0x9ffc0000004b0d70) at cpu_switch+0xd0
 sched_switch(0xe00000011e201200, 0xe000000114cdfb00, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe00000011e201200, 0x9ffc0000004e9a60) at mi_switch+0x3f0
 sleepq_switch(0xe000000114f670d8, 0x5c, 0xe00000011e201578, 0xe00000011e201200) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe000000114f670d8, 0x5c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe000000114f670d8, 0x5c, 0x9ffc000000b04218, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
 _sleep(0xe000000114f670d8, 0xe000000114f67318, 0x5c, 0x9ffc000000b04218, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe0000000139c6bb4, 0xa000000090585360, 0xa000000090585378, 0x0, 0xe000000114f671e0) at pipe_read+0x740
 dofileread(0xe00000011e201200, 0x0, 0xe0000000139c6b90, 0xa000000090585360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe00000011e201200, 0x0, 0xa000000090585360) at kern_readv+0xa0
 sys_read(0xe00000011e201200, 0xa0000000905854e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
 syscall(0xe00000011e416000, 0x1208e0000, 0x2000, 0xe00000011e201200, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cc1 pid 56101 tid 100412 td 0xe000000114cdfb00
 cpu_switch(0xe000000114cdfb00, 0xe000000010b92480, 0xe0000000122ae900, 0x9ffc0000004b0d70) at cpu_switch+0xd0
 sched_switch(0xe000000114cdfb00, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
 mi_switch(0x103, 0x0, 0xe000000114cdfb00, 0x9ffc0000004f60e0) at mi_switch+0x3f0
 turnstile_wait(0xe0000000122ae900, 0xe000000011c48d80, 0x0, 0x9ffc000000b014b8, 0xe000000114cdfb00) at turnstile_wait+0x960
 __rw_wlock_hard(0x9ffc000000e27d98, 0xe000000114cdfb00, 0x0, 0x9ffc000000bf691c, 0x9ffc000000e27d80) at __rw_wlock_hard+0x4c0
 _rw_wlock_cookie(0x9ffc000000e27d80, 0x9ffc000000b3f248, 0x68a, 0x9ffc00000097ebe0, 0x815) at _rw_wlock_cookie+0x150
 pmap_enter(0xe0000001149efb40, 0x12128a000, 0x2, 0xe0000040fb51baa0, 0x3, 0x0, 0x9ffc000000b3f248, 0xe00000011be6dd18) at pmap_enter+0x40
 vm_fault_hold(0xe0000001149efa08, 0x12128a000, 0x2, 0x0, 0x0) at vm_fault_hold+0x2c70
 vm_fault(0xe0000001149efa08, 0x12128a000, 0x2, 0x0, 0xe000000114cdfb00, 0x9ffc000000987ce0, 0x716, 0xe000000114cbeee0) at vm_fault+0xf0
 trap(0x14, 0xa0000000904cd400) at trap+0x9e0
 ivt_Data_TLB() at ivt_Data_TLB+0x1d0
 
 Tracing command cc pid 56100 tid 100438 td 0xe00000011e29f200
 
 Tracing command sh pid 56099 tid 100368 td 0xe00000001267ad80
 
 Tracing command make pid 56082 tid 100437 td 0xe00000011e29f680
 
 Tracing command sh pid 56081 tid 100371 td 0xe000000012aac480
 
 Tracing command make pid 56080 tid 100391 td 0xe000000013ada480
 
 Tracing command cat pid 55825 tid 100411 td 0xe000000114db0000
 
 Tracing command sh pid 55823 tid 100424 td 0xe000000114db1200
 
 Tracing command find pid 55822 tid 100407 td 0xe000000114cded80
 
 Tracing command sh pid 55818 tid 100384 td 0xe000000013a33200
 
 Tracing command mail pid 55817 tid 100432 td 0xe00000011e29e000
 
 Tracing command sh pid 55816 tid 100393 td 0xe000000013a33b00
 
 Tracing command sh pid 55809 tid 100431 td 0xe00000011c59b680
 
 Tracing command lockf pid 55808 tid 100430 td 0xe00000011c59bb00
 
 Tracing command sh pid 55806 tid 100382 td 0xe000000012aecd80
 
 Tracing command sh pid 55805 tid 100425 td 0xe000000114db0d80
 
 Tracing command mail pid 55608 tid 100423 td 0xe00000011c59a480
 
 Tracing command sh pid 55607 tid 100400 td 0xe000000013adb200
 
 Tracing command sh pid 55600 tid 100403 td 0xe000000114c47200
 
 Tracing command lockf pid 55599 tid 100401 td 0xe000000013adad80
 
 Tracing command sh pid 55597 tid 100398 td 0xe000000013adbb00
 
 Tracing command cron pid 55595 tid 100418 td 0xe000000114db1680
 
 Tracing command sh pid 55313 tid 100414 td 0xe000000114cde480
 
 Tracing command make pid 55312 tid 100413 td 0xe000000114cdf680
 
 Tracing command rsync pid 53021 tid 100387 td 0xe00000001286c900
 
 Tracing command rsync pid 53016 tid 100416 td 0xe000000114c47b00
 
 Tracing command tcsh pid 53013 tid 100417 td 0xe000000114db1b00
 
 Tracing command sshd pid 53012 tid 100408 td 0xe000000114cde900
 
 Tracing command sshd pid 53002 tid 100392 td 0xe000000013ada000
 
 Tracing command sh pid 50963 tid 100399 td 0xe000000013adb680
 
 Tracing command make pid 50962 tid 100396 td 0xe000000013a32000
 
 Tracing command sh pid 50961 tid 100419 td 0xe000000114cdf200
 
 Tracing command sh pid 50069 tid 100402 td 0xe000000114c47680
 
 Tracing command make pid 50068 tid 100405 td 0xe000000114c46900
 
 Tracing command sh pid 50066 tid 100370 td 0xe000000012aac900
 
 Tracing command make pid 1006 tid 100395 td 0xe000000013a32480
 
 Tracing command sh pid 1005 tid 100410 td 0xe000000114db0480
 
 Tracing command top pid 23692 tid 100409 td 0xe000000013a33680
 
 Tracing command make pid 6794 tid 100372 td 0xe000000012aac000
 
 Tracing command sh pid 6793 tid 100388 td 0xe00000001286c480
 
 Tracing command make pid 6788 tid 100380 td 0xe000000012aed680
 
 Tracing command sh pid 4715 tid 100375 td 0xe000000012aec000
 
 Tracing command su pid 4714 tid 100383 td 0xe000000012aec900
 
 Tracing command sh pid 4634 tid 100352 td 0xe000000012459b00
 
 Tracing command tcsh pid 4506 tid 100377 td 0xe000000012aad680
 
 Tracing command sshd pid 4505 tid 100366 td 0xe00000001267b200
 
 Tracing command sshd pid 4503 tid 100355 td 0xe000000012584000
 
 Tracing command nginx pid 4432 tid 100378 td 0xe000000012aad200
 
 Tracing command nginx pid 4431 tid 100379 td 0xe000000012aedb00
 
 Tracing command sh pid 4404 tid 100060 td 0xe000000012334000
 
 Tracing command su pid 4403 tid 100373 td 0xe00000001286db00
 
 Tracing command tcsh pid 4398 tid 100376 td 0xe000000012aadb00
 
 Tracing command sshd pid 4397 tid 100367 td 0xe00000001267b680
 
 Tracing command sshd pid 4395 tid 100374 td 0xe000000012aec480
 
 Tracing command sh pid 2622 tid 100353 td 0xe000000012459680
 
 Tracing command sh pid 988 tid 100365 td 0xe00000001267bb00
 
 Tracing command su pid 987 tid 100360 td 0xe00000001286c000
 
 Tracing command tcsh pid 983 tid 100059 td 0xe000000012334480
 
 Tracing command sshd pid 982 tid 100063 td 0xe0000000122ad200
 
 Tracing command sshd pid 980 tid 100356 td 0xe00000001267a480
 
 Tracing command login pid 979 tid 100049 td 0xe000000011973200
 
 Tracing command getty pid 978 tid 100050 td 0xe000000011972d80
 
 Tracing command inetd pid 958 tid 100061 td 0xe0000000122adb00
 
 Tracing command cron pid 920 tid 100062 td 0xe0000000122ad680
 
 Tracing command sendmail pid 916 tid 100069 td 0xe0000000122ac000
 
 Tracing command sendmail pid 913 tid 100361 td 0xe00000001286d680
 
 Tracing command sshd pid 910 tid 100350 td 0xe000000012584480
 
 Tracing command ntpd pid 851 tid 100065 td 0xe000000012335680
 
 Tracing command syslogd pid 773 tid 100359 td 0xe000000012585680
 
 Tracing command devd pid 635 tid 100357 td 0xe00000001267a000
 
 Tracing command adjkerntz pid 134 tid 100351 td 0xe00000001245a000
 
 Tracing command zfskern pid 40 tid 100349 td 0xe000000012584900
 
 Tracing command zfskern pid 40 tid 100348 td 0xe000000012584d80
 
 Tracing command zfskern pid 40 tid 100346 td 0xe000000012458d80
 
 Tracing command zfskern pid 40 tid 100068 td 0xe000000012334900
 
 Tracing command zfskern pid 40 tid 100064 td 0xe000000012335b00
 
 Tracing command softdepflush pid 21 tid 100058 td 0xe0000000122ac480
 
 Tracing command vnlru pid 20 tid 100057 td 0xe0000000122ac900
 
 Tracing command syncer pid 19 tid 100056 td 0xe0000000122acd80
 
 Tracing command bufdaemon pid 18 tid 100055 td 0xe000000011c48480
 
 Tracing command pagezero pid 17 tid 100054 td 0xe000000011c48900
 
 Tracing command vmdaemon pid 9 tid 100053 td 0xe000000011c48d80
 
 Tracing command pagedaemon pid 8 tid 100052 td 0xe000000011c49200
 
 Tracing command xpt_thrd pid 7 tid 100051 td 0xe000000011c49680
 
 Tracing command sctp_iterator pid 4 tid 100048 td 0xe000000011973680
 
 Tracing command mpt_recovery1 pid 3 tid 100044 td 0xe000000011125b00
 
 Tracing command mpt_recovery0 pid 2 tid 100042 td 0xe000000011972480
 
 Tracing command usb pid 16 tid 100039 td 0xe000000011124480
 
 Tracing command usb pid 16 tid 100038 td 0xe000000011124900
 
 Tracing command usb pid 16 tid 100037 td 0xe000000011124d80
 
 Tracing command usb pid 16 tid 100036 td 0xe000000011125200
 
 Tracing command usb pid 16 tid 100034 td 0xe000000010d00d80
 
 Tracing command usb pid 16 tid 100033 td 0xe000000010d01200
 
 Tracing command usb pid 16 tid 100032 td 0xe000000010d01680
 
 Tracing command usb pid 16 tid 100031 td 0xe000000010d01b00
 
 Tracing command usb pid 16 tid 100029 td 0xe000000010c2f680
 
 Tracing command usb pid 16 tid 100028 td 0xe000000010c2fb00
 
 Tracing command usb pid 16 tid 100027 td 0xe000000010d00000
 
 Tracing command usb pid 16 tid 100026 td 0xe000000010d00480
 
 Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000010c2e000
 
 Tracing command rand_harvestq pid 14 tid 100013 td 0xe000000010bb1680
 
 Tracing command geom pid 13 tid 100011 td 0xe000000010bbc000
 
 Tracing command geom pid 13 tid 100010 td 0xe000000010b93680
 
 Tracing command geom pid 13 tid 100009 td 0xe000000010b93b00
 
 Tracing command intr pid 12 tid 100046 td 0xe000000011c48000
 
 Tracing command intr pid 12 tid 100045 td 0xe000000011125680
 
 Tracing command intr pid 12 tid 100043 td 0xe000000011972000
 
 Tracing command intr pid 12 tid 100041 td 0xe000000011972900
 
 Tracing command intr pid 12 tid 100040 td 0xe000000011124000
 
 Tracing command intr pid 12 tid 100035 td 0xe000000010d00900
 
 Tracing command intr pid 12 tid 100030 td 0xe000000010c2f200
 
 Tracing command intr pid 12 tid 100025 td 0xe000000010bbdb00
 
 Tracing command intr pid 12 tid 100021 td 0xe000000010c2ed80
 
 Tracing command intr pid 12 tid 100020 td 0xe000000010bbc480
 
 Tracing command intr pid 12 tid 100018 td 0xe000000010bbcd80
 
 Tracing command intr pid 12 tid 100017 td 0xe000000010bbd200
 
 Tracing command intr pid 12 tid 100008 td 0xe000000010bb0000
 
 Tracing command intr pid 12 tid 100007 td 0xe000000010bb0480
 
 Tracing command intr pid 12 tid 100006 td 0xe000000010bb0900
 
 Tracing command intr pid 12 tid 100005 td 0xe000000010b92000
 
 Tracing command idle pid 11 tid 100004 td 0xe000000010b92480
 
 Tracing command idle pid 11 tid 100003 td 0xe000000010b92900
 
 Tracing command init pid 1 tid 100002 td 0xe000000010b92d80
 
 Tracing command audit pid 10 tid 100001 td 0xe000000010b93200
 
 Tracing command kernel pid 0 tid 100390 td 0xe000000013a32d80
 
 Tracing command kernel pid 0 tid 100389 td 0xe000000013ada900
 
 Tracing command kernel pid 0 tid 100386 td 0xe00000001286cd80
 
 Tracing command kernel pid 0 tid 100385 td 0xe00000001286d200
 
 Tracing command kernel pid 0 tid 100354 td 0xe00000001267a900
 
 Tracing command kernel pid 0 tid 100347 td 0xe000000012585200
 
 Tracing command kernel pid 0 tid 100345 td 0xe000000012458000
 
 Tracing command kernel pid 0 tid 100344 td 0xe000000012451680
 
 Tracing command kernel pid 0 tid 100343 td 0xe000000012450900
 
 Tracing command kernel pid 0 tid 100342 td 0xe000000012450000
 
 Tracing command kernel pid 0 tid 100341 td 0xe000000012443680
 
 Tracing command kernel pid 0 tid 100340 td 0xe000000012440000
 
 Tracing command kernel pid 0 tid 100339 td 0xe000000012442000
 
 Tracing command kernel pid 0 tid 100338 td 0xe00000001243c900
 
 Tracing command kernel pid 0 tid 100337 td 0xe000000012440900
 
 Tracing command kernel pid 0 tid 100336 td 0xe00000001243b200
 
 Tracing command kernel pid 0 tid 100335 td 0xe00000001243d200
 
 Tracing command kernel pid 0 tid 100334 td 0xe000000012439b00
 
 Tracing command kernel pid 0 tid 100333 td 0xe00000001243c000
 
 Tracing command kernel pid 0 tid 100332 td 0xe000000012438900
 
 Tracing command kernel pid 0 tid 100331 td 0xe00000001243a900
 
 Tracing command kernel pid 0 tid 100330 td 0xe000000012437200
 
 Tracing command kernel pid 0 tid 100329 td 0xe000000012439200
 
 Tracing command kernel pid 0 tid 100328 td 0xe000000012435b00
 
 Tracing command kernel pid 0 tid 100327 td 0xe000000012438000
 
 Tracing command kernel pid 0 tid 100326 td 0xe000000012434900
 
 Tracing command kernel pid 0 tid 100325 td 0xe000000012436900
 
 Tracing command kernel pid 0 tid 100324 td 0xe00000001242f200
 
 Tracing command kernel pid 0 tid 100323 td 0xe000000012435200
 
 Tracing command kernel pid 0 tid 100322 td 0xe00000001242db00
 
 Tracing command kernel pid 0 tid 100321 td 0xe00000001242d200
 
 Tracing command kernel pid 0 tid 100320 td 0xe00000001242c480
 
 Tracing command kernel pid 0 tid 100319 td 0xe000000012425b00
 
 Tracing command kernel pid 0 tid 100318 td 0xe000000012425200
 
 Tracing command kernel pid 0 tid 100317 td 0xe000000012424480
 
 Tracing command kernel pid 0 tid 100316 td 0xe000000012423b00
 
 Tracing command kernel pid 0 tid 100315 td 0xe000000012422d80
 
 Tracing command kernel pid 0 tid 100314 td 0xe000000012422000
 
 Tracing command kernel pid 0 tid 100313 td 0xe00000001241f680
 
 Tracing command kernel pid 0 tid 100312 td 0xe00000001241e480
 
 Tracing command kernel pid 0 tid 100311 td 0xe000000012422900
 
 Tracing command kernel pid 0 tid 100310 td 0xe00000001241cd80
 
 Tracing command kernel pid 0 tid 100309 td 0xe00000001241ed80
 
 Tracing command kernel pid 0 tid 100308 td 0xe000000012451b00
 
 Tracing command kernel pid 0 tid 100307 td 0xe00000001245a480
 
 Tracing command kernel pid 0 tid 100306 td 0xe000000012458900
 
 Tracing command kernel pid 0 tid 100305 td 0xe000000012458480
 
 Tracing command kernel pid 0 tid 100304 td 0xe000000012450d80
 
 Tracing command kernel pid 0 tid 100303 td 0xe000000012450480
 
 Tracing command kernel pid 0 tid 100302 td 0xe000000012442d80
 
 Tracing command kernel pid 0 tid 100301 td 0xe000000012451200
 
 Tracing command kernel pid 0 tid 100300 td 0xe000000012443b00
 
 Tracing command kernel pid 0 tid 100299 td 0xe000000012443200
 
 Tracing command kernel pid 0 tid 100298 td 0xe000000012441b00
 
 Tracing command kernel pid 0 tid 100297 td 0xe000000012441680
 
 Tracing command kernel pid 0 tid 100296 td 0xe000000012442900
 
 Tracing command kernel pid 0 tid 100295 td 0xe000000012442480
 
 Tracing command kernel pid 0 tid 100294 td 0xe000000012440d80
 
 Tracing command kernel pid 0 tid 100293 td 0xe000000012440480
 
 Tracing command kernel pid 0 tid 100292 td 0xe00000001243cd80
 
 Tracing command kernel pid 0 tid 100291 td 0xe000000012441200
 
 Tracing command kernel pid 0 tid 100290 td 0xe00000001243db00
 
 Tracing command kernel pid 0 tid 100289 td 0xe00000001243d680
 
 Tracing command kernel pid 0 tid 100288 td 0xe00000001243bb00
 
 Tracing command kernel pid 0 tid 100287 td 0xe00000001243b680
 
 Tracing command kernel pid 0 tid 100286 td 0xe00000001243a000
 
 Tracing command kernel pid 0 tid 100285 td 0xe00000001243c480
 
 Tracing command kernel pid 0 tid 100284 td 0xe00000001243ad80
 
 Tracing command kernel pid 0 tid 100283 td 0xe00000001243a480
 
 Tracing command kernel pid 0 tid 100282 td 0xe000000012438d80
 
 Tracing command kernel pid 0 tid 100281 td 0xe000000012438480
 
 Tracing command kernel pid 0 tid 100280 td 0xe000000012436d80
 
 Tracing command kernel pid 0 tid 100279 td 0xe000000012439680
 
 Tracing command kernel pid 0 tid 100278 td 0xe000000012437b00
 
 Tracing command kernel pid 0 tid 100277 td 0xe000000012437680
 
 Tracing command kernel pid 0 tid 100276 td 0xe000000012436000
 
 Tracing command kernel pid 0 tid 100275 td 0xe000000012435680
 
 Tracing command kernel pid 0 tid 100274 td 0xe000000012434000
 
 Tracing command kernel pid 0 tid 100273 td 0xe000000012436480
 
 Tracing command kernel pid 0 tid 100272 td 0xe000000012434d80
 
 Tracing command kernel pid 0 tid 100271 td 0xe000000012434480
 
 Tracing command kernel pid 0 tid 100270 td 0xe00000001242ed80
 
 Tracing command kernel pid 0 tid 100269 td 0xe00000001242e900
 
 Tracing command kernel pid 0 tid 100268 td 0xe00000001242fb00
 
 Tracing command kernel pid 0 tid 100267 td 0xe00000001242f680
 
 Tracing command kernel pid 0 tid 100266 td 0xe00000001242e000
 
 Tracing command kernel pid 0 tid 100265 td 0xe00000001242d680
 
 Tracing command kernel pid 0 tid 100264 td 0xe00000001242c000
 
 Tracing command kernel pid 0 tid 100263 td 0xe00000001242e480
 
 Tracing command kernel pid 0 tid 100262 td 0xe00000001242cd80
 
 Tracing command kernel pid 0 tid 100261 td 0xe00000001242c900
 
 Tracing command kernel pid 0 tid 100260 td 0xe000000012424d80
 
 Tracing command kernel pid 0 tid 100259 td 0xe000000012424900
 
 Tracing command kernel pid 0 tid 100258 td 0xe000000012423200
 
 Tracing command kernel pid 0 tid 100257 td 0xe000000012425680
 
 Tracing command kernel pid 0 tid 100256 td 0xe000000012424000
 
 Tracing command kernel pid 0 tid 100255 td 0xe000000012423680
 
 Tracing command kernel pid 0 tid 100254 td 0xe000000012422480
 
 Tracing command kernel pid 0 tid 100253 td 0xe00000001241fb00
 
 Tracing command kernel pid 0 tid 100252 td 0xe00000001241e900
 
 Tracing command kernel pid 0 tid 100251 td 0xe00000001241e000
 
 Tracing command kernel pid 0 tid 100250 td 0xe00000001241f200
 
 Tracing command kernel pid 0 tid 100249 td 0xe00000001241c900
 
 Tracing command kernel pid 0 tid 100248 td
 
 
 options	CONFIG_AUTOGENERATED
 ident	UZI
 machine	ia64
 cpu	ITANIUM2
 makeoptions	MODULES_OVERRIDE=geom/geom_part opensolaris zfs acl_nfs4 acl_posix1e
 makeoptions	DEBUG=-g
 options	USB_DEBUG
 options	TEXTDUMP_VERBOSE
 options	TEXTDUMP_PREFERRED
 options	DIAGNOSTIC
 options	DEBUG_VFS_LOCKS
 options	DEBUG_LOCKS
 options	MALLOC_DEBUG_MAXZONES=8
 options	WITNESS_SKIPSPIN
 options	WITNESS
 options	INVARIANT_SUPPORT
 options	INVARIANTS
 options	DEADLKRES
 options	GDB
 options	DDB
 options	KDB_TRACE
 options	KDB
 options	_KPOSIX_PRIORITY_SCHEDULING
 options	QUOTA
 options	UFS_GJOURNAL
 options	UFS_DIRHASH
 options	UFS_ACL
 options	SYSVSHM
 options	SYSVSEM
 options	SYSVMSG
 options	STACK
 options	SOFTUPDATES
 options	SMP
 options	SCTP
 options	SCSI_DELAY=5000
 options	SCHED_ULE
 options	PSEUDOFS
 options	PROCFS
 options	PROCDESC
 options	PRINTF_BUFR_SIZE=128
 options	PREEMPTION
 options	P1003_1B_SEMAPHORES
 options	NFS_ROOT
 options	NFSD
 options	NFSLOCKD
 options	NFSCL
 options	MSDOSFS
 options	MD_ROOT
 options	MAC
 options	KTRACE
 options	INET6
 options	INET
 options	INCLUDE_CONFIG_FILE
 options	GEOM_LABEL
 options	FFS
 options	TMPFS
 options	NULLFS
 options	FDESCFS
 options	COMPAT_FREEBSD7
 options	CD9660
 options	CAPABILITIES
 options	CAPABILITY_MODE
 options	AUDIT
 options	NEW_PCIB
 options	GEOM_PART_MBR
 options	GEOM_PART_GPT
 options	GEOM_PART_BSD
 device	acpi
 device	io
 device	mem
 device	uart_ns8250
 device	miibus
 device	pci
 device	scbus
 device	usb
 device	mpt
 device	cd
 device	da
 device	pass
 device	ehci
 device	ohci
 device	uhci
 device	umass
 device	bge
 device	fxp
 device	ether
 device	faith
 device	gif
 device	loop
 device	md
 device	puc
 device	random
 device	tun
 device	uart
 device	vlan
 device	firmware
 
 
 GDB: debug ports: uart
 GDB: current port: uart
 KDB: debugger backends: ddb gdb
 KDB: current backend: ddb
 Copyright (c) 1992-2013 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 10.0-CURRENT #12 r255488M: Tue Oct 22 15:42:23 BST 2013
     root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI ia64
 gcc version 4.2.1 20070831 patched [FreeBSD]
 WARNING: WITNESS option enabled, expect reduced performance.
 WARNING: DIAGNOSTIC option enabled, expect reduced performance.
 CPU: Madison (1500 MHz Itanium 2)
   Origin = "GenuineIntel"  Revision = 5
   Features = 0x1<LB>
 real memory  = 6442450944 (6144 MB)
 avail memory = 6286475264 (5995 MB)
 FPSWA Revision = 0x10012, Entry = 0xe0000040ffe60050
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP)
 cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0
 random: <Software, Yarrow> initialized
 Event timer "ITC" frequency 1500000000 Hz quality 1000
 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603)
 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603)
 acpi0: <HP> on motherboard
 acpi0: Power Button (fixed)
 acpi0: Sleep Button (fixed)
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
 acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0
 acpi_tz0: <Thermal Zone> on acpi0
 pcib0: <ACPI Host-PCI bridge> on acpi0
 pcib0: could not evaluate _ADR - AE_NOT_FOUND
 pci0: <ACPI PCI bus> on pcib0
 ohci0: <NEC uPD 9210 USB controller> mem 0x80023000-0x80023fff irq 16 at device 1.0 on pci0
 usbus0 on ohci0
 ohci1: <NEC uPD 9210 USB controller> mem 0x80022000-0x80022fff irq 17 at device 1.1 on pci0
 usbus1 on ohci1
 ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x80021000-0x800210ff irq 18 at device 1.2 on pci0
 usbus2: EHCI version 0.95
 usbus2 on ehci0
 pci0: <mass storage, ATA> at device 2.0 (no driver attached)
 fxp0: <Intel 82550C Pro/100 Ethernet> port 0xd00-0xd3f mem 0x80020000-0x80020fff,0x80000000-0x8001ffff irq 20 at device 3.0 on pci0
 miibus0: <MII bus> on fxp0
 inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
 inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
 fxp0: Ethernet address: 00:11:0a:31:e6:0a
 pcib1: <ACPI Host-PCI bridge> on acpi0
 pcib1: could not evaluate _ADR - AE_NOT_FOUND
 pci32: <ACPI PCI bus> on pcib1
 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2100-0x21ff mem 0x90840000-0x9084ffff,0x90830000-0x9083ffff irq 27 at device 1.0 on pci32
 mpt0: MPI Version=1.2.12.0
 mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0x90820000-0x9082ffff,0x90810000-0x9081ffff irq 28 at device 1.1 on pci32
 mpt1: MPI Version=1.2.12.0
 bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x000105> mem 0x90800000-0x9080ffff irq 29 at device 2.0 on pci32
 bge0: CHIP ID 0x00000105; ASIC REV 0x00; CHIP REV 0x01; PCI on PCI-X 66 MHz; 64bit
 miibus1: <MII bus> on bge0
 brgphy0: <BCM5701 1000BASE-T media interface> PHY 1 on miibus1
 brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
 bge0: Ethernet address: 00:11:0a:31:46:06
 pcib2: <ACPI Host-PCI bridge> on acpi0
 pcib2: could not evaluate _ADR - AE_NOT_FOUND
 pci64: <ACPI PCI bus> on pcib2
 pcib3: <ACPI Host-PCI bridge> on acpi0
 pcib3: could not evaluate _ADR - AE_NOT_FOUND
 pci96: <ACPI PCI bus> on pcib3
 pcib4: <ACPI Host-PCI bridge> on acpi0
 pcib4: could not evaluate _ADR - AE_NOT_FOUND
 pci128: <ACPI PCI bus> on pcib4
 pcib5: <ACPI Host-PCI bridge> on acpi0
 pcib5: could not evaluate _ADR - AE_NOT_FOUND
 pci192: <ACPI PCI bus> on pcib5
 pcib6: <ACPI Host-PCI bridge> on acpi0
 pcib6: could not evaluate _ADR - AE_NOT_FOUND
 pci224: <ACPI PCI bus> on pcib6
 uart0: <HP Auxiliary Diva Serial Port> mem 0xf4051000-0xf405100f irq 82 at device 1.0 on pci224
 puc0: <HP Diva Serial [GSP] Multiport UART - Everest SP2> mem 0xf4050000-0xf4050fff,0xf4020000-0xf403ffff irq 82 at device 1.1 on pci224
 uart1: <Non-standard ns8250 class UART with FIFOs> at port 1 on puc0
 uart1: console (9600,n,8,1)
 uart2: <Non-standard ns8250 class UART with FIFOs> at port 2 on puc0
 uart3: <Non-standard ns8250 class UART with FIFOs> at port 3 on puc0
 vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf0000000-0xf3ffffff,0xf4040000-0xf404ffff at device 2.0 on pci224
 uart4: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0
 uart5: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0
 uart5: debug port (9600,n,8,1)
 Timecounters tick every 1.000 msec
 usbus0: 12Mbps Full Speed USB v1.0
 usbus1: 12Mbps Full Speed USB v1.0
 usbus2: 480Mbps High Speed USB v2.0
 ugen1.1: <NEC> at usbus1
 ugen2.1: <NEC> at usbus2
 uhub0: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
 uhub1: <NEC EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
 ugen0.1: <NEC> at usbus0
 uhub2: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
 uhub0: 2 ports with 2 removable, self powered
 uhub2: 3 ports with 3 removable, self powered
 uhub1: 5 ports with 5 removable, self powered
 WARNING: WITNESS option enabled, expect reduced performance.
 WARNING: DIAGNOSTIC option enabled, expect reduced performance.
 da0 at mpt0 bus 0 scbus0 target 0 lun 0
 da0: <HP 73.4G MAX3073NC HPC1> Fixed Direct Access SCSI-3 device 
 da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
 da0: Command Queueing enabled
 da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
 da1 at mpt0 bus 0 scbus0 target 1 lun 0
 da1: <SEAGATE ST373454LC D404> Fixed Direct Access SCSI-3 device 
 da1: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
 da1: Command Queueing enabled
 da1: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C)
 da2 at mpt1 bus 0 scbus1 target 2 lun 0
 da2: <SEAGATE ST373454LC D402> Fixed Direct Access SCSI-3 device 
 da2: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
 da2: Command Queueing enabled
 da2: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C)
 Trying to mount root from ufs:/dev/da0p2 [rw]...
 <118>Setting hostuuid: a90e62bf-ce84-11e1-9fc5-00110a31e60a.
 <118>Setting hostid: 0x688171f3.
 ZFS filesystem version: 5
 ZFS storage pool version: features support (5000)
 <118>Starting ddb.
 <118>Entropy harvesting: interrupts ethernet point_to_point kickstart.
 <118>Starting file system checks:
 <118>/dev/da0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
 <118>/dev/da0p2: clean, 8350171 free (97659 frags, 1031564 blocks, 0.3% fragmentation)
 <118>Mounting local file systems:.
 <118>Writing entropy file:.
 <118>Setting hostname: mech-as221.men.bris.ac.uk.
 <118>Starting Network: lo0 fxp0 bge0.
 <118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 <118>	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
 <118>	inet6 ::1 prefixlen 128 
 <118>	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
 <118>	inet 127.0.0.1 netmask 0xff000000 
 <118>	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 <118>fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 <118>	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 <118>	ether 00:11:0a:31:e6:0a
 <118>	inet 10.10.10.13 netmask 0xffffff00 broadcast 10.10.10.255 
 <118>	inet6 fe80::211:aff:fe31:e60a%fxp0 prefixlen 64 scopeid 0x1 
 <118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 <118>	media: Ethernet autoselect (100baseTX <full-duplex>)
 <118>	status: active
 <118>bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 <118>	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
 <118>	ether 00:11:0a:31:46:06
 <118>	inet 137.222.187.221 netmask 0xffffff00 broadcast 137.222.187.255 
 <118>	inet6 fe80::211:aff:fe31:4606%bge0 prefixlen 64 scopeid 0x2 
 <118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 <118>	media: Ethernet autoselect (none)
 <118>	status: no carrier
 <118>Starting devd.
 <118>add net default: gateway 137.222.187.250
 <118>add net fe80::: gateway ::1
 <118>add net ff02::: gateway ::1
 <118>add net ::ffff:0.0.0.0: gateway ::1
 <118>add net ::0.0.0.0: gateway ::1
 <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/gcc46 /usr/local/lib/gcc48 /usr/local/lib/graphviz
 <118>Creating and/or trimming log files.
 <118>Starting syslogd.
 <118>No core dumps found.
 <118>Clearing /tmp (X related).
 <118>Recovering vi editor sessions:.
 <118>Updating motd:.
 <118>Mounting late file systems:.
 <118>Starting ntpd.
 <118>Starting bsdstats.
 <118>Posting monthly OS statistics to rpt.bsdstats.org
 <118>Performing sanity check on sshd configuration.
 <118>Starting sshd.
 <118>Starting cron.
 <118>Local package initialization:cd: /var/portbuild/: No such file or directory
 <118>.
 <118>Starting inetd.
 <118>Starting background file system checks in 60 seconds.
 <118>
 <118>Tue Oct 22 16:19:22 BST 2013
 <118>Oct 22 16:19:55 mech-as221 su: mexas to root on /dev/pts/0
 lock order reversal:
  1st 0xa00000005f11b698 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3059
  2nd 0xe00000001232a400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000c9ec0) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc0000004d0260) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000bfb620, 0x9ffc000000506110) at kdb_backtrace+0xc0
 _witness_debugger(0x1, 0x9ffc000000b03df8, 0x9ffc000000509ae0, 0xb9d, 0x9ffc000000b31318) at _witness_debugger+0x60
 witness_checkorder(0xe00000001232a400, 0x9ffc000000b03748, 0x9ffc000000b31318, 0x11c, 0x0) at witness_checkorder+0x15b0
 _sx_xlock(0xe00000001232a400, 0x0, 0x9ffc000000b31318, 0x11c) at _sx_xlock+0x120
 ufsdirhash_acquire(0xe000000012a14dc8, 0xe00000001232a400, 0x9ffc0000008d0000, 0x38b) at ufsdirhash_acquire+0x70
 ufsdirhash_remove(0xe000000012a14dc8, 0xa0000000620e3914, 0x5914, 0xa00000009028b1e8) at ufsdirhash_remove+0x20
 ufs_dirremove(0xe000000012a16c30, 0xe000000012a8d1b8, 0x0, 0x0) at ufs_dirremove+0x380
 ufs_remove(0xa00000009028b380, 0xe000000012a8d1b8, 0x9ffc0000009a27f0) at ufs_remove+0xe0
 VOP_REMOVE_APV(0x9ffc000000bc32e8, 0xa00000009028b380, 0x0, 0xa00000009028b390, 0xa00000009028b388, 0x9ffc0000005c0110, 0xa1c, 0x9ffc000000dd2220) at VOP_REMOVE_APV+0x2a0
 kern_unlinkat(0xe000000012aacd80, 0xffffffffffffff9c, 0x12151c700, 0x0, 0x0) at kern_unlinkat+0x3f0
 kern_unlink(0xe000000012aacd80, 0x12151c700, 0x0) at kern_unlink+0x40
 sys_unlink(0xe000000012aacd80, 0xa00000009028b4e8, 0x9ffc000000986c80, 0x48d) at sys_unlink+0x30
 syscall(0xe000000012875280, 0x12151cb42, 0x5, 0xe000000012aacd80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 <118>Oct 22 16:27:08 mech-as221 login: ROOT LOGIN (root) ON ttyu1
 Expensive timeout(9) function: 0x9ffc000000b4e130(0) 0.030631349 s
 <118>Oct 22 19:28:59 mech-as221 su: BAD SU mexas to root on /dev/pts/1
 <118>Oct 22 19:29:06 mech-as221 su: mexas to root on /dev/pts/1
 <118>Oct 22 20:21:15 mech-as221 su: mexas to root on /dev/pts/2
 panic: deadlkres: possible deadlock detected for 0xe000000012aed200, blocked for 900014 ticks
 
 cpuid = 1
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000c9ec0) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc0000004d0260) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000bfb620, 0x9ffc0000004573a0, 0x40c, 0x9ffc000000dd2220) at kdb_backtrace+0xc0
 vpanic(0x9ffc000000aee760, 0xa00000008fdd1518) at vpanic+0x260
 panic(0x9ffc000000aee760, 0x9ffc000000aee9f8, 0xe000000012aed200, 0xdbbae) at panic+0x80
 deadlkres(0xdbbae, 0xe000000012aed200, 0x9ffc000000af13d0, 0x9ffc000000aee6f0) at deadlkres+0x420
 fork_exit(0x9ffc000000b456a0, 0x0, 0xa00000008fdd1550) at fork_exit+0x120
 enter_userland() at enter_userland
 KDB: enter: panic

From: Anton Shterenlikht <mexas@bris.ac.uk>
To: bug-followup@freebsd.org
Cc:  
Subject: RE: kern/183007: ia64: panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks
Date: Thu, 24 Oct 2013 10:01:38 +0100 (BST)

 Using "call doadump", and then crashinfo
 gives this core.txt file:
 
 mech-as221.men.bris.ac.uk dumped core - see ./vmcore.6
 
 Thu Oct 24 09:57:05 BST 2013
 
 FreeBSD mech-as221.men.bris.ac.uk 10.0-CURRENT FreeBSD 10.0-CURRENT #12 r255488M: Tue Oct 22 15:42:23 BST 2013     root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI  ia64
 
 panic: deadlkres: possible deadlock detected for 0xe000000011c49200, blocked for 901124 ticks
 
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "ia64-marcel-freebsd"...
 Cannot access memory at address 0xe0000040fffbdfb8
 (kgdb) #0  0x0000000000000000 in ?? ()
 (kgdb) 
 
 ------------------------------------------------------------------------
 ps -axl
 
  UID   PID  PPID CPU PRI NI   VSZ  RSS MWCHAN           STAT TT       TIME COMMAND
    0     0     0   0  -8  0     0    0 -                DLs   -  121:04.40 [kernel]
    0     1     0   0  20  0 10368 1112 wait             DLs   -    0:00.03 [init]
    0     2     0   0 -16  0     0    0 idle             DL    -    0:00.00 [mpt_recovery0]
    0     3     0   0 -16  0     0    0 idle             DL    -    0:00.00 [mpt_recovery1]
    0     4     0   0 -16  0     0    0 waiting_         DL    -    0:00.00 [sctp_iterator]
    0     7     0   0 -16  0     0    0 ccb_scan         DL    -    0:00.00 [xpt_thrd]
    0     8     0   0 -16  0     0    0 pmap pv          LL    -    0:00.48 [pagedaemon]
    0     9     0   0 -16  0     0    0 -                RL    -    0:00.00 [vmdaemon]
    0    10     0   0 -16  0     0    0 audit_wo         DL    -    0:00.00 [audit]
    0    11     0   0 155  0     0    0 -                RL    -  252:01.23 [idle]
    0    12     0   0 -76  0     0    0 -                WL    -    1:54.21 [intr]
    0    13     0   0  -8  0     0    0 -                DL    -    1:06.94 [geom]
    0    14     0   0 -16  0     0    0 -                DL    -    0:06.59 [rand_harvestq]
    0    15     0   0 -16  0     0    0 tzpoll           DL    -    0:00.18 [acpi_thermal]
    0    16     0   0 -68  0     0    0 -                DL    -    0:00.35 [usb]
    0    17     0   0 155  0     0    0 pgzero           DL    -    0:00.00 [pagezero]
    0    18     0   0 -16  0     0    0 psleep           DL    -    0:00.13 [bufdaemon]
    0    19     0   0  16  0     0    0 syncer           DL    -    1:13.82 [syncer]
    0    20     0   0 -16  0     0    0 vlruwt           DL    -    0:15.43 [vnlru]
    0    21     0   0 -16  0     0    0 sdflush          DL    -    0:00.14 [softdepflush]
    0    40     0   0  -8  0     0    0 e000000012252a18 DL    -    4:05.62 [zfskern]
    0   153     1   0  52  0 12064 1704 pause            DWs   -    0:00.00 [adjkerntz]
    0   654     1   0  20  0 13856 1616 select           Ds    -    0:00.02 [devd]
    0   792     1   0  20  0 12336 3032 select           Ds    -    0:00.09 [syslogd]
    0   873     1   0  20  0 17288 4688 select           Ds    -    0:01.33 [ntpd]
    0   932     1   0  20  0 22736 4672 select           Ds    -    0:00.02 [sshd]
    0   935     1   0  20  0 24280 6368 -                Rs    -    0:00.50 [sendmail]
   25   938     1   0  20  0 18752 4232 pause            DWs   -    0:00.00 [sendmail]
    0   942     1   0  20  0 12488 3272 pmap pv          LLs   -    0:00.13 [cron]
    0   978     1   0  52  0 12624 2296 select           Ds    -    0:00.00 [inetd]
    0  1000     1   0  52  0 12312 3376 ttydcd           D     -    0:00.01 [getty]
    0  1001     1   0  20  0 14128 4976 wait             DWs   -    0:00.00 [login]
    0  1002   932   0  32  0 26304 9440 select           Ds    -    0:00.11 [sshd]
 1001  1004  1002   0  20  0 26304 6784 select           D     -    0:13.09 [sshd]
 1001  1005  1004   0  20  0 14536 7264 pause            DWs   -    0:00.00 [tcsh]
    0  1012  1005   0  20  0 14104 4792 wait             DW    -    0:00.00 [su]
    0  1015  1012   0  33  0 13352 5168 ttyin            D+    -    0:00.16 [sh]
    0  1309  1001   0  20  0 13352 5200 ttyin            D+    -    0:00.17 [sh]
    0 96150   932   0  34  0 26304 9632 select           Ds    -    0:00.00 [sshd]
 1001 96152 96150   0  20  0 26304 6736 select           D     -    0:00.00 [sshd]
 1001 96153 96152   0  52  0 14536 6016 pause            DWs   -    0:00.00 [tcsh]
 1001 96156 96153   0  52  0 12920 6272 vm objec         L     -    0:00.00 [rsync]
 1001 96157 96156   0  20  0 21112 7888 select           D     -    0:00.00 [rsync]
    0 96187   942   0  20  0 12488 3320 piperd           D     -    0:00.00 [cron]
    0 96189 96187   0  25  0 13352 4992 wait             DWs   -    0:00.00 [sh]
    0 96191 96189   0  25  0  7960 3112 wait             DW    -    0:00.00 [lockf]
    0 96192 96191   0  52  0 13352 4992 wait             DW    -    0:00.00 [sh]
    0 96199 96192   0  52  0 13352 2464 wait             DW    -    0:00.00 [sh]
    0 96200 96192   0  24  0 12248 3448 piperd           D     -    0:00.00 [mail]
    0 96282 96199   0  52  0 13352 4864 wait             DW    -    0:00.00 [sh]
    0 96283 96282   0  52  0 13352 4992 wait             DW    -    0:00.00 [sh]
    0 96285 96283   0  52  0  7960 3112 wait             DW    -    0:00.00 [lockf]
    0 96286 96285   0  52  0 13352 4992 wait             DW    -    0:00.00 [sh]
    0 96293 96286   0  52  0 13352 2472 wait             DW    -    0:00.00 [sh]
    0 96294 96286   0  52  0 12248 3440 piperd           D     -    0:00.00 [mail]
    0 96295 96293   0  52  0 13352 5136 wait             DW    -    0:00.00 [sh]
    0 96299 96295   0  20  0 12136 4160 vm page          L     -    0:00.00 [find]
    0 96300 96295   0  52  0 13352   24 wait             DW    -    0:00.00 [sh]
    0 96302 96300   0  52  0 12072 4176 piperd           D     -    0:00.00 [cat]
 
 ------------------------------------------------------------------------
 vmstat -s
 
  42430424 cpu context switches
  99421538 device interrupts
   4215788 software interrupts
  76657815 traps
 140567023 system calls
        22 kernel threads created
     19953  fork() calls
    176205 vfork() calls
         0 rfork() calls
         0 swap pager pageins
         0 swap pager pages paged in
         0 swap pager pageouts
         0 swap pager pages paged out
      5015 vnode pager pageins
     18555 vnode pager pages paged in
         0 vnode pager pageouts
         0 vnode pager pages paged out
         0 page daemon wakeups
         0 pages examined by the page daemon
         0 pages reactivated
   2207587 copy-on-write faults
       340 copy-on-write optimized faults
  72307484 zero fill pages zeroed
         0 zero fill pages prezeroed
       213 intransit blocking page faults
  75448333 total VM faults taken
     22715 page faults requiring I/O
         0 pages affected by kernel thread creation
    703913 pages affected by  fork()
   4057227 pages affected by vfork()
         0 pages affected by rfork()
         0 pages cached
  84394336 pages freed
         0 pages freed by daemon
         0 pages freed by exiting processes
      4609 pages active
    433449 pages inactive
      2026 pages in VM cache
    319481 pages wired down
     12179 pages free
      8192 bytes per page
 216313539 total name lookups
           cache hits (90% pos + 5% neg) system 0% per-directory
           deletions 0%, falsehits 0%, toolong 0%
 
 ------------------------------------------------------------------------
 vmstat -m
 
 vmstat: memstat_kvm_malloc: invalid kernel virtual address
          Type InUse MemUse HighUse Requests  Size(s)
 
 ------------------------------------------------------------------------
 vmstat -z
 
 vmstat: memstat_kvm_uma: KVM short read
 ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
 
 
 
 ------------------------------------------------------------------------
 vmstat -i
 
 interrupt                   total       rate
 clock                    85559128     425667
 uart0                        4936         24
 mpt0                      2060488      10251
 mpt1                      1367156       6801
 fxp0                        10067         50
 bge0                      1075390       5350
 Total                    90077165     448145
 
 ------------------------------------------------------------------------
 pstat -T
 
 110/40000 files
 0M/8166M swap space
 
 ------------------------------------------------------------------------
 pstat -s
 
 Device          1K-blocks     Used    Avail Capacity
 /dev/da0p3        8362912        0  8362912     0%
 
 ------------------------------------------------------------------------
 iostat
 
 iostat: kvm_read(_tk_nin): invalid kernel virtual address
 iostat: disabling TTY statistics
              da0              da1              da2             cpu
   KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
  13.99 3311 45.25  30.71 7104 213.07  31.32 6971 213.22  13  0 46  0 40
 
 ------------------------------------------------------------------------
 ipcs -a
 
 ipcs: shmsegs: invalid kernel virtual address
 Message Queues:
 T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   
 
 
 ------------------------------------------------------------------------
 ipcs -T
 
 msginfo:
 	msgmax:        16384	(max characters in a message)
 	msgmni:           40	(# of message queues)
 	msgmnb:         2048	(max characters in a message queue)
 	msgtql:           40	(max # of messages in system)
 	msgssz:            8	(size of a message segment)
 	msgseg:         2048	(# of message segments in system)
 
 shminfo:
 	shmmax:   1073741824	(max shared memory segment size)
 	shmmin:            1	(min shared memory segment size)
 	shmmni:          192	(max number of shared memory identifiers)
 	shmseg:          128	(max shared memory segments per process)
 	shmall:       131072	(max amount of shared memory in pages)
 
 seminfo:
 	semmni:           50	(# of semaphore identifiers)
 	semmns:          340	(# of semaphores in system)
 	semmnu:          150	(# of undo structures in system)
 	semmsl:          340	(max # of semaphores per id)
 	semopm:          100	(max # of operations per semop call)
 	semume:           50	(max # of undo entries per process)
 	semusz:          632	(size in bytes of undo structure)
 	semvmx:        32767	(semaphore maximum value)
 	semaem:        16384	(adjust on exit max value)
 
 
 ------------------------------------------------------------------------
 nfsstat
 
 Client Info:
 Rpc Counts:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
         0         0         0         0         0         0         0         0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0         0         0         0         0         0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         0         0         0         0
 Rpc Info:
  TimedOut   Invalid X Replies   Retries  Requests
         0         0         0         0         0
 Cache Info:
 Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
         0         0         0         0         0         0         0         0
 BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
         0         0         0         0         0         0         0         0
 
 Server Info:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
         0         0         0         0         0         0         0         0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0         0         0         0         0         0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         0         0         0         0
 Server Ret-Failed
                 0
 Server Faults
             0
 Server Cache Stats:
    Inprog      Idem  Non-idem    Misses
         0         0         0         0
 Server Write Gathering:
  WriteOps  WriteRPC   Opsaved
         0         0         0
 
 ------------------------------------------------------------------------
 netstat -s
 
 tcp:
 	282995 packets sent
 		223922 data packets (62082838 bytes)
 		695403 data packets (599610 bytes) retransmitted
 		618576506 data packets unnecessarily retransmitted
 		0 resends initiated by MTU discovery
 		56310 ack-only packets (202761 delayed)
 		0 URG only packets
 		0 window probe packets
 		9036 window update packets
 		1592 control packets
 	124318 packets received
 		33423 acks (for 15875451 bytes)
 		385 duplicate acks
 		1288 acks for unsent data
 		94315 packets (110672333 bytes) received in-sequence
 		4 completely duplicate packets (739 bytes)
 		1496 old duplicate packets
 		1349 packets with some dup. data (568 bytes duped)
 		198374 out-of-order packets (100900200 bytes)
 		3 packets (1 byte) of data after window
 		470899 window probes
 		1496 window update packets
 		77109 packets received after close
 		3 discarded for bad checksums
 		0 discarded for bad header offset fields
 		3 discarded because packet too short
 		3 discarded due to memory problems
 	198359 connection requests
 	6710 connection accepts
 	339 bad connection attempts
 	0 listen queue overflows
 	1 ignored RSTs in the window
 	336 connections established (including accepts)
 	540 connections closed (including 1469 drops)
 		246 connections updated cached RTT on close
 		360 connections updated cached RTT variance on close
 		310 connections updated cached ssthresh on close
 	1 embryonic connection dropped
 	493485 segments updated rtt (of 73632 attempts)
 	1542 retransmit timeouts
 		55014893 connections dropped by rexmit timeout
 	56738 persist timeouts
 		0 connections dropped by persist timeout
 	0 Connections (fin_wait_2) dropped because of timeout
 	17 keepalive timeouts
 		259207 keepalive probes sent
 		0 connections dropped by keepalive
 	13190 correct ACK header predictions
 	85152 correct data packet header predictions
 	99 syncache entries added
 		2 retransmitted
 		1 dupsyn
 		0 dropped
 		115 completed
 		0 bucket overflow
 		0 cache overflow
 		0 reset
 		0 stale
 		0 aborted
 		0 badack
 		0 unreach
 		0 zone failures
 	124 cookies sent
 	115 cookies received
 	3 hostcache entries added
 		0 bucket overflow
 	0 SACK recovery episodes
 	0 segment rexmits in SACK recovery episodes
 	0 byte rexmits in SACK recovery episodes
 	6 SACK options (SACK blocks) received
 	0 SACK options (SACK blocks) sent
 	0 SACK scoreboard overflow
 	0 packets with ECN CE bit set
 	0 packets with ECN ECT(0) bit set
 	0 packets with ECN ECT(1) bit set
 	0 successful ECN handshakes
 	0 times ECN reduced the congestion window
 udp:
 	10210 datagrams received
 	0 with incomplete header
 	0 with bad data length field
 	0 with bad checksum
 	0 with no checksum
 	0 dropped due to no socket
 	9413 broadcast/multicast datagrams undelivered
 	0 dropped due to full socket buffers
 	0 not for hashed pcb
 	797 delivered
 	2060 datagrams output
 	0 times multicast source filter matched
 ip:
 	133621 total packets received
 	0 bad header checksums
 	0 with size smaller than minimum
 	0 with data size < data length
 	1 with ip length > max ip packet size
 	0 with header length < data size
 	298 with data length < header length
 	638 with bad options
 	1523 with incorrect version number
 	734107 fragments received
 	462943 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 packets reassembled ok
 	132252 packets for this host
 	1369 packets for unknown/unsupported protocol
 	0 packets forwarded (0 packets fast forwarded)
 	0 packets not forwardable
 	0 packets received for unknown multicast group
 	0 redirects sent
 	285289 packets sent from this host
 	220 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	339 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	1687 tunneling packets that can't find gif
 	75478 datagrams with bad address in header
 icmp:
 	0 calls to icmp_error
 	0 errors not generated in response to an icmp message
 	Output histogram:
 		echo reply: 2
 		domain name request: 9
 	1 message with bad code fields
 	0 messages less than the minimum length
 	0 messages with bad checksum
 	0 messages with bad length
 	1369 multicast echo requests ignored
 	0 multicast timestamp requests ignored
 	Input histogram:
 		echo reply: 306
 		echo: 1371
 		IPv6 where-are-you: 297
 	2 message responses generated
 	0 invalid return addresses
 	0 no return routes
 igmp:
 	0 messages received
 	0 messages received with too few bytes
 	0 messages received with wrong TTL
 	0 messages received with bad checksum
 	0 V1/V2 membership queries received
 	0 V3 membership queries received
 	0 membership queries received with invalid field(s)
 	0 general queries received
 	0 group queries received
 	0 group-source queries received
 	0 group-source queries dropped
 	0 membership reports received
 	0 membership reports received with invalid field(s)
 	0 membership reports received for groups to which we belong
 	0 V3 reports received without Router Alert
 	0 membership reports sent
 arp:
 	27 ARP requests sent
 	7 ARP replies sent
 	18168 ARP requests received
 	6 ARP replies received
 	56996 ARP packets received
 	0 total packets dropped due to no ARP entry
 	24 ARP entrys timed out
 	0 Duplicate IPs seen
 ip6:
 	0 total packets received
 	0 with size smaller than minimum
 	0 with data size < data length
 	0 with bad options
 	0 with incorrect version number
 	5 fragments received
 	0 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 fragments that exceeded limit
 	0 packets reassembled ok
 	5 packets for this host
 	0 packets forwarded
 	0 packets not forwardable
 	0 redirects sent
 	3 packets sent from this host
 	0 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	14 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	0 packets that violated scope rules
 	0 multicast packets which we don't join
 	Input histogram:
 		ISOIP: 734405
 	Mbuf statistics:
 		0 one mbuf
 		two or more mbuf:
 			(null)= 1
 		1 one ext mbuf
 		0 two or more ext mbuf
 	0 packets whose headers are not contiguous
 	0 tunneling packets that can't find gif
 	0 packets discarded because of too many headers
 	0 failures of source address selection
 	Source addresses selection rule applied:
 icmp6:
 	0 calls to icmp6_error
 	0 errors not generated in response to an icmp6 message
 	0 errors not generated because of rate limitation
 	Output histogram:
 		neighbor solicitation: 2
 	0 messages with bad code fields
 	0 messages < minimum length
 	0 bad checksums
 	0 messages with bad length
 	Input histogram:
 		#232: 99
 		#233: 115
 		#234: 3
 	Histogram of error messages to be generated:
 		0 no route
 		0 administratively prohibited
 		0 beyond scope
 		0 address unreachable
 		0 port unreachable
 		0 packet too big
 		0 time exceed transit
 		0 time exceed reassembly
 		0 erroneous header field
 		0 unrecognized next header
 		0 unrecognized option
 		0 redirect
 		0 unknown
 	0 message responses generated
 	0 messages with too many ND options
 	0 messages with bad ND options
 	0 bad neighbor solicitation messages
 	0 bad neighbor advertisement messages
 	0 bad router solicitation messages
 	0 bad router advertisement messages
 	1 bad redirect message
 	0 path MTU changes
 rip6:
 	0 messages received
 	0 checksum calculations on inbound
 	9 messages with bad checksum
 	0 messages dropped due to no socket
 	0 multicast messages dropped due to no socket
 	1 message dropped due to full socket buffers
 	18446744073709551606 delivered
 	0 datagrams output
 
 ------------------------------------------------------------------------
 netstat -m
 
 netstat: memstat_kvm_all: invalid kernel virtual address
 
 ------------------------------------------------------------------------
 netstat -idW
 
 Name      Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
 fxp0     1500 <Link#1>      00:11:0a:31:e6:0a    10070     0     0        0     0     0    0 
 fxp0     1500 10.10.10.0    10.10.10.13           1666     -     -        0     -     -    - 
 fxp0     1500 fe80::211:aff fe80::211:aff:fe3        0     -     -        0     -     -    - 
 bge0     1500 <Link#2>      00:11:0a:31:46:06   914712     0     0   748267     0     0    0 
 bge0     1500 137.222.187.0 mech-as221          845982     -     -   748186     -     -    - 
 bge0     1500 fe80::211:aff fe80::211:aff:fe3        0     -     -        2     -     -    - 
 lo0     16384 <Link#3>                              46     0     0       46     0     0    0 
 lo0     16384 localhost     ::1                      0     -     -        0     -     -    - 
 lo0     16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    - 
 lo0     16384 your-net      localhost               46     -     -       46     -     -    - 
 
 ------------------------------------------------------------------------
 netstat -anr
 
 Routing tables
 
 Internet:
 Destination        Gateway            Flags    Refs      Use  Netif Expire
 default            137.222.187.250    UGS         0   547946   bge0
 10.10.10.0/24      link#1             U           0        0   fxp0
 10.10.10.13        link#1             UHS         0        0    lo0
 127.0.0.1          link#3             UH          0       46    lo0
 137.222.187.0/24   link#2             U           0   200240   bge0
 137.222.187.221    link#2             UHS         0        0    lo0
 
 Internet6:
 Destination                       Gateway                       Flags      Netif Expire
 ::/96                             ::1                           UGRS        lo0
 ::1                               link#3                        UH          lo0
 ::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
 fe80::/10                         ::1                           UGRS        lo0
 fe80::%fxp0/64                    link#1                        U          fxp0
 fe80::211:aff:fe31:e60a%fxp0      link#1                        UHS         lo0
 fe80::%bge0/64                    link#2                        U          bge0
 fe80::211:aff:fe31:4606%bge0      link#2                        UHS         lo0
 fe80::%lo0/64                     link#3                        U           lo0
 fe80::1%lo0                       link#3                        UHS         lo0
 ff01::%fxp0/32                    fe80::211:aff:fe31:e60a%fxp0  U          fxp0
 ff01::%bge0/32                    fe80::211:aff:fe31:4606%bge0  U          bge0
 ff01::%lo0/32                     ::1                           U           lo0
 ff02::/16                         ::1                           UGRS        lo0
 ff02::%fxp0/32                    fe80::211:aff:fe31:e60a%fxp0  U          fxp0
 ff02::%bge0/32                    fe80::211:aff:fe31:4606%bge0  U          bge0
 ff02::%lo0/32                     ::1                           U           lo0
 
 ------------------------------------------------------------------------
 netstat -anA
 
 netstat: kvm_read: Invalid argument
 netstat: kvm_read: Invalid argument
 Active Internet connections (including servers)
 Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
 e000000013417d18 udp4       0      0 127.0.0.1.123      *.*                
 e000000013418000 udp6       0      0 fe80::1%lo0.123    *.*                
 e000000013418188 udp6       0      0 ::1.123            *.*                
 e000000013418310 udp6       0      0 fe80::211:aff:fe.1 *.*                
 e000000013418498 udp4       0      0 137.222.187.221.12 *.*                
 e000000013418620 udp6       0      0 fe80::211:aff:fe.1 *.*                
 e0000000134187a8 udp4       0      0 10.10.10.13.123    *.*                
 e000000013418930 udp6       0      0 *.123              *.*                
 e000000013418ab8 udp4       0      0 *.123              *.*                
 e00000001340b880 udp4       0      0 *.514              *.*                
 e00000001340ba08 udp6       0      0 *.514              *.*                
 Active UNIX domain sockets
 Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
 e0000000135dec30 dgram       0      0        0 e0000000135cb860        0 e0000000135df0e0
 e0000000135df0e0 dgram       0      0        0 e0000000135cb860        0 e0000000135df3b0
 e0000000135df2c0 dgram       0      0        0 e0000000135df4a0        0        0
 e0000000135df3b0 dgram       0      0        0 e0000000135cb860        0 e0000000135cb770
 e0000000135cb770 dgram       0      0        0 e0000000135cb860        0        0
 e0000000135cb860 dgram       0      0 e0000000133b09c0        0 e0000000135dec30        0 /var/run/logpriv
 e0000000135df4a0 dgram       0      0 e0000000135ccea0        0 e0000000135df2c0        0 /var/run/log
 
 ------------------------------------------------------------------------
 netstat -aL
 
 netstat: kvm_read: Invalid argument
 netstat: kvm_read: Invalid argument
 Current listen queue sizes (qlen/incqlen/maxqlen)
 Proto Listen         Local Address         
 
 ------------------------------------------------------------------------
 fstat
 
 Segmentation fault
 
 ------------------------------------------------------------------------
 dmesg
 
 dmesg: kvm_read: 
 
 ------------------------------------------------------------------------
 kernel config
 
 options	CONFIG_AUTOGENERATED
 ident	UZI
 machine	ia64
 cpu	ITANIUM2
 makeoptions	MODULES_OVERRIDE=geom/geom_part opensolaris zfs acl_nfs4 acl_posix1e
 makeoptions	DEBUG=-g
 options	USB_DEBUG
 options	TEXTDUMP_VERBOSE
 options	TEXTDUMP_PREFERRED
 options	DIAGNOSTIC
 options	DEBUG_VFS_LOCKS
 options	DEBUG_LOCKS
 options	MALLOC_DEBUG_MAXZONES=8
 options	WITNESS_SKIPSPIN
 options	WITNESS
 options	INVARIANT_SUPPORT
 options	INVARIANTS
 options	DEADLKRES
 options	GDB
 options	DDB
 options	KDB_TRACE
 options	KDB
 options	_KPOSIX_PRIORITY_SCHEDULING
 options	QUOTA
 options	UFS_GJOURNAL
 options	UFS_DIRHASH
 options	UFS_ACL
 options	SYSVSHM
 options	SYSVSEM
 options	SYSVMSG
 options	STACK
 options	SOFTUPDATES
 options	SMP
 options	SCTP
 options	SCSI_DELAY=5000
 options	SCHED_ULE
 options	PSEUDOFS
 options	PROCFS
 options	PROCDESC
 options	PRINTF_BUFR_SIZE=128
 options	PREEMPTION
 options	P1003_1B_SEMAPHORES
 options	NFS_ROOT
 options	NFSD
 options	NFSLOCKD
 options	NFSCL
 options	MSDOSFS
 options	MD_ROOT
 options	MAC
 options	KTRACE
 options	INET6
 options	INET
 options	INCLUDE_CONFIG_FILE
 options	GEOM_LABEL
 options	FFS
 options	TMPFS
 options	NULLFS
 options	FDESCFS
 options	COMPAT_FREEBSD7
 options	CD9660
 options	CAPABILITIES
 options	CAPABILITY_MODE
 options	AUDIT
 options	NEW_PCIB
 options	GEOM_PART_MBR
 options	GEOM_PART_GPT
 options	GEOM_PART_BSD
 device	acpi
 device	io
 device	mem
 device	uart_ns8250
 device	miibus
 device	pci
 device	scbus
 device	usb
 device	mpt
 device	cd
 device	da
 device	pass
 device	ehci
 device	ohci
 device	uhci
 device	umass
 device	bge
 device	fxp
 device	ether
 device	faith
 device	gif
 device	loop
 device	md
 device	puc
 device	random
 device	tun
 device	uart
 device	vlan
 device	firmware
 
 ------------------------------------------------------------------------
 ddb capture buffer
 
 
Responsible-Changed-From-To: freebsd-bugs->freebsd-ia64 
Responsible-Changed-By: marcel 
Responsible-Changed-When: Sun Dec 8 03:39:22 UTC 2013 
Responsible-Changed-Why:  
Assign. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=183007 
State-Changed-From-To: open->closed 
State-Changed-By: marcel 
State-Changed-When: Mon Jan 20 18:25:33 UTC 2014 
State-Changed-Why:  
The use of kernel option DIAGNOSTICS was triggering enough runtime 
checking that under system load the deadlock resolver got confused. 
Removal of the kernel option avoided the problem. Since this is not 
ia64-specific and also not necessarily a problem that can be fixed, 
close the PR out. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=183007 
>Unformatted:
