From mexas@bristol.ac.uk  Wed Oct 23 09:03:59 2013
Return-Path: <mexas@bristol.ac.uk>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTP id 54F38B0F
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 23 Oct 2013 09:03:59 +0000 (UTC)
	(envelope-from mexas@bristol.ac.uk)
Received: from eu1sys200aog101.obsmtp.com (eu1sys200aog101.obsmtp.com [207.126.144.111])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id D96552E31
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 23 Oct 2013 09:03:57 +0000 (UTC)
Received: from mail-wi0-f171.google.com ([209.85.212.171]) (using TLSv1) by eu1sys200aob101.postini.com ([207.126.147.11]) with SMTP
	ID DSNKUmeQ4qQG1tVNR6ALRQ9DI2IWXQHJKPCh@postini.com; Wed, 23 Oct 2013 09:03:57 UTC
Received: by mail-wi0-f171.google.com with SMTP id h11so6976185wiv.4
        for <FreeBSD-gnats-submit@freebsd.org>; Wed, 23 Oct 2013 02:03:30 -0700 (PDT)
Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241])
        by mx.google.com with ESMTPSA id s4sm14732627wiy.1.2013.10.23.02.03.27
        for <FreeBSD-gnats-submit@freebsd.org>
        (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Wed, 23 Oct 2013 02:03:28 -0700 (PDT)
Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1])
	by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9N93Q8U078489
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO)
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 23 Oct 2013 10:03:26 +0100 (BST)
	(envelope-from mexas@mech-cluster241.men.bris.ac.uk)
Received: (from mexas@localhost)
	by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9N93QPm078488;
	Wed, 23 Oct 2013 10:03:26 +0100 (BST)
	(envelope-from mexas)
Message-Id: <201310230903.r9N93QPm078488@mech-cluster241.men.bris.ac.uk>
Date: Wed, 23 Oct 2013 10:03:26 +0100 (BST)
From: Anton Shterenlikht <mexas@bris.ac.uk>
Sender: Anton Shterenlikht <mexas@bristol.ac.uk>
Reply-To: Anton Shterenlikht <mexas@bris.ac.uk>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: panic:  uma_zfree: Freeing to non free bucket index.
X-Send-Pr-Version: 3.114
X-GNATS-Notify:

>Number:         183227
>Category:       ia64
>Synopsis:       panic:  uma_zfree: Freeing to non free bucket index.
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    marcel
>State:          patched
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 23 09:10:00 UTC 2013
>Closed-Date:    
>Last-Modified:  Sun Apr 20 02:03:46 UTC 2014
>Originator:     Anton Shterenlikht
>Release:        FreeBSD 11.0-CURRENT ia64
>Organization:
University of Bristol	
>Environment:

FreeBSD mech-as221.men.bris.ac.uk 10.0-CURRENT FreeBSD 10.0-CURRENT #12 r255488M: Tue Oct 22 15:42:23 BST 2013     root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI  ia64

	
>Description:

I managed to obtain a texdump with
# ddb scripts
lockinfo=show locks; show alllocks; show lockedvnods
kdb.enter.panic=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; textdump dump

So below are: panic.txt, version.txt, config.txt, ddb.txt, msgbuf.txt

# panic.txt #

uma_zfree: Freeing to non free bucket index.

# version.txt #

FreeBSD 10.0-CURRENT #12 r255488M: Tue Oct 22 15:42:23 BST 2013
    root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI

# config.txt

options	CONFIG_AUTOGENERATED
ident	UZI
machine	ia64
cpu	ITANIUM2
makeoptions	MODULES_OVERRIDE=geom/geom_part opensolaris zfs acl_nfs4 acl_posix1e
makeoptions	DEBUG=-g
options	USB_DEBUG
options	TEXTDUMP_VERBOSE
options	TEXTDUMP_PREFERRED
options	DIAGNOSTIC
options	DEBUG_VFS_LOCKS
options	DEBUG_LOCKS
options	MALLOC_DEBUG_MAXZONES=8
options	WITNESS_SKIPSPIN
options	WITNESS
options	INVARIANT_SUPPORT
options	INVARIANTS
options	DEADLKRES
options	GDB
options	DDB
options	KDB_TRACE
options	KDB
options	_KPOSIX_PRIORITY_SCHEDULING
options	QUOTA
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SYSVSHM
options	SYSVSEM
options	SYSVMSG
options	STACK
options	SOFTUPDATES
options	SMP
options	SCTP
options	SCSI_DELAY=5000
options	SCHED_ULE
options	PSEUDOFS
options	PROCFS
options	PROCDESC
options	PRINTF_BUFR_SIZE=128
options	PREEMPTION
options	P1003_1B_SEMAPHORES
options	NFS_ROOT
options	NFSD
options	NFSLOCKD
options	NFSCL
options	MSDOSFS
options	MD_ROOT
options	MAC
options	KTRACE
options	INET6
options	INET
options	INCLUDE_CONFIG_FILE
options	GEOM_LABEL
options	FFS
options	TMPFS
options	NULLFS
options	FDESCFS
options	COMPAT_FREEBSD7
options	CD9660
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_GPT
options	GEOM_PART_BSD
device	acpi
device	io
device	mem
device	uart_ns8250
device	miibus
device	pci
device	scbus
device	usb
device	mpt
device	cd
device	da
device	pass
device	ehci
device	ohci
device	uhci
device	umass
device	bge
device	fxp
device	ether
device	faith
device	gif
device	loop
device	md
device	puc
device	random
device	tun
device	uart
device	vlan
device	firmware


# ddb.txt #

db:0:kdb.enter.panic>  run lockinfo
db:1:lockinfo> show locks
shared lockmgr zfs (zfs) r = 0 (0xe000000012d87658) locked @ /usr/src/sys/kern/vfs_subr.c:2099
db:1:locks>  show alllocks
Process 4293 (svn) thread 0xe00000001291ed80 (100363)
shared lockmgr zfs (zfs) r = 0 (0xe000000012d87658) locked @ /usr/src/sys/kern/vfs_subr.c:2099
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.panic>  show pcpu
cpuid        = 1
dynamic pcpu = 0x400005fc2ad00
curthread    = 0xe00000001291ed80: pid 4293 "svn"
curpcb       = 0
fpcurthread  = none
idlethread   = 0xe000000010b92480: tid 100004 "idle: cpu1"
MD: vhpt       = 0xe000004040600000
MD: lid        = 0x1000000
MD: clock      = 0x1f81ddc9eb9c
MD: clock_mode = 2
MD: clock_load = 0
MD: stats      = 0xe000000010b6ea30
MD: pmap       = 0xe000000010b9f9b8
spin locks held:
db:0:kdb.enter.panic>  bt
Tracing pid 4293 tid 100363 td 0xe00000001291ed80
kdb_enter(0x9ffc000000af8d60, 0x9ffc000000af8d60, 0x9ffc0000004573f0, 0x40c) at kdb_enter+0x92
vpanic(0x9ffc000000b33428, 0xa00000009026ae88, 0x100, 0x9ffc000000bfb3c8) at vpanic+0x2b0
kassert_panic(0x9ffc000000b33428, 0x38, 0xe00000001e7eff90, 0xe0000040ffbdfb00) at kassert_panic+0x120
uma_zfree_arg(0xe0000040ffbdc380, 0xe00000001e7efac0, 0xe00000001e7eff90) at uma_zfree_arg+0x2d0
free(0xe00000001e7efac0, 0xa0000000016eda30) at free+0x170
zfs_kmem_free(0xe00000001e7efac0, 0xa00000000155f270) at zfs_kmem_free+0x30
mze_destroy(0xe00000002d458a58, 0xa00000000155fd20) at mze_destroy+0x70
zap_evict(0xe00000002fd2d110, 0xe00000002d458a00, 0xa000000001445940) at zap_evict+0x70
dbuf_evict_user(0xe00000002fd2d110, 0xe00000002fd2d250, 0xe00000002fd2d248) at dbuf_evict_user+0x130
dbuf_clear(0xe00000002fd2d110, 0xe00000002fd2d188, 0x0, 0xe00000002fcfe490, 0xe000000012dc62a0, 0xa00000000144f830) at dbuf_clear+0xc0
dbuf_hold_impl(0xe000000012dc9540, 0x0, 0x0, 0x0, 0x0, 0xa00000009026aee0) at dbuf_hold_impl+0x1d0
dbuf_hold(0xe000000012dc9540, 0x0, 0x0, 0xa000000001463fa0) at dbuf_hold+0x40
dmu_buf_hold(0xe0000000124d8800, 0xa, 0x0, 0x0, 0xa00000009026af00, 0x1) at dmu_buf_hold+0x120
zap_lockdir(0xe0000000124d8800, 0x9, 0x0, 0x1, 0x1, 0x0, 0xa00000009026af40) at zap_lockdir+0x50
zap_lookup_norm(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0x0, 0x0) at zap_lookup_norm+0x60
zap_lookup(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0xa0000000015b5800, 0xb1e) at zap_lookup+0x60
zfs_dirent_lock(0xa00000009026afa0, 0xe000000012699140, 0xa00000009026afe0, 0xa00000009026afa8, 0x6, 0x0, 0x0, 0xe000000021c2d880) at zfs_dirent_lock+0xbb0
zfs_dirlook(0xe000000012699140, 0xa00000009026afe0, 0xa00000009026b220, 0x0, 0x0, 0x0) at zfs_dirlook+0x240
zfs_lookup(0xe000000012d875f0, 0xa00000009026afe0, 0xa00000009026b220, 0xa00000009026b248, 0x0, 0xe000000010b80f00, 0xe00000001291ed80, 0x0) at zfs_lookup+0x7a0
zfs_freebsd_lookup(0xa00000009026b0e0, 0xa00000009026b248, 0xa00000009026b280) at zfs_freebsd_lookup+0x100
VOP_CACHEDLOOKUP_APV(0xa0000000016bda00, 0xa00000009026b0e0, 0x0, 0x9ffc000000b41b68, 0x9ffc00000058c860) at VOP_CACHEDLOOKUP_APV+0x270
vfs_cache_lookup(0xe000000012d875f0, 0x0, 0xa00000009026b248) at vfs_cache_lookup+0x1f0
VOP_LOOKUP_APV(0xa0000000016bda00, 0xa00000009026b118, 0x0, 0x9ffc000000b41bc8) at VOP_LOOKUP_APV+0x270
lookup(0xa00000009026b1c0, 0x200000, 0xe000000012d875f0, 0x0, 0xa00000009026b250) at lookup+0xe20
namei(0xa00000009026b1c0, 0xe00000001291a000, 0x4000144, 0x0, 0xa00000009026b230, 0xa00000009026b278, 0xa00000009026b270, 0xa00000009026b220) at namei+0xb50
kern_statat_vnhook(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320, 0x0) at kern_statat_vnhook+0xf0
kern_statat(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_statat+0x40
kern_stat(0xe00000001291ed80, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_stat+0x40
sys_stat(0xe00000001291ed80, 0xa00000009026b4e8, 0x9ffc000000986c80, 0x48d) at sys_stat+0x30
syscall(0xe00000001291a000, 0x7fffffffffffe1f0, 0x1209936e0, 0xe00000001291ed80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
db:0:kdb.enter.panic>  ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
 4293  4265  4265     0  R+      CPU 1                       svn
 4265  4204  4265     0  S+      wait     0xe0000000122a3720 sh
 4205  1004  4205     0  S+      ttyin    0xe000000010c348a8 sh
 4204  4203  4204     0  S+      wait     0xe000000012224000 sh
 4203  4187  4203  1001  S+      wait     0xe0000000122b6de0 su
 4187  4186  4187  1001  Ss+     pause    0xe000000012a4f7c8 tcsh
 4186  4184  4184  1001  S       select   0xe0000000122eaf40 sshd
 4184   935  4184     0  Ss      select   0xe0000000122eae40 sshd
 1004     1  1004     0  Ss+     wait     0xe000000012a6c4a0 login
 1003     1     1     0  S       ttydcd   0xe000000010c34ce8 getty
  982     1   982     0  Ss      select   0xe0000000122eb0c0 inetd
  945     1   945     0  Ss      nanslp   0x9ffc000000c23588 cron
  941     1   941    25  Ss      pause    0xe0000000122a29e8 sendmail
  938     1   938     0  Ss      select   0xe000000010831540 sendmail
  935     1   935     0  Ss      select   0xe0000000122eb6c0 sshd
  876     1   876     0  Ss      select   0xe0000000108316c0 ntpd
  795     1   795     0  Ss      select   0xe0000000122eb740 syslogd
  657     1   657     0  Ss      select   0xe000000010831740 devd
  156     1   156     0  Ss      pause    0xe000000012224548 adjkerntz
   40     0     0     0  DL      (threaded)                  [zfskern]
100348                   D       zio->io_ 0xe00000001245e6e8 [txg_thread_enter]
100347                   D       tx->tx_q 0xe000000012475a38 [txg_thread_enter]
100345                   D       spa->spa 0xe0000000124049f8 [trim tank]
100068                   D       l2arc_fe 0xa0000000016d1ec0 [l2arc_feed_thread]
100067                   D       arc_recl 0xa0000000016c20a0 [arc_reclaim_thread]
   21     0     0     0  DL      sdflush  0x9ffc000000bfc808 [softdepflush]
   20     0     0     0  DL      vlruwt   0xe000000012224de0 [vnlru]
   19     0     0     0  DL      syncer   0x9ffc000000e18e90 [syncer]
   18     0     0     0  DL      psleep   0x9ffc000000bfbba0 [bufdaemon]
   17     0     0     0  DL      pgzero   0x9ffc000000bfcbac [pagezero]
    9     0     0     0  DL      psleep   0x9ffc000000bfcb14 [vmdaemon]
    8     0     0     0  DL      psleep   0x9ffc000000bfcb44 [pagedaemon]
    7     0     0     0  DL      ccb_scan 0x9ffc000000bfcea0 [xpt_thrd]
    4     0     0     0  DL      waiting_ 0x9ffc000000e1b530 [sctp_iterator]
    3     0     0     0  DL      idle     0xa000000000a12000 [mpt_recovery1]
    2     0     0     0  DL      idle     0xa0000000009fa000 [mpt_recovery0]
   16     0     0     0  DL      (threaded)                  [usb]
100039                   D       -        0xa0000000009f6e18 [usbus2]
100038                   D       -        0xa0000000009f6dc0 [usbus2]
100037                   D       -        0xa0000000009f6d68 [usbus2]
100036                   D       -        0xa0000000009f6d10 [usbus2]
100034                   D       -        0xa0000000009ed460 [usbus1]
100033                   D       -        0xa0000000009ed408 [usbus1]
100032                   D       -        0xa0000000009ed3b0 [usbus1]
100031                   D       -        0xa0000000009ed358 [usbus1]
100029                   D       -        0xa0000000009e9460 [usbus0]
100028                   D       -        0xa0000000009e9408 [usbus0]
100027                   D       -        0xa0000000009e93b0 [usbus0]
100026                   D       -        0xa0000000009e9358 [usbus0]
   15     0     0     0  DL      tzpoll   0x9ffc000000bfa5c8 [acpi_thermal]
   14     0     0     0  DL      -        0x9ffc000000bfa840 [rand_harvestq]
   13     0     0     0  DL      (threaded)                  [geom]
100011                   D       -        0x9ffc000000bfad68 [g_down]
100010                   D       -        0x9ffc000000bfad60 [g_up]
100009                   D       -        0x9ffc000000bfad50 [g_event]
   12     0     0     0  WL      (threaded)                  [intr]
100046                   I                                   [swi0: uart uart+++]
100045                   I                                   [irq29: bge0]
100043                   I                                   [irq28: mpt1]
100041                   I                                   [irq27: mpt0]
100040                   I                                   [irq20: fxp0]
100035                   I                                   [irq18: ehci0]
100030                   I                                   [irq17: ohci1]
100025                   I                                   [irq16: ohci0]
100021                   I                                   [swi6: task queue]
100020                   I                                   [swi6: Giant taskq]
100018                   I                                   [swi5: fast taskq]
100017                   I                                   [swi2: cambio]
100008                   I                                   [swi4: clock]
100007                   I                                   [swi4: clock]
100006                   I                                   [swi3: vm]
100005                   I                                   [swi1: netisr 0]
   11     0     0     0  RL      (threaded)                  [idle]
100004                   CanRun                              [idle: cpu1]
100003                   Run     CPU 0                       [idle: cpu0]
    1     0     1     0  SLs     wait     0xe000000010b90de0 [init]
   10     0     0     0  DL      audit_wo 0x9ffc000000e1fbc8 [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100380                   D       -        0xe000000012685600 [zil_clean]
100379                   D       -        0xe000000012686800 [zil_clean]
100356                   D       -        0xe0000000122f2700 [zil_clean]
100355                   D       -        0xe000000012686a00 [zil_clean]
100354                   D       -        0xe0000000122f2b00 [zil_clean]
100353                   D       -        0xe0000000125ed900 [zil_clean]
100352                   D       -        0xe000000012686d00 [zil_clean]
100346                   D       -        0xe0000000122f9400 [zfs_vn_rele_taskq]
100344                   D       -        0xe0000000122f8600 [zio_ioctl_intr]
100343                   D       -        0xe000000011c3d700 [zio_ioctl_issue]
100342                   D       -        0xe0000000122f8900 [zio_claim_intr]
100341                   D       -        0xe0000000122f8800 [zio_claim_issue]
100340                   D       -        0xe0000000122f9700 [zio_free_intr]
100339                   D       -        0xe0000000122f9200 [zio_free_issue_99]
100338                   D       -        0xe0000000122f9200 [zio_free_issue_98]
100337                   D       -        0xe0000000122f9200 [zio_free_issue_97]
100336                   D       -        0xe0000000122f9200 [zio_free_issue_96]
100335                   D       -        0xe0000000122f9200 [zio_free_issue_95]
100334                   D       -        0xe0000000122f9200 [zio_free_issue_94]
100333                   D       -        0xe0000000122f9200 [zio_free_issue_93]
100332                   D       -        0xe0000000122f9200 [zio_free_issue_92]
100331                   D       -        0xe0000000122f9200 [zio_free_issue_91]
100330                   D       -        0xe0000000122f9200 [zio_free_issue_90]
100329                   D       -        0xe0000000122f9200 [zio_free_issue_89]
100328                   D       -        0xe0000000122f9200 [zio_free_issue_88]
100327                   D       -        0xe0000000122f9200 [zio_free_issue_87]
100326                   D       -        0xe0000000122f9200 [zio_free_issue_86]
100325                   D       -        0xe0000000122f9200 [zio_free_issue_85]
100324                   D       -        0xe0000000122f9200 [zio_free_issue_84]
100323                   D       -        0xe0000000122f9200 [zio_free_issue_83]
100322                   D       -        0xe0000000122f9200 [zio_free_issue_82]
100321                   D       -        0xe0000000122f9200 [zio_free_issue_81]
100320                   D       -        0xe0000000122f9200 [zio_free_issue_80]
100319                   D       -        0xe0000000122f9200 [zio_free_issue_79]
100318                   D       -        0xe0000000122f9200 [zio_free_issue_78]
100317                   D       -        0xe0000000122f9200 [zio_free_issue_77]
100316                   D       -        0xe0000000122f9200 [zio_free_issue_76]
100315                   D       -        0xe0000000122f9200 [zio_free_issue_75]
100314                   D       -        0xe0000000122f9200 [zio_free_issue_74]
100313                   D       -        0xe0000000122f9200 [zio_free_issue_73]
100312                   D       -        0xe0000000122f9200 [zio_free_issue_72]
100311                   D       -        0xe0000000122f9200 [zio_free_issue_71]
100310                   D       -        0xe0000000122f9200 [zio_free_issue_70]
100309                   D       -        0xe0000000122f9200 [zio_free_issue_69]
100308                   D       -        0xe0000000122f9200 [zio_free_issue_68]
100307                   D       -        0xe0000000122f9200 [zio_free_issue_67]
100306                   D       -        0xe0000000122f9200 [zio_free_issue_66]
100305                   D       -        0xe0000000122f9200 [zio_free_issue_65]
100304                   D       -        0xe0000000122f9200 [zio_free_issue_64]
100303                   D       -        0xe0000000122f9200 [zio_free_issue_63]
100302                   D       -        0xe0000000122f9200 [zio_free_issue_62]
100301                   D       -        0xe0000000122f9200 [zio_free_issue_61]
100300                   D       -        0xe0000000122f9200 [zio_free_issue_60]
100299                   D       -        0xe0000000122f9200 [zio_free_issue_59]
100298                   D       -        0xe0000000122f9200 [zio_free_issue_58]
100297                   D       -        0xe0000000122f9200 [zio_free_issue_57]
100296                   D       -        0xe0000000122f9200 [zio_free_issue_56]
100295                   D       -        0xe0000000122f9200 [zio_free_issue_55]
100294                   D       -        0xe0000000122f9200 [zio_free_issue_54]
100293                   D       -        0xe0000000122f9200 [zio_free_issue_53]
100292                   D       -        0xe0000000122f9200 [zio_free_issue_52]
100291                   D       -        0xe0000000122f9200 [zio_free_issue_51]
100290                   D       -        0xe0000000122f9200 [zio_free_issue_50]
100289                   D       -        0xe0000000122f9200 [zio_free_issue_49]
100288                   D       -        0xe0000000122f9200 [zio_free_issue_48]
100287                   D       -        0xe0000000122f9200 [zio_free_issue_47]
100286                   D       -        0xe0000000122f9200 [zio_free_issue_46]
100285                   D       -        0xe0000000122f9200 [zio_free_issue_45]
100284                   D       -        0xe0000000122f9200 [zio_free_issue_44]
100283                   D       -        0xe0000000122f9200 [zio_free_issue_43]
100282                   D       -        0xe0000000122f9200 [zio_free_issue_42]
100281                   D       -        0xe0000000122f9200 [zio_free_issue_41]
100280                   D       -        0xe0000000122f9200 [zio_free_issue_40]
100279                   D       -        0xe0000000122f9200 [zio_free_issue_39]
100278                   D       -        0xe0000000122f9200 [zio_free_issue_38]
100277                   D       -        0xe0000000122f9200 [zio_free_issue_37]
100276                   D       -        0xe0000000122f9200 [zio_free_issue_36]
100275                   D       -        0xe0000000122f9200 [zio_free_issue_35]
100274                   D       -        0xe0000000122f9200 [zio_free_issue_34]
100273                   D       -        0xe0000000122f9200 [zio_free_issue_33]
100272                   D       -        0xe0000000122f9200 [zio_free_issue_32]
100271                   D       -        0xe0000000122f9200 [zio_free_issue_31]
100270                   D       -        0xe0000000122f9200 [zio_free_issue_30]
100269                   D       -        0xe0000000122f9200 [zio_free_issue_29]
100268                   D       -        0xe0000000122f9200 [zio_free_issue_28]
100267                   D       -        0xe0000000122f9200 [zio_free_issue_27]
100266                   D       -        0xe0000000122f9200 [zio_free_issue_26]
100265                   D       -        0xe0000000122f9200 [zio_free_issue_25]
100264                   D       -        0xe0000000122f9200 [zio_free_issue_24]
100263                   D       -        0xe0000000122f9200 [zio_free_issue_23]
100262                   D       -        0xe0000000122f9200 [zio_free_issue_22]
100261                   D       -        0xe0000000122f9200 [zio_free_issue_21]
100260                   D       -        0xe0000000122f9200 [zio_free_issue_20]
100259                   D       -        0xe0000000122f9200 [zio_free_issue_19]
100258                   D       -        0xe0000000122f9200 [zio_free_issue_18]
100257                   D       -        0xe0000000122f9200 [zio_free_issue_17]
100256                   D       -        0xe0000000122f9200 [zio_free_issue_16]
100255                   D       -        0xe0000000122f9200 [zio_free_issue_15]
100254                   D       -        0xe0000000122f9200 [zio_free_issue_14]
100253                   D       -        0xe0000000122f9200 [zio_free_issue_13]
100252                   D       -        0xe0000000122f9200 [zio_free_issue_12]
100251                   D       -        0xe0000000122f9200 [zio_free_issue_11]
100250                   D       -        0xe0000000122f9200 [zio_free_issue_10]
100249                   D       -        0xe0000000122f9200 [zio_free_issue_9]
100248                   D       -        0xe0000000122f9200 [zio_free_issue_8]
100247                   D       -        0xe0000000122f9200 [zio_free_issue_7]
100246                   D       -        0xe0000000122f9200 [zio_free_issue_6]
100245                   D       -        0xe0000000122f9200 [zio_free_issue_5]
100244                   D       -        0xe0000000122f9200 [zio_free_issue_4]
100243                   D       -        0xe0000000122f9200 [zio_free_issue_3]
100242                   D       -        0xe0000000122f9200 [zio_free_issue_2]
100241                   D       -        0xe0000000122f9200 [zio_free_issue_1]
100240                   D       -        0xe0000000122f9200 [zio_free_issue_0]
100239                   D       -        0xe0000000122f9100 [zio_write_intr_high]
100238                   D       -        0xe0000000122f9100 [zio_write_intr_high]
100237                   D       -        0xe0000000122f9100 [zio_write_intr_high]
100236                   D       -        0xe0000000122f9100 [zio_write_intr_high]
100235                   D       -        0xe0000000122f9100 [zio_write_intr_high]
100234                   D       -        0xe0000000122f9000 [zio_write_intr_7]
100233                   D       -        0xe0000000122f9000 [zio_write_intr_6]
100232                   D       -        0xe0000000122f9000 [zio_write_intr_5]
100231                   D       -        0xe0000000122f9000 [zio_write_intr_4]
100230                   D       -        0xe0000000122f9000 [zio_write_intr_3]
100229                   D       -        0xe0000000122f9000 [zio_write_intr_2]
100228                   D       -        0xe0000000122f9000 [zio_write_intr_1]
100227                   D       -        0xe0000000122f9000 [zio_write_intr_0]
100226                   D       -        0xe0000000122f8f00 [zio_write_issue_hig]
100225                   D       -        0xe0000000122f8f00 [zio_write_issue_hig]
100224                   D       -        0xe0000000122f8f00 [zio_write_issue_hig]
100223                   D       -        0xe0000000122f8f00 [zio_write_issue_hig]
100222                   D       -        0xe0000000122f8f00 [zio_write_issue_hig]
100221                   D       -        0xe0000000122f8e00 [zio_write_issue_1]
100220                   D       -        0xe0000000122f8e00 [zio_write_issue_0]
100219                   D       -        0xe0000000122f8d00 [zio_read_intr_1]
100218                   D       -        0xe0000000122f8d00 [zio_read_intr_0]
100217                   D       -        0xe0000000122f8c00 [zio_read_issue_7]
100216                   D       -        0xe0000000122f8c00 [zio_read_issue_6]
100215                   D       -        0xe0000000122f8c00 [zio_read_issue_5]
100214                   D       -        0xe0000000122f8c00 [zio_read_issue_4]
100213                   D       -        0xe0000000122f8c00 [zio_read_issue_3]
100212                   D       -        0xe0000000122f8c00 [zio_read_issue_2]
100211                   D       -        0xe0000000122f8c00 [zio_read_issue_1]
100210                   D       -        0xe0000000122f8c00 [zio_read_issue_0]
100209                   D       -        0xe0000000122f8b00 [zio_null_intr]
100208                   D       -        0xe0000000122f8a00 [zio_null_issue]
100066                   D       -        0xe0000000122f9900 [system_taskq_1]
100065                   D       -        0xe0000000122f9900 [system_taskq_0]
100047                   D       -        0x9ffc000000c22320 [deadlkres]
100023                   D       -        0xe000000010bc7300 [kqueue taskq]
100022                   D       -        0xe000000010bc7400 [ffs_trim taskq]
100019                   D       -        0xe000000010bc7700 [thread taskq]
100016                   D       -        0xe000000010bc7a00 [acpi_task_2]
100015                   D       -        0xe000000010bc7a00 [acpi_task_1]
100014                   D       -        0xe000000010bc7a00 [acpi_task_0]
100012                   D       -        0xe000000010bc7d00 [firmware taskq]
100000                   D       swapin   0x9ffc000000c15ed8 [swapper]
db:0:kdb.enter.panic>  alltrace

Tracing command svn pid 4293 tid 100363 td 0xe00000001291ed80
kdb_enter(0x9ffc000000af8d60, 0x9ffc000000af8d60, 0x9ffc0000004573f0, 0x40c) at kdb_enter+0x92
vpanic(0x9ffc000000b33428, 0xa00000009026ae88, 0x100, 0x9ffc000000bfb3c8) at vpanic+0x2b0
kassert_panic(0x9ffc000000b33428, 0x38, 0xe00000001e7eff90, 0xe0000040ffbdfb00) at kassert_panic+0x120
uma_zfree_arg(0xe0000040ffbdc380, 0xe00000001e7efac0, 0xe00000001e7eff90) at uma_zfree_arg+0x2d0
free(0xe00000001e7efac0, 0xa0000000016eda30) at free+0x170
zfs_kmem_free(0xe00000001e7efac0, 0xa00000000155f270) at zfs_kmem_free+0x30
mze_destroy(0xe00000002d458a58, 0xa00000000155fd20) at mze_destroy+0x70
zap_evict(0xe00000002fd2d110, 0xe00000002d458a00, 0xa000000001445940) at zap_evict+0x70
dbuf_evict_user(0xe00000002fd2d110, 0xe00000002fd2d250, 0xe00000002fd2d248) at dbuf_evict_user+0x130
dbuf_clear(0xe00000002fd2d110, 0xe00000002fd2d188, 0x0, 0xe00000002fcfe490, 0xe000000012dc62a0, 0xa00000000144f830) at dbuf_clear+0xc0
dbuf_hold_impl(0xe000000012dc9540, 0x0, 0x0, 0x0, 0x0, 0xa00000009026aee0) at dbuf_hold_impl+0x1d0
dbuf_hold(0xe000000012dc9540, 0x0, 0x0, 0xa000000001463fa0) at dbuf_hold+0x40
dmu_buf_hold(0xe0000000124d8800, 0xa, 0x0, 0x0, 0xa00000009026af00, 0x1) at dmu_buf_hold+0x120
zap_lockdir(0xe0000000124d8800, 0x9, 0x0, 0x1, 0x1, 0x0, 0xa00000009026af40) at zap_lockdir+0x50
zap_lookup_norm(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0x0, 0x0) at zap_lookup_norm+0x60
zap_lookup(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0xa0000000015b5800, 0xb1e) at zap_lookup+0x60
zfs_dirent_lock(0xa00000009026afa0, 0xe000000012699140, 0xa00000009026afe0, 0xa00000009026afa8, 0x6, 0x0, 0x0, 0xe000000021c2d880) at zfs_dirent_lock+0xbb0
zfs_dirlook(0xe000000012699140, 0xa00000009026afe0, 0xa00000009026b220, 0x0, 0x0, 0x0) at zfs_dirlook+0x240
zfs_lookup(0xe000000012d875f0, 0xa00000009026afe0, 0xa00000009026b220, 0xa00000009026b248, 0x0, 0xe000000010b80f00, 0xe00000001291ed80, 0x0) at zfs_lookup+0x7a0
zfs_freebsd_lookup(0xa00000009026b0e0, 0xa00000009026b248, 0xa00000009026b280) at zfs_freebsd_lookup+0x100
VOP_CACHEDLOOKUP_APV(0xa0000000016bda00, 0xa00000009026b0e0, 0x0, 0x9ffc000000b41b68, 0x9ffc00000058c860) at VOP_CACHEDLOOKUP_APV+0x270
vfs_cache_lookup(0xe000000012d875f0, 0x0, 0xa00000009026b248) at vfs_cache_lookup+0x1f0
VOP_LOOKUP_APV(0xa0000000016bda00, 0xa00000009026b118, 0x0, 0x9ffc000000b41bc8) at VOP_LOOKUP_APV+0x270
lookup(0xa00000009026b1c0, 0x200000, 0xe000000012d875f0, 0x0, 0xa00000009026b250) at lookup+0xe20
namei(0xa00000009026b1c0, 0xe00000001291a000, 0x4000144, 0x0, 0xa00000009026b230, 0xa00000009026b278, 0xa00000009026b270, 0xa00000009026b220) at namei+0xb50
kern_statat_vnhook(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320, 0x0) at kern_statat_vnhook+0xf0
kern_statat(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_statat+0x40
kern_stat(0xe00000001291ed80, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_stat+0x40
sys_stat(0xe00000001291ed80, 0xa00000009026b4e8, 0x9ffc000000986c80, 0x48d) at sys_stat+0x30
syscall(0xe00000001291a000, 0x7fffffffffffe1f0, 0x1209936e0, 0xe00000001291ed80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 4265 tid 100060 td 0xe0000000122ac000
cpu_switch(0xe0000000122ac000, 0xe000000010b92900, 0x9ffc000000c738b8, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe0000000122ac000, 0xe000000010b92900, 0x9ffc000000c3d818, 0x9ffc000000c3d800) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000122ac000, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe0000000122a3720, 0x6c, 0xe0000000122ac378, 0xe0000000122ac000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000122a3720, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000122a3720, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
_sleep(0xe0000000122a3720, 0xe0000000122a3820, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000000122ac000, 0x7, 0x0, 0xa00000008fe3b390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000000122ac000, 0xffffffff, 0xa00000008fe3b390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000000122ac000, 0xa00000008fe3b4e8, 0xa00000008fe3b390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe0000000122a3720, 0x7fffffffffffe790, 0x0, 0xe0000000122ac000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 4205 tid 100371 td 0xe00000001270ad80
cpu_switch(0xe00000001270ad80, 0xe000000010b92480, 0x9ffc000000c75820, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001270ad80, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001270ad80, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe000000010c348a8, 0x0, 0xe00000001270b0f8, 0xe00000001270ad80) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000010c348a8, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000010c348a8, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30
_cv_wait_sig(0xe000000010c348a8, 0xe000000010c34808, 0xe00000001270ad80, 0x0) at _cv_wait_sig+0x4d0
tty_wait(0xe000000010c34800, 0xe000000010c348a8, 0x2, 0x9ffc000000b06ef8, 0x9ffc00000053a020) at tty_wait+0xb0
ttydisc_read(0xe000000010c34800, 0xa000000090293360, 0x0, 0x0) at ttydisc_read+0x4c0
ttydev_read(0xe000000010c34800, 0xa000000090293360, 0x0, 0x9ffc000000246cc0) at ttydev_read+0xe0
devfs_read_f(0xe000000012376cd0, 0xa000000090293360, 0xe000000010b82700, 0x0, 0xe00000001270ad80) at devfs_read_f+0xf0
dofileread(0xe00000001270ad80, 0x0, 0xe000000012376cd0, 0xa000000090293360, 0x0, 0x0) at dofileread+0x150
kern_readv(0xe00000001270ad80, 0x0, 0xa000000090293360) at kern_readv+0xa0
sys_read(0xe00000001270ad80, 0xa0000000902934e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
syscall(0xe000000012a4e000, 0x7fffffffffffec50, 0x1, 0xe00000001270ad80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sh pid 4204 tid 100050 td 0xe000000011972d80
cpu_switch(0xe000000011972d80, 0xe000000010b92480, 0x9ffc000000c73f20, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe000000011972d80, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000011972d80, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe000000012224000, 0x6c, 0xe0000000119730f8, 0xe000000011972d80) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012224000, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012224000, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
_sleep(0xe000000012224000, 0xe000000012224100, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe000000011972d80, 0x7, 0x0, 0xa00000008fdeb390, 0x36, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe000000011972d80, 0xffffffff, 0xa00000008fdeb390, 0x6, 0x0) at kern_wait+0xe0
sys_wait4(0xe000000011972d80, 0xa00000008fdeb4e8, 0xa00000008fdeb390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000012224000, 0x7fffffffffffe8b0, 0x6, 0xe000000011972d80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command su pid 4203 tid 100062 td 0xe0000000122adb00
cpu_switch(0xe0000000122adb00, 0xe000000010b92480, 0x9ffc000000c74b28, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe0000000122adb00, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000122adb00, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe0000000122b6de0, 0x6c, 0xe0000000122ade78, 0xe0000000122adb00) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000122b6de0, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000122b6de0, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
_sleep(0xe0000000122b6de0, 0xe0000000122b6ee0, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe0000000122adb00, 0x0, 0x106c, 0xa00000008fe4b390, 0x32, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe0000000122adb00, 0x106c, 0xa00000008fe4b390, 0x2, 0x0) at kern_wait+0xe0
sys_wait4(0xe0000000122adb00, 0xa00000008fe4b4e8, 0xa00000008fe4b390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe0000000122b6de0, 0x7fffffffffffe4c0, 0x2, 0xe0000000122adb00, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command tcsh pid 4187 tid 100375 td 0xe00000001298c480
cpu_switch(0xe00000001298c480, 0xe000000010b92480, 0x9ffc000000c73ef8, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001298c480, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001298c480, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe000000012a4f7c8, 0x74, 0xe00000001298c7f8, 0xe00000001298c480) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012a4f7c8, 0x74, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012a4f7c8, 0x74, 0x9ffc000000ac89c8, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
_sleep(0xe000000012a4f7c8, 0xe000000012a4f820, 0x74, 0x9ffc000000ac89c8, 0x0, 0x0, 0x100) at _sleep+0x7b0
kern_sigsuspend(0xe00000001298c480, 0x0, 0xe000000012a4f7c8) at kern_sigsuspend+0x140
sys_sigsuspend(0xe00000001298c480, 0xa0000000902b34e8, 0x9ffc000000986c80, 0x48d) at sys_sigsuspend+0x60
syscall(0xe000000012a4f720, 0x0, 0x10004f5d0, 0xe00000001298c480, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sshd pid 4186 tid 100370 td 0xe00000001270b200
cpu_switch(0xe00000001270b200, 0xe000000010b92480, 0x9ffc000000c75a78, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001270b200, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001270b200, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe0000000122eaf40, 0x0, 0xe00000001270b578, 0xe00000001270b200) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000122eaf40, 0x0, 0x9ffc000000b00730, 0xe0000000122eaf20) at sleepq_catch_signals+0x190
sleepq_timedwait_sig(0xe0000000122eaf40, 0x0, 0x9ffc0000003bf0b0, 0x9ffc0000003bf0f0, 0x713) at sleepq_timedwait_sig+0x30
_cv_timedwait_sig_sbt(0xe0000000122eaf40, 0xe0000000122eaf20, 0x5a62b60182b9, 0x12c0000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0
seltdwait(0xe0000000122eaf00, 0x5a62b60182b9, 0x12c0000000, 0xe0000000122eaf38, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0xf0
kern_select(0xe00000001270b200, 0x0, 0x121416228, 0x121416230, 0x0, 0x5a62b60182b9, 0xa) at kern_select+0xb40
sys_select(0xe00000001270b200, 0xa00000009028b4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe000000012a4e4a0, 0x121416228, 0x121416230, 0xe00000001270b200, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sshd pid 4184 tid 100061 td 0xe00000001233a000
cpu_switch(0xe00000001233a000, 0xe000000010b92480, 0x9ffc000000c75a50, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001233a000, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001233a000, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe0000000122eae40, 0x0, 0xe00000001233a378, 0xe00000001233a000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000122eae40, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000122eae40, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30
_cv_wait_sig(0xe0000000122eae40, 0xe0000000122eae20, 0xe00000001233a000, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0
seltdwait(0xe0000000122eae00, 0xffffffffffffffff, 0x0, 0xe0000000122eae38) at seltdwait+0x110
sys_poll(0xe00000001233a000, 0xa00000008fe434e8, 0xe000000012963170, 0xa00000008fe43290) at sys_poll+0x620
syscall(0xe0000000122b7280, 0x1, 0xffffffffffffffff, 0xe00000001233a000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command login pid 1004 tid 100373 td 0xe00000001298c900
cpu_switch(0xe00000001298c900, 0xe000000010b92480, 0x9ffc000000c744c0, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001298c900, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001298c900, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe000000012a6c4a0, 0x6c, 0xe00000001298cc78, 0xe00000001298c900) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012a6c4a0, 0x6c, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000012a6c4a0, 0x6c, 0x9ffc000000b2fb20, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
_sleep(0xe000000012a6c4a0, 0xe000000012a6c5a0, 0x6c, 0x9ffc000000b2fb20, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
kern_wait6(0xe00000001298c900, 0x0, 0x106d, 0xa0000000902a3390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
kern_wait(0xe00000001298c900, 0x106d, 0xa0000000902a3390, 0x0, 0x0) at kern_wait+0xe0
sys_wait4(0xe00000001298c900, 0xa0000000902a34e8, 0xa0000000902a3390, 0x9ffc000000986c80) at sys_wait4+0x70
syscall(0xe000000012a6c4a0, 0x7fffffffffffece0, 0x0, 0xe00000001298c900, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command getty pid 1003 tid 100358 td 0xe000000012531b00
cpu_switch(0xe000000012531b00, 0xe000000010b92480, 0x9ffc000000c74ec0, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe000000012531b00, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012531b00, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe000000010c34ce8, 0x0, 0xe000000012531e78, 0xe000000012531b00) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000010c34ce8, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe000000010c34ce8, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30
_cv_wait_sig(0xe000000010c34ce8, 0xe000000010c34c08, 0xe000000012531b00, 0x0) at _cv_wait_sig+0x4d0
tty_wait(0xe000000010c34c00, 0xe000000010c34ce8, 0x1, 0x9ffc000000b06ef8, 0x9ffc00000052f250, 0x694, 0xe00000001235d800) at tty_wait+0xb0
ttydev_open(0xe000000010bb2200, 0x9ffc000000b06f30, 0x2000, 0x9ffc000000b07018, 0xe000000010c34c00) at ttydev_open+0x710
devfs_open(0xa000000090243158, 0x0, 0xa000000090243118) at devfs_open+0x2c0
VOP_OPEN_APV(0x9ffc000000b866e0, 0xa000000090243158, 0x0, 0x9ffc000000b41b10) at VOP_OPEN_APV+0x270
vn_open_vnode(0xe000000012a3a000, 0x3, 0xe0000000122e4200, 0xe000000012531b00, 0xe0000000123536d0) at vn_open_vnode+0x2c0
vn_open_cred(0xa000000090243288, 0xa000000090243390, 0x980, 0x0, 0xe0000000122e4200, 0xe0000000123536d0) at vn_open_cred+0x520
vn_open(0xa000000090243288, 0xa000000090243390, 0x980, 0xe0000000123536d0, 0x9ffc0000005c80d0, 0x695, 0x0) at vn_open+0x50
kern_openat(0xe000000012531b00, 0xffffffffffffff9c, 0x10001ca90, 0x0, 0x0, 0x7980) at kern_openat+0x490
kern_open(0xe000000012531b00, 0x10001ca90, 0x0, 0x2, 0x7980) at kern_open+0x40
sys_open(0xe000000012531b00, 0xa0000000902434e8, 0x9ffc000000986c80, 0x48d) at sys_open+0x40
syscall(0xe0000000126cc000, 0x2, 0x100007980, 0xe000000012531b00, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command inetd pid 982 tid 100372 td 0xe000000012a68480
cpu_switch(0xe000000012a68480, 0xe00000001291ed80, 0x9ffc000000c746a0, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe000000012a68480, 0xe00000001291ed80, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000012a68480, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe0000000122eb0c0, 0x0, 0xe000000012a687f8, 0xe000000012a68480) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000122eb0c0, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000122eb0c0, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30
_cv_wait_sig(0xe0000000122eb0c0, 0xe0000000122eb0a0, 0xe000000012a68480, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0
seltdwait(0xe0000000122eb080, 0xffffffffffffffff, 0x0, 0xe0000000122eb0b8, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0x110
kern_select(0xe000000012a68480, 0x0, 0x7fffffffffffe0e0, 0x0, 0x0, 0xffffffffffffffff, 0x6) at kern_select+0xb40
sys_select(0xe000000012a68480, 0xa00000009029b4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe00000001291b720, 0x7fffffffffffe0e0, 0x0, 0xe000000012a68480, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command cron pid 945 tid 100359 td 0xe00000001291e000
cpu_switch(0xe00000001291e000, 0xe000000010b92900, 0x9ffc000000c752a8, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001291e000, 0xe000000010b92900, 0x9ffc000000c3d818, 0x9ffc000000c3d800) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001291e000, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0x9ffc000000c23588, 0x6c, 0xe00000001291e378, 0xe00000001291e000) at sleepq_switch+0x2e0
sleepq_catch_signals(0x9ffc000000c23588, 0x6c, 0x9ffc000000b00730, 0xe00000001291e390) at sleepq_catch_signals+0x190
sleepq_timedwait_sig(0x9ffc000000c23588, 0x6c, 0x3bff63d09, 0x9ffc0000004697a0, 0x998) at sleepq_timedwait_sig+0x30
_sleep(0x9ffc000000c23588, 0x0, 0x6c, 0x9ffc000000afa9f0, 0x1, 0x3bff63d09, 0x200) at _sleep+0x750
kern_nanosleep(0xe00000001291e000, 0x3bff63d096, 0xa00000009024b380) at kern_nanosleep+0x2a0
sys_nanosleep(0xe00000001291e000, 0xa00000009024b4e8, 0x0, 0x9ffc000000986c80) at sys_nanosleep+0xb0
syscall(0xe0000000122a2de0, 0x7fffffffffffec28, 0x0, 0xe00000001291e000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sendmail pid 941 tid 100365 td 0xe00000001270bb00
cpu_switch(0xe00000001270bb00, 0xe000000010b92480, 0x9ffc000000c75348, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001270bb00, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001270bb00, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe0000000122a29e8, 0x74, 0xe00000001270be78, 0xe00000001270bb00) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000122a29e8, 0x74, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000122a29e8, 0x74, 0x9ffc000000ac89c8, 0x100, 0x9ffc000000469800) at sleepq_wait_sig+0x30
_sleep(0xe0000000122a29e8, 0xe0000000122a2a40, 0x74, 0x9ffc000000ac89c8, 0x0, 0x0, 0x100) at _sleep+0x7b0
kern_sigsuspend(0xe00000001270bb00, 0x0, 0xe0000000122a29e8) at kern_sigsuspend+0x140
sys_sigsuspend(0xe00000001270bb00, 0xa00000009027b4e8, 0x9ffc000000986c80, 0x48d) at sys_sigsuspend+0x60
syscall(0xe0000000122a2940, 0xe00000001270bb08, 0x9ffc000000983510, 0xe00000001270bb00, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sendmail pid 938 tid 100357 td 0xe00000001270a000
cpu_switch(0xe00000001270a000, 0xe000000010b92480, 0x9ffc000000c74268, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001270a000, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001270a000, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe000000010831540, 0x0, 0xe00000001270a378, 0xe00000001270a000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000010831540, 0x0, 0x9ffc000000b00730, 0xe000000010831520) at sleepq_catch_signals+0x190
sleepq_timedwait_sig(0xe000000010831540, 0x0, 0x9ffc0000003bf0b0, 0x9ffc0000003bf0f0, 0x713) at sleepq_timedwait_sig+0x30
_cv_timedwait_sig_sbt(0xe000000010831540, 0xe000000010831520, 0x59e8045668c1, 0x50000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0
seltdwait(0xe000000010831500, 0x59e8045668c1, 0x50000000, 0xe000000010831538, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0xf0
kern_select(0xe00000001270a000, 0x0, 0x7fffffffffffc3c8, 0x0, 0x0, 0x59e8045668c1, 0x4) at kern_select+0xb40
sys_select(0xe00000001270a000, 0xa00000009023b4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe0000000126cc4a0, 0x7fffffffffffc3c8, 0x0, 0xe00000001270a000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command sshd pid 935 tid 100063 td 0xe0000000122ad680
cpu_switch(0xe0000000122ad680, 0xe000000010b92480, 0x9ffc000000c74790, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe0000000122ad680, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe0000000122ad680, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe0000000122eb6c0, 0x0, 0xe0000000122ad9f8, 0xe0000000122ad680) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000122eb6c0, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000122eb6c0, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30
_cv_wait_sig(0xe0000000122eb6c0, 0xe0000000122eb6a0, 0xe0000000122ad680, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0
seltdwait(0xe0000000122eb680, 0xffffffffffffffff, 0x0, 0xe0000000122eb6b8, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0x110
kern_select(0xe0000000122ad680, 0x0, 0x121416198, 0x0, 0x0, 0xffffffffffffffff, 0x5) at kern_select+0xb40
sys_select(0xe0000000122ad680, 0xa00000008fe534e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe0000000122b6940, 0x121416198, 0x0, 0xe0000000122ad680, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command ntpd pid 876 tid 100362 td 0xe00000001291f200
cpu_switch(0xe00000001291f200, 0xe000000010b92480, 0x9ffc000000c75690, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001291f200, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001291f200, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe0000000108316c0, 0x0, 0xe00000001291f578, 0xe00000001291f200) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000108316c0, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000108316c0, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30
_cv_wait_sig(0xe0000000108316c0, 0xe0000000108316a0, 0xe00000001291f200, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0
seltdwait(0xe000000010831680, 0xffffffffffffffff, 0x0, 0xe0000000108316b8, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0x110
kern_select(0xe00000001291f200, 0x0, 0x7fffffffffffec88, 0x0, 0x0, 0xffffffffffffffff, 0x1e) at kern_select+0xb40
sys_select(0xe00000001291f200, 0xa0000000902634e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe00000001291a4a0, 0x7fffffffffffec88, 0x0, 0xe00000001291f200, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command syslogd pid 795 tid 100064 td 0xe000000011c49b00
cpu_switch(0xe000000011c49b00, 0xe000000010b92900, 0x9ffc000000c75bb8, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe000000011c49b00, 0xe000000010b92900, 0x9ffc000000c3d818, 0x9ffc000000c3d800) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000011c49b00, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe0000000122eb740, 0x0, 0xe000000011c49e78, 0xe000000011c49b00) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe0000000122eb740, 0x0, 0x9ffc000000b00730, 0x9ffc000000433850) at sleepq_catch_signals+0x190
sleepq_wait_sig(0xe0000000122eb740, 0x0, 0x9ffc0000003bfeb0, 0x611, 0x9ffc0000003bfef0) at sleepq_wait_sig+0x30
_cv_wait_sig(0xe0000000122eb740, 0xe0000000122eb720, 0xe000000011c49b00, 0x0, 0x9ffc000000c188c0) at _cv_wait_sig+0x4d0
seltdwait(0xe0000000122eb700, 0xffffffffffffffff, 0x0, 0xe0000000122eb738, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0x110
kern_select(0xe000000011c49b00, 0x0, 0x120866168, 0x0, 0x0, 0xffffffffffffffff, 0x9) at kern_select+0xb40
sys_select(0xe000000011c49b00, 0xa00000008fe5b4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe0000000122a3280, 0x120866168, 0x0, 0xe000000011c49b00, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command devd pid 657 tid 100349 td 0xe00000001245c000
cpu_switch(0xe00000001245c000, 0xe000000010b92480, 0x9ffc000000c742b8, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe00000001245c000, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe00000001245c000, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe000000010831740, 0x0, 0xe00000001245c378, 0xe00000001245c000) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000010831740, 0x0, 0x9ffc000000b00730, 0xe000000010831720) at sleepq_catch_signals+0x190
sleepq_timedwait_sig(0xe000000010831740, 0x0, 0x9ffc0000003bf0b0, 0x9ffc0000003bf0f0, 0x713) at sleepq_timedwait_sig+0x30
_cv_timedwait_sig_sbt(0xe000000010831740, 0xe000000010831720, 0x59f0a1c6c699, 0x3c0000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0
seltdwait(0xe000000010831700, 0x59f0a1c6c699, 0x3c0000000, 0xe000000010831738, 0x9ffc000000b03f48, 0x9ffc00000050f230) at seltdwait+0xf0
kern_select(0xe00000001245c000, 0x0, 0x7fffffffffffcd00, 0x0, 0x0, 0x59f0a1c6c699, 0x5) at kern_select+0xb40
sys_select(0xe00000001245c000, 0xa0000000901fb4e8, 0x9ffc000000986c80, 0x48d) at sys_select+0xb0
syscall(0xe0000000126cd280, 0x7fffffffffffcd00, 0x0, 0xe00000001245c000, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return

Tracing command adjkerntz pid 156 tid 100049 td 0xe000000011973200
cpu_switch(0xe000000011973200, 0xe000000010b92480, 0x9ffc000000c73728, 0x9ffc0000004b0d70) at cpu_switch+0xd0
sched_switch(0xe000000011973200, 0xe000000010b92480, 0x9ffc000000c3e518, 0x9ffc000000c3e500) at sched_switch+0x890
mi_switch(0x104, 0x0, 0xe000000011973200, 0x9ffc0000004e9a60) at mi_switch+0x3f0
sleepq_switch(0xe000000012224548, 0x74, 0xe000000011973578, 0xe000000011973200) at sleepq_switch+0x2e0
sleepq_catch_signals(0xe000000012224548, 0x74, 0x9ffc000000b00730, 0x9ffc000000b00af0) at sleepq_catch_signals+0x190
sle

# msbuf.txt #

GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #12 r255488M: Tue Oct 22 15:42:23 BST 2013
    root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI ia64
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
CPU: Madison (1500 MHz Itanium 2)
  Origin = "GenuineIntel"  Revision = 5
  Features = 0x1<LB>
real memory  = 6442450944 (6144 MB)
avail memory = 6286475264 (5995 MB)
FPSWA Revision = 0x10012, Entry = 0xe0000040ffe60050
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP)
cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0
random: <Software, Yarrow> initialized
Event timer "ITC" frequency 1500000000 Hz quality 1000
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603)
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603)
acpi0: <HP> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0
acpi_tz0: <Thermal Zone> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pcib0: could not evaluate _ADR - AE_NOT_FOUND
pci0: <ACPI PCI bus> on pcib0
ohci0: <NEC uPD 9210 USB controller> mem 0x80023000-0x80023fff irq 16 at device 1.0 on pci0
usbus0 on ohci0
ohci1: <NEC uPD 9210 USB controller> mem 0x80022000-0x80022fff irq 17 at device 1.1 on pci0
usbus1 on ohci1
ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x80021000-0x800210ff irq 18 at device 1.2 on pci0
usbus2: EHCI version 0.95
usbus2 on ehci0
pci0: <mass storage, ATA> at device 2.0 (no driver attached)
fxp0: <Intel 82550C Pro/100 Ethernet> port 0xd00-0xd3f mem 0x80020000-0x80020fff,0x80000000-0x8001ffff irq 20 at device 3.0 on pci0
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
fxp0: Ethernet address: 00:11:0a:31:e6:0a
pcib1: <ACPI Host-PCI bridge> on acpi0
pcib1: could not evaluate _ADR - AE_NOT_FOUND
pci32: <ACPI PCI bus> on pcib1
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2100-0x21ff mem 0x90840000-0x9084ffff,0x90830000-0x9083ffff irq 27 at device 1.0 on pci32
mpt0: MPI Version=1.2.12.0
mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0x90820000-0x9082ffff,0x90810000-0x9081ffff irq 28 at device 1.1 on pci32
mpt1: MPI Version=1.2.12.0
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x000105> mem 0x90800000-0x9080ffff irq 29 at device 2.0 on pci32
bge0: CHIP ID 0x00000105; ASIC REV 0x00; CHIP REV 0x01; PCI on PCI-X 66 MHz; 64bit
miibus1: <MII bus> on bge0
brgphy0: <BCM5701 1000BASE-T media interface> PHY 1 on miibus1
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 00:11:0a:31:46:06
pcib2: <ACPI Host-PCI bridge> on acpi0
pcib2: could not evaluate _ADR - AE_NOT_FOUND
pci64: <ACPI PCI bus> on pcib2
pcib3: <ACPI Host-PCI bridge> on acpi0
pcib3: could not evaluate _ADR - AE_NOT_FOUND
pci96: <ACPI PCI bus> on pcib3
pcib4: <ACPI Host-PCI bridge> on acpi0
pcib4: could not evaluate _ADR - AE_NOT_FOUND
pci128: <ACPI PCI bus> on pcib4
pcib5: <ACPI Host-PCI bridge> on acpi0
pcib5: could not evaluate _ADR - AE_NOT_FOUND
pci192: <ACPI PCI bus> on pcib5
pcib6: <ACPI Host-PCI bridge> on acpi0
pcib6: could not evaluate _ADR - AE_NOT_FOUND
pci224: <ACPI PCI bus> on pcib6
uart0: <HP Auxiliary Diva Serial Port> mem 0xf4051000-0xf405100f irq 82 at device 1.0 on pci224
puc0: <HP Diva Serial [GSP] Multiport UART - Everest SP2> mem 0xf4050000-0xf4050fff,0xf4020000-0xf403ffff irq 82 at device 1.1 on pci224
uart1: <Non-standard ns8250 class UART with FIFOs> at port 1 on puc0
uart1: console (9600,n,8,1)
uart2: <Non-standard ns8250 class UART with FIFOs> at port 3 on puc0
vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf0000000-0xf3ffffff,0xf4040000-0xf404ffff at device 2.0 on pci224
uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0
uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0
uart4: debug port (9600,n,8,1)
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
ugen1.1: <NEC> at usbus1
ugen2.1: <NEC> at usbus2
uhub0: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub1: <NEC EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen0.1: <NEC> at usbus0
uhub2: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub0: 2 ports with 2 removable, self powered
uhub2: 3 ports with 3 removable, self powered
uhub1: 5 ports with 5 removable, self powered
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
da2 at mpt1 bus 0 scbus1 target 2 lun 0
da2: <SEAGATE ST373454LC D402> Fixed Direct Access SCSI-3 device 
da2: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
da2: Command Queueing enabled
da2: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C)
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <HP 73.4G MAX3073NC HPC1> Fixed Direct Access SCSI-3 device 
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <SEAGATE ST373454LC D404> Fixed Direct Access SCSI-3 device 
da1: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
da1: Command Queueing enabled
da1: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C)
Trying to mount root from ufs:/dev/da0p2 [rw]...
WARNING: / was not properly dismounted
<118>Setting hostuuid: a90e62bf-ce84-11e1-9fc5-00110a31e60a.
<118>Setting hostid: 0x688171f3.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
<118>Starting ddb.
<118>Entropy harvesting: interrupts ethernet point_to_point kickstart.
<118>Starting file system checks:
<118>/dev/da0p2: UNREF FILE I=6288403  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=174 MTIME=Oct 22 20:59 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288425  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=1290 MTIME=Oct 23 00:30 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288428  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=444 MTIME=Oct 23 02:58 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288429  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=1886 MTIME=Oct 23 02:41 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288430  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=283 MTIME=Oct 23 02:41 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288431  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=288 MTIME=Oct 23 02:43 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288432  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=2052 MTIME=Oct 23 02:43 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288433  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=468 MTIME=Oct 23 03:03 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE  I=6288436  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=0 MTIME=Oct 23 03:03 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288438  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=192 MTIME=Oct 23 03:03 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288443  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=0 MTIME=Oct 23 03:01 2013  (CLEARED)
<118>/dev/da0p2: UNREF FILE I=6288445  OWNER=root MODE=100600
<118>/dev/da0p2: SIZE=0 MTIME=Oct 23 03:01 2013  (CLEARED)
<118>/dev/da0p2: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
<118>/dev/da0p2: SUMMARY INFORMATION BAD (SALVAGED)
<118>/dev/da0p2: BLK(S) MISSING IN BIT MAPS (SALVAGED)
<118>/dev/da0p2: 672211 files, 6449503 used, 24016134 free (101838 frags, 2989287 blocks, 0.3% fragmentation)
<118>Mounting local file systems:.
<118>Writing entropy file:.
<118>Setting hostname: mech-as221.men.bris.ac.uk.
<118>Starting Network: lo0 fxp0 bge0.
<118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
<118>	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
<118>	inet6 ::1 prefixlen 128 
<118>	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
<118>	inet 127.0.0.1 netmask 0xff000000 
<118>	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
<118>fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
<118>	ether 00:11:0a:31:e6:0a
<118>	inet 10.10.10.13 netmask 0xffffff00 broadcast 10.10.10.255 
<118>	inet6 fe80::211:aff:fe31:e60a%fxp0 prefixlen 64 scopeid 0x1 
<118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118>	media: Ethernet autoselect (100baseTX <full-duplex>)
<118>	status: active
<118>bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
<118>	ether 00:11:0a:31:46:06
<118>	inet 137.222.187.221 netmask 0xffffff00 broadcast 137.222.187.255 
<118>	inet6 fe80::211:aff:fe31:4606%bge0 prefixlen 64 scopeid 0x2 
<118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118>	media: Ethernet autoselect (none)
<118>	status: no carrier
<118>Starting devd.
<118>add net default: gateway 137.222.187.250
<118>add net fe80::: gateway ::1
<118>add net ff02::: gateway ::1
<118>add net ::ffff:0.0.0.0: gateway ::1
<118>add net ::0.0.0.0: gateway ::1
<118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/gcc46 /usr/local/lib/gcc48 /usr/local/lib/graphviz
<118>Creating and/or trimming log files.
<118>Starting syslogd.
<118>savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe000000012aed200, blocked for 900014 ticks
<118>Oct 23 03:21:29 mech-as221 savecore: reboot after panic: deadlkres: possible deadlock detected for 0xe000000012aed200, blocked for 900014 ticks
<118>savecore: writing core to /var/crash/textdump.tar.4
<118>/var/crash/vmcore.4 not found
<118>Clearing /tmp (X related).
<118>Recovering vi editor sessions:.
<118>Updating motd:.
<118>Mounting late file systems:.
<118>Starting ntpd.
<118>Starting bsdstats.
<118>Posting monthly OS statistics to rpt.bsdstats.org
<118>Performing sanity check on sshd configuration.
<118>Starting sshd.
<118>Starting cron.
<118>Local package initialization:cd: /var/portbuild/: No such file or directory
<118>.
<118>Starting inetd.
<118>Starting background file system checks in 60 seconds.
<118>
<118>Wed Oct 23 03:21:34 BST 2013
Expensive timeout(9) function: 0x9ffc000000b4e130(0) 0.030631053 s
Expensive timeout(9) function: 0x9ffc000000b4e130(0) 0.092184233 s
<118>Oct 23 09:22:05 mech-as221 su: BAD SU mexas to root on /dev/pts/0
<118>Oct 23 09:22:13 mech-as221 su: mexas to root on /dev/pts/0
<118>Oct 23 09:22:26 mech-as221 login: ROOT LOGIN (root) ON ttyu1
panic: uma_zfree: Freeing to non free bucket index.
cpuid = 1
KDB: stack backtrace:
db_trace_self(0x9ffc0000000c9ec0) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc0000004d0260) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000bfb620, 0x9ffc0000004573a0, 0x40c, 0x9ffc000000dd2220) at kdb_backtrace+0xc0
vpanic(0x9ffc000000b33428, 0xa00000009026ae88, 0x9ffc000000c21820, 0x9ffc000000bfb3c8) at vpanic+0x260
kassert_panic(0x9ffc000000b33428, 0x38, 0xe00000001e7eff90, 0xe0000040ffbdfb00) at kassert_panic+0x120
uma_zfree_arg(0xe0000040ffbdc380, 0xe00000001e7efac0, 0xe00000001e7eff90) at uma_zfree_arg+0x2d0
free(0xe00000001e7efac0, 0xa0000000016eda30) at free+0x170
zfs_kmem_free(0xe00000001e7efac0, 0xa00000000155f270) at zfs_kmem_free+0x30
mze_destroy(0xe00000002d458a58, 0xa00000000155fd20) at mze_destroy+0x70
zap_evict(0xe00000002fd2d110, 0xe00000002d458a00, 0xa000000001445940) at zap_evict+0x70
dbuf_evict_user(0xe00000002fd2d110, 0xe00000002fd2d250, 0xe00000002fd2d248) at dbuf_evict_user+0x130
dbuf_clear(0xe00000002fd2d110, 0xe00000002fd2d188, 0x0, 0xe00000002fcfe490, 0xe000000012dc62a0, 0xa00000000144f830) at dbuf_clear+0xc0
dbuf_hold_impl(0xe000000012dc9540, 0x0, 0x0, 0x0, 0x0, 0xa00000009026aee0) at dbuf_hold_impl+0x1d0
dbuf_hold(0xe000000012dc9540, 0x0, 0x0, 0xa000000001463fa0) at dbuf_hold+0x40
dmu_buf_hold(0xe0000000124d8800, 0xa, 0x0, 0x0, 0xa00000009026af00, 0x1) at dmu_buf_hold+0x120
zap_lockdir(0xe0000000124d8800, 0x9, 0x0, 0x1, 0x1, 0x0, 0xa00000009026af40) at zap_lockdir+0x50
zap_lookup_norm(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0x0, 0x0) at zap_lookup_norm+0x60
zap_lookup(0xe0000000124d8800, 0x9, 0xa00000009026afe0, 0x8, 0x1, 0xa00000009026af88, 0xa0000000015b5800, 0xb1e) at zap_lookup+0x60
zfs_dirent_lock(0xa00000009026afa0, 0xe000000012699140, 0xa00000009026afe0, 0xa00000009026afa8, 0x6, 0x0, 0x0, 0xe000000021c2d880) at zfs_dirent_lock+0xbb0
zfs_dirlook(0xe000000012699140, 0xa00000009026afe0, 0xa00000009026b220, 0x0, 0x0, 0x0) at zfs_dirlook+0x240
zfs_lookup(0xe000000012d875f0, 0xa00000009026afe0, 0xa00000009026b220, 0xa00000009026b248, 0x0, 0xe000000010b80f00, 0xe00000001291ed80, 0x0) at zfs_lookup+0x7a0
zfs_freebsd_lookup(0xa00000009026b0e0, 0xa00000009026b248, 0xa00000009026b280) at zfs_freebsd_lookup+0x100
VOP_CACHEDLOOKUP_APV(0xa0000000016bda00, 0xa00000009026b0e0, 0x0, 0x9ffc000000b41b68, 0x9ffc00000058c860) at VOP_CACHEDLOOKUP_APV+0x270
vfs_cache_lookup(0xe000000012d875f0, 0x0, 0xa00000009026b248) at vfs_cache_lookup+0x1f0
VOP_LOOKUP_APV(0xa0000000016bda00, 0xa00000009026b118, 0x0, 0x9ffc000000b41bc8) at VOP_LOOKUP_APV+0x270
lookup(0xa00000009026b1c0, 0x200000, 0xe000000012d875f0, 0x0, 0xa00000009026b250) at lookup+0xe20
namei(0xa00000009026b1c0, 0xe00000001291a000, 0x4000144, 0x0, 0xa00000009026b230, 0xa00000009026b278, 0xa00000009026b270, 0xa00000009026b220) at namei+0xb50
kern_statat_vnhook(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320, 0x0) at kern_statat_vnhook+0xf0
kern_statat(0xe00000001291ed80, 0x0, 0xffffffffffffff9c, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_statat+0x40
kern_stat(0xe00000001291ed80, 0x121d2f760, 0x0, 0xa00000009026b320) at kern_stat+0x40
sys_stat(0xe00000001291ed80, 0xa00000009026b4e8, 0x9ffc000000986c80, 0x48d) at sys_stat+0x30
syscall(0xe00000001291a000, 0x7fffffffffffe1f0, 0x1209936e0, 0xe00000001291ed80, 0x0, 0x0, 0x9ffc000000983500, 0x8) at syscall+0x5e0
epc_syscall_return() at epc_syscall_return
KDB: enter: panic



The only local modification is this patch, suggested by kib@
to collect more debug data:

# svn diff /usr/src/
Index: /usr/src/sys/kern/uipc_syscalls.c
===================================================================
--- /usr/src/sys/kern/uipc_syscalls.c   (revision 255488)
+++ /usr/src/sys/kern/uipc_syscalls.c   (working copy)
@@ -2079,7 +2079,7 @@
        VM_OBJECT_WUNLOCK(obj);
        KASSERT(error != 0 || (m->wire_count > 0 && m->valid ==
            VM_PAGE_BITS_ALL),
-           ("wrong page state m %p", m));
+           ("wrong page state m %p off %#jx xfsize %d", m, off, xfsize));
        return (error);
 }
 
# 
	
>How-To-Repeat:

The panic is triggered by

poudriere ports -c
or
poudriere jails -c

	
>Fix:

	


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-ia64 
Responsible-Changed-By: marcel 
Responsible-Changed-When: Mon Jan 20 18:30:32 UTC 2014 
Responsible-Changed-Why:  
Assign. This is actually caused by a PMAP bug on ia64. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=183227 
State-Changed-From-To: open->patched 
State-Changed-By: marcel 
State-Changed-When: Mon Jan 20 18:38:22 UTC 2014 
State-Changed-Why:  
Fix committed on head (see r260914). The PR will be closed when the fix is 
MFC'd to stable/10. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=183227 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/183227: commit references a PR
Date: Mon, 20 Jan 2014 18:37:47 +0000 (UTC)

 Author: marcel
 Date: Mon Jan 20 18:37:35 2014
 New Revision: 260914
 URL: http://svnweb.freebsd.org/changeset/base/260914
 
 Log:
   In pmap_set_pte(), make sure to enforce ordering by inserting a memory
   fence. Under system load, the CPU has been found to change the order
   by which the stores are made visible. When the tag is made visible
   before the other TLB values, other CPUs may use the invalid TLB values
   and do bad things.
   
   While here (i.e. not a fix) don't return errors from pmap_remove_vhpt()
   to callers of pmap_remove_pte(). Those callers don't check the return
   value and as such don't do what is needed to keep a consistent state.
   More importantly, pmap_remove_vhpt() can't really have an error without
   it indicating something unintended. Using KASSERT is therefore better.
   
   PR:		182999, 183227
 
 Modified:
   head/sys/ia64/ia64/pmap.c
 
 Modified: head/sys/ia64/ia64/pmap.c
 ==============================================================================
 --- head/sys/ia64/ia64/pmap.c	Mon Jan 20 18:15:06 2014	(r260913)
 +++ head/sys/ia64/ia64/pmap.c	Mon Jan 20 18:37:35 2014	(r260914)
 @@ -1303,6 +1303,8 @@ pmap_set_pte(struct ia64_lpte *pte, vm_o
  
  	pte->itir = PAGE_SHIFT << 2;
  
 +	ia64_mf();
 +
  	pte->tag = ia64_ttag(va);
  }
  
 @@ -1321,8 +1323,8 @@ pmap_remove_pte(pmap_t pmap, struct ia64
  	 * First remove from the VHPT.
  	 */
  	error = pmap_remove_vhpt(va);
 -	if (error)
 -		return (error);
 +	KASSERT(error == 0, ("%s: pmap_remove_vhpt returned %d",
 +	    __func__, error));
  
  	pmap_invalidate_page(va);
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: Anton Shterenlikht <mexas@bris.ac.uk>
To: bug-followup@freebsd.org
Cc:  
Subject: ia64/183227: panic: uma_zfree: Freeing to non free bucket index.
Date: Tue, 25 Mar 2014 05:15:42 -0700 (PDT)

 This seems to be the same panic, but now in r262690.
 
 # cat /var/crash/info.last 
 Dump header from device /dev/da0p3
   Architecture: ia64
   Architecture Version: 1
   Dump Length: 73728B (0 MB)
   Blocksize: 512
   Dumptime: Tue Mar 25 03:39:36 2014
   Hostname: mech-as221.men.bris.ac.uk
   Magic: FreeBSD Text Dump
   Version String: FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014
     root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI
   Panic String: uma_zfree: Freeing to non free bucket index.
   Dump Parity: 730754683
   Bounds: 3
   Dump Status: good
 # 
 
 # cat /var/crash/panic.txt 
 uma_zfree: Freeing to non free bucket index.
 # cat /var/crash/version.txt 
 FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014
     root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI
 # 
 
 *************************************
 msgbuf.txt
 *************************************
 
 GDB: debug ports: uart
 GDB: current port: uart
 KDB: debugger backends: ddb gdb
 KDB: current backend: ddb
 Copyright (c) 1992-2014 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 11.0-CURRENT #13 r262690M: Thu Mar 20 15:39:46 GMT 2014
     root@mech-as221.men.bris.ac.uk:/usr/obj/usr/src/sys/UZI ia64
 gcc version 4.2.1 20070831 patched [FreeBSD]
 WARNING: WITNESS option enabled, expect reduced performance.
 CPU: Madison (1500 MHz Itanium 2)
   Origin = "GenuineIntel"  Revision = 5
   Features = 0x1<LB>
 real memory  = 6442450944 (6144 MB)
 avail memory = 6291316736 (5999 MB)
 FPSWA Revision = 0x10012, Entry = 0xe0000040ffe60050
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP)
 cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0
 random: <Software, Yarrow> initialized
 Event timer "ITC" frequency 1500000000 Hz quality 1000
 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603)
 ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603)
 acpi0: <HP> on motherboard
 acpi0: Power Button (fixed)
 acpi0: Sleep Button (fixed)
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
 acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0
 acpi_tz0: <Thermal Zone> on acpi0
 pcib0: <ACPI Host-PCI bridge> on acpi0
 pcib0: could not evaluate _ADR - AE_NOT_FOUND
 pci0: <ACPI PCI bus> on pcib0
 ohci0: <NEC uPD 9210 USB controller> mem 0x80023000-0x80023fff irq 16 at device 1.0 on pci0
 usbus0 on ohci0
 ohci1: <NEC uPD 9210 USB controller> mem 0x80022000-0x80022fff irq 17 at device 1.1 on pci0
 usbus1 on ohci1
 ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x80021000-0x800210ff irq 18 at device 1.2 on pci0
 usbus2: EHCI version 0.95
 usbus2 on ehci0
 pci0: <mass storage, ATA> at device 2.0 (no driver attached)
 pci0: <network, ethernet> at device 3.0 (no driver attached)
 pcib1: <ACPI Host-PCI bridge> on acpi0
 pcib1: could not evaluate _ADR - AE_NOT_FOUND
 pci32: <ACPI PCI bus> on pcib1
 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2100-0x21ff mem 0x90840000-0x9084ffff,0x90830000-0x9083ffff irq 27 at device 1.0 on pci32
 mpt0: MPI Version=1.2.12.0
 mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0x90820000-0x9082ffff,0x90810000-0x9081ffff irq 28 at device 1.1 on pci32
 mpt1: MPI Version=1.2.12.0
 bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x000105> mem 0x90800000-0x9080ffff irq 29 at device 2.0 on pci32
 bge0: CHIP ID 0x00000105; ASIC REV 0x00; CHIP REV 0x01; PCI on PCI-X 66 MHz; 64bit
 miibus0: <MII bus> on bge0
 brgphy0: <BCM5701 1000BASE-T media interface> PHY 1 on miibus0
 brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
 bge0: Ethernet address: 00:11:0a:31:46:06
 pcib2: <ACPI Host-PCI bridge> on acpi0
 pcib2: could not evaluate _ADR - AE_NOT_FOUND
 pci64: <ACPI PCI bus> on pcib2
 pcib3: <ACPI Host-PCI bridge> on acpi0
 pcib3: could not evaluate _ADR - AE_NOT_FOUND
 pci96: <ACPI PCI bus> on pcib3
 pcib4: <ACPI Host-PCI bridge> on acpi0
 pcib4: could not evaluate _ADR - AE_NOT_FOUND
 pci128: <ACPI PCI bus> on pcib4
 pcib5: <ACPI Host-PCI bridge> on acpi0
 pcib5: could not evaluate _ADR - AE_NOT_FOUND
 pci192: <ACPI PCI bus> on pcib5
 pcib6: <ACPI Host-PCI bridge> on acpi0
 pcib6: could not evaluate _ADR - AE_NOT_FOUND
 pci224: <ACPI PCI bus> on pcib6
 uart0: <HP Auxiliary Diva Serial Port> mem 0xf4051000-0xf405100f irq 82 at device 1.0 on pci224
 puc0: <HP Diva Serial [GSP] Multiport UART - Everest SP2> mem 0xf4050000-0xf4050fff,0xf4020000-0xf403ffff irq 82 at device 1.1 on pci224
 uart1: <Non-standard ns8250 class UART with FIFOs> at port 1 on puc0
 uart1: console (9600,n,8,1)
 uart2: <Non-standard ns8250 class UART with FIFOs> at port 3 on puc0
 vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf0000000-0xf3ffffff,0xf4040000-0xf404ffff at device 2.0 on pci224
 vgapci0: Boot video device
 uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0
 uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0
 uart4: debug port (9600,n,8,1)
 Timecounters tick every 1.000 msec
 usbus0: 12Mbps Full Speed USB v1.0
 usbus1: 12Mbps Full Speed USB v1.0
 usbus2: 480Mbps High Speed USB v2.0
 ugen0.1: <NEC> at usbus0
 ugen1.1: <NEC> at usbus1
 uhub0: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
 uhub1: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
 ugen2.1: <NEC> at usbus2
 uhub2: <NEC EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
 uhub1: 2 ports with 2 removable, self powered
 uhub0: 3 ports with 3 removable, self powered
 random: unblocking device.
 uhub2: 5 ports with 5 removable, self powered
 da0 at mpt0 bus 0 scbus0 target 0 lun 0
 da0: <HP 73.4G MAX3073NC HPC1> Fixed Direct Access SCSI-3 device 
 da0: Serial Number X1015960    0613
 da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
 da0: Command Queueing enabled
 da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
 da1 at mpt0 bus 0 scbus0 target 1 lun 0
 da1: <SEAGATE ST373454LC D404> Fixed Direct Access SCSI-3 device 
 da1: Serial Number 3KP345CH
 da1: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
 da1: Command Queueing enabled
 da1: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C)
 da2 at mpt1 bus 0 scbus1 target 2 lun 0
 da2: <SEAGATE ST373454LC D402> Fixed Direct Access SCSI-3 device 
 da2: Serial Number 3KP0WEXZ
 da2: 320.000MB/s transfers (160.000MHz, offset 63, 16bit)
 da2: Command Queueing enabled
 da2: 70007MB (143374650 512 byte sectors: 255H 63S/T 8924C)
 WARNING: WITNESS option enabled, expect reduced performance.
 Trying to mount root from ufs:/dev/da0p2 [rw]...
 WARNING: / was not properly dismounted
 <118>Setting hostuuid: a90e62bf-ce84-11e1-9fc5-00110a31e60a.
 <118>Setting hostid: 0x688171f3.
 ZFS filesystem version: 5
 ZFS storage pool version: features support (5000)
 <118>Starting ddb.
 <118>Entropy harvesting: interrupts ethernet point_to_point swi.
 <118>Starting file system checks:
 <118>/dev/da0p2: UNREF FILE I=6288401  OWNER=root MODE=10644
 <118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:19 2014  (CLEARED)
 <118>/dev/da0p2: UNREF FILE I=6288429  OWNER=root MODE=10644
 <118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:18 2014  (CLEARED)
 <118>/dev/da0p2: UNREF FILE I=6288431  OWNER=root MODE=10644
 <118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:18 2014  (CLEARED)
 <118>/dev/da0p2: UNREF FILE I=6288432  OWNER=root MODE=10644
 <118>/dev/da0p2: SIZE=0 MTIME=Mar 21 13:19 2014  (CLEARED)
 <118>/dev/da0p2: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
 <118>/dev/da0p2: SUMMARY INFORMATION BAD (SALVAGED)
 <118>/dev/da0p2: BLK(S) MISSING IN BIT MAPS (SALVAGED)
 <118>/dev/da0p2: 761703 files, 11337625 used, 19128012 free (156372 frags, 2371455 blocks, 0.5% fragmentation)
 <118>Mounting local file systems:.
 <118>Writing entropy file:.
 <118>Setting hostname: mech-as221.men.bris.ac.uk.
 <118>Starting Network: lo0 bge0.
 <118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 <118>	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
 <118>	inet6 ::1 prefixlen 128 
 <118>	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
 <118>	inet 127.0.0.1 netmask 0xff000000 
 <118>	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 <118>bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 <118>	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
 <118>	ether 00:11:0a:31:46:06
 <118>	inet 137.222.187.221 netmask 0xffffff00 broadcast 137.222.187.255 
 <118>	inet6 fe80::211:aff:fe31:4606%bge0 prefixlen 64 scopeid 0x1 
 <118>	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 <118>	media: Ethernet autoselect (none)
 <118>	status: no carrier
 <118>Starting devd.
 <118>Starting pflogd: 
 <118>add net default: gateway 137.222.187.250
 <118>add net fe80::: gateway ::1
 <118>add net ff02::: gateway ::1
 <118>add net ::ffff:0.0.0.0: gateway ::1
 <118>add net ::0.0.0.0: gateway ::1
 <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/gcc47 /usr/local/lib/gcc48 /usr/local/lib/graphviz
 <118>Creating and/or trimming log files.
 <118>Starting syslogd.
 <118>savecore: reboot
 <118>Mar 21 14:10:17 mech-as221 savecore: reboot
 <118>savecore: writing core to /var/crash/textdump.tar.2
 <118>/var/crash/vmcore.2 not found
 <118>Starting casperd.
 <118>Clearing /tmp (X related).
 <118>Recovering vi editor sessions:.
 <118>Updating motd:.
 <118>Mounting late file systems:.
 <118>Starting ntpd.
 <118>Performing sanity check on nginx configuration:
 <118>nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
 <118>nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
 <118>Starting nginx.
 <118>Starting bsdstats.
 <118>Posting monthly OS statistics to rpt.bsdstats.org
 <118>Performing sanity check on sshd configuration.
 <118>Starting sshd.
 <118>Starting sendmail_submit.
 <118>Starting sendmail_msp_queue.
 <118>Starting cron.
 <118>Local package initialization:cd: /var/portbuild/: No such file or directory
 <118>.
 <118>Starting inetd.
 <118>Starting background file system checks in 60 seconds.
 <118>
 <118>Fri Mar 21 14:10:22 GMT 2014
 <118>Mar 21 14:11:03 mech-as221 login: ROOT LOGIN (root) ON ttyu1
 lock order reversal:
  1st 0xa00000005f6bf058 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3081
  2nd 0xe000000015952400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0
 _witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000b31b08) at _witness_debugger+0x60
 witness_checkorder(0xe000000015952400, 0x9ffc000000b03808, 0x9ffc000000b31b08, 0x11c, 0x0) at witness_checkorder+0x15c0
 _sx_xlock(0xe000000015952400, 0x0, 0x9ffc000000b31b08, 0x11c) at _sx_xlock+0x120
 ufsdirhash_acquire(0xe0000000162cdb90) at ufsdirhash_acquire+0x70
 ufsdirhash_newblk(0xe0000000162cdb90, 0xa200, 0x0, 0x9ffc0000008d59a0, 0xda1, 0xa000000090471030) at ufsdirhash_newblk+0x20
 ufs_direnter(0xe0000000162f29c0, 0x0, 0xa000000090471068, 0xa000000090471310, 0x0, 0x1) at ufs_direnter+0x530
 ufs_rename(0xa000000090471240, 0xa000000090471068, 0x0) at ufs_rename+0x1590
 VOP_RENAME_APV(0x9ffc000000bc2740, 0xa000000090471368, 0x0, 0x0, 0x9ffc0000005c1f10, 0x797, 0x9ffc000000dd1720) at VOP_RENAME_APV+0x230
 kern_renameat(0xa000000090471310, 0xa000000090471368, 0x0, 0xffffffffffffff9c, 0x12210c238, 0x0) at kern_renameat+0x630
 kern_rename(0xe000000012613b00, 0x12210c1d8, 0x12210c238, 0x0) at kern_rename+0x40
 sys_rename(0xe000000012613b00, 0xa0000000904714e8, 0x9ffc000000986c80, 0x48d) at sys_rename+0x30
 syscall(0xe000000012ade940, 0x12210c238, 0x12210c108, 0xe000000012613b00, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 lock order reversal:
  1st 0xe0000000129d47b8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:851
  2nd 0xe000000016c63658 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2101
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0
 _witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000b0e198) at _witness_debugger+0x60
 witness_checkorder(0xe000000016c63658, 0x9ffc000000b03808, 0x9ffc000000b0e198, 0x835, 0xe000000016c63720, 0x9ffc000000ae1498) at witness_checkorder+0x15c0
 __lockmgr_args(0xe000000016c63658, 0x80500, 0xe000000016c63720, 0x9ffc000000ae1498, 0x60, 0x33, 0x9ffc000000b0e198, 0x835) at __lockmgr_args+0x10a0
 vop_stdlock(0xa0000000905790e0, 0x9ffc0000009a0e70, 0x38a) at vop_stdlock+0x80
 VOP_LOCK1_APV(0x9ffc000000bad388, 0xa0000000905790c0, 0xe000000012ef7680) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000016c635f0, 0x80500, 0x9ffc000000b0e198, 0x835) at _vn_lock+0x130
 vget(0xe000000016c635f0, 0x80500, 0xe000000012ef7680, 0x9ffc000000242ca0) at vget+0x100
 devfs_allocv(0xe000000103d57300, 0xe000000012a683c8, 0x80500, 0xa000000090579110) at devfs_allocv+0x1b0
 devfs_root(0xe000000012a683c8, 0x80000, 0xa000000090579160, 0xe00000001b080190, 0x9ffc00000059dea0, 0x17b7, 0x8) at devfs_root+0x60
 vfs_donmount(0xe000000012ef7680, 0x0, 0xe000000012aa7600) at vfs_donmount+0x2c50
 sys_nmount(0xe000000012ef7680, 0x0, 0x0, 0x9ffc000000986c80) at sys_nmount+0xd0
 syscall(0xe000000012ef1280, 0x8, 0x0, 0xe000000012ef7680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 lock order reversal:
  1st 0xe000000016c47658 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:851
  2nd 0xe000000016c562d8 ufs (ufs) @ /usr/src/sys/fs/nullfs/null_vnops.c:620
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0
 _witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000ae6b28) at _witness_debugger+0x60
 witness_checkorder(0xe000000016c562d8, 0x9ffc000000b03808, 0x9ffc000000ae6b28, 0x26c, 0xe000000016c563a0, 0x9ffc000000aecf70) at witness_checkorder+0x15c0
 __lockmgr_args(0xe000000016c562d8, 0x80500, 0xe000000016c563a0, 0x9ffc000000aecf70, 0x60, 0x33, 0x9ffc000000ae6b28, 0x26c) at __lockmgr_args+0x10a0
 ffs_lock(0xa0000000904d10e0, 0xe000000016c562d8, 0x80500) at ffs_lock+0x160
 VOP_LOCK1_APV(0x9ffc000000bc1e00, 0xa0000000904d10c0, 0x9ffc00000034ebd0, 0x0) at VOP_LOCK1_APV+0x230
 null_lock(0xa0000000904d10f0, 0x9ffc000000ae6b28, 0xe000000016c65860) at null_lock+0x1a0
 VOP_LOCK1_APV(0x9ffc000000b8aa18, 0xa0000000904d10f0, 0x2) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe000000016c65860, 0x80400, 0x9ffc000000ae6a38, 0x118) at _vn_lock+0x130
 nullfs_root(0xe000000012a66a28, 0x80000, 0xa0000000904d1160, 0xe000000016c65860, 0x9ffc00000059dea0, 0x17b7, 0x8) at nullfs_root+0x90
 vfs_donmount(0xe000000012a52d80, 0x0, 0xe000000012aa7600) at vfs_donmount+0x2c50
 sys_nmount(0xe000000012a52d80, 0x0, 0x0, 0x9ffc000000986c80) at sys_nmount+0xd0
 syscall(0xe000000012e17280, 0x8, 0x0, 0xe000000012a52d80, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 <118>Mar 24 08:49:56 mech-as221 su: BAD SU mexas to root on /dev/pts/0
 <118>Mar 24 08:50:01 mech-as221 su: mexas to root on /dev/pts/0
 <118>Mar 24 11:14:02 mech-as221 su: mexas to root on /dev/pts/0
 <118>Mar 24 13:19:28 mech-as221 su: mexas to root on /dev/pts/0
 lock order reversal:
  1st 0xe0000000170162d8 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:1720
  2nd 0xe0000040876f6548 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2101
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc0000004ff860) at kdb_backtrace+0xc0
 _witness_debugger(0x1, 0x9ffc000000b03ec0, 0x9ffc000000503240, 0xb9d, 0x9ffc000000b0e198) at _witness_debugger+0x60
 witness_checkorder(0xe0000040876f6548, 0x9ffc000000b03808, 0x9ffc000000b0e198, 0x835, 0xe0000040876f6610, 0xa000000001362370) at witness_checkorder+0x15c0
 __lockmgr_args(0xe0000040876f6548, 0x80500, 0xe0000040876f6610, 0xa000000001362370, 0x60, 0x33, 0x9ffc000000b0e198, 0x835) at __lockmgr_args+0x10a0
 vop_stdlock(0xa00000008ff0f4d0, 0x9ffc0000009a0e70, 0x38a) at vop_stdlock+0x80
 VOP_LOCK1_APV(0x9ffc000000bad388, 0xa00000008ff0f4b0, 0x9ffc000000f0d470) at VOP_LOCK1_APV+0x230
 _vn_lock(0xe0000040876f64e0, 0x80500, 0x9ffc000000b0e198, 0x835) at _vn_lock+0x130
 vget(0xe0000040876f64e0, 0x80500, 0xe0000000122b8900, 0x0) at vget+0x100
 vfs_msync(0xe0000001076c52e8, 0x4, 0xe0000040876f64e0, 0x9ffc0000005b4dd0) at vfs_msync+0x120
 sync_fsync(0x9ffc000000b0e198, 0x9ffc000000f65a08, 0x240000) at sync_fsync+0x210
 VOP_FSYNC_APV(0x9ffc000000badcc8, 0xa00000008ff0f528, 0x0, 0x9ffc000000b42028, 0x9ffc0000005b6390) at VOP_FSYNC_APV+0x270
 sched_sync(0xe000000017016278, 0xe0000000170163d8, 0xe000000017016270, 0xe0000000121cfd00) at sched_sync+0x650
 fork_exit(0x9ffc000000b4bcb0, 0x0, 0xa00000008ff0f550) at fork_exit+0x120
 enter_userland() at enter_userland
 panic: uma_zfree: Freeing to non free bucket index.
 cpuid = 1
 KDB: stack backtrace:
 db_trace_self(0x9ffc0000000ce350) at db_trace_self+0x40
 db_trace_self_wrapper(0x9ffc0000004c96c0) at db_trace_self_wrapper+0x70
 kdb_backtrace(0x9ffc000000bfa6d8, 0x9ffc000000450850, 0x40c, 0x9ffc000000dd1720) at kdb_backtrace+0xc0
 vpanic(0x9ffc000000b33c90, 0xa00000008ff874f8, 0x9ffc000000c2dcd0, 0x9ffc000000bfa490) at vpanic+0x260
 kassert_panic(0x9ffc000000b33c90, 0x1f8, 0xe00000001a82df90, 0xe0000040ffbd6480) at kassert_panic+0x120
 uma_zfree_arg(0xe0000040ffbd9340, 0xe00000001a82d400, 0xe00000001a82df90) at uma_zfree_arg+0x2d0
 free(0xe00000001a82d400, 0xa0000000013f1a30) at free+0x150
 zfs_kmem_free(0xe00000001a82d400, 0xa000000001285e10, 0x309) at zfs_kmem_free+0x30
 zio_buf_free(0xe00000001a82d400, 0x200, 0x0) at zio_buf_free+0xc0
 zio_pop_transforms(0xe0000040d8cf2730, 0xe0000040664bd380, 0xe0000040664bd388, 0xe0000040d8cf2870, 0xe0000040d8cf2920, 0xe0000040d8cf2930, 0xa000000001295cf0) at zio_pop_transforms+0x170
 zio_done(0xe0000040d8cf2730, 0xe0000040d8cf29f0, 0x0) at zio_done+0x580
 zio_execute(0xe0000040d8cf2730, 0x200000, 0x2f0002, 0xa000000001295740, 0x40e, 0x40e) at zio_execute+0x2f0
 zio_notify_parent(0xe0000040d8cf2730, 0xe0000040875bcac8, 0xe0000040d8cf29b8, 0x0, 0xe0000040d8cf2a30, 0xa0000000012966c0, 0x898) at zio_notify_parent+0x290
 zio_done(0xe0000040875bcac8, 0xe000000024aa2ab0, 0x0) at zio_done+0xf50
 zio_execute(0xe0000040875bcac8) at zio_execute+0x2f0
 taskq_run_ent(0xe0000040875bce28, 0x9ffc0000004e8450, 0x50e, 0xa0000000012ba120) at taskq_run_ent+0x50
 taskqueue_run_locked(0xe000000010bba600, 0xe0000040875bce08, 0x1, 0xe000000010bba648, 0x9ffc000000b01290, 0xe000000010bba65c, 0xe000000010bba608) at taskqueue_run_locked+0x1f0
 taskqueue_thread_loop(0xe000000010bba600, 0xe000000010bba660, 0xe000000010bba65c, 0xe000000010bba630) at taskqueue_thread_loop+0x130
 fork_exit(0x9ffc000000b5b530, 0xe0000000108486e0, 0xa00000008ff87550) at fork_exit+0x120
 enter_userland() at enter_userland
 KDB: enter: panic
 
 ******************************
 ddb.txt
 ******************************
 
 db:0:kdb.enter.panic>  run lockinfo
 db:1:lockinfo> show locks
 db:1:locks>  show alllocks
 Process 80735 (cc1plus) thread 0xe000000012a50d80 (100357)
 shared lockmgr zfs (zfs) r = 0 (0xe0000000155c1178) locked @ /usr/src/sys/kern/vfs_subr.c:2101
 Process 76331 (find) thread 0xe000000012e5fb00 (100376)
 shared lockmgr zfs (zfs) r = 0 (0xe000000015ec82d8) locked @ /usr/src/sys/kern/vfs_lookup.c:518
 db:1:alllocks>  show lockedvnods
 Locked vnodes
 db:0:kdb.enter.panic>  show pcpu
 cpuid        = 1
 dynamic pcpu = 0x400005fc29800
 curthread    = 0xe0000000121dbb00: pid 0 "zio_read_intr"
 curpcb       = 0
 fpcurthread  = none
 idlethread   = 0xe000000010b95680: tid 100004 "idle: cpu1"
 MD: vhpt       = 0xe00000003f100000
 MD: lid        = 0x1000000
 MD: clock      = 0x1a4179a98ef4b
 MD: clock_mode = 2
 MD: clock_load = 0
 MD: stats      = 0xe000000010b6ea30
 MD: pmap       = 0x9ffc000000c22dc0
 spin locks held:
 db:0:kdb.enter.panic>  bt
 Tracing pid 0 tid 100213 td 0xe0000000121dbb00
 kdb_enter(0x9ffc000000af8ed8, 0x9ffc000000af8ed8, 0x9ffc0000004508a0, 0x40c) at kdb_enter+0x92
 vpanic(0x9ffc000000b33c90, 0xa00000008ff874f8, 0x100, 0x9ffc000000bfa490) at vpanic+0x2b0
 kassert_panic(0x9ffc000000b33c90, 0x1f8, 0xe00000001a82df90, 0xe0000040ffbd6480) at kassert_panic+0x120
 uma_zfree_arg(0xe0000040ffbd9340, 0xe00000001a82d400, 0xe00000001a82df90) at uma_zfree_arg+0x2d0
 free(0xe00000001a82d400, 0xa0000000013f1a30) at free+0x150
 zfs_kmem_free(0xe00000001a82d400, 0xa000000001285e10, 0x309) at zfs_kmem_free+0x30
 zio_buf_free(0xe00000001a82d400, 0x200, 0x0) at zio_buf_free+0xc0
 zio_pop_transforms(0xe0000040d8cf2730, 0xe0000040664bd380, 0xe0000040664bd388, 0xe0000040d8cf2870, 0xe0000040d8cf2920, 0xe0000040d8cf2930, 0xa000000001295cf0) at zio_pop_transforms+0x170
 zio_done(0xe0000040d8cf2730, 0xe0000040d8cf29f0, 0x0) at zio_done+0x580
 zio_execute(0xe0000040d8cf2730, 0x200000, 0x2f0002, 0xa000000001295740, 0x40e, 0x40e) at zio_execute+0x2f0
 zio_notify_parent(0xe0000040d8cf2730, 0xe0000040875bcac8, 0xe0000040d8cf29b8, 0x0, 0xe0000040d8cf2a30, 0xa0000000012966c0, 0x898) at zio_notify_parent+0x290
 zio_done(0xe0000040875bcac8, 0xe000000024aa2ab0, 0x0) at zio_done+0xf50
 zio_execute(0xe0000040875bcac8) at zio_execute+0x2f0
 taskq_run_ent(0xe0000040875bce28, 0x9ffc0000004e8450, 0x50e, 0xa0000000012ba120) at taskq_run_ent+0x50
 taskqueue_run_locked(0xe000000010bba600, 0xe0000040875bce08, 0x1, 0xe000000010bba648, 0x9ffc000000b01290, 0xe000000010bba65c, 0xe000000010bba608) at taskqueue_run_locked+0x1f0
 taskqueue_thread_loop(0xe000000010bba600, 0xe000000010bba660, 0xe000000010bba65c, 0xe000000010bba630) at taskqueue_thread_loop+0x130
 fork_exit(0x9ffc000000b5b530, 0xe0000000108486e0, 0xa00000008ff87550) at fork_exit+0x120
 enter_userland() at enter_userland
 db:0:kdb.enter.panic>  ps
   pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
 80756 80740   588     0  RJ      CPU 0                       c++
 80740 15731   588     0  SJ      wait     0xe000000104544de0 sh
 80736 80734   588     0  SJ      piperd   0xe00000001235d810 as
 80735 80734   588     0  DJ      zio->io_ 0xe0000040bebaf518 cc1plus
 80734 47211   588     0  SJ      wait     0xe0000040810f1280 c++
 80733 49525   588     0  S       nanslp   0x9ffc000000c2fa39 sleep
 76333 76330 64193     0  S       piperd   0xe00000001235ce70 wc
 76332 76330 64193     0  S       piperd   0xe000000018bed5a8 tee
 76331 76330 64193     0  D       zio->io_ 0xe0000000124db180 find
 76330 76326 64193     0  S       wait     0xe000000012ef1280 sh
 76326 64478 64193     0  S       piperd   0xe0000040810075a8 sh
 64479 64471 64193     0  S       piperd   0xe000000018bed340 mail
 64478 64471 64193     0  S       wait     0xe0000040811c7280 sh
 64471 64470 64193     0  SW      wait     0xe0000001045384a0 sh
 64470 64468 64193     0  SW      wait     0xe0000000122b7280 lockf
 64468 64467 64193     0  SW      wait     0xe000000012add720 sh
 64467 64203 64193     0  SW      wait     0xe000000178f0c940 sh
 64204 64196 64193     0  S       piperd   0xe00000001235d0d8 mail
 64203 64196 64193     0  SW      wait     0xe00000010452ede0 sh
 64196 64195 64193     0  SW      wait     0xe000000018bdd720 sh
 64195 64193 64193     0  SW      wait     0xe000000012e17720 lockf
 64193 64191 64193     0  SWs     wait     0xe000000012ef0de0 sh
 64191   786   786     0  S       piperd   0xe000000012e06738 cron
 47211 92677   588     0  SJ      wait     0xe0000001045444a0 make
 92677 92676   588     0  SWJ     wait     0xe00000010452f720 make
 92676 92652   588     0  SWJ     wait     0xe000000012ef44a0 sh
 92652 92650   588     0  SWJ     wait     0xe00000010452e940 make
 92650 59384   588     0  SW      wait     0xe000000018bdc940 sh
 59715 59384   588     0  S       piperd   0xe000000012e070d8 tee
 59384  3381   588     0  S       select   0xe0000000123ec4c0 sh
 15731 87774   588     0  SJ      wait     0xe000004080a1ade0 gmake
 87774 87772   588     0  SWJ     wait     0xe0000000122b0000 gmake
 87772 87688   588     0  SWJ     wait     0xe0000001045a5720 sh
 87688 87686   588     0  SWJ     wait     0xe000004080a1b280 make
 87686 63244   588     0  SW      wait     0xe0000001044e4940 sh
 63399 63244   588     0  S       piperd   0xe000000018becc08 tee
 63244  3381   588     0  S       select   0xe0000000123eb5c0 sh
 49525  3381   588     0  S       wait     0xe000000012ade940 sh
  3381   588   588     0  S       select   0xe0000000123eb940 sh
   593   587   593    25  Ss      piperd   0xe00000010459ac08 sendmail
   588   587   588     0  SWs     wait     0xe0000000122b6de0 sh
   587   786   786     0  S       piperd   0xe00000010459a268 cron
   875     1   875     0  Ss+     ttyin    0xe000000011c30ca8 getty
   844     1     1     0  S       ttydcd   0xe000000011c310e8 getty
   821     1   821     0  Ss      select   0xe0000000122505c0 inetd
   786     1   786     0  Ss      nanslp   0x9ffc000000c2fa38 cron
   782     1   782    25  Ss      pause    0xe000000012adc9e8 sendmail
   779     1   779     0  Ss      select   0xe0000000123ec140 sendmail
   776     1   776     0  Ss      select   0xe0000000123ec3c0 sshd
   723   721   721    80  S       kqread   0xe000000010bbb300 nginx
   721     1   721     0  SWs     pause    0xe000000010b93328 nginx
   702     1   702     0  Ss      select   0xe000000012250fc0 ntpd
   642     1   642     0  Ss      select   0xe000000012250dc0 casperd
   641     1    22     0  S+      select   0xe000000012250c40 casperd
   612     1   612     0  Ss      select   0xe000000012250d40 syslogd
   486     1   486     0  Ss      select   0xe000000012250e40 devd
   135     1   135     0  SWs     pause    0xe000000012adce88 adjkerntz
    40     0     0     0  DL      (threaded)                  [zfskern]
 100341                   D       tx->tx_s 0xe0000000125d8208 [txg_thread_enter]
 100340                   D       tx->tx_q 0xe0000000125d8228 [txg_thread_enter]
 100334                   D       spa->spa 0xe0000000123daac0 [trim tank]
 100067                   D       l2arc_fe 0xa0000000013d5540 [l2arc_feed_thread]
 100066                   D       arc_recl 0xa0000000013c5720 [arc_reclaim_thread]
    21     0     0     0  DL      sdflush  0x9ffc000000bfb890 [softdepflush]
    20     0     0     0  DL      vlruwt   0xe0000000122b0de0 [vnlru]
    19     0     0     0  DL      syncer   0x9ffc000000f65b10 [syncer]
    18     0     0     0  DL      psleep   0x9ffc000000bfac70 [bufdaemon]
    17     0     0     0  DL      pgzero   0x9ffc000000bfbbfc [pagezero]
     9     0     0     0  DL      psleep   0x9ffc000000bfbb64 [vmdaemon]
     8     0     0     0  DL      psleep   0x9ffc000000bfbb94 [pagedaemon]
     5     0     0     0  DL      waiting_ 0x9ffc000000f681f0 [sctp_iterator]
     4     0     0     0  DL      idle     0xa000000000a22000 [mpt_recovery1]
     3     0     0     0  DL      idle     0xa000000000a0a000 [mpt_recovery0]
    16     0     0     0  DL      (threaded)                  [usb]
 100039                   D       -        0xa000000000a06e18 [usbus2]
 100038                   D       -        0xa000000000a06dc0 [usbus2]
 100037                   D       -        0xa000000000a06d68 [usbus2]
 100036                   D       -        0xa000000000a06d10 [usbus2]
 100034                   D       -        0xa0000000009fd460 [usbus1]
 100033                   D       -        0xa0000000009fd408 [usbus1]
 100032                   D       -        0xa0000000009fd3b0 [usbus1]
 100031                   D       -        0xa0000000009fd358 [usbus1]
 100029                   D       -        0xa0000000009f9460 [usbus0]
 100028                   D       -        0xa0000000009f9408 [usbus0]
 100027                   D       -        0xa0000000009f93b0 [usbus0]
 100026                   D       -        0xa0000000009f9358 [usbus0]
    15     0     0     0  DL      tzpoll   0x9ffc000000bf9670 [acpi_thermal]
     2     0     0     0  DL      (threaded)                  [cam]
 100051                   D       ccb_scan 0x9ffc000000bfbf38 [scanner]
 100015                   D       -        0x9ffc000000bfc080 [doneq0]
    14     0     0     0  DL      -        0x9ffc000000bf98e8 [rand_harvestq]
    13     0     0     0  DL      (threaded)                  [geom]
 100011                   D       -        0x9ffc000000bf9e28 [g_down]
 100010                   D       -        0x9ffc000000bf9e20 [g_up]
 100009                   D       -        0x9ffc000000bf9e10 [g_event]
    12     0     0     0  WL      (threaded)                  [intr]
 100045                   I                                   [swi0: uart uart+++]
 100044                   I                                   [irq29: bge0]
 100042                   I                                   [irq28: mpt1]
 100040                   I                                   [irq27: mpt0]
 100035                   I                                   [irq18: ehci0]
 100030                   I                                   [irq17: ohci1]
 100025                   I                                   [irq16: ohci0]
 100023                   I                                   [swi6: Giant taskq]
 100021                   I                                   [swi5: fast taskq]
 100014                   I                                   [swi6: task queue]
 100008                   I                                   [swi1: netisr 0]
 100007                   I                                   [swi3: vm]
 100006                   I                                   [swi4: clock (1)]
 100005                   I                                   [swi4: clock (0)]
    11     0     0     0  RL      (threaded)                  [idle]
 100004                   CanRun                              [idle: cpu1]
 100003                   CanRun                              [idle: cpu0]
     1     0     1     0  SLs     wait     0xe000000010b924a0 [init]
    10     0     0     0  DL      audit_wo 0x9ffc000000f6c888 [audit]
     0     0     0     0  RLs     (threaded)                  [kernel]
 100073                   D       -        0xe00000001225b600 [zil_clean]
 100075                   D       -        0xe00000001225a600 [zil_clean]
 100081                   D       -        0xe00000405fb09100 [zil_clean]
 100092                   D       -        0xe00000010471b300 [zil_clean]
 100467                   D       -        0xe00000010471b900 [zil_clean]
 100356                   D       -        0xe00000001225bb00 [zil_clean]
 100355                   D       -        0xe00000001225bc00 [zil_clean]
 100354                   D       -        0xe000000010bbb000 [zil_clean]
 100353                   D       -        0xe000000010bbaf00 [zil_clean]
 100352                   D       -        0xe00000001225bd00 [zil_clean]
 100351                   D       -        0xe000000010bbae00 [zil_clean]
 100345                   D       -        0xe000000010bbac00 [zil_clean]
 100339                   D       -        0xe00000001225c300 [metaslab_group_task]
 100338                   D       -        0xe00000001225c400 [metaslab_group_task]
 100337                   D       -        0xe00000001225c500 [zfs_vn_rele_taskq]
 100336                   D       -        0xe00000001225c600 [metaslab_group_task]
 100335                   D       -        0xe00000001225c700 [metaslab_group_task]
 100333                   D       -        0xe00000001225c800 [zio_ioctl_intr]
 100332                   D       -        0xe00000001225c900 [zio_ioctl_issue]
 100331                   D       -        0xe00000001225ca00 [zio_claim_intr]
 100330                   D       -        0xe00000001225cb00 [zio_claim_issue]
 100329                   D       -        0xe00000001225cc00 [zio_free_intr]
 100328                   D       -        0xe00000001225cd00 [zio_free_issue_7_11]
 100327                   D       -        0xe00000001225cd00 [zio_free_issue_7_10]
 100326                   D       -        0xe00000001225cd00 [zio_free_issue_7_9]
 100325                   D       -        0xe00000001225cd00 [zio_free_issue_7_8]
 100324                   D       -        0xe00000001225cd00 [zio_free_issue_7_7]
 100323                   D       -        0xe00000001225cd00 [zio_free_issue_7_6]
 100322                   D       -        0xe00000001225cd00 [zio_free_issue_7_5]
 100321                   D       -        0xe00000001225cd00 [zio_free_issue_7_4]
 100320                   D       -        0xe00000001225cd00 [zio_free_issue_7_3]
 100319                   D       -        0xe00000001225cd00 [zio_free_issue_7_2]
 100318                   D       -        0xe00000001225cd00 [zio_free_issue_7_1]
 100317                   D       -        0xe00000001225cd00 [zio_free_issue_7_0]
 100316                   D       -        0xe00000001225ce00 [zio_free_issue_6_11]
 100315                   D       -        0xe00000001225ce00 [zio_free_issue_6_10]
 100314                   D       -        0xe00000001225ce00 [zio_free_issue_6_9]
 100313                   D       -        0xe00000001225ce00 [zio_free_issue_6_8]
 100312                   D       -        0xe00000001225ce00 [zio_free_issue_6_7]
 100311                   D       -        0xe00000001225ce00 [zio_free_issue_6_6]
 100310                   D       -        0xe00000001225ce00 [zio_free_issue_6_5]
 100309                   D       -        0xe00000001225ce00 [zio_free_issue_6_4]
 100308                   D       -        0xe00000001225ce00 [zio_free_issue_6_3]
 100307                   D       -        0xe00000001225ce00 [zio_free_issue_6_2]
 100306                   D       -        0xe00000001225ce00 [zio_free_issue_6_1]
 100305                   D       -        0xe00000001225ce00 [zio_free_issue_6_0]
 100304                   D       -        0xe00000001225cf00 [zio_free_issue_5_11]
 100303                   D       -        0xe00000001225cf00 [zio_free_issue_5_10]
 100302                   D       -        0xe00000001225cf00 [zio_free_issue_5_9]
 100301                   D       -        0xe00000001225cf00 [zio_free_issue_5_8]
 100300                   D       -        0xe00000001225cf00 [zio_free_issue_5_7]
 100299                   D       -        0xe00000001225cf00 [zio_free_issue_5_6]
 100298                   D       -        0xe00000001225cf00 [zio_free_issue_5_5]
 100297                   D       -        0xe00000001225cf00 [zio_free_issue_5_4]
 100296                   D       -        0xe00000001225cf00 [zio_free_issue_5_3]
 100295                   D       -        0xe00000001225cf00 [zio_free_issue_5_2]
 100294                   D       -        0xe00000001225cf00 [zio_free_issue_5_1]
 100293                   D       -        0xe00000001225cf00 [zio_free_issue_5_0]
 100292                   D       -        0xe00000001225d000 [zio_free_issue_4_11]
 100291                   D       -        0xe00000001225d000 [zio_free_issue_4_10]
 100290                   D       -        0xe00000001225d000 [zio_free_issue_4_9]
 100289                   D       -        0xe00000001225d000 [zio_free_issue_4_8]
 100288                   D       -        0xe00000001225d000 [zio_free_issue_4_7]
 100287                   D       -        0xe00000001225d000 [zio_free_issue_4_6]
 100286                   D       -        0xe00000001225d000 [zio_free_issue_4_5]
 100285                   D       -        0xe00000001225d000 [zio_free_issue_4_4]
 100284                   D       -        0xe00000001225d000 [zio_free_issue_4_3]
 100283                   D       -        0xe00000001225d000 [zio_free_issue_4_2]
 100282                   D       -        0xe00000001225d000 [zio_free_issue_4_1]
 100281                   D       -        0xe00000001225d000 [zio_free_issue_4_0]
 100280                   D       -        0xe00000001225d100 [zio_free_issue_3_11]
 100279                   D       -        0xe00000001225d100 [zio_free_issue_3_10]
 100278                   D       -        0xe00000001225d100 [zio_free_issue_3_9]
 100277                   D       -        0xe00000001225d100 [zio_free_issue_3_8]
 100276                   D       -        0xe00000001225d100 [zio_free_issue_3_7]
 100275                   D       -        0xe00000001225d100 [zio_free_issue_3_6]
 100274                   D       -        0xe00000001225d100 [zio_free_issue_3_5]
 100273                   D       -        0xe00000001225d100 [zio_free_issue_3_4]
 100272                   D       -        0xe00000001225d100 [zio_free_issue_3_3]
 100271                   D       -        0xe00000001225d100 [zio_free_issue_3_2]
 100270                   D       -        0xe00000001225d100 [zio_free_issue_3_1]
 100269                   D       -        0xe00000001225d100 [zio_free_issue_3_0]
 100268                   D       -        0xe00000001225d200 [zio_free_issue_2_11]
 100267                   D       -        0xe00000001225d200 [zio_free_issue_2_10]
 100266                   D       -        0xe00000001225d200 [zio_free_issue_2_9]
 100265                   D       -        0xe00000001225d200 [zio_free_issue_2_8]
 100264                   D       -        0xe00000001225d200 [zio_free_issue_2_7]
 100263                   D       -        0xe00000001225d200 [zio_free_issue_2_6]
 100262                   D       -        0xe00000001225d200 [zio_free_issue_2_5]
 100261                   D       -        0xe00000001225d200 [zio_free_issue_2_4]
 100260                   D       -        0xe00000001225d200 [zio_free_issue_2_3]
 100259                   D       -        0xe00000001225d200 [zio_free_issue_2_2]
 100258                   D       -        0xe00000001225d200 [zio_free_issue_2_1]
 100257                   D       -        0xe00000001225d200 [zio_free_issue_2_0]
 100256                   D       -        0xe00000001225d300 [zio_free_issue_1_11]
 100255                   D       -        0xe00000001225d300 [zio_free_issue_1_10]
 100254                   D       -        0xe00000001225d300 [zio_free_issue_1_9]
 100253                   D       -        0xe00000001225d300 [zio_free_issue_1_8]
 100252                   D       -        0xe00000001225d300 [zio_free_issue_1_7]
 100251                   D       -        0xe00000001225d300 [zio_free_issue_1_6]
 100250                   D       -        0xe00000001225d300 [zio_free_issue_1_5]
 100249                   D       -        0xe00000001225d300 [zio_free_issue_1_4]
 100248                   D       -        0xe00000001225d300 [zio_free_issue_1_3]
 100247                   D       -        0xe00000001225d300 [zio_free_issue_1_2]
 100246                   D       -        0xe00000001225d300 [zio_free_issue_1_1]
 100245                   D       -        0xe00000001225d300 [zio_free_issue_1_0]
 100244                   D       -        0xe000000010bbab00 [zio_free_issue_0_11]
 100243                   D       -        0xe000000010bbab00 [zio_free_issue_0_10]
 100242                   D       -        0xe000000010bbab00 [zio_free_issue_0_9]
 100241                   D       -        0xe000000010bbab00 [zio_free_issue_0_8]
 100240                   D       -        0xe000000010bbab00 [zio_free_issue_0_7]
 100239                   D       -        0xe000000010bbab00 [zio_free_issue_0_6]
 100238                   D       -        0xe000000010bbab00 [zio_free_issue_0_5]
 100237                   D       -        0xe000000010bbab00 [zio_free_issue_0_4]
 100236                   D       -        0xe000000010bbab00 [zio_free_issue_0_3]
 100235                   D       -        0xe000000010bbab00 [zio_free_issue_0_2]
 100234                   D       -        0xe000000010bbab00 [zio_free_issue_0_1]
 100233                   D       -        0xe000000010bbab00 [zio_free_issue_0_0]
 100232                   D       -        0xe000000010bbaa00 [zio_write_intr_high]
 100231                   D       -        0xe000000010bbaa00 [zio_write_intr_high]
 100230                   D       -        0xe000000010bbaa00 [zio_write_intr_high]
 100229                   D       -        0xe000000010bbaa00 [zio_write_intr_high]
 100228                   D       -        0xe000000010bbaa00 [zio_write_intr_high]
 100227                   D       -        0xe000000010bba900 [zio_write_intr_7]
 100226                   D       -        0xe000000010bba900 [zio_write_intr_6]
 100225                   D       -        0xe000000010bba900 [zio_write_intr_5]
 100224                   D       -        0xe000000010bba900 [zio_write_intr_4]
 100223                   D       -        0xe000000010bba900 [zio_write_intr_3]
 100222                   D       -        0xe000000010bba900 [zio_write_intr_2]
 100221                   D       -        0xe000000010bba900 [zio_write_intr_1]
 100220                   D       -        0xe000000010bba900 [zio_write_intr_0]
 100219                   D       -        0xe000000010bba800 [zio_write_issue_hig]
 100218                   D       -        0xe000000010bba800 [zio_write_issue_hig]
 100217                   D       -        0xe000000010bba800 [zio_write_issue_hig]
 100216                   D       -        0xe000000010bba800 [zio_write_issue_hig]
 100215                   D       -        0xe000000010bba800 [zio_write_issue_hig]
 100214                   D       -        0xe000000010bba700 [zio_write_issue]
 100213                   Run     CPU 1                       [zio_read_intr]
 100212                   D       -        0xe000000010bba500 [zio_read_issue_7]
 100211                   D       -        0xe000000010bba500 [zio_read_issue_6]
 100210                   D       -        0xe000000010bba500 [zio_read_issue_5]
 100209                   D       -        0xe000000010bba500 [zio_read_issue_4]
 100208                   D       -        0xe000000010bba500 [zio_read_issue_3]
 100207                   D       -        0xe000000010bba500 [zio_read_issue_2]
 100206                   D       -        0xe000000010bba500 [zio_read_issue_1]
 100205                   D       -        0xe000000010bba500 [zio_read_issue_0]
 100204                   D       -        0xe000000010bba400 [zio_null_intr]
 100203                   D       -        0xe000000010bba300 [zio_null_issue]
 100201                   D       -        0xe000000010bb9d00 [metaslab_group_task]
 100200                   D       -        0xe000000010bb9e00 [metaslab_group_task]
 100065                   D       -        0xe00000001225db00 [system_taskq_1]
 100064                   D       -        0xe00000001225db00 [system_taskq_0]
 100050                   D       -        0xe000000010bbbb00 [CAM taskq]
 100046                   D       -        0x9ffc000000c2e7d1 [deadlkres]
 100022                   D       -        0xe000000010bbb500 [thread taskq]
 100020                   D       -        0xe000000010bbb700 [kqueue taskq]
 100019                   D       -        0xe000000010bbb800 [ffs_trim taskq]
 100018                   D       -        0xe000000010bbb900 [acpi_task_2]
 100017                   D       -        0xe000000010bbb900 [acpi_task_1]
 100016                   D       -        0xe000000010bbb900 [acpi_task_0]
 100012                   D       -        0xe000000010bbbd00 [firmware taskq]
 100000                   D       swapin   0x9ffc000000c22388 [swapper]
 db:0:kdb.enter.panic>  alltrace
 
 Tracing command c++ pid 80756 tid 101342 td 0xe000000107665b00
 ia64_ih_stop(0x9ffc00000096cb40, 0x40b) at ia64_ih_stop+0x80
 ia64_handle_intr(0xa000000090749400) at ia64_handle_intr+0x110
 ivt_External_Interrupt() at ivt_External_Interrupt+0x30
 
 Tracing command sh pid 80740 tid 100405 td 0xe000000012a52000
 cpu_switch(0xe000000012a52000, 0xe000000010b95b00, 0x9ffc000000c814a8, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000000012a52000, 0xe000000010b95b00, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012a52000, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe000000104544de0, 0x6c, 0xe000000012a52378, 0xe000000012a52000) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe000000104544de0, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe000000104544de0, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
 _sleep(0xe000000104544de0, 0xe000000104544ee0, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
 kern_wait6(0xe000000012a52000, 0x7, 0x0, 0xa0000000905a9390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
 kern_wait(0xe000000012a52000, 0xffffffff, 0xa0000000905a9390, 0x0, 0x0) at kern_wait+0xe0
 sys_wait4(0xe000000012a52000, 0xa0000000905a94e8, 0xa0000000905a9390, 0x9ffc000000986c80) at sys_wait4+0x70
 syscall(0xe000000104544de0, 0x7fffffffffffb2f0, 0x0, 0xe000000012a52000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command as pid 80736 tid 100390 td 0xe000000018be8900
 cpu_switch(0xe000000018be8900, 0xe000000010bb0d80, 0x9ffc000000c818e0, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000000018be8900, 0xe000000010bb0d80, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000018be8900, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe00000001235d810, 0x5c, 0xe000000018be8c78, 0xe000000018be8900) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe00000001235d810, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe00000001235d810, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
 _sleep(0xe00000001235d810, 0xe00000001235da50, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe000000012360f24, 0xa000000090559360, 0xa000000090559378, 0x0, 0xe00000001235d918) at pipe_read+0x740
 dofileread(0xe000000018be8900, 0x0, 0xe000000012360f00, 0xa000000090559360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000018be8900, 0x0, 0xa000000090559360) at kern_readv+0xa0
 sys_read(0xe000000018be8900, 0xa0000000905594e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
 syscall(0xe000000018bdc000, 0x1208de000, 0x2000, 0xe000000018be8900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command cc1plus pid 80735 tid 100357 td 0xe000000012a50d80
 cpu_switch(0xe000000012a50d80, 0xe000000010b97680, 0x9ffc000000c81ea8, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000000012a50d80, 0xe000000010b97680, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012a50d80, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe0000040bebaf518, 0x0, 0xe000000012a510f8, 0xe000000012a50d80) at sleepq_switch+0x2e0
 sleepq_wait(0xe0000040bebaf518, 0x0, 0x9ffc000000b008c0, 0xe000000012a50d80, 0x9ffc0000003b9520) at sleepq_wait+0xb0
 _cv_wait(0xe0000040bebaf518, 0xe0000040bebaf4f8, 0xe000000012a50d80, 0x0) at _cv_wait+0x4d0
 zio_wait(0xe0000040bebaf1f8, 0xe0000040bebaf4e8, 0xe0000040bebaf4f8, 0xe0000040bebaf518, 0xa00000000113e640, 0x91a, 0xa0000000012ba120) at zio_wait+0x150
 dbuf_read(0xe00000408e9383f0, 0xe0000040bebaf1f8, 0xa, 0xa000000090450ea8) at dbuf_read+0xd70
 dmu_buf_hold(0xe00000408e9383f0, 0xa, 0x0, 0x0, 0xa000000090450ee0, 0x1) at dmu_buf_hold+0x1d0
 zap_lockdir(0xe000000012aab400, 0x5f652, 0x0, 0x1, 0x1, 0x0, 0xa000000090450f20) at zap_lockdir+0x50
 zap_lookup_norm(0xe000000012aab400, 0x5f652, 0xa000000090450fc0, 0x8, 0x1, 0xa000000090450f68, 0x0, 0x0) at zap_lookup_norm+0x60
 zap_lookup(0xe000000012aab400, 0x5f652, 0xa000000090450fc0, 0x8, 0x1, 0xa000000090450f68, 0xa0000000012b3eb0, 0xb1e) at zap_lookup+0x60
 zfs_dirent_lock(0xa000000090450f80, 0xe00000001ce0aa10, 0xa000000090450fc0, 0xa000000090450f88, 0x6, 0x0, 0x0, 0xe00000001b934f00) at zfs_dirent_lock+0xbb0
 zfs_dirlook(0xe00000001ce0aa10, 0xa000000090450fc0, 0xa0000000904512e8, 0x0, 0x0, 0x0) at zfs_dirlook+0x240
 zfs_lookup(0xe0000000155c1110, 0xa000000090450fc0, 0xa0000000904512e8, 0xa000000090451310, 0x0, 0xe000004082290200, 0xe000000012a50d80, 0x0) at zfs_lookup+0x7a0
 zfs_freebsd_lookup(0xa0000000904510c0, 0xa000000090451310, 0xa000000090451348) at zfs_freebsd_lookup+0x100
 VOP_CACHEDLOOKUP_APV(0xa0000000013c0e88, 0xa0000000904510c0, 0x0, 0x9ffc000000b42308, 0x9ffc000000587490) at VOP_CACHEDLOOKUP_APV+0x270
 vfs_cache_lookup(0xe0000000155c1110, 0x0, 0xa000000090451310) at vfs_cache_lookup+0x1f0
 VOP_LOOKUP_APV(0xa0000000013c0e88, 0xa0000000904510f8, 0x0, 0x9ffc000000b42368) at VOP_LOOKUP_APV+0x270
 lookup(0xa000000090451288, 0x200000, 0xe0000000155c1110, 0x0, 0xa000000090451318) at lookup+0xe30
 namei(0xa000000090451288, 0xe0000000123c2000, 0x9ffc0000008fe220, 0x0, 0xa0000000904512f8) at namei+0xb40
 vn_open_cred(0xa000000090451288, 0xa000000090451390, 0x1a4, 0x0, 0xe000004082290200, 0xe00000001233ecd0) at vn_open_cred+0x500
 vn_open(0xa000000090451288, 0xa000000090451390, 0x1a4, 0xe00000001233ecd0, 0x9ffc0000005c3ee0, 0x695, 0x0) at vn_open+0x50
 kern_openat(0xe000000012a50d80, 0xffffffffffffff9c, 0x1214529e0, 0x0, 0x0, 0x1b6) at kern_openat+0x490
 kern_open(0xe000000012a50d80, 0x1214529e0, 0x0, 0x8000, 0x1b6) at kern_open+0x40
 sys_open(0xe000000012a50d80, 0xa0000000904514e8, 0x9ffc000000986c80, 0x48d) at sys_open+0x40
 syscall(0xe0000000123c2000, 0x8000, 0x1b6, 0xe000000012a50d80, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command c++ pid 80734 tid 100487 td 0xe000004081208480
 cpu_switch(0xe000004081208480, 0xe000000018be8900, 0x9ffc000000c81070, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000004081208480, 0xe000000018be8900, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000004081208480, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe0000040810f1280, 0x6c, 0xe0000040812087f8, 0xe000004081208480) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000040810f1280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000040810f1280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
 _sleep(0xe0000040810f1280, 0xe0000040810f1380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
 kern_wait6(0xe000004081208480, 0x0, 0x13b5f, 0xa000000090711390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
 kern_wait(0xe000004081208480, 0x13b5f, 0xa000000090711390, 0x0, 0x0) at kern_wait+0xe0
 sys_wait4(0xe000004081208480, 0xa0000000907114e8, 0xa000000090711390, 0x9ffc000000986c80) at sys_wait4+0x70
 syscall(0xe0000040810f1280, 0x1208060d8, 0x0, 0xe000004081208480, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sleep pid 80733 tid 100414 td 0xe000000104522900
 cpu_switch(0xe000000104522900, 0xe000000012613b00, 0x9ffc000000c81818, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000000104522900, 0xe000000012613b00, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000104522900, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0x9ffc000000c2fa39, 0x6c, 0xe000000104522c78, 0xe000000104522900) at sleepq_switch+0x2e0
 sleepq_catch_signals(0x9ffc000000c2fa39, 0x6c, 0x9ffc000000b008c0, 0xe000000104522c90) at sleepq_catch_signals+0x190
 sleepq_timedwait_sig(0x9ffc000000c2fa39, 0x6c, 0x20000000, 0x9ffc000000462c50, 0x998) at sleepq_timedwait_sig+0x30
 _sleep(0x9ffc000000c2fa39, 0x0, 0x6c, 0x9ffc000000afaba8, 0x1, 0x20000000, 0x200) at _sleep+0x750
 kern_nanosleep(0xe000000104522900, 0x200000000, 0xa0000000905f1380) at kern_nanosleep+0x2a0
 sys_nanosleep(0xe000000104522900, 0xa0000000905f14e8, 0x0, 0x9ffc000000986c80) at sys_nanosleep+0xb0
 syscall(0xe00000010452f280, 0x7fffffffffffeb40, 0x7fffffffffffeb10, 0xe000000104522900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command wc pid 76333 tid 100374 td 0xe000000012a52900
 cpu_switch(0xe000000012a52900, 0xe000000010b95680, 0x9ffc000000c81750, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000000012a52900, 0xe000000010b95680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012a52900, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe00000001235ce70, 0x5c, 0xe000000012a52c78, 0xe000000012a52900) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe00000001235ce70, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe00000001235ce70, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
 _sleep(0xe00000001235ce70, 0xe00000001235d0b0, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe00000001233ff14, 0xa0000000904d9360, 0xa0000000904d9378, 0x0, 0xe00000001235cf78) at pipe_read+0x740
 dofileread(0xe000000012a52900, 0x0, 0xe00000001233fef0, 0xa0000000904d9360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012a52900, 0x0, 0xa0000000904d9360) at kern_readv+0xa0
 sys_read(0xe000000012a52900, 0xa0000000904d94e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
 syscall(0xe000000012e16de0, 0x7ffffffffffeece0, 0x10000, 0xe000000012a52900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command tee pid 76332 tid 100451 td 0xe000004081060000
 cpu_switch(0xe000004081060000, 0xe000000010b97680, 0x9ffc000000c80d28, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000004081060000, 0xe000000010b97680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000004081060000, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe000000018bed5a8, 0x5c, 0xe000004081060378, 0xe000004081060000) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe000000018bed5a8, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe000000018bed5a8, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
 _sleep(0xe000000018bed5a8, 0xe000000018bed7e8, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe000000018c3aac4, 0xa000000090701360, 0xa000000090701378, 0x0, 0xe000000018bed6b0) at pipe_read+0x740
 dofileread(0xe000004081060000, 0x0, 0xe000000018c3aaa0, 0xa000000090701360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000004081060000, 0x0, 0xa000000090701360) at kern_readv+0xa0
 sys_read(0xe000004081060000, 0xa0000000907014e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
 syscall(0xe0000040811c6000, 0x120804000, 0x2000, 0xe000004081060000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command find pid 76331 tid 100376 td 0xe000000012e5fb00
 cpu_switch(0xe000000012e5fb00, 0xe000000010b95b00, 0x9ffc000000c80148, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000000012e5fb00, 0xe000000010b95b00, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012e5fb00, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe0000000124db180, 0x0, 0xe000000012e5fe78, 0xe000000012e5fb00) at sleepq_switch+0x2e0
 sleepq_wait(0xe0000000124db180, 0x0, 0x9ffc000000b008c0, 0xe000000012e5fb00, 0x9ffc0000003b9520) at sleepq_wait+0xb0
 _cv_wait(0xe0000000124db180, 0xe0000000124db160, 0xe000000012e5fb00, 0x0) at _cv_wait+0x4d0
 zio_wait(0xe0000000124dae60, 0xe0000000124db150, 0xe0000000124db160, 0xe0000000124db180, 0xa00000000113e640, 0x91a, 0xa0000000012ba120) at zio_wait+0x150
 dbuf_read(0xe0000001078e07e0, 0xe0000000124dae60, 0x22, 0xa0000000904e8ed8, 0xe0000001078e0800, 0xe0000001078e0830) at dbuf_read+0xd70
 dnode_hold_impl(0xe00000001290cc00, 0x4129e, 0x1, 0xa00000000135da58, 0xa0000000904e8f00) at dnode_hold_impl+0x210
 dnode_hold(0xe00000001290cc00, 0x4129e, 0xa00000000135da58, 0xa0000000904e8f00) at dnode_hold+0x40
 dmu_bonus_hold(0xe00000001290cc00, 0x4129e, 0x0, 0xa0000000904e8f10) at dmu_bonus_hold+0x40
 sa_buf_hold(0xe00000001290cc00, 0x4129e, 0x0, 0xa0000000904e8f10, 0xa000000001276000, 0xd20) at sa_buf_hold+0x40
 zfs_zget(0xe000000012a5a000, 0x4129e, 0xa0000000904e8fa8, 0x8, 0x1, 0xe000000012a5a6a8, 0xa0000000012b3eb0, 0xa00000000137fcd8) at zfs_zget+0x120
 zfs_dirent_lock(0xa0000000904e8fa0, 0xe00000010319a2e0, 0xa0000000904e8fe0, 0xa0000000904e8fa8, 0x6, 0x0, 0x0, 0xe00000001b8d7000) at zfs_dirent_lock+0xc10
 zfs_dirlook(0xe00000010319a2e0, 0xa0000000904e8fe0, 0xa0000000904e9220, 0x0, 0x0, 0x0) at zfs_dirlook+0x240
 zfs_lookup(0xe000000015ec8270, 0xa0000000904e8fe0, 0xa0000000904e9220, 0xa0000000904e9248, 0x0, 0xe000004082290100, 0xe000000012e5fb00, 0x0) at zfs_lookup+0x7a0
 zfs_freebsd_lookup(0xa0000000904e90e0, 0xa0000000904e9248, 0xa0000000904e9280) at zfs_freebsd_lookup+0x100
 VOP_CACHEDLOOKUP_APV(0xa0000000013c0e88, 0xa0000000904e90e0, 0x0, 0x9ffc000000b42308, 0x9ffc000000587490) at VOP_CACHEDLOOKUP_APV+0x270
 vfs_cache_lookup(0xe000000015ec8270, 0x0, 0xa0000000904e9248) at vfs_cache_lookup+0x1f0
 VOP_LOOKUP_APV(0xa0000000013c0e88, 0xa0000000904e9118, 0x0, 0x9ffc000000b42368) at VOP_LOOKUP_APV+0x270
 lookup(0xa0000000904e91c0, 0x200000, 0xe000000015ec8270, 0x0, 0xa0000000904e9250) at lookup+0xe30
 namei(0xa0000000904e91c0, 0xa0000000904e91f8, 0x4000104, 0x0, 0xa0000000904e9230, 0xa0000000904e9278, 0xa0000000904e9270, 0xa0000000904e9220) at namei+0xb40
 kern_statat_vnhook(0xe000000012e5fb00, 0x200, 0xffffff9c, 0x120873ac8, 0x0, 0xa0000000904e9320, 0x0) at kern_statat_vnhook+0xf0
 kern_statat(0xe000000012e5fb00, 0x200, 0xffffff9c, 0x120873ac8, 0x0, 0xa0000000904e9320) at kern_statat+0x40
 sys_fstatat(0xe000000012e5fb00, 0xa0000000904e94e8, 0x9ffc000000986c80, 0x48d) at sys_fstatat+0x50
 syscall(0xe000000012e164a0, 0x120873ac8, 0x120873a50, 0xe000000012e5fb00, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 76330 tid 100394 td 0xe000000012ef7680
 cpu_switch(0xe000000012ef7680, 0xe000000010b97680, 0x9ffc000000c81070, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000000012ef7680, 0xe000000010b97680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012ef7680, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe000000012ef1280, 0x6c, 0xe000000012ef79f8, 0xe000000012ef7680) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe000000012ef1280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe000000012ef1280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
 _sleep(0xe000000012ef1280, 0xe000000012ef1380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
 kern_wait6(0xe000000012ef7680, 0x7, 0x0, 0xa000000090579390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
 kern_wait(0xe000000012ef7680, 0xffffffff, 0xa000000090579390, 0x0, 0x0) at kern_wait+0xe0
 sys_wait4(0xe000000012ef7680, 0xa0000000905794e8, 0xa000000090579390, 0x9ffc000000986c80) at sys_wait4+0x70
 syscall(0xe000000012ef1280, 0x7fffffffffffe4d0, 0x0, 0xe000000012ef7680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 76326 tid 100386 td 0xe000000012e5e000
 cpu_switch(0xe000000012e5e000, 0xe000000010b95b00, 0x9ffc000000c81c28, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000000012e5e000, 0xe000000010b95b00, 0x9ffc000000c49c98, 0x9ffc000000c49c80) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012e5e000, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe0000040810075a8, 0x5c, 0xe000000012e5e378, 0xe000000012e5e000) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000040810075a8, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000040810075a8, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
 _sleep(0xe0000040810075a8, 0xe0000040810077e8, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe000000018c3a2f4, 0xa000000090539360, 0xa000000090539378, 0x0, 0xe0000040810076b0) at pipe_read+0x740
 dofileread(0xe000000012e5e000, 0x3, 0xe000000018c3a2d0, 0xa000000090539360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012e5e000, 0x3, 0xa000000090539360) at kern_readv+0xa0
 sys_read(0xe000000012e5e000, 0xa0000000905394e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
 syscall(0xe000000018bdd280, 0x7fffffffffffe800, 0x80, 0xe000000012e5e000, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command mail pid 64479 tid 100398 td 0xe000000012ef6900
 cpu_switch(0xe000000012ef6900, 0xe0000001045b5680, 0x9ffc000000c81098, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe000000012ef6900, 0xe0000001045b5680, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe000000012ef6900, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe000000018bed340, 0x5c, 0xe000000012ef6c78, 0xe000000012ef6900) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe000000018bed340, 0x5c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe000000018bed340, 0x5c, 0x9ffc000000b04328, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
 _sleep(0xe000000018bed340, 0xe000000018bed580, 0x5c, 0x9ffc000000b04328, 0x0, 0x0, 0x100) at _sleep+0x7b0
 pipe_read(0xe000004069110d44, 0xa000000090401360, 0xa000000090401378, 0x0, 0xe000000018bed448) at pipe_read+0x740
 dofileread(0xe000000012ef6900, 0x0, 0xe000004069110d20, 0xa000000090401360, 0x0, 0x0) at dofileread+0x150
 kern_readv(0xe000000012ef6900, 0x0, 0xa000000090401360) at kern_readv+0xa0
 sys_read(0xe000000012ef6900, 0xa0000000904014e8, 0x9ffc000000986c80, 0x48d) at sys_read+0x100
 syscall(0xe000000012ef04a0, 0x12080e000, 0x2000, 0xe000000012ef6900, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 64478 tid 100447 td 0xe0000001045b5680
 cpu_switch(0xe0000001045b5680, 0xe000000010bb0d80, 0x9ffc000000c81f70, 0x9ffc0000004aa1a0) at cpu_switch+0xd0
 sched_switch(0xe0000001045b5680, 0xe000000010bb0d80, 0x9ffc000000c4a998, 0x9ffc000000c4a980) at sched_switch+0x890
 mi_switch(0x104, 0x0, 0xe0000001045b5680, 0x9ffc0000004e3290) at mi_switch+0x3f0
 sleepq_switch(0xe0000040811c7280, 0x6c, 0xe0000001045b59f8, 0xe0000001045b5680) at sleepq_switch+0x2e0
 sleepq_catch_signals(0xe0000040811c7280, 0x6c, 0x9ffc000000b008c0, 0x9ffc000000b00c80) at sleepq_catch_signals+0x190
 sleepq_wait_sig(0xe0000040811c7280, 0x6c, 0x9ffc000000b30338, 0x100, 0x9ffc000000462cb0) at sleepq_wait_sig+0x30
 _sleep(0xe0000040811c7280, 0xe0000040811c7380, 0x6c, 0x9ffc000000b30338, 0x0, 0x0, 0x100, 0x0) at _sleep+0x7b0
 kern_wait6(0xe0000001045b5680, 0x7, 0x0, 0xa0000000906e1390, 0x30, 0x0, 0x0) at kern_wait6+0xba0
 kern_wait(0xe0000001045b5680, 0xffffffff, 0xa0000000906e1390, 0x0, 0x0) at kern_wait+0xe0
 sys_wait4(0xe0000001045b5680, 0xa0000000906e14e8, 0xa0000000906e1390, 0x9ffc000000986c80) at sys_wait4+0x70
 syscall(0xe0000040811c7280, 0x7fffffffffffe6e0, 0x0, 0xe0000001045b5680, 0x0, 0x0, 0x9ffc000000982660, 0x8) at syscall+0x5e0
 epc_syscall_return() at epc_syscall_return
 
 Tracing command sh pid 64471 tid 100401 td 0xe000000104523680
 
 Tracing command lockf pid 64470 tid 100359 td 0xe000000012a50480
 
 Tracing command sh pid 64468 tid 100364 td 0xe000000012a26900
 
 Tracing command sh pid 64467 tid 101341 td 0xe000000107665200
 
 Tracing command mail pid 64204 tid 100446 td 0xe0000001045b5b00
 
 Tracing command sh pid 64203 tid 100421 td 0xe000000104522480
 
 Tracing command sh pid 64196 tid 100385 td 0xe000000012e5e480
 
 Tracing command lockf pid 64195 tid 100372 td 0xe000000012a53200
 
 Tracing command sh pid 64193 tid 100395 td 0xe000000012ef7200
 
 Tracing command cron pid 64191 tid 100488 td 0xe000004081208000
 
 Tracing command make pid 47211 tid 100407 td 0xe000000012ee9680
 
 Tracing command make pid 92677 tid 100412 td 0xe000000104522d80
 
 Tracing command sh pid 92676 tid 100382 td 0xe000000012ee8480
 
 Tracing command make pid 92652 tid 100422 td 0xe000000104522000
 
 Tracing command sh pid 92650 tid 100388 td 0xe000000012a27680
 
 Tracing command tee pid 59715 tid 100419 td 0xe00000010458e480
 
 Tracing command sh pid 59384 tid 100363 td 0xe000000012a26d80
 
 Tracing command gmake pid 15731 tid 100434 td 0xe000000104591200
 
 Tracing command gmake pid 87774 tid 100371 td 0xe000000012a53680
 
 Tracing command sh pid 87772 tid 100438 td 0xe000004080a2fb00
 
 Tracing command make pid 87688 tid 100433 td 0xe000000104591680
 
 Tracing command sh pid 87686 tid 100420 td 0xe00000010458e000
 
 Tracing command tee pid 63399 tid 100387 td 0xe000000012a27b00
 
 Tracing command sh pid 63244 tid 100365 td 0xe000000012a26480
 
 Tracing command sh pid 49525 tid 100361 td 0xe000000012613b00
 
 Tracing command sh pid 3381 tid 100383 td 0xe000000012ee8000
 
 Tracing command sendmail pid 593 tid 100368 td 0xe000000012a99680
 
 Tracing command sh pid 588 tid 100360 td 0xe000000012a50000
 
 Tracing command cron pid 587 tid 100445 td 0xe000000107664000
 
 Tracing command getty pid 875 tid 100377 td 0xe000000012e5f680
 
 Tracing command getty pid 844 tid 100059 td 0xe000000011123680
 
 Tracing command inetd pid 821 tid 100370 td 0xe000000012a53b00
 
 Tracing command cron pid 786 tid 100344 td 0xe00000001243ed80
 
 Tracing command sendmail pid 782 tid 100367 td 0xe000000012a99b00
 
 Tracing command sendmail pid 779 tid 100049 td 0xe0000000121da480
 
 Tracing command sshd pid 776 tid 100048 td 0xe0000000121da900
 
 Tracing command nginx pid 723 tid 100060 td 0xe000000011123200
 
 Tracing command nginx pid 721 tid 100061 td 0xe000000011122d80
 
 Tracing command ntpd pid 702 tid 100362 td 0xe000000012a27200
 
 Tracing command casperd pid 642 tid 100343 td 0xe00000001243f200
 
 Tracing command casperd pid 641 tid 100358 td 0xe000000012a50900
 
 Tracing command syslogd pid 612 tid 100369 td 0xe000000012a99200
 
 Tracing command devd pid 486 tid 100062 td 0xe0000000123be480
 
 Tracing command adjkerntz pid 135 tid 100366 td 0xe000000012a26000
 
 Tracing command zfskern pid 40 tid 100341 td 0xe000000012612480
 
 Tracing command zfskern pid 40 tid 100340 td 0xe000000012612900
 
 Tracing command zfskern pid 40 tid 100334 td 0xe000000012438d80
 
 Tracing command zfskern pid 40 tid 100067 td 0xe0000000123b8d80
 
 Tracing command zfskern pid 40 tid 100066 td 0xe0000000123b9200
 
 Tracing command softdepflush pid 21 tid 100058 td 0xe0000000122b8000
 
 Tracing command vnlru pid 20 tid 100057 td 0xe0000000122b8480
 
 Tracing command syncer pid 19 tid 100056 td 0xe0000000122b8900
 
 Tracing command bufdaemon pid 18 tid 100055 td 0xe0000000122b8d80
 
 Tracing command pagezero pid 17 tid 100054 td 0xe0000000122b9200
 
 Tracing command vmdaemon pid 9 tid 100053 td 0xe0000000122b9680
 
 Tracing command pagedaemon pid 8 tid 100052 td 0xe0000000122b9b00
 
 Tracing command sctp_iterator pid 5 tid 100047 td 0xe0000000121dad80
 
 Tracing command mpt_recovery1 pid 4 tid 100043 td 0xe000000010d06d80
 
 Tracing command mpt_recovery0 pid 3 tid 100041 td 0xe000000010d07680
 
 Tracing command usb pid 16 tid 100039 td 0xe000000011122000
 
 Tracing command usb pid 16 tid 100038 td 0xe000000011122480
 
 Tracing command usb pid 16 tid 100037 td 0xe000000011122900
 
 Tracing command usb pid 16 tid 100036 td 0xe000000010c33b00
 
 Tracing command usb pid 16 tid 100034 td 0xe000000010d04480
 
 Tracing command usb pid 16 tid 100033 td 0xe000000010d04900
 
 Tracing command usb pid 16 tid 100032 td 0xe000000010d04d80
 
 Tracing command usb pid 16 tid 100031 td 0xe000000010d05200
 
 Tracing command usb pid 16 tid 100029 td 0xe000000010d05b00
 
 Tracing command usb pid 16 tid 100028 td 0xe000000010d06000
 
 Tracing command usb pid 16 tid 100027 td 0xe000000010bb1200
 
 Tracing command usb pid 16 tid 100026 td 0xe000000010bb1680
 
 Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000010c32000
 
 Tracing command cam pid 2 tid 100051 td 0xe000000011123b00
 
 Tracing command cam pid 2 tid 100015 td 0xe000000010b97680
 
 Tracing command rand_harvestq pid 14 tid 100013 td 0xe000000010bb0000
 
 Tracing command geom pid 13 tid 100011 td 0xe000000010bb0900
 
 Tracing command geom pid 13 tid 100010 td 0xe000000010bb0d80
 
 Tracing command geom pid 13 tid 100009 td 0xe000000010b94000
 
 Tracing command intr pid 12 tid 100045 td 0xe000000010d06480
 
 Tracing command intr pid 12 tid 100044 td 0xe000000010d06900
 
 Tracing command intr pid 12 tid 100042 td 0xe000000010d07200
 
 Tracing command intr pid 12 tid 100040 td 0xe000000010d07b00
 
 Tracing command intr pid 12 tid 100035 td 0xe000000010d04000
 
 Tracing command intr pid 12 tid 100030 td 0xe000000010d05680
 
 Tracing command intr pid 12 tid 100025 td 0xe000000010bb1b00
 
 Tracing command intr pid 12 tid 100023 td 0xe000000010c3248
State-Changed-From-To: patched->patched 
State-Changed-By: linimon 
State-Changed-When: Sun Apr 20 01:48:45 UTC 2014 
State-Changed-Why:  
over to committer as MFC reminder. 


Responsible-Changed-From-To: freebsd-ia64->marcel 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sun Apr 20 01:48:45 UTC 2014 
Responsible-Changed-Why:  

http://www.freebsd.org/cgi/query-pr.cgi?pr=183227 
>Unformatted:
