From nobody@FreeBSD.org  Tue Jun 15 20:38:38 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D99321065678
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 15 Jun 2010 20:38:38 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id C70938FC13
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 15 Jun 2010 20:38:38 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o5FKccwh024100
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 15 Jun 2010 20:38:38 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o5FKccvn024098;
	Tue, 15 Jun 2010 20:38:38 GMT
	(envelope-from nobody)
Message-Id: <201006152038.o5FKccvn024098@www.freebsd.org>
Date: Tue, 15 Jun 2010 20:38:38 GMT
From: Dmitry Pryanishnikov <lynx.ripe@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010)
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         147890
>Category:       kern
>Synopsis:       [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kib
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jun 15 20:40:01 UTC 2010
>Closed-Date:    Mon Jun 28 11:27:06 UTC 2010
>Last-Modified:  Mon Jun 28 11:27:06 UTC 2010
>Originator:     Dmitry Pryanishnikov
>Release:        RELENG_8
>Organization:
Home
>Environment:
FreeBSD lynx.homenet 8.0-STABLE FreeBSD 8.0-STABLE #0: Tue Jun 15 15:05:05 EEST 2010     root@lynx.homenet:/databig/obj/databig/ftp/RELENG_8/src.100418/sys/lynx  i386

>Description:

After upgrade RELENG_8 as of 10.04.2010 -> 14.06.2010 my desktop (previously running w/o problems for days) started to lock pretty fast (within 1-2 hours of uptime). I've narrowed down the time window of the problem appearance: kernel built from the sources as of date=2010.04.18.00.00.00 runs OK, one as of date=2010.04.20.00.00.00 already locks. Nothing "special" is required in order to trigger the condition: after boot I just log in as a non-root (via xdm - it starts xfce4 environment), then start the Thunar File Manager and GQview. Then I switch to text console (crash dump often doesn't complete when console is running X) and just do 'ps axww|grep D' from time to time. Eventually processes thunar-vfs-mime-cleaner-1 (stuck in uninterruptible wait state) start to accumulate. In this state, simple 'sync' also hangs the issuing shell, and (the most painful result) file operations (file creations/moves/...) don't get synched to the disk. Then deadlock resolver (after debug.deadl
 kres.slptime_threshold seconds) detects the deadlock, which allows to get crash dump. I've got valid dumps both for kernel built from 14.06.2010 and from 20.04.2010 sources, they look similar. Here is the crash of the latter:

panic: deadlkres: possible deadlock detected for 0xcb52e4a0, blocked for 294235 ticks

(I've lowered debug.deadlkres.slptime_threshold manually after detecting hung thunar-vfs-mime-cleaner-1 processes)

(kgdb) print *(struct thread *)0xcb52e4a0
$1 = {td_lock = 0xc07db49c, td_proc = 0xcb5002a8, td_plist = {tqe_next = 0x0, tqe_prev = 0xcb5002b0}, td_runq = {tqe_next = 0x0, tqe_prev = 0xc07ce834}, td_slpq = {tqe_next = 0xcb52db90, 
    tqe_prev = 0xc7383018}, td_lockq = {tqe_next = 0x0, tqe_prev = 0x0}, td_cpuset = 0xc702ce38, td_sel = 0x0, td_sleepqueue = 0x0, td_turnstile = 0xc7ca2c00, td_umtxq = 0xcabc3940, td_tid = 100186, 
  td_sigqueue = {sq_signals = {__bits = {0, 0, 0, 0}}, sq_kill = {__bits = {0, 0, 0, 0}}, sq_list = {tqh_first = 0x0, tqh_last = 0xcb52e500}, sq_proc = 0xcb5002a8, sq_flags = 1}, td_flags = 4, 
  td_inhibitors = 2, td_pflags = 0, td_dupfd = -1, td_sqqueue = 0, td_wchan = 0xcb3a9df4, td_wmesg = 0xc0735016 "ufs", td_lastcpu = 2 '\002', td_oncpu = 255 '&#1066;', td_owepreempt = 0 '\0', td_tsqueue = 0 '\0', 
  td_locks = 0, td_rw_rlocks = 0, td_lk_slocks = 0, td_blocked = 0x0, td_lockname = 0x0, td_contested = {lh_first = 0x0}, td_sleeplocks = 0xc0908618, td_intr_nesting_level = 0, td_pinned = 0, 
  td_ucred = 0xc7668580, td_estcpu = 0, td_slptick = 703512, td_blktick = 0, td_ru = {ru_utime = {tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 0}, ru_maxrss = 224, ru_ixrss = 12, ru_idrss = 4, 
    ru_isrss = 128, ru_minflt = 145, ru_majflt = 0, ru_nswap = 0, ru_inblock = 0, ru_oublock = 0, ru_msgsnd = 0, ru_msgrcv = 0, ru_nsignals = 0, ru_nvcsw = 1, ru_nivcsw = 10}, td_incruntime = 0, 
  td_runtime = 23985873, td_pticks = 0, td_sticks = 0, td_iticks = 0, td_uticks = 0, td_intrval = 0, td_oldsigmask = {__bits = {0, 0, 0, 0}}, td_sigmask = {__bits = {0, 0, 0, 0}}, td_generation = 11, 
  td_sigstk = {ss_sp = 0x0, ss_size = 0, ss_flags = 4}, td_xsig = 0, td_profil_addr = 0, td_profil_ticks = 0, td_name = "initial thread\000-cle", td_fpop = 0x0, td_dbgflags = 0, td_ng_outbound = 0, td_osd = {
    osd_nslots = 0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, td_rqindex = 32 ' ', td_base_pri = 128 '\200', td_priority = 128 '\200', td_pri_class = 3 '\003', td_user_pri = 128 '\200', 
  td_base_user_pri = 128 '\200', td_pcb = 0xeb6a3d90, td_state = TDS_INHIBITED, td_retval = {0, 1}, td_slpcallout = {c_links = {sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0, 
    c_arg = 0x0, c_func = 0, c_lock = 0x0, c_flags = 16, c_cpu = 0}, td_frame = 0xeb6a3d38, td_kstack_obj = 0xcb4ca3b8, td_kstack = 3949600768, td_kstack_pages = 2, td_unused1 = 0x0, td_unused2 = 0, 
  td_unused3 = 0, td_critnest = 1, td_md = {md_spinlock_count = 1, md_saved_flags = 582}, td_sched = 0xcb52e6bc, td_ar = 0x0, td_syscalls = 11802, td_lprof = {{lh_first = 0x0}, {lh_first = 0x0}}, 
  td_dtrace = 0x0, td_errno = 0, td_vnet = 0x0, td_vnet_lpush = 0x0}


(kgdb) print *(struct proc *)0xcb5002a8
$2 = {p_list = {le_next = 0xcabb3d48, le_prev = 0xcabb1000}, p_threads = {tqh_first = 0xcb52e4a0, tqh_last = 0xcb52e4a8}, p_slock = {lock_object = {lo_name = 0xc073b098 "process slock", lo_flags = 720896, 
      lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, p_ucred = 0xc7668580, p_fd = 0xc781b900, p_fdtol = 0x0, p_stats = 0xc781ce00, p_limit = 0xc7826500, p_limco = {c_links = {sle = {sle_next = 0x0}, tqe = {
        tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0, c_arg = 0x0, c_func = 0, c_lock = 0xcb500330, c_flags = 0, c_cpu = 0}, p_sigacts = 0xcabea000, p_flag = 268451840, p_state = PRS_NORMAL, p_pid = 2044, 
  p_hash = {le_next = 0x0, le_prev = 0xc6fe9ff0}, p_pglist = {le_next = 0xcabb3d48, le_prev = 0xc7800db8}, p_pptr = 0xc7043000, p_sibling = {le_next = 0xcabb3d48, le_prev = 0xc7043084}, p_children = {
    lh_first = 0x0}, p_mtx = {lock_object = {lo_name = 0xc073b08b "process lock", lo_flags = 21168128, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, p_ksi = 0xc750f410, p_sigqueue = {sq_signals = {__bits = {
        0, 0, 0, 0}}, sq_kill = {__bits = {0, 0, 0, 0}}, sq_list = {tqh_first = 0x0, tqh_last = 0xcb500368}, sq_proc = 0xcb5002a8, sq_flags = 1}, p_oppid = 0, p_vmspace = 0xc76febc8, p_swtick = 703502, 
  p_realtimer = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}, p_ru = {ru_utime = {tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 0}, ru_maxrss = 0, 
    ru_ixrss = 0, ru_idrss = 0, ru_isrss = 0, ru_minflt = 0, ru_majflt = 0, ru_nswap = 0, ru_inblock = 0, ru_oublock = 0, ru_msgsnd = 0, ru_msgrcv = 0, ru_nsignals = 0, ru_nvcsw = 0, ru_nivcsw = 0}, p_rux = {
    rux_runtime = 23985873, rux_uticks = 0, rux_sticks = 1, rux_iticks = 0, rux_uu = 0, rux_su = 9993, rux_tu = 9993}, p_crux = {rux_runtime = 0, rux_uticks = 0, rux_sticks = 0, rux_iticks = 0, rux_uu = 0, 
    rux_su = 0, rux_tu = 0}, p_profthreads = 0, p_exitthreads = 0, p_traceflag = 0, p_tracevp = 0x0, p_tracecred = 0x0, p_textvp = 0xcb524218, p_lock = 0, p_sigiolst = {slh_first = 0x0}, p_sigparent = 20, 
  p_sig = 0, p_code = 0, p_stops = 0, p_stype = 0, p_step = 0 '\0', p_pfsflags = 0 '\0', p_nlminfo = 0x0, p_aioinfo = 0x0, p_singlethread = 0x0, p_suspcount = 0, p_xthread = 0x0, p_boundary_count = 0, 
  p_pendingcnt = 0, p_itimers = 0x0, p_magic = 3203398350, p_osrel = 800504, p_comm = "thunar-vfs-mime-cle", p_pgrp = 0xc7bd7040, p_sysent = 0xc07a4220, p_args = 0xc7bd6d40, p_cpulimit = 9223372036854775807, 
  p_nice = 0 '\0', p_fibnum = 0, p_xstat = 0, p_klist = {kl_list = {slh_first = 0x0}, kl_lock = 0xc04d47e0 <knlist_mtx_lock>, kl_unlock = 0xc04d4810 <knlist_mtx_unlock>, 
    kl_assert_locked = 0xc04d4b60 <knlist_mtx_assert_locked>, kl_assert_unlocked = 0xc04d4b30 <knlist_mtx_assert_unlocked>, kl_lockarg = 0xcb500330}, p_numthreads = 1, p_md = {md_ldt = 0x0}, p_itcallout = {
    c_links = {sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0, c_arg = 0x0, c_func = 0, c_lock = 0x0, c_flags = 16, c_cpu = 0}, p_acflag = 0, p_peers = 0x0, p_leader = 0xcb5002a8, 
  p_emuldata = 0x0, p_label = 0x0, p_sched = 0xcb500550, p_ktr = {stqh_first = 0x0, stqh_last = 0xcb500538}, p_mqnotifier = {lh_first = 0x0}, p_dtrace = 0x0, p_pwait = {cv_description = 0xc073b8cc "ppwait", 
    cv_waiters = 0}}


I'm ready to run any additional kgdb commands against those crash dumps, to try patches on this desktop etc...

Here is the kernel config:

ident		LYNX
makeoptions	DEBUG=-g		#Build kernel with gdb(1) debug symbols
options 	INCLUDE_CONFIG_FILE     # Include this file in kernel
cpu		I686_CPU		# aka Pentium Pro(tm)
options		SCHED_ULE
options		SMP			# Symmetric MultiProcessor Kernel
options 	PREEMPTION		# Enable kernel thread preemption
options		MUTEX_DEBUG
options 	WITNESS
device		apic			# I/O apic
options 	COMPAT_FREEBSD4
options 	COMPAT_FREEBSD5
options 	COMPAT_FREEBSD6
options 	COMPAT_FREEBSD7
options 	SYSVSHM
options 	SYSVSEM
options 	SYSVMSG
options 	KDB
options 	KDB_TRACE
options 	DDB
options 	DDB_NUMSYM
options 	BREAK_TO_DEBUGGER	# BREAK on a serial console goes to DDB
options 	SYSCTL_DEBUG
options 	KTRACE			#kernel tracing
options 	INVARIANTS
options 	INVARIANT_SUPPORT
options		DEADLKRES		#software deadlock resolver thread
options 	INET			#Internet communications protocols
options 	INET6			#IPv6 communications protocols
options		IPSEC			#IP security (requires device crypto)
options		IPSEC_NAT_T		#NAT-T support, UDP encap of ESP
device		ether			#Generic Ethernet
device		loop			#Network loopback device
device		bpf			#Berkeley packet filter
device		enc			#IPsec interface
options 	IPFIREWALL		#firewall
options 	IPFIREWALL_VERBOSE	#enable logging to syslogd(8)
options 	IPFIREWALL_VERBOSE_LIMIT=100	#limit verbosity
options 	IPFIREWALL_FORWARD	#packet destination changes
options		IPFIREWALL_NAT		#ipfw kernel nat support
options		LIBALIAS		# libalias library, performing NAT
options 	BPF_JITTER		#support for BPF just-in-time compiler
options 	FFS			#Fast filesystem
options 	SOFTUPDATES
options 	UFS_EXTATTR
options 	UFS_ACL
options 	UFS_DIRHASH
options		UFS_GJOURNAL		# Gjournal-based UFS journaling support.
options 	QUOTA			#enable disk quotas
options 	SUIDDIR			#The suiddir hack
options 	_KPOSIX_PRIORITY_SCHEDULING
options 	P1003_1B_SEMAPHORES
device		pty		#Pseudo ttys
options 	MSGBUF_SIZE=131072
device		crypto		# core crypto support
device		pci
device		atkbdc		# AT keyboard controller
device		atkbd		# AT keyboard
device		psm		# PS/2 mouse
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
device		vga		# VGA video card driver
device		splash		# Splash screen and screen saver support
options		X86BIOS		# x86 real mode BIOS emulator, required by atkbdc/dpms/vesa
options 	PRINTF_BUFR_SIZE=128	# Prevent printf output being interspersed.
device		sc
options 	SC_HISTORY_SIZE=20000
options 	SC_TWOBUTTON_MOUSE
device		uart			# Generic UART driver
device		ata
device		atadisk		# ATA disk drives
options 	ATA_STATIC_ID	# Static device numbering
device		scbus		# SCSI bus (required for SCSI)
device		da		# Direct Access (disks)
device		cd		# CD
device		pass		# Passthrough device (direct SCSI access)
options		SND_PCM_64	# Process 32bit samples through 64bit integer

And the loader.conf:

autoboot_delay="5"		# Delay in seconds before autobooting
beastie_disable="YES"		# Turn the beastie boot menu on and off
kernel="kernel"			# /boot sub-directory containing kernel and modules
kern.hz="1000"			# Set the kernel interval timer rate
kern.maxdsiz="1G"		# Set the max data size
kern.maxssiz="1G"		# Set the max stack size
kern.maxtsiz="1G"		# Set the max text size
kern.ipc.semmni=80		# Number of semaphore identifiers
kern.ipc.semmns=480		# Maximum number of semaphores in the system
hw.ata.atapi_dma=1		# Using DMA for ATAPI devices
hw.snd.verbose=2		# Make /dev/sndstat more verbose
hw.snd.maxautovchans=4		# Enable software sound mixer
hint.uart.0.flags="0x10"	# device is potential system console
debug.minidump=1		# Use minidumps
debug.witness.watch=0		# Turn off the witness
ipdivert_load="YES"
accf_data_load="YES"		# Wait for data accept filter
accf_dns_load="YES"		# Wait for full DNS request accept filter
accf_http_load="YES"		# Wait for full HTTP request accept filter
if_tap_load="YES"		# Ethernet tunnel software network interface
if_tun_load="YES"		# Tunnel driver (user process ppp)
if_vlan_load="YES"		# IEEE 802.1Q VLAN network interface
miibus_load="YES"		# miibus support, needed for some drivers
if_em_load="YES"		# Intel(R) PRO/1000 Gigabit Ethernet
snd_hda_load="YES"		# Intel High Definition Audio
usb_load="YES"			# USB subsystem
uhci_load="YES"			# UHCI USB Host Controller driver
ehci_load="YES"			# USB Enhanced Host Controller driver
ugen_load="YES"			# USB generic device, if all else fails ...
uhid_load="YES"			# Human Interface Devices
ukbd_load="YES"			# Keyboard
ulpt_load="YES"			# Printer
ums_load="YES"			# Mouse
umass_load="YES"		# Mass Storage Devices
ucom_load="YES"			# USB tty support:
uplcom_load="YES"		#  Prolific PL-2303/2303X/2303HX serial adapters
uftdi_load="YES"		#  FTDI FT2232C, FT8U100AX and FT8U232AM chips
uark_load="YES"			#  Arkmicro Technologies ARK3116 based USB serial adapter
snd_uaudio_load="YES"		# USB audio
umodem_load="YES"		# Modems
random_load="YES"
cpufreq_load="YES"
coretemp_load="YES"
ichwd_load="YES"
smbus_load="YES"
smb_load="YES"
ichsmb_load="YES"
speaker_load="YES"
atapicam_load="YES"
fdc_load="YES"
drm_load="YES"
radeon_load="YES"
vesa_load="YES"
acpi_video_load="YES"
firewire_load="YES"



>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: brucec 
Responsible-Changed-When: Wed Jun 16 00:20:49 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=147890 

From: Bruce Cran <bruce@cran.org.uk>
To: bug-followup@freebsd.org,
 lynx.ripe@gmail.com
Cc:  
Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -&gt; 20.04.2010)
Date: Wed, 16 Jun 2010 14:03:30 +0100

 Could you run the commands shown in  
 http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-
 handbook/kerneldebug-deadlocks.html to get more information to help debug the 
 problem please?
 
 -- 
 Bruce

From: Dmitry Pryanishnikov <lynx.ripe@gmail.com>
To: Bruce Cran <bruce@cran.org.uk>
Cc: bug-followup@freebsd.org
Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in 
	RELENG_8 (18.04.2010 -&gt; 20.04.2010)
Date: Thu, 17 Jun 2010 01:12:17 +0300

 Hello!
 
 2010/6/16 Bruce Cran <bruce@cran.org.uk>:
 > Could you run the commands shown in
 > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html
 > to get more information to help debug the problem please?
 
   Done - I've rebuilt the kernel from the sources as of 20.04.2010
 with the missing debugging options, repeated the problem, turned
 ddb(4) capture buffer on, issued recommended ddb commands, then called
 'doadump'; then rebooted and extracted capture buffer contents from
 the crashdump using ddb(8).
 
   BTW, is there a way to get this information _without repeating the
 problem_ - using only a crashdump from it's previous occurrence? All
 data required for analysis should already be there. I suspect not
 every deadlock is easy repeatable (like this one)...
 
 db> ps
 
   pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
  2143  1846  2143     0  S+      ufs      0xcd466dc8 sync
  2131     1  1911  1000  S       ufs      0xcd466dc8 initial thread
  1979     1  1911  1000  S       select   0xc804fd64 initial thread
  1968     1  1918  1000  S       select   0xcdb825e4 initial thread
  1965  1943  1939  1000  S       (threaded)          soffice.bin
 100193                   S       select   0xc75c2e64 soffice.bin
 100192                   S       accept   0xcdb93b82 soffice.bin
 100191                   S       ucond    0xcdb83500 soffice.bin
 100190                   S       ucond    0xcdb83540 soffice.bin
 100167                   S       select   0xcdb83264 initial thread
  1962  1928  1911  1000  S       select   0xc80503a4 initial thread
  1961  1928  1911  1000  S       select   0xcdb832a4 initial thread
  1960  1928  1911  1000  S       select   0xc80505e4 initial thread
  1953     1  1911  1000  S       select   0xc8050da4 xfce4-settings-help
  1952     1  1951  1000  S       ufs      0xcddc65b8 initial thread
  1943  1939  1939  1000  S       wait     0xc80802a8 sh
  1939     1  1939  1000  Ss      wait     0xc80dad48 sh
  1935     1  1911  1000  S       select   0xcdb82e64 xfsettingsd
  1932     1  1911  1000  S       (threaded)          xfdesktop
 100149                   S       select   0xcdb829e4 initial thread
  1930     1  1911  1000  S       (threaded)          Thunar
 100196                   S       piperd   0xc804d930 Thunar
 100128                   S       select   0xc75cf124 initial thread
  1928     1  1911  1000  S       select   0xc8050764 initial thread
  1926     1  1911  1000  S       select   0xc8050164 initial thread
  1920     1  1918  1000  S       select   0xc8050b24 initial thread
  1918     1  1918  1000  Ss      select   0xc8050924 dbus-daemon
  1917     1  1911  1000  S       select   0xc75b8824 dbus-launch
  1911  1842  1911  1000  Ss      select   0xc75b91a4 initial thread
  1862  1828  1862     0  S+      ttyin    0xc7379870 bash
  1846  1829  1846     0  S+      wait     0xc7688550 bash
  1842  1836  1842     0  Ss      wait     0xc781c550 xdm
  1839  1836  1839     0  Ss      select   0xc80507e4 initial thread
  1837     1     1     0  S       ttydcd   0xc7390490 getty
  1836     1     1     0  S       pause    0xc80d7058 xdm
  1835     1  1835     0  Ss+     ttyin    0xc71f7a70 getty
  1834     1  1834     0  Ss+     ttyin    0xc71f7e70 getty
  1833     1  1833     0  Ss+     ttyin    0xc7379070 getty
  1832     1  1832     0  Ss+     ttyin    0xc7379470 getty
  1831     1  1831     0  Ss+     ttyin    0xc71f7c70 getty
  1830     1  1830     0  Ss+     ttyin    0xc7379270 getty
  1829     1  1829     0  Ss+     wait     0xc768aaa0 login
  1828     1  1828     0  Ss+     wait     0xc80d7d48 login
  1802     1  1802     0  Ss      select   0xc7394de4 inetd
  1770     1  1770     0  Ss      select   0xc804f524 moused
  1724     1  1724     0  Ss      nanslp   0xc07cf224 cron
  1717     1  1717    25  Ss      pause    0xc7c42058 sendmail
  1713     1  1713     0  Ss      select   0xc7577ce4 sendmail
  1707     1  1707     0  Ss      select   0xc75cf8e4 sshd
  1621     1  1621   556  Ss      select   0xc80501a4 dbus-daemon
  1416     1  1416     0  Ss      select   0xc75b7324 ntpd
  1380  1374  1374     0  S       piperd   0xc804d188 logger
  1379  1374  1374     0  S       select   0xc7577aa4 ssh
  1374     1  1374     0  Ss      wait     0xc7f6a7f8 sh
  1200     1  1200    53  Ss      (threaded)          named
 100140                   S       kqread   0xc703e380 named
 100139                   S       ucond    0xc75c4e40 named
 100138                   S       ucond    0xc75c4b00 named
 100137                   S       ucond    0xc75990c0 named
 100136                   S       ucond    0xc7394d80 named
 100135                   S       ucond    0xc7381800 named
 100121                   S       sigwait  0xe9633be4 named
  1101     1  1101     0  Ss      select   0xc75b79e4 syslogd
   700     1   700     0  Ss      select   0xc75cf2a4 devd
   100     1   100     0  Ss      pause    0xc71b6850 adjkerntz
    20     0     0     0  SL      sdflush  0xc0949920 [softdepflush]
    19     0     0     0  SL      vlruwt   0xc759f550 [vnlru]
    18     0     0     0  SL      ufs      0xcd466dc8 [syncer]
    17     0     0     0  SL      psleep   0xc093b348 [bufdaemon]
    16     0     0     0  SL      pgzero   0xc094a5f4 [pagezero]
    15     0     0     0  SL      psleep   0xc094a21c [vmdaemon]
    14     0     0     0  SL      psleep   0xc094a1e4 [pagedaemon]
     9     0     0     0  SL      ccb_scan 0xc07b8dd4 [xpt_thrd]
     8     0     0     0  SL      -        0xc71f763c [fdc0]
     7     0     0     0  SL      -        0xc72a0000 [fw0_probe]
    13     0     0     0  SL      (threaded)          usb
 100053                   D       -        0xc7284d0c [usbus4]
 100052                   D       -        0xc7284cdc [usbus4]
 100051                   D       -        0xc7284cac [usbus4]
 100050                   D       -        0xc7284c7c [usbus4]
 100049                   D       -        0xc726ddac [usbus3]
 100048                   D       -        0xc726dd7c [usbus3]
 100047                   D       -        0xc726dd4c [usbus3]
 100046                   D       -        0xc726dd1c [usbus3]
 100045                   D       -        0xc725cdac [usbus2]
 100044                   D       -        0xc725cd7c [usbus2]
 100043                   D       -        0xc725cd4c [usbus2]
 100042                   D       -        0xc725cd1c [usbus2]
 100040                   D       -        0xc7248dac [usbus1]
 100039                   D       -        0xc7248d7c [usbus1]
 100038                   D       -        0xc7248d4c [usbus1]
 100037                   D       -        0xc7248d1c [usbus1]
 100035                   D       -        0xc721ddac [usbus0]
 100034                   D       -        0xc721dd7c [usbus0]
 100033                   D       -        0xc721dd4c [usbus0]
 100032                   D       -        0xc721dd1c [usbus0]
     6     0     0     0  SL      crypto_r 0xc094962c [crypto returns]
     5     0     0     0  SL      crypto_w 0xc0949608 [crypto]
    12     0     0     0  SL      -        0xc07cf084 [yarrow]
     4     0     0     0  SL      -        0xc07cce64 [g_down]
     3     0     0     0  SL      -        0xc07cce60 [g_up]
     2     0     0     0  SL      -        0xc07cce58 [g_event]
    11     0     0     0  WL      (threaded)          intr
 100172                   I                           [irq256: vgapci0]
 100060                   I                           [swi0: uart]
 100059                   I                           [irq12: psm0]
 100058                   I                           [irq1: atkbd0]
 100056                   I                           [irq14: ata0]
 100041                   I                           [irq18: fwohci0+]
 100036                   I                           [irq19: uhci1++]
 100031                   I                           [irq23: uhci0 ehci0]
 100029                   I                           [irq16: atapci0+]
 100028                   I                           [irq257: hdac0]
 100027                   I                           [irq9: acpi0]
 100025                   I                           [swi6: task queue]
 100024                   I                           [swi6: Giant taskq]
 100019                   I                           [swi5: +]
 100018                   I                           [swi2: cambio]
 100011                   I                           [swi4: clock]
 100010                   I                           [swi4: clock]
 100009                   I                           [swi4: clock]
 100008                   I                           [swi4: clock]
 100007                   I                           [swi3: vm]
 100006                   I                           [swi1: netisr 0]
    10     0     0     0  RL      (threaded)          idle
 100005                   Run     CPU 0               [idle: cpu0]
 100004                   CanRun                      [idle: cpu1]
 100003                   Run     CPU 2               [idle: cpu2]
 100002                   Run     CPU 3               [idle: cpu3]
     1     0     1     0  SLs     wait     0xc7043000 [init]
     0     0     0     0  RLs     (threaded)          kernel
 100120                   D       -        0xc75991c0 [dummynet]
 100061                   Run     CPU 1               [deadlkres]
 100054                   D       -        0xc727ecc0 [fw0_taskq]
 100030                   D       -        0xc7204300 [em0 taskq]
 100026                   D       -        0xc71a4ac0 [kqueue taskq]
 100023                   D       -        0xc71a5040 [acpi_task_2]
 100022                   D       -        0xc71a5040 [acpi_task_1]
 100021                   D       -        0xc71a5040 [acpi_task_0]
 100020                   D       -        0xc71a5080 [thread taskq]
 100000                   D       sched    0xc07ccf00 [swapper]
 db> trace
 
 Tracing pid 0 tid 100061 td 0xc7385000
 kdb_enter(c0740afa,c0740afa,c073a07e,c6bcbcd0,1,...) at 0xc053618a =
 kdb_enter+0x3a
 panic(c073a07e,c07166d9,c7c47940,3574e,c7c425d8,...) at 0xc05077f6 = panic+0x136
 deadlkres(0,c6bcbd38,c073bc0b,343,c07ccf00,...) at 0xc04c4a1a = deadlkres+0x27a
 fork_exit(c04c47a0,0,c6bcbd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6bcbd70, ebp = 0 ---
 db> show pcpu
 
 cpuid        = 1
 dynamic pcpu    = 0x60a4f00
 curthread    = 0xc7385000: pid 0 "deadlkres"
 curpcb       = 0xc6bcbd90
 fpcurthread  = none
 idlethread   = 0xc70446f0: pid 10 "idle: cpu1"
 APIC ID      = 1
 currentldt   = 0x50
 spin locks held:
 db> show allpcpu
 
 Current CPU: 1
 
 cpuid        = 0
 dynamic pcpu    = 0x86cf00
 curthread    = 0xc70444a0: pid 10 "idle: cpu0"
 curpcb       = 0xc6883d90
 fpcurthread  = none
 idlethread   = 0xc70444a0: pid 10 "idle: cpu0"
 APIC ID      = 0
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 1
 dynamic pcpu    = 0x60a4f00
 curthread    = 0xc7385000: pid 0 "deadlkres"
 curpcb       = 0xc6bcbd90
 fpcurthread  = none
 idlethread   = 0xc70446f0: pid 10 "idle: cpu1"
 APIC ID      = 1
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 2
 dynamic pcpu    = 0x60a7f00
 curthread    = 0xc7044940: pid 10 "idle: cpu2"
 curpcb       = 0xc687dd90
 fpcurthread  = none
 idlethread   = 0xc7044940: pid 10 "idle: cpu2"
 APIC ID      = 2
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 3
 dynamic pcpu    = 0x60aaf00
 curthread    = 0xc7044b90: pid 10 "idle: cpu3"
 curpcb       = 0xc687ad90
 fpcurthread  = none
 idlethread   = 0xc7044b90: pid 10 "idle: cpu3"
 APIC ID      = 3
 currentldt   = 0x50
 spin locks held:
 
 db> show locks
 
 db> show alllocks
 
 Process 2131 (thunar-vfs-mime-cle) thread 0xc7c47940 (100108)
 Process 1952 (gam_server) thread 0xc7385b90 (100092)
 Process 18 (syncer) thread 0xc7383000 (100067)
 db> show lockedvnods
 
 Locked vnodes
 db> alltrace
 
 
 Tracing command sync pid 2143 tid 100114 td 0xc7c43940
 sched_switch(c7c43940,0,104,191,28b4be6b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7c43940,0,c0745097,260,50,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 = sleepq_wait+0x63
 __lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f =
 __lockmgr_args+0xb1f
 ffs_lock(e95fbb74,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1
 = ffs_lock+0xa1
 VOP_LOCK1_APV(c07a47c0,e95fbb74,c7c439e4,c07b7a20,cd466d70,...) at
 0xc070c945 = VOP_LOCK1_APV+0xb5
 _vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 = _vn_lock+0x78
 vget(cd466d70,80100,c7c43940,39a,c763b870,...) at 0xc05969db = vget+0xbb
 qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 = qsync+0x197
 ffs_sync(c763b870,2,c074fa1e,93,c763b870,...) at 0xc067a139 = ffs_sync+0x349
 sync(c7c43940,e95fbcf8,c,c,286,...) at 0xc059d2e5 = sync+0x145
 syscall(e95fbd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (36, FreeBSD ELF32, sync), eip = 0x480d6f5f, esp =
 0xbfbfec4c, ebp = 0xbfbfec58 ---
 
 Tracing command thunar-vfs-mime-cle pid 2131 tid 100108 td 0xc7c47940
 sched_switch(c7c47940,0,104,191,2fe57af9,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7c47940,0,c0745097,260,50,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 = sleepq_wait+0x63
 __lockmgr_args(cd466dc8,400400,cd466e30,0,0,...) at 0xc04f087f =
 __lockmgr_args+0xb1f
 ffs_lock(e95e3878,e95e3878,c04f808c,400400,cd466d70,...) at 0xc067e9f1
 = ffs_lock+0xa1
 VOP_LOCK1_APV(c07a47c0,e95e3878,e95e3920,c07b7a20,cd466d70,...) at
 0xc070c945 = VOP_LOCK1_APV+0xb5
 _vn_lock(cd466d70,400400,c076358b,15c,e95e38bc,...) at 0xc05a2fd8 =
 _vn_lock+0x78
 ufs_accessx(e95e3920,c0772d8f,0,1,cd466d70,...) at 0xc068bce4 = ufs_accessx+0xa4
 VOP_ACCESSX_APV(c07a47c0,e95e3920,cddc6560,c07b7780,cd466d70,...) at
 0xc070abd5 = VOP_ACCESSX_APV+0xc5
 ufs_delete_denied(c7c47940,48f4da,80000,e95e39c4,e95e39d4,...) at
 0xc06859e4 = ufs_delete_denied+0xb4
 ufs_lookup_(0,e95e3a0c,c070b115,e95e3a2c,c0772e9e,...) at 0xc06862f1 =
 ufs_lookup_+0x811
 ufs_lookup(e95e3a2c,c0772e9e,cd466d70,cd466d70,e95e3bf8,...) at
 0xc068661e = ufs_lookup+0x1e
 VOP_CACHEDLOOKUP_APV(c07a47c0,e95e3a2c,e95e3bf8,e95e3be4,caefd180,...)
 at 0xc070b115 = VOP_CACHEDLOOKUP_APV+0xc5
 vfs_cache_lookup(e95e3ab8,c074f40e,c07b7660,200000,e95e3bcc,...) at
 0xc0583d66 = vfs_cache_lookup+0xd6
 VOP_LOOKUP_APV(c07a47c0,e95e3ab8,c074dc99,30b,e95e3be4,...) at
 0xc070dac5 = VOP_LOOKUP_APV+0xe5
 lookup(e95e3bcc,c7819000,0,c5,c7c42550,...) at 0xc058ad13 = lookup+0x5c3
 namei(e95e3bcc,c7c479e4,c090d068,c07906bc,c076408d,...) at 0xc058ba22
 = namei+0x4c2
 kern_renameat(c7c47940,ffffff9c,28528280,ffffff9c,2852d3c0,...) at
 0xc059aa97 = kern_renameat+0x57
 kern_rename(c7c47940,28528280,2852d3c0,0,e95e3d2c,...) at 0xc059ae46 =
 kern_rename+0x36
 rename(c7c47940,e95e3cf8,c0770332,c07484d2,282,...) at 0xc059ae79 = rename+0x29
 syscall(e95e3d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (128, FreeBSD ELF32, rename), eip = 0x281bea9b, esp =
 0xbfbfe35c, ebp = 0xbfbfebd8 ---
 
 Tracing command gqview pid 1979 tid 100180 td 0xc80db000
 sched_switch(c80db000,0,104,191,1c15aa7b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80db000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(eb716aac,c04f811a,c804fd50,0,c80db000,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(c804fd64,0,eb716adc,101,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _cv_timedwait_sig(c804fd64,c804fd50,bb9,621,eb716b70,...) at
 0xc04c5690 = _cv_timedwait_sig+0x250
 seltdwait(eb716c60,eb716c68,52f,c80db000,eb716b60,...) at 0xc054b5ba =
 seltdwait+0x8a
 poll(c80db000,eb716cf8,c0770332,c0747b95,200282,...) at 0xc054ba80 = poll+0x300
 syscall(eb716d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28b27b0f, esp =
 0xbfbfea9c, ebp = 0xbfbfeab8 ---
 
 Tracing command gconfd-2 pid 1968 tid 100147 td 0xc8025940
 sched_switch(c8025940,0,104,191,5bd77f79,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8025940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(e96a1aac,c04f811a,cdb825d0,0,c8025940,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(cdb825e4,0,e96a1adc,101,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _cv_timedwait_sig(cdb825e4,cdb825d0,7531,621,e96a1b90,...) at
 0xc04c5690 = _cv_timedwait_sig+0x250
 seltdwait(e96a1c60,e96a1c68,52f,c8025940,e96a1b60,...) at 0xc054b5ba =
 seltdwait+0x8a
 poll(c8025940,e96a1cf8,c0770332,c0747b95,286,...) at 0xc054ba80 = poll+0x300
 syscall(e96a1d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28463b0f, esp =
 0xbfbfea1c, ebp = 0xbfbfea38 ---
 
 Tracing command soffice.bin pid 1965 tid 100193 td 0xcdb9e250
 sched_switch(cdb9e250,0,104,191,dbfe1e6b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(cdb9e250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c75c2e50,0,c073f15b,cdb9e250,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c75c2e64,0,eb73dadc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c75c2e64,c75c2e50,c074742e,621,eb73db68,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c7fc202c,c074742e,52f,cdb9e250,eb73db60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(cdb9e250,eb73dcf8,c0770332,c072d816,286,...) at 0xc054ba80 = poll+0x300
 syscall(eb73dd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2a8a1b0f, esp =
 0xbf6fbebc, ebp = 0xbf6fbed8 ---
 
 Tracing command soffice.bin pid 1965 tid 100192 td 0xcdb9e4a0
 sched_switch(cdb9e4a0,0,104,191,c99bad73,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,58,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(cdb9e4a0,0,c0745097,1a0,58,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(cdb93b82,58,c0747ac1,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(cdb93b82,c093b1b0,158,c0747ac1,0,...) at 0xc0510b84 = _sleep+0x354
 kern_accept(cdb9e4a0,8,0,0,0,...) at 0xc05751f0 = kern_accept+0x120
 accept(cdb9e4a0,eb73acf8,c0770332,c0748721,286,...) at 0xc0575661 = accept+0x41
 syscall(eb73ad38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (30, FreeBSD ELF32, accept), eip = 0x2a89209f, esp =
 0xbf7fc52c, ebp = 0xbf7fc548 ---
 
 Tracing command soffice.bin pid 1965 tid 100191 td 0xcdb9e6f0
 sched_switch(cdb9e6f0,0,104,191,b926aaea,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(cdb9e6f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(d6f06,c053fd30,cdb9e6f0,0,100,...) at 0xc0540377
 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(cdb83500,0,c074235e,100,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _sleep(cdb83500,c07d2a44,100,c074235e,d6f06,...) at 0xc0510b4e = _sleep+0x31e
 __umtx_op_cv_wait(cdb9e6f0,eb737cf8,eb737d2c,c06f41e5,cdb9e6f0,...) at
 0xc051ff78 = __umtx_op_cv_wait+0x558
 _umtx_op(cdb9e6f0,eb737cf8,c0770332,c0747f0c,202,...) at 0xc051b0c7 =
 _umtx_op+0x27
 syscall(eb737d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2a808227, esp =
 0xbf8fddbc, ebp = 0xbf8fdde8 ---
 
 Tracing command soffice.bin pid 1965 tid 100190 td 0xcdb9e940
 sched_switch(cdb9e940,0,104,191,646d55a2,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(cdb9e940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(2711,c053fd30,cdb9e940,1,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(cdb83540,0,c074235e,100,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _sleep(cdb83540,c07d2084,100,c074235e,2711,...) at 0xc0510b4e = _sleep+0x31e
 __umtx_op_cv_wait(cdb9e940,eb734cf8,eb734d2c,c06f41e5,cdb9e940,...) at
 0xc051ff78 = __umtx_op_cv_wait+0x558
 _umtx_op(cdb9e940,eb734cf8,c0770332,c0747f0c,202,...) at 0xc051b0c7 =
 _umtx_op+0x27
 syscall(eb734d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2a808227, esp =
 0xbf9feeac, ebp = 0xbf9feed8 ---
 
 Tracing command soffice.bin pid 1965 tid 100167 td 0xc8027940
 sched_switch(c8027940,0,104,191,ba406799,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8027940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(e96e7aac,c04f811a,cdb83250,0,c8027940,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(cdb83264,0,e96e7adc,101,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _cv_timedwait_sig(cdb83264,cdb83250,a7ade,621,e96e7b88,...) at
 0xc04c5690 = _cv_timedwait_sig+0x250
 seltdwait(e96e7c60,e96e7c68,52f,c8027940,e96e7b60,...) at 0xc054b5ba =
 seltdwait+0x8a
 poll(c8027940,e96e7cf8,c0770332,c0747b95,200286,...) at 0xc054ba80 = poll+0x300
 syscall(e96e7d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2a8a1b0f, esp =
 0xbfbfe4bc, ebp = 0xbfbfe4d8 ---
 
 Tracing command xfce4-xkb-plugin pid 1962 tid 100156 td 0xc80836f0
 sched_switch(c80836f0,0,104,191,5b37d11a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80836f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c8050390,0,c073f15b,c80836f0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c80503a4,0,e96c6adc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c80503a4,c8050390,c074742e,621,e96c6b70,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c75bea2c,c074742e,52f,c80836f0,e96c6b60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c80836f0,e96c6cf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300
 syscall(e96c6d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28ad3b0f, esp =
 0xbfbfe99c, ebp = 0xbfbfe9b8 ---
 
 Tracing command xfce4-mixer-plugin pid 1961 tid 100173 td 0xc80dd250
 sched_switch(c80dd250,0,104,191,413877a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80dd250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,cdb83290,0,c073f15b,c80dd250,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(cdb832a4,0,eb701adc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(cdb832a4,cdb83290,c074742e,621,eb701b68,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c758352c,c074742e,52f,c80dd250,eb701b60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c80dd250,eb701cf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300
 syscall(eb701d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2900cb0f, esp =
 0xbfbfe99c, ebp = 0xbfbfe9b8 ---
 
 Tracing command xfce4-menu-plugin pid 1960 tid 100104 td 0xc781e250
 sched_switch(c781e250,0,104,191,4d4ebc5e,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c781e250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c80505d0,0,c073f15b,c781e250,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c80505e4,0,e95abadc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c80505e4,c80505d0,c074742e,621,e95abb70,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c75c522c,c074742e,52f,c781e250,e95abb60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c781e250,e95abcf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300
 syscall(e95abd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28de2b0f, esp =
 0xbfbfe9ac, ebp = 0xbfbfe9c8 ---
 
 Tracing command xfce4-settings-help pid 1953 tid 100158 td 0xc8083250
 sched_switch(c8083250,0,104,191,5b3922a3,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8083250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c8050d90,0,c073f15b,c8083250,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c8050da4,0,e96ccadc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c8050da4,c8050d90,c074742e,621,e96ccb80,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c7fc562c,c074742e,52f,c8083250,e96ccb60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c8083250,e96cccf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300
 syscall(e96ccd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x288b3b0f, esp =
 0xbfbfea3c, ebp = 0xbfbfea58 ---
 
 Tracing command gam_server pid 1952 tid 100092 td 0xc7385b90
 sched_switch(c7385b90,0,104,191,2fe5a1ff,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7385b90,0,c0745097,260,cddc65b8,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(cddc65b8,50,c07393ba,4,1,...) at 0xc0540f63 = sleepq_wait+0x63
 __lockmgr_args(cddc65b8,200100,cddc6620,0,0,...) at 0xc04f0215 =
 __lockmgr_args+0x4b5
 ffs_lock(e957a7e0,c054944b,c074db65,200100,cddc6560,...) at 0xc067e9f1
 = ffs_lock+0xa1
 VOP_LOCK1_APV(c07a47c0,e957a7e0,c7385c34,c07b7a20,cddc6560,...) at
 0xc070c945 = VOP_LOCK1_APV+0xb5
 _vn_lock(cddc6560,200100,c074e6ce,82b,4,...) at 0xc05a2fd8 = _vn_lock+0x78
 vget(cddc6560,200100,c7385b90,50,0,...) at 0xc05969db = vget+0xbb
 vfs_hash_get(c763b870,48f4da,200000,c7385b90,e957a960,...) at
 0xc0589b5d = vfs_hash_get+0xed
 ffs_vgetf(c763b870,48f4da,200000,e957a960,0,...) at 0xc0679149 = ffs_vgetf+0x49
 ffs_vget(c763b870,48f4da,200000,e957a960,e957a970,...) at 0xc067968e =
 ffs_vget+0x2e
 ufs_lookup_(0,e957a9a8,c070b115,e957a9c8,c0772e9e,...) at 0xc06865a9 =
 ufs_lookup_+0xac9
 ufs_lookup(e957a9c8,c0772e9e,cd466d70,cd466d70,e957ab8c,...) at
 0xc068661e = ufs_lookup+0x1e
 VOP_CACHEDLOOKUP_APV(c07a47c0,e957a9c8,e957ab8c,e957ab78,caefd180,...)
 at 0xc070b115 = VOP_CACHEDLOOKUP_APV+0xc5
 vfs_cache_lookup(e957aa54,c074f40e,c07b7660,200000,e957ab60,...) at
 0xc0583d66 = vfs_cache_lookup+0xd6
 VOP_LOOKUP_APV(c07a47c0,e957aa54,c074dc99,30b,e957ab78,...) at
 0xc070dac5 = VOP_LOOKUP_APV+0xe5
 lookup(e957ab60,c7692c00,0,c5,c7688000,...) at 0xc058ad13 = lookup+0x5c3
 namei(e957ab60,c802a5a0,c757ac00,e957ac14,c04d8bde,...) at 0xc058ba22
 = namei+0x4c2
 kern_statat_vnhook(c7385b90,0,ffffff9c,2846ad00,0,...) at 0xc059b3f2 =
 kern_statat_vnhook+0x72
 kern_statat(c7385b90,0,ffffff9c,2846ad00,0,...) at 0xc059b54c = kern_statat+0x3c
 kern_stat(c7385b90,2846ad00,0,e957ac1c,c7385b90,...) at 0xc059b696 =
 kern_stat+0x36
 stat(c7385b90,e957acf8,c0770332,c07483c6,246,...) at 0xc059b73f = stat+0x2f
 syscall(e957ad38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (188, FreeBSD ELF32, stat), eip = 0x28386a23, esp =
 0xbfbfe98c, ebp = 0xbfbfea28 ---
 
 Tracing command sh pid 1943 tid 100148 td 0xc80256f0
 sched_switch(c80256f0,0,104,191,e36a908,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80256f0,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c80802a8,5c,c0747a51,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c80802a8,c8080330,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354
 kern_wait(c80256f0,ffffffff,e96a4c78,2,0,...) at 0xc04dbde6 = kern_wait+0xb46
 wait4(c80256f0,e96a4cf8,c0770332,c07482ab,216,...) at 0xc04dbe5b = wait4+0x3b
 syscall(e96a4d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815dbdb, esp =
 0xbfbfe9ec, ebp = 0xbfbfea08 ---
 
 Tracing command sh pid 1939 tid 100184 td 0xc80844a0
 sched_switch(c80844a0,0,104,191,c21d95d3,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80844a0,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c80dad48,5c,c0747a51,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c80dad48,c80dadd0,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354
 kern_wait(c80844a0,ffffffff,eb722c78,2,0,...) at 0xc04dbde6 = kern_wait+0xb46
 wait4(c80844a0,eb722cf8,c0770332,c0747a30,216,...) at 0xc04dbe5b = wait4+0x3b
 syscall(eb722d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815dbdb, esp =
 0xbfbfe99c, ebp = 0xbfbfe9b8 ---
 
 Tracing command xfsettingsd pid 1935 tid 100075 td 0xc7681250
 sched_switch(c7681250,0,104,191,b7150b60,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7681250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,cdb82e50,0,c073f15b,c7681250,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(cdb82e64,0,e9536adc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(cdb82e64,cdb82e50,c074742e,621,e9536b70,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c7fc492c,c074742e,52f,c7681250,e9536b60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c7681250,e9536cf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300
 syscall(e9536d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x287c0b0f, esp =
 0xbfbfeacc, ebp = 0xbfbfeae8 ---
 
 Tracing command xfdesktop pid 1932 tid 100149 td 0xc80254a0
 sched_switch(c80254a0,0,104,191,3355019d,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80254a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(e96a7aac,c04f811a,cdb829d0,0,c80254a0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(cdb829e4,0,e96a7adc,101,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _cv_timedwait_sig(cdb829e4,cdb829d0,3e9,621,e96a7b88,...) at
 0xc04c5690 = _cv_timedwait_sig+0x250
 seltdwait(e96a7c60,e96a7c68,52f,c80254a0,e96a7b60,...) at 0xc054b5ba =
 seltdwait+0x8a
 poll(c80254a0,e96a7cf8,c0770332,c0747b95,286,...) at 0xc054ba80 = poll+0x300
 syscall(e96a7d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28eb2b0f, esp =
 0xbfbfe6bc, ebp = 0xbfbfe6d8 ---
 
 Tracing command Thunar pid 1930 tid 100196 td 0xc781f6f0
 sched_switch(c781f6f0,0,104,191,2c27b007,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c781f6f0,0,c0745097,1a0,4c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c804d930,4c,c0747637,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c804d930,c804daa0,14c,c0747637,0,...) at 0xc0510b84 = _sleep+0x354
 pipe_read(c76f4690,eb746c5c,caefd180,0,c781f6f0,...) at 0xc054efa7 =
 pipe_read+0x3f7
 dofileread(eb746c5c,ffffffff,ffffffff,0,c76f4690,...) at 0xc054bf26 =
 dofileread+0x96
 kern_readv(c781f6f0,d,eb746c5c,eb746c7c,1,...) at 0xc054c258 = kern_readv+0x58
 read(c781f6f0,eb746cf8,c,c,200286,...) at 0xc054c36f = read+0x4f
 syscall(eb746d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28a68593, esp =
 0xbf5faeec, ebp = 0xbf5faf08 ---
 
 Tracing command Thunar pid 1930 tid 100128 td 0xc7f70940
 sched_switch(c7f70940,0,104,191,6c161109,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7f70940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(e964faac,c04f811a,c75cf110,0,c7f70940,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(c75cf124,0,e964fadc,101,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _cv_timedwait_sig(c75cf124,c75cf110,3e9,621,e964fb80,...) at
 0xc04c5690 = _cv_timedwait_sig+0x250
 seltdwait(e964fc60,e964fc68,52f,c7f70940,e964fb60,...) at 0xc054b5ba =
 seltdwait+0x8a
 poll(c7f70940,e964fcf8,c0770332,c0747b95,200286,...) at 0xc054ba80 = poll+0x300
 syscall(e964fd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28a0eb0f, esp =
 0xbfbfeadc, ebp = 0xbfbfeaf8 ---
 
 Tracing command xfce4-panel pid 1928 tid 100159 td 0xc8083000
 sched_switch(c8083000,0,104,191,5b37bb60,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8083000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(e96cfaac,c04f811a,c8050750,0,c8083000,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(c8050764,0,e96cfadc,101,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _cv_timedwait_sig(c8050764,c8050750,3db,621,e96cfb78,...) at
 0xc04c5690 = _cv_timedwait_sig+0x250
 seltdwait(e96cfc60,e96cfc68,52f,c8083000,e96cfb60,...) at 0xc054b5ba =
 seltdwait+0x8a
 poll(c8083000,e96cfcf8,c0770332,c0747b95,200282,...) at 0xc054ba80 = poll+0x300
 syscall(e96cfd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28d53b0f, esp =
 0xbfbfea3c, ebp = 0xbfbfea58 ---
 
 Tracing command xfwm4 pid 1926 tid 100071 td 0xc7681b90
 sched_switch(c7681b90,0,104,191,5b378290,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7681b90,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c8050150,0,c073f15b,c7681b90,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c8050164,0,e9525adc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c8050164,c8050150,c074742e,621,e9525b78,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c75c592c,c074742e,52f,c7681b90,e9525b60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c7681b90,e9525cf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300
 syscall(e9525d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28de9b0f, esp =
 0xbfbfe6ac, ebp = 0xbfbfe6c8 ---
 
 Tracing command xfconfd pid 1920 tid 100141 td 0xc8026940
 sched_switch(c8026940,0,104,191,40a1f3b2,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8026940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c8050b10,0,c073f15b,c8026940,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c8050b24,0,e968fadc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c8050b24,c8050b10,c074742e,621,e968fb70,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c75c5e2c,c074742e,52f,c8026940,e968fb60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c8026940,e968fcf8,c0770332,c07737a6,286,...) at 0xc054ba80 = poll+0x300
 syscall(e968fd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x283e8b0f, esp =
 0xbfbfe9ec, ebp = 0xbfbfea08 ---
 
 Tracing command dbus-daemon pid 1918 tid 100072 td 0xc7681940
 sched_switch(c7681940,0,104,191,b454ff12,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7681940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c8050910,0,c073f15b,c7681940,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c8050924,0,e9529adc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c8050924,c8050910,c074742e,621,e9529bb8,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c808962c,c074742e,52f,c7681940,e9529b60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c7681940,e9529cf8,c0770332,c0747f0c,282,...) at 0xc054ba80 = poll+0x300
 syscall(e9529d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x281a3b0f, esp =
 0xbfbfe7ac, ebp = 0xbfbfe7c8 ---
 
 Tracing command dbus-launch pid 1917 tid 100183 td 0xc80846f0
 sched_switch(c80846f0,0,104,191,3a99ea08,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80846f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c75b8810,0,c073f15b,c80846f0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c75b8824,0,eb71fa80,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c75b8824,c75b8810,c074742e,621,c76915e8,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c76915e8,d8,caefd180,c80846f0,c074b1b4,...) at 0xc054b5d2 =
 seltdwait+0xa2
 kern_select(c80846f0,6,bfbfeac8,0,bfbfe8c8,0,20,bfbfe600,cdb76000) at
 0xc054c884 = kern_select+0x504
 select(c80846f0,eb71fcf8,c0770332,c072d816,282,...) at 0xc054cbd6 = select+0x66
 syscall(eb71fd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x282f6513, esp =
 0xbfbfe5fc, ebp = 0xbfbfe628 ---
 
 Tracing command xfce4-session pid 1911 tid 100179 td 0xc80db250
 sched_switch(c80db250,0,104,191,b715754d,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80db250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c75b9190,0,c073f15b,c80db250,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c75b91a4,0,eb713adc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c75b91a4,c75b9190,c074742e,621,eb713ba8,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c783272c,c074742e,52f,c80db250,eb713b60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c80db250,eb713cf8,c0770332,c072d816,286,...) at 0xc054ba80 = poll+0x300
 syscall(eb713d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28df6b0f, esp =
 0xbfbfeadc, ebp = 0xbfbfeaf8 ---
 
 Tracing command bash pid 1862 tid 100091 td 0xc7680000
 sched_switch(c7680000,0,104,191,eb8a2f02,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7680000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,c7680000,c7680000,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7379870,0,c0749892,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c7379870,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 =
 _cv_wait_sig+0x240
 tty_wait(c7379800,c7379870,e9576c5c,1,0,...) at 0xc0559721 = tty_wait+0x71
 ttydisc_read(c7379800,e9576c5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c
 ttydev_read(c738fa00,e9576c5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa
 devfs_read_f(c7f4fa10,e9576c5c,caefd780,0,c7680000,...) at 0xc04a6c1e
 = devfs_read_f+0x7e
 dofileread(e9576c5c,ffffffff,ffffffff,0,c7f4fa10,...) at 0xc054bf26 =
 dofileread+0x96
 kern_readv(c7680000,0,e9576c5c,e9576c7c,1,...) at 0xc054c258 = kern_readv+0x58
 read(c7680000,e9576cf8,c0770332,c074792b,246,...) at 0xc054c36f = read+0x4f
 syscall(e9576d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x8127ca3, esp =
 0xbfbfe08c, ebp = 0xbfbfe0b8 ---
 
 Tracing command bash pid 1846 tid 100088 td 0xc7680250
 sched_switch(c7680250,0,104,191,1efe8d41,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7680250,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7688550,5c,c0747a51,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c7688550,c76885d8,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354
 kern_wait(c7680250,ffffffff,e956ac78,6,0,...) at 0xc04dbde6 = kern_wait+0xb46
 wait4(c7680250,e956acf8,c0770332,c074792b,286,...) at 0xc04dbe5b = wait4+0x3b
 syscall(e956ad38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x80e26df, esp =
 0xbfbfea2c, ebp = 0xbfbfea48 ---
 
 Tracing command xdm pid 1842 tid 100095 td 0xc76bd6f0
 sched_switch(c76bd6f0,0,104,191,cfae5dd4,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76bd6f0,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c781c550,5c,c0747a51,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c781c550,c781c5d8,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354
 kern_wait(c76bd6f0,ffffffff,e9590c78,0,0,...) at 0xc04dbde6 = kern_wait+0xb46
 wait4(c76bd6f0,e9590cf8,c0770332,c074792b,282,...) at 0xc04dbe5b = wait4+0x3b
 syscall(e9590d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2831fbdb, esp =
 0xbfbfe8cc, ebp = 0xbfbfe8e8 ---
 
 Tracing command Xorg pid 1839 tid 100081 td 0xc7680940
 sched_switch(c7680940,0,104,191,5b384353,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7680940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(e954ea50,c04f811a,c80507d0,0,c7680940,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(c80507e4,0,e954ea80,101,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _cv_timedwait_sig(c80507e4,c80507d0,3b737,621,c8085038,...) at
 0xc04c5690 = _cv_timedwait_sig+0x250
 seltdwait(e954ec2c,e954ec34,c7fcb900,c7680940,c074b1b4,...) at
 0xc054b5ba = seltdwait+0x8a
 kern_select(c7680940,1c,81e8a00,0,0,e954ec74,20,f3,7c830) at
 0xc054c884 = kern_select+0x504
 select(c7680940,e954ecf8,c0770332,c0747a40,200246,...) at 0xc054cbd6 =
 select+0x66
 syscall(e954ed38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x285ad513, esp =
 0xbfbfeb0c, ebp = 0xbfbfed78 ---
 
 Tracing command getty pid 1837 tid 100160 td 0xc8082b90
 sched_switch(c8082b90,0,104,191,1cef9f31,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8082b90,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c7390404,0,c073f15b,c8082b90,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7390490,0,e96d296c,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c7390490,c7390404,c07491e9,4fd,c7390400,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 tty_wait(c7390400,c7390490,0,4e,caf0d560,...) at 0xc0559721 = tty_wait+0x71
 ttydev_open(c738e600,3,2000,c8082b90,e96d2a08,...) at 0xc055b4a2 =
 ttydev_open+0x372
 devfs_open(e96d2ac0,c0772e58,caf0d560,0,e96d2ba8,...) at 0xc04a8768 =
 devfs_open+0xe8
 VOP_OPEN_APV(c07862c0,e96d2ac0,e96d2a4c,c054944b,0,...) at 0xc070afc5
 = VOP_OPEN_APV+0xc5
 vn_open_cred(e96d2ba8,e96d2c60,1,0,c7041400,...) at 0xc05a29a6 =
 vn_open_cred+0x456
 vn_open(e96d2ba8,e96d2c60,1,c76f4000,c054944b,...) at 0xc05a2acb = vn_open+0x3b
 kern_openat(c8082b90,ffffff9c,804fbc0,0,3,...) at 0xc05a1947 = kern_openat+0xe7
 kern_open(c8082b90,804fbc0,0,2,3,...) at 0xc05a1e45 = kern_open+0x35
 open(c8082b90,e96d2cf8,c,c,282,...) at 0xc05a1e80 = open+0x30
 syscall(e96d2d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (5, FreeBSD ELF32, open), eip = 0x2817ea83, esp =
 0xbfbfedac, ebp = 0xbfbfedd8 ---
 
 Tracing command xdm pid 1836 tid 100168 td 0xc80ddb90
 sched_switch(c80ddb90,0,104,191,c8a6f9b1,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80ddb90,0,c0745097,1a0,68,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c80d7058,68,c0740d48,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c80d7058,c80d7088,168,c0740d48,0,...) at 0xc0510b84 = _sleep+0x354
 kern_sigsuspend(c80ddb90,0,0,0,0,...) at 0xc050b7ae = kern_sigsuspend+0xae
 sigsuspend(c80ddb90,e96eacf8,c0770332,c074792b,246,...) at 0xc050b8bd
 = sigsuspend+0x4d
 syscall(e96ead38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (4, FreeBSD ELF32, write), eip = 0x28307d9b, esp =
 0xbfbfe85c, ebp = 0xbfbfe918 ---
 
 Tracing command getty pid 1835 tid 100078 td 0xc76c1250
 sched_switch(c76c1250,0,104,191,1cef0682,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76c1250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,c76c1250,c76c1250,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c71f7a70,0,c0749892,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c71f7a70,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 =
 _cv_wait_sig+0x240
 tty_wait(c71f7a00,c71f7a70,e9542c5c,1,0,...) at 0xc0559721 = tty_wait+0x71
 ttydisc_read(c71f7a00,e9542c5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c
 ttydev_read(c738f300,e9542c5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa
 devfs_read_f(c7691428,e9542c5c,c7041400,0,c76c1250,...) at 0xc04a6c1e
 = devfs_read_f+0x7e
 dofileread(e9542c5c,ffffffff,ffffffff,0,c7691428,...) at 0xc054bf26 =
 dofileread+0x96
 kern_readv(c76c1250,0,e9542c5c,e9542c7c,1,...) at 0xc054c258 = kern_readv+0x58
 read(c76c1250,e9542cf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f
 syscall(e9542d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp =
 0xbfbfedbc, ebp = 0xbfbfede8 ---
 
 Tracing command getty pid 1834 tid 100085 td 0xc76c06f0
 sched_switch(c76c06f0,0,104,191,1d0ddcfc,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76c06f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,c76c06f0,c76c06f0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c71f7e70,0,c0749892,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c71f7e70,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 =
 _cv_wait_sig+0x240
 tty_wait(c71f7e00,c71f7e70,e955ec5c,1,0,...) at 0xc0559721 = tty_wait+0x71
 ttydisc_read(c71f7e00,e955ec5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c
 ttydev_read(c738f400,e955ec5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa
 devfs_read_f(c767b5b0,e955ec5c,c7041400,0,c76c06f0,...) at 0xc04a6c1e
 = devfs_read_f+0x7e
 dofileread(e955ec5c,ffffffff,ffffffff,0,c767b5b0,...) at 0xc054bf26 =
 dofileread+0x96
 kern_readv(c76c06f0,0,e955ec5c,e955ec7c,1,...) at 0xc054c258 = kern_readv+0x58
 read(c76c06f0,e955ecf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f
 syscall(e955ed38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp =
 0xbfbfedbc, ebp = 0xbfbfede8 ---
 
 Tracing command getty pid 1833 tid 100089 td 0xc76c0250
 sched_switch(c76c0250,0,104,191,1ce387c1,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76c0250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,c76c0250,c76c0250,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7379070,0,c0749892,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c7379070,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 =
 _cv_wait_sig+0x240
 tty_wait(c7379000,c7379070,e956ec5c,1,0,...) at 0xc0559721 = tty_wait+0x71
 ttydisc_read(c7379000,e956ec5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c
 ttydev_read(c738f500,e956ec5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa
 devfs_read_f(c8085930,e956ec5c,c7041400,0,c76c0250,...) at 0xc04a6c1e
 = devfs_read_f+0x7e
 dofileread(e956ec5c,ffffffff,ffffffff,0,c8085930,...) at 0xc054bf26 =
 dofileread+0x96
 kern_readv(c76c0250,0,e956ec5c,e956ec7c,1,...) at 0xc054c258 = kern_readv+0x58
 read(c76c0250,e956ecf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f
 syscall(e956ed38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp =
 0xbfbfedbc, ebp = 0xbfbfede8 ---
 
 Tracing command getty pid 1832 tid 100083 td 0xc76c0940
 sched_switch(c76c0940,0,104,191,1ceaf1fb,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76c0940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,c76c0940,c76c0940,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7379470,0,c0749892,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c7379470,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 =
 _cv_wait_sig+0x240
 tty_wait(c7379400,c7379470,e9556c5c,1,0,...) at 0xc0559721 = tty_wait+0x71
 ttydisc_read(c7379400,e9556c5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c
 ttydev_read(c738f600,e9556c5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa
 devfs_read_f(c76f4070,e9556c5c,c7041400,0,c76c0940,...) at 0xc04a6c1e
 = devfs_read_f+0x7e
 dofileread(e9556c5c,ffffffff,ffffffff,0,c76f4070,...) at 0xc054bf26 =
 dofileread+0x96
 kern_readv(c76c0940,0,e9556c5c,e9556c7c,1,...) at 0xc054c258 = kern_readv+0x58
 read(c76c0940,e9556cf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f
 syscall(e9556d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp =
 0xbfbfedbc, ebp = 0xbfbfede8 ---
 
 Tracing command getty pid 1831 tid 100127 td 0xc7f70b90
 sched_switch(c7f70b90,0,104,191,1d0d5c35,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7f70b90,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,c7f70b90,c7f70b90,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c71f7c70,0,c0749892,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c71f7c70,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 =
 _cv_wait_sig+0x240
 tty_wait(c71f7c00,c71f7c70,e964bc5c,1,0,...) at 0xc0559721 = tty_wait+0x71
 ttydisc_read(c71f7c00,e964bc5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c
 ttydev_read(c738f700,e964bc5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa
 devfs_read_f(c7691af0,e964bc5c,c7041400,0,c7f70b90,...) at 0xc04a6c1e
 = devfs_read_f+0x7e
 dofileread(e964bc5c,ffffffff,ffffffff,0,c7691af0,...) at 0xc054bf26 =
 dofileread+0x96
 kern_readv(c7f70b90,0,e964bc5c,e964bc7c,1,...) at 0xc054c258 = kern_readv+0x58
 read(c7f70b90,e964bcf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f
 syscall(e964bd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp =
 0xbfbfedbc, ebp = 0xbfbfede8 ---
 
 Tracing command getty pid 1830 tid 100130 td 0xc7f704a0
 sched_switch(c7f704a0,0,104,191,1cfa1c5b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7f704a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,c7f704a0,c7f704a0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7379270,0,c0749892,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c7379270,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 =
 _cv_wait_sig+0x240
 tty_wait(c7379200,c7379270,e9657c5c,1,0,...) at 0xc0559721 = tty_wait+0x71
 ttydisc_read(c7379200,e9657c5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c
 ttydev_read(c738f800,e9657c5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa
 devfs_read_f(c767b380,e9657c5c,c7041400,0,c7f704a0,...) at 0xc04a6c1e
 = devfs_read_f+0x7e
 dofileread(e9657c5c,ffffffff,ffffffff,0,c767b380,...) at 0xc054bf26 =
 dofileread+0x96
 kern_readv(c7f704a0,0,e9657c5c,e9657c7c,1,...) at 0xc054c258 = kern_readv+0x58
 read(c7f704a0,e9657cf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f
 syscall(e9657d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp =
 0xbfbfedbc, ebp = 0xbfbfede8 ---
 
 Tracing command login pid 1829 tid 100094 td 0xc76bd940
 sched_switch(c76bd940,0,104,191,4cf2740b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76bd940,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c768aaa0,5c,c0747a51,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c768aaa0,c768ab28,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354
 kern_wait(c76bd940,736,e9583c78,0,0,...) at 0xc04dbde6 = kern_wait+0xb46
 wait4(c76bd940,e9583cf8,c,c,216,...) at 0xc04dbe5b = wait4+0x3b
 syscall(e9583d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28120bdb, esp =
 0xbfbfed2c, ebp = 0xbfbfed48 ---
 
 Tracing command login pid 1828 tid 100163 td 0xc80824a0
 sched_switch(c80824a0,0,104,191,5be7b466,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80824a0,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c80d7d48,5c,c0747a51,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c80d7d48,c80d7dd0,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354
 kern_wait(c80824a0,746,e96dbc78,0,0,...) at 0xc04dbde6 = kern_wait+0xb46
 wait4(c80824a0,e96dbcf8,c,c,216,...) at 0xc04dbe5b = wait4+0x3b
 syscall(e96dbd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28120bdb, esp =
 0xbfbfed2c, ebp = 0xbfbfed48 ---
 
 Tracing command inetd pid 1802 tid 100103 td 0xc781e4a0
 sched_switch(c781e4a0,0,104,191,d72096,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c781e4a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c7394dd0,0,c073f15b,c781e4a0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7394de4,0,e95a8a80,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c7394de4,c7394dd0,c074742e,621,c767b0a8,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c767b0a8,58,c7041400,c781e4a0,c0765b58,...) at 0xc054b5d2 =
 seltdwait+0xa2
 kern_select(c781e4a0,7,bfbfdfc0,0,0,0,20,bfbfdf80,c781a000) at
 0xc054c884 = kern_select+0x504
 select(c781e4a0,e95a8cf8,c0770332,c0747aa4,202,...) at 0xc054cbd6 = select+0x66
 syscall(e95a8d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x281a3513, esp =
 0xbfbfdf7c, ebp = 0xbfbfee08 ---
 
 Tracing command moused pid 1770 tid 100090 td 0xc76c0000
 sched_switch(c76c0000,0,104,191,ab40f8dd,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76c0000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c804f510,0,c073f15b,c76c0000,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c804f524,0,e9572a80,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c804f524,c804f510,c074742e,621,c7edb690,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c7edb690,58,c7041400,c76c0000,0,...) at 0xc054b5d2 = seltdwait+0xa2
 kern_select(c76c0000,6,bfbfea34,0,0,0,20,bfbfe9c0,c76bb000) at
 0xc054c884 = kern_select+0x504
 select(c76c0000,e9572cf8,c0770332,c0752f28,246,...) at 0xc054cbd6 = select+0x66
 syscall(e9572d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x281ac513, esp =
 0xbfbfe9bc, ebp = 0xbfbfeb58 ---
 
 Tracing command cron pid 1724 tid 100097 td 0xc76bd250
 sched_switch(c76bd250,0,104,191,9e5cb557,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76bd250,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(dea9,c053fd30,c76bd250,1,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(c07cf224,5c,c0741eb4,100,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _sleep(c07cf224,0,15c,c0741eb4,dea9,...) at 0xc0510b4e = _sleep+0x31e
 kern_nanosleep(c76bd250,e9596c68,e9596c70,39,0,...) at 0xc0517f21 =
 kern_nanosleep+0xc1
 nanosleep(c76bd250,e9596cf8,c0770332,c0747f0c,206,...) at 0xc051995f =
 nanosleep+0x6f
 syscall(e9596d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2817bf17, esp =
 0xbfbfec8c, ebp = 0xbfbfecb8 ---
 
 Tracing command sendmail pid 1717 tid 100110 td 0xc7c474a0
 sched_switch(c7c474a0,0,104,191,e0648a6b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7c474a0,0,c0745097,1a0,68,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7c42058,68,c0740d48,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c7c42058,c7c42088,168,c0740d48,0,...) at 0xc0510b84 = _sleep+0x354
 kern_sigsuspend(c7c474a0,0,0,0,0,...) at 0xc050b7ae = kern_sigsuspend+0xae
 sigsuspend(c7c474a0,e95ebcf8,c0770332,c074792b,202,...) at 0xc050b8bd
 = sigsuspend+0x4d
 syscall(e95ebd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (4, FreeBSD ELF32, write), eip = 0x2832fd9b, esp =
 0xbfbfda9c, ebp = 0xbfbfdac8 ---
 
 Tracing command sendmail pid 1713 tid 100087 td 0xc76c04a0
 sched_switch(c76c04a0,0,104,191,95606d23,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76c04a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(e9566a50,c04f811a,c7577cd0,0,c76c04a0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(c7577ce4,0,e9566a80,101,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _cv_timedwait_sig(c7577ce4,c7577cd0,1389,621,c80ea188,...) at
 0xc04c5690 = _cv_timedwait_sig+0x250
 seltdwait(e9566c2c,e9566c34,c75cb480,c76c04a0,c159b014,...) at
 0xc054b5ba = seltdwait+0x8a
 kern_select(c76c04a0,8,bfbfd8bc,0,0,e9566c74,20,5,0) at 0xc054c884 =
 kern_select+0x504
 select(c76c04a0,e9566cf8,c0770332,c0747e6b,202,...) at 0xc054cbd6 = select+0x66
 syscall(e9566d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x283d6513, esp =
 0xbfbfd02c, ebp = 0xbfbfdac8 ---
 
 Tracing command sshd pid 1707 tid 100086 td 0xc76804a0
 sched_switch(c76804a0,0,104,191,9bb47272,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76804a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c75cf8d0,0,c073f15b,c76804a0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c75cf8e4,0,e9562a80,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c75cf8e4,c75cf8d0,c074742e,621,c7674b60,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c7674b60,58,c75cb180,c76804a0,c076500d,...) at 0xc054b5d2 =
 seltdwait+0xa2
 kern_select(c76804a0,5,286090ac,0,0,0,20,bfbfe2a0,c76887f8) at
 0xc054c884 = kern_select+0x504
 select(c76804a0,e9562cf8,c,c,206,...) at 0xc054cbd6 = select+0x66
 syscall(e9562d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9513, esp =
 0xbfbfe29c, ebp = 0xbfbfee38 ---
 
 Tracing command dbus-daemon pid 1621 tid 100170 td 0xc80dd6f0
 sched_switch(c80dd6f0,0,104,191,850a5cc,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80dd6f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c8050190,0,c073f15b,c80dd6f0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c80501a4,0,e96f0adc,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c80501a4,c8050190,c074742e,621,e96f0b78,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c7589e2c,c074742e,52f,c80dd6f0,e96f0b60,...) at 0xc054b5d2 =
 seltdwait+0xa2
 poll(c80dd6f0,e96f0cf8,c0770332,c0747f0c,282,...) at 0xc054ba80 = poll+0x300
 syscall(e96f0d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x281a3b0f, esp =
 0xbfbfe95c, ebp = 0xbfbfe978 ---
 
 Tracing command ntpd pid 1416 tid 100113 td 0xc7c43b90
 sched_switch(c7c43b90,0,104,191,6810e037,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7c43b90,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c75b7310,0,c073f15b,c7c43b90,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c75b7324,0,e95f7a80,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c75b7324,c75b7310,c074742e,621,c76f4d90,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c76f4d90,58,c7041400,c7c43b90,0,...) at 0xc054b5d2 = seltdwait+0xa2
 kern_select(c7c43b90,20,bfbfed18,0,0,0,20,bfbfecf0,c7c417f8) at
 0xc054c884 = kern_select+0x504
 select(c7c43b90,e95f7cf8,16,16,212,...) at 0xc054cbd6 = select+0x66
 syscall(e95f7d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x28354513, esp =
 0xbfbfecec, ebp = 0xbfbfedb8 ---
 
 Tracing command logger pid 1380 tid 100076 td 0xc7681000
 sched_switch(c7681000,0,104,191,bc02b312,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7681000,0,c0745097,1a0,4c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c804d188,4c,c0747637,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c804d188,c804d2f8,14c,c0747637,0,...) at 0xc0510b84 = _sleep+0x354
 pipe_read(c7edb850,e953ac5c,c7041400,0,c7681000,...) at 0xc054efa7 =
 pipe_read+0x3f7
 dofileread(e953ac5c,ffffffff,ffffffff,0,c7edb850,...) at 0xc054bf26 =
 dofileread+0x96
 kern_readv(c7681000,0,e953ac5c,e953ac7c,1,...) at 0xc054c258 = kern_readv+0x58
 read(c7681000,e953acf8,c0770332,c0747c21,216,...) at 0xc054c36f = read+0x4f
 syscall(e953ad38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2817d593, esp =
 0xbfbfe93c, ebp = 0xbfbfe958 ---
 
 Tracing command ssh pid 1379 tid 100155 td 0xc8083940
 sched_switch(c8083940,0,104,191,bdba131a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8083940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c7577a90,0,c073f15b,c8083940,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7577aa4,0,e96b9a80,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c7577aa4,c7577a90,c074742e,621,c767bb98,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c767bb98,58,c7f7db00,c8083940,246,...) at 0xc054b5d2 = seltdwait+0xa2
 kern_select(c8083940,5,285010b0,285010b4,0,0,20,bfbfbf30,c807f000) at
 0xc054c884 = kern_select+0x504
 select(c8083940,e96b9cf8,c0770332,c073d197,206,...) at 0xc054cbd6 = select+0x66
 syscall(e96b9d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x28381513, esp =
 0xbfbfbf2c, ebp = 0xbfbfbfd8 ---
 
 Tracing command sh pid 1374 tid 100129 td 0xc7f706f0
 sched_switch(c7f706f0,0,104,191,db29b8dc,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7f706f0,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7f6a7f8,5c,c0747a51,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c7f6a7f8,c7f6a880,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354
 kern_wait(c7f706f0,ffffffff,e9653c78,2,0,...) at 0xc04dbde6 = kern_wait+0xb46
 wait4(c7f706f0,e9653cf8,c0770332,c0747a30,216,...) at 0xc04dbe5b = wait4+0x3b
 syscall(e9653d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815dbdb, esp =
 0xbfbfea4c, ebp = 0xbfbfea68 ---
 
 Tracing command named pid 1200 tid 100140 td 0xc8026b90
 sched_switch(c8026b90,0,104,191,6e86129a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,58,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8026b90,0,c0745097,1a0,58,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c703e380,58,c073b7b7,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c703e380,c703e380,158,c073b7b7,0,...) at 0xc0510b84 = _sleep+0x354
 kern_kevent(c8026b90,9,0,40,e968cc5c,...) at 0xc04d8514 = kern_kevent+0x364
 kevent(c8026b90,e968ccf8,c0770332,c072d816,246,...) at 0xc04d8e4b = kevent+0x19b
 syscall(e968cd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2849f07b, esp =
 0xbf4f9e0c, ebp = 0xbf4f9f98 ---
 
 Tracing command named pid 1200 tid 100139 td 0xc8027000
 sched_switch(c8027000,0,104,191,a96622fc,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8027000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(119d12,c053fd30,c8027000,2,100,...) at 0xc0540377
 = sleepq_catch_signals+0xb7
 sleepq_timedwait_sig(c75c4e40,0,c074235e,100,0,...) at 0xc0540daa =
 sleepq_timedwait_sig+0x1a
 _sleep(c75c4e40,c07d18cc,100,c074235e,119d12,...) at 0xc0510b4e = _sleep+0x31e
 __umtx_op_cv_wait(c8027000,e9689cf8,e9689d2c,c06f41e5,c8027000,...) at
 0xc051ff78 = __umtx_op_cv_wait+0x558
 _umtx_op(c8027000,e9689cf8,c0770332,c0747f0c,202,...) at 0xc051b0c7 =
 _umtx_op+0x27
 syscall(e9689d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp =
 0xbf5faddc, ebp = 0xbf5fae08 ---
 
 Tracing command named pid 1200 tid 100138 td 0xc8027250
 sched_switch(c8027250,0,104,191,fd997d29,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c8027250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c75c4b00,0,c074235e,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c75c4b00,c07d1a6c,100,c074235e,0,...) at 0xc0510b84 = _sleep+0x354
 __umtx_op_cv_wait(c8027250,e9686cf8,e9686d2c,c06f41e5,c8027250,...) at
 0xc051fe96 = __umtx_op_cv_wait+0x476
 _umtx_op(c8027250,e9686cf8,c0770332,c0747b73,202,...) at 0xc051b0c7 =
 _umtx_op+0x27
 syscall(e9686d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp =
 0xbf6fbe9c, ebp = 0xbf6fbec8 ---
 
 Tracing command named pid 1200 tid 100137 td 0xc80274a0
 sched_switch(c80274a0,0,104,191,6a100e2d,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80274a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c75990c0,0,c074235e,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c75990c0,c07d1a6c,100,c074235e,0,...) at 0xc0510b84 = _sleep+0x354
 __umtx_op_cv_wait(c80274a0,e9683cf8,e9683d2c,c06f41e5,c80274a0,...) at
 0xc051fe96 = __umtx_op_cv_wait+0x476
 _umtx_op(c80274a0,e9683cf8,c0770332,c0747b73,202,...) at 0xc051b0c7 =
 _umtx_op+0x27
 syscall(e9683d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp =
 0xbf7fce9c, ebp = 0xbf7fcec8 ---
 
 Tracing command named pid 1200 tid 100136 td 0xc80276f0
 sched_switch(c80276f0,0,104,191,fdbd72ee,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c80276f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7394d80,0,c074235e,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c7394d80,c07d1a6c,100,c074235e,0,...) at 0xc0510b84 = _sleep+0x354
 __umtx_op_cv_wait(c80276f0,e9680cf8,e9680d2c,c06f41e5,c80276f0,...) at
 0xc051fe96 = __umtx_op_cv_wait+0x476
 _umtx_op(c80276f0,e9680cf8,c0770332,c0747b73,202,...) at 0xc051b0c7 =
 _umtx_op+0x27
 syscall(e9680d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp =
 0xbf8fde9c, ebp = 0xbf8fdec8 ---
 
 Tracing command named pid 1200 tid 100135 td 0xc7385940
 sched_switch(c7385940,0,104,191,6e8997e7,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7385940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7381800,0,c074235e,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c7381800,c07d1a6c,100,c074235e,0,...) at 0xc0510b84 = _sleep+0x354
 __umtx_op_cv_wait(c7385940,e967dcf8,e967dd2c,c06f41e5,c7385940,...) at
 0xc051fe96 = __umtx_op_cv_wait+0x476
 _umtx_op(c7385940,e967dcf8,c0770332,c0747b73,202,...) at 0xc051b0c7 =
 _umtx_op+0x27
 syscall(e967dd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp =
 0xbf9fee9c, ebp = 0xbf9feec8 ---
 
 Tracing command named pid 1200 tid 100121 td 0xc7f71b90
 sched_switch(c7f71b90,0,104,191,4dc27c2b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7f71b90,0,c0745097,1a0,68,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(e9633be4,68,c0740d4e,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(e9633be4,c7f6bdd0,168,c0740d4e,0,...) at 0xc0510b84 = _sleep+0x354
 kern_sigtimedwait(c7f71b90,4003,0,0,0,...) at 0xc050bb4c =
 kern_sigtimedwait+0x28c
 sigwait(c7f71b90,e9633cf8,c0770332,c0748721,200282,...) at 0xc050bfb7
 = sigwait+0x77
 syscall(e9633d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (429, FreeBSD ELF32, sigwait), eip = 0x284146db, esp =
 0xbfbfec9c, ebp = 0xbfbfecc8 ---
 
 Tracing command syslogd pid 1101 tid 100123 td 0xc7f716f0
 sched_switch(c7f716f0,0,104,191,ddc5cff9,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7f716f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c75b79d0,0,c073f15b,c7f716f0,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c75b79e4,0,e963ba80,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c75b79e4,c75b79d0,c074742e,621,c7675e00,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c7675e00,58,c7041400,c7f716f0,0,...) at 0xc054b5d2 = seltdwait+0xa2
 kern_select(c7f716f0,b,282310ac,0,0,0,20,bfbfe260,c7f6b7f8) at
 0xc054c884 = kern_select+0x504
 select(c7f716f0,e963bcf8,c0770332,c0747a40,246,...) at 0xc054cbd6 = select+0x66
 syscall(e963bd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x28192513, esp =
 0xbfbfe25c, ebp = 0xbfbfedc8 ---
 
 Tracing command devd pid 700 tid 100101 td 0xc781e940
 sched_switch(c781e940,0,104,191,a3e23ce5,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c781e940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_catch_signals(c04f811a,c75cf290,0,c073f15b,c781e940,...) at
 0xc0540377 = sleepq_catch_signals+0xb7
 sleepq_wait_sig(c75cf2a4,0,e95a2a80,101,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _cv_wait_sig(c75cf2a4,c75cf290,c074742e,621,c767b578,...) at
 0xc04c5ce0 = _cv_wait_sig+0x240
 seltdwait(c767b578,58,c7041400,c781e940,c05494ec,...) at 0xc054b5d2 =
 seltdwait+0xa2
 kern_select(c781e940,6,bfbfe9a0,0,0,0,20,bfbfe970,c781a550) at
 0xc054c884 = kern_select+0x504
 select(c781e940,e95a2cf8,c0770332,c07484d2,206,...) at 0xc054cbd6 = select+0x66
 syscall(e95a2d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x8088343, esp =
 0xbfbfe96c, ebp = 0xbfbfee48 ---
 
 Tracing command adjkerntz pid 100 tid 100073 td 0xc76816f0
 sched_switch(c76816f0,0,104,191,5022b0eb,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c76816f0,0,c0745097,1a0,68,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c71b6850,68,c0740d48,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c71b6850,c71b6880,168,c0740d48,0,...) at 0xc0510b84 = _sleep+0x354
 kern_sigsuspend(c76816f0,0,0,0,0,...) at 0xc050b7ae = kern_sigsuspend+0xae
 sigsuspend(c76816f0,e952dcf8,c0770332,c0747e6b,202,...) at 0xc050b8bd
 = sigsuspend+0x4d
 syscall(e952dd38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (4, FreeBSD ELF32, write), eip = 0x280d6d9b, esp =
 0xbfbfed6c, ebp = 0xbfbfee38 ---
 
 Tracing command softdepflush pid 20 tid 100069 td 0xc727d940
 sched_switch(c727d940,0,104,191,ebf6f65,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,44,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727d940,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_timedwait(c0949920,44,c0761839,0,0,...) at 0xc0540e6b =
 sleepq_timedwait+0x6b
 _sleep(c0949920,c09498c4,44,c0761839,3e8,...) at 0xc0510b69 = _sleep+0x339
 softdep_flush(0,e7332d38,c073bc0b,343,c759f2a8,...) at 0xc0674b44 =
 softdep_flush+0x244
 fork_exit(c0674900,0,e7332d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe7332d70, ebp = 0 ---
 
 Tracing command vnlru pid 19 tid 100068 td 0xc727db90
 sched_switch(c727db90,0,104,191,de62f5e,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727db90,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_timedwait(c759f550,50,c074f6cf,0,0,...) at 0xc0540e6b =
 sleepq_timedwait+0x6b
 _sleep(c759f550,c093b5d8,250,c074f6cf,3e8,...) at 0xc0510b69 = _sleep+0x339
 vnlru_proc(0,e732fd38,c073bc0b,343,c759f550,...) at 0xc0599157 = vnlru_proc+0xe7
 fork_exit(c0599070,0,e732fd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe732fd70, ebp = 0 ---
 
 Tracing command syncer pid 18 tid 100067 td 0xc7383000
 sched_switch(c7383000,0,104,191,e660f504,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7383000,0,c0745097,260,50,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 = sleepq_wait+0x63
 __lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f =
 __lockmgr_args+0xb1f
 ffs_lock(e732cb34,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1
 = ffs_lock+0xa1
 VOP_LOCK1_APV(c07a47c0,e732cb34,c73830a4,c07b7a20,cd466d70,...) at
 0xc070c945 = VOP_LOCK1_APV+0xb5
 _vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 = _vn_lock+0x78
 vget(cd466d70,80100,c7383000,39a,c763b870,...) at 0xc05969db = vget+0xbb
 qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 = qsync+0x197
 ffs_sync(c763b870,3,c074e6ce,d6b,c763b870,...) at 0xc067a139 = ffs_sync+0x349
 sync_fsync(e732cc7c,c0772c36,c76b8778,e732cc7c,c76b86b8,...) at
 0xc059812f = sync_fsync+0x18f
 VOP_FSYNC_APV(c0799bc0,e732cc7c,c074e6ce,6a5,c7383000,...) at
 0xc070a545 = VOP_FSYNC_APV+0xc5
 sync_vnode(c093b618,c093b604,3e8,6d4,4e20,...) at 0xc059840b = sync_vnode+0x16b
 sched_sync(0,e732cd38,c073bc0b,343,c759f7f8,...) at 0xc0598753 =
 sched_sync+0x273
 fork_exit(c05984e0,0,e732cd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe732cd70, ebp = 0 ---
 
 Tracing command bufdaemon pid 17 tid 100066 td 0xc7383250
 sched_switch(c7383250,0,104,191,3b094933,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,44,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7383250,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_timedwait(c093b348,44,c074cb15,0,0,...) at 0xc0540e6b =
 sleepq_timedwait+0x6b
 _sleep(c093b348,c093b34c,44,c074cb15,3e8,...) at 0xc0510b69 = _sleep+0x339
 buf_daemon(0,e7329d38,c073bc0b,343,c759faa0,...) at 0xc05807b8 =
 buf_daemon+0x138
 fork_exit(c0580680,0,e7329d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe7329d70, ebp = 0 ---
 
 Tracing command pagezero pid 16 tid 100065 td 0xc73834a0
 sched_switch(c73834a0,0,104,191,d7d993b1,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c73834a0,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_timedwait(c094a5f4,0,c07678c2,0,0,...) at 0xc0540e6b =
 sleepq_timedwait+0x6b
 _sleep(c094a5f4,c094a1b0,0,c07678c2,493e0,...) at 0xc0510b69 = _sleep+0x339
 vm_pagezero(0,e7326d38,c073bc0b,343,c759fd48,...) at 0xc06b42dc =
 vm_pagezero+0xdc
 fork_exit(c06b4200,0,e7326d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe7326d70, ebp = 0 ---
 
 Tracing command vmdaemon pid 15 tid 100064 td 0xc73836f0
 sched_switch(c73836f0,0,104,191,237688ca,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c73836f0,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_wait(c094a21c,68,c074cb15,0,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _sleep(c094a21c,c094a220,68,c074cb15,0,...) at 0xc0510b9b = _sleep+0x36b
 vm_daemon(0,e7323d38,c073bc0b,343,c70432a8,...) at 0xc06ae699 = vm_daemon+0x59
 fork_exit(c06ae640,0,e7323d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe7323d70, ebp = 0 ---
 
 Tracing command pagedaemon pid 14 tid 100063 td 0xc7383940
 sched_switch(c7383940,0,104,191,a04fab1a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,44,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7383940,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_timedwait(c094a1e4,44,c074cb15,0,0,...) at 0xc0540e6b =
 sleepq_timedwait+0x6b
 _sleep(c094a1e4,c094a1b0,44,c074cb15,1388,...) at 0xc0510b69 = _sleep+0x339
 vm_pageout(0,e7320d38,c073bc0b,343,c7043550,...) at 0xc06af79b =
 vm_pageout+0x2bb
 fork_exit(c06af4e0,0,e7320d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe7320d70, ebp = 0 ---
 
 Tracing command xpt_thrd pid 9 tid 100062 td 0xc7383b90
 sched_switch(c7383b90,0,104,191,1e90655d,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7383b90,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_wait(c07b8dd4,4c,c072889c,0,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _sleep(c07b8dd4,c07b8e18,4c,c072889c,0,...) at 0xc0510b9b = _sleep+0x36b
 xpt_scanner_thread(0,c6bced38,c073bc0b,343,c70437f8,...) at 0xc043fb87
 = xpt_scanner_thread+0x47
 fork_exit(c043fb40,0,c6bced38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6bced70, ebp = 0 ---
 
 Tracing command fdc0 pid 8 tid 100057 td 0xc72446f0
 sched_switch(c72446f0,0,104,191,14b3b3f0,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c72446f0,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_timedwait(c71f763c,4c,c0ca82e8,0,0,...) at 0xc0540e6b =
 sleepq_timedwait+0x6b
 _sleep(c71f763c,c71f76f0,4c,c0ca82e8,3e8,...) at 0xc0510b69 = _sleep+0x339
 fdc_thread(c71f7600,c6bbed38,c073bc0b,343,c7043aa0,...) at 0xc0ca610d
 = fdc_thread+0x89d
 fork_exit(c0ca5870,c71f7600,c6bbed38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6bbed70, ebp = 0 ---
 
 Tracing command fw0_probe pid 7 tid 100055 td 0xc7244b90
 sched_switch(c7244b90,0,104,191,415a8d7b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7244b90,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c72a0000,5c,c0d5ba7e,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c72a0000,c72a4488,15c,c0d5ba7e,0,...) at 0xc0510b84 = _sleep+0x354
 fw_bus_probe_thread(c72a0000,c6b9ed38,c073bc0b,343,c7043d48,...) at
 0xc0d4eaab = fw_bus_probe_thread+0x9b
 fork_exit(c0d4ea10,c72a0000,c6b9ed38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b9ed70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100053 td 0xc727c250
 sched_switch(c727c250,0,104,191,1d799992,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727c250,0,c0745097,260,c727c250,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c7284d0c,0,c6b94cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c7284d0c,c7284dac,c0b6b100,6c,c7284d14,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c7284d04,c6b94d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c7284d04,c6b94d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b94d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100052 td 0xc727c4a0
 sched_switch(c727c4a0,0,104,191,e6af0580,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727c4a0,0,c0745097,260,c727c4a0,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c7284cdc,0,c6b91cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c7284cdc,c7284dac,c0b6b100,6c,c7284ce4,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c7284cd4,c6b91d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c7284cd4,c6b91d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b91d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100051 td 0xc727c6f0
 sched_switch(c727c6f0,0,104,191,1d7457f7,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727c6f0,0,c0745097,260,c727c6f0,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c7284cac,0,c6b8ecbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c7284cac,c7284dac,c0b6b100,6c,c7284cb4,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c7284ca4,c6b8ed38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c7284ca4,c6b8ed38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b8ed70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100050 td 0xc727c940
 sched_switch(c727c940,0,104,191,1d744207,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727c940,0,c0745097,260,c727c940,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c7284c7c,0,c6b8bcbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c7284c7c,c7284dac,c0b6b100,6c,c7284c84,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c7284c74,c6b8bd38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c7284c74,c6b8bd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b8bd70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100049 td 0xc727cb90
 sched_switch(c727cb90,0,104,191,1d7428a5,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727cb90,0,c0745097,260,c727cb90,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c726ddac,0,c6b86cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c726ddac,c726de4c,c0b6b100,6c,c726ddb4,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c726dda4,c6b86d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c726dda4,c6b86d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b86d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100048 td 0xc727d000
 sched_switch(c727d000,0,104,191,63226d58,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727d000,0,c0745097,260,c727d000,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c726dd7c,0,c6b83cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c726dd7c,c726de4c,c0b6b100,6c,c726dd84,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c726dd74,c6b83d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c726dd74,c6b83d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b83d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100047 td 0xc727d250
 sched_switch(c727d250,0,104,191,1d6f01d4,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727d250,0,c0745097,260,c727d250,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c726dd4c,0,c6b80cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c726dd4c,c726de4c,c0b6b100,6c,c726dd54,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c726dd44,c6b80d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c726dd44,c6b80d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b80d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100046 td 0xc727d4a0
 sched_switch(c727d4a0,0,104,191,1d6ee90b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727d4a0,0,c0745097,260,c727d4a0,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c726dd1c,0,c6b7dcbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c726dd1c,c726de4c,c0b6b100,6c,c726dd24,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c726dd14,c6b7dd38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c726dd14,c6b7dd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b7dd70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100045 td 0xc71bd940
 sched_switch(c71bd940,0,104,191,1d6ecd9f,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71bd940,0,c0745097,260,c71bd940,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c725cdac,0,c6b79cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c725cdac,c725ce4c,c0b6b100,6c,c725cdb4,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c725cda4,c6b79d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c725cda4,c6b79d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b79d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100044 td 0xc71bdb90
 sched_switch(c71bdb90,0,104,191,6505c938,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71bdb90,0,c0745097,260,c71bdb90,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c725cd7c,0,c6b76cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c725cd7c,c725ce4c,c0b6b100,6c,c725cd84,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c725cd74,c6b76d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c725cd74,c6b76d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b76d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100043 td 0xc7243000
 sched_switch(c7243000,0,104,191,1d699d50,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7243000,0,c0745097,260,c7243000,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c725cd4c,0,c6b73cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c725cd4c,c725ce4c,c0b6b100,6c,c725cd54,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c725cd44,c6b73d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c725cd44,c6b73d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b73d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100042 td 0xc7243250
 sched_switch(c7243250,0,104,191,1d69877b,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7243250,0,c0745097,260,c7243250,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c725cd1c,0,c6b70cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c725cd1c,c725ce4c,c0b6b100,6c,c725cd24,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c725cd14,c6b70d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c725cd14,c6b70d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b70d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100040 td 0xc72436f0
 sched_switch(c72436f0,0,104,191,1d69708f,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c72436f0,0,c0745097,260,c72436f0,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c7248dac,0,c6b69cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c7248dac,c7248e4c,c0b6b100,6c,c7248db4,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c7248da4,c6b69d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c7248da4,c6b69d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b69d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100039 td 0xc7243940
 sched_switch(c7243940,0,104,191,650581e3,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7243940,0,c0745097,260,c7243940,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c7248d7c,0,c6b66cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c7248d7c,c7248e4c,c0b6b100,6c,c7248d84,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c7248d74,c6b66d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c7248d74,c6b66d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b66d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100038 td 0xc7243b90
 sched_switch(c7243b90,0,104,191,1d63f67e,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7243b90,0,c0745097,260,c7243b90,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c7248d4c,0,c6b63cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c7248d4c,c7248e4c,c0b6b100,6c,c7248d54,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c7248d44,c6b63d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c7248d44,c6b63d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b63d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100037 td 0xc7244000
 sched_switch(c7244000,0,104,191,1d63dff5,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7244000,0,c0745097,260,c7244000,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c7248d1c,0,c6b60cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c7248d1c,c7248e4c,c0b6b100,6c,c7248d24,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c7248d14,c6b60d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c7248d14,c6b60d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b60d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100035 td 0xc72444a0
 sched_switch(c72444a0,0,104,191,1d63c44a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c72444a0,0,c0745097,260,c72444a0,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c721ddac,0,c6b59cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c721ddac,c721de4c,c0b6b100,6c,c721ddb4,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c721dda4,c6b59d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c721dda4,c6b59d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b59d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100034 td 0xc71bc000
 sched_switch(c71bc000,0,104,191,6322a5c5,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71bc000,0,c0745097,260,c71bc000,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c721dd7c,0,c6b56cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c721dd7c,c721de4c,c0b6b100,6c,c721dd84,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c721dd74,c6b56d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c721dd74,c6b56d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b56d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100033 td 0xc71bc250
 sched_switch(c71bc250,0,104,191,1d5e122f,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71bc250,0,c0745097,260,c71bc250,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c721dd4c,0,c6b53cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c721dd4c,c721de4c,c0b6b100,6c,c721dd54,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c721dd44,c6b53d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c721dd44,c6b53d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b53d70, ebp = 0 ---
 
 Tracing command usb pid 13 tid 100032 td 0xc71bc4a0
 sched_switch(c71bc4a0,0,104,191,1d5dface,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71bc4a0,0,c0745097,260,c71bc4a0,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c721dd1c,0,c6b50cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _cv_wait(c721dd1c,c721de4c,c0b6b100,6c,c721dd24,...) at 0xc04c62c0 =
 _cv_wait+0x240
 usb_process(c721dd14,c6b50d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10
 = usb_process+0x170
 fork_exit(c0b61aa0,c721dd14,c6b50d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b50d70, ebp = 0 ---
 
 Tracing command crypto returns pid 6 tid 100017 td 0xc71b7940
 sched_switch(c71b7940,0,104,191,1d5dd82d,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71b7940,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_wait(c094962c,5c,c075f35c,0,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _sleep(c094962c,c094963c,5c,c075f35c,0,...) at 0xc0510b9b = _sleep+0x36b
 crypto_ret_proc(0,c68a8d38,c073bc0b,343,c71b62a8,...) at 0xc06507bc =
 crypto_ret_proc+0x1cc
 fork_exit(c06505f0,0,c68a8d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68a8d70, ebp = 0 ---
 
 Tracing command crypto pid 5 tid 100016 td 0xc71b7b90
 sched_switch(c71b7b90,0,104,191,1d5dbc82,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71b7b90,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_wait(c0949608,5c,c075f560,0,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _sleep(c0949608,c0949618,5c,c075f560,0,...) at 0xc0510b9b = _sleep+0x36b
 crypto_proc(0,c68a5d38,c073bc0b,343,c71b6550,...) at 0xc0651c91 =
 crypto_proc+0x311
 fork_exit(c0651980,0,c68a5d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68a5d70, ebp = 0 ---
 
 Tracing command yarrow pid 12 tid 100015 td 0xc7046250
 sched_switch(c7046250,0,104,191,73a5fb5a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7046250,0,c0745097,283,2,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_timedwait(c07cf084,0,c0bc5e24,2,0,...) at 0xc0540e6b =
 sleepq_timedwait+0x6b
 _sleep(c07cf084,0,0,c0bc5e24,64,...) at 0xc0510b69 = _sleep+0x339
 pause(c0bc5e24,64,c0bc5e26,111,0,...) at 0xc0510cf7 = pause+0x47
 random_kthread(0,c68a2d38,c073bc0b,343,c7042000,...) at 0xc0bbf0a6 =
 random_kthread+0x1b6
 fork_exit(c0bbeef0,0,c68a2d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68a2d70, ebp = 0 ---
 
 Tracing command g_down pid 4 tid 100014 td 0xc70464a0
 sched_switch(c70464a0,0,104,191,23c1cf6a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c70464a0,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_wait(c07cce64,4c,c0758b0d,0,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _sleep(c07cce64,c07ccdc8,24c,c0758b0d,0,...) at 0xc0510b9b = _sleep+0x36b
 g_io_schedule_down(c70464a0,0,c07380d3,74,0,...) at 0xc04af026 =
 g_io_schedule_down+0x56
 g_down_procbody(0,c689fd38,c073bc0b,343,c70422a8,...) at 0xc04af68d =
 g_down_procbody+0x8d
 fork_exit(c04af600,0,c689fd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc689fd70, ebp = 0 ---
 
 Tracing command g_up pid 3 tid 100013 td 0xc70466f0
 sched_switch(c70466f0,0,104,191,23de6e8a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c70466f0,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_wait(c07cce60,4c,c0758b0d,0,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _sleep(c07cce60,c07ccde8,24c,c0758b0d,0,...) at 0xc0510b9b = _sleep+0x36b
 g_io_schedule_up(c70466f0,0,c07380d3,5d,0,...) at 0xc04aefbe =
 g_io_schedule_up+0x11e
 g_up_procbody(0,c689cd38,c073bc0b,343,c7042550,...) at 0xc04af71d =
 g_up_procbody+0x8d
 fork_exit(c04af690,0,c689cd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc689cd70, ebp = 0 ---
 
 Tracing command g_event pid 2 tid 100012 td 0xc7046940
 sched_switch(c7046940,0,104,191,6c429b16,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7046940,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_timedwait(c07cce58,4c,c0758b0d,0,0,...) at 0xc0540e6b =
 sleepq_timedwait+0x6b
 _sleep(c07cce58,0,4c,c0758b0d,64,...) at 0xc0510b69 = _sleep+0x339
 g_event_procbody(0,c6899d38,c073bc0b,343,c70427f8,...) at 0xc04af7eb =
 g_event_procbody+0xcb
 fork_exit(c04af720,0,c6899d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6899d70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100172 td 0xc781d6f0
 sched_switch(c781d6f0,0,109,191,498f3eb,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c71ea570,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c7fce100,eb6fed38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c7fce100,eb6fed38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xeb6fed70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100060 td 0xc7385250
 fork_trampoline() at 0xc06d6b78 = fork_trampoline
 
 Tracing command intr pid 11 tid 100059 td 0xc73854a0
 sched_switch(c73854a0,0,109,191,a41e83fa,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c7040cf0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c737e7e0,c6bc4d38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c737e7e0,c6bc4d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6bc4d70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100058 td 0xc73856f0
 sched_switch(c73856f0,0,109,191,f1f97bb8,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c7041370,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c737e9a0,c6bc1d38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c737e9a0,c6bc1d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6bc1d70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100056 td 0xc7244940
 sched_switch(c7244940,0,109,191,23dde220,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c7040bf0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c736e120,c6ba4d38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c736e120,c6ba4d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6ba4d70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100041 td 0xc72434a0
 sched_switch(c72434a0,0,109,191,1d325785,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c70409f0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c71f4b80,c6b6dd38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c71f4b80,c6b6dd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b6dd70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100036 td 0xc7244250
 sched_switch(c7244250,0,109,191,ff498233,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c7040970,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c71f49b0,c6b5dd38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c71f49b0,c6b5dd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b5dd70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100031 td 0xc71bc6f0
 fork_trampoline() at 0xc06d6b78 = fork_trampoline
 
 Tracing command intr pid 11 tid 100029 td 0xc71bcb90
 fork_trampoline() at 0xc06d6b78 = fork_trampoline
 
 Tracing command intr pid 11 tid 100028 td 0xc71bd000
 sched_switch(c71bd000,0,109,191,f83d1055,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c71dadf0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c71ec000,c690bd38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c71ec000,c690bd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc690bd70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100027 td 0xc71bd250
 fork_trampoline() at 0xc06d6b78 = fork_trampoline
 
 Tracing command intr pid 11 tid 100025 td 0xc71bd6f0
 sched_switch(c71bd6f0,0,109,191,a3db484d,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c71537f0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c70832d0,c68c0d38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c70832d0,c68c0d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68c0d70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100024 td 0xc70846f0
 fork_trampoline() at 0xc06d6b78 = fork_trampoline
 
 Tracing command intr pid 11 tid 100019 td 0xc71b74a0
 sched_switch(c71b74a0,0,109,191,9cffb807,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c71538f0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c7083310,c68aed38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c7083310,c68aed38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68aed70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100018 td 0xc71b76f0
 sched_switch(c71b76f0,0,109,191,f12cfc2e,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c71539f0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c7083340,c68abd38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c7083340,c68abd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68abd70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100011 td 0xc7046b90
 sched_switch(c7046b90,0,109,191,30988d8a,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c7040470,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c7083790,c6896d38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c7083790,c6896d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6896d70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100010 td 0xc7084000
 sched_switch(c7084000,0,109,191,7170b6,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c70404f0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c70837a0,c6893d38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c70837a0,c6893d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6893d70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100009 td 0xc7084250
 sched_switch(c7084250,0,109,191,77674170,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c7040570,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c70837b0,c6890d38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c70837b0,c6890d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6890d70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100008 td 0xc70844a0
 sched_switch(c70844a0,0,109,191,78233802,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c70405f0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c70837c0,c688dd38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c70837c0,c688dd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc688dd70, ebp = 0 ---
 
 Tracing command intr pid 11 tid 100007 td 0xc7044000
 fork_trampoline() at 0xc06d6b78 = fork_trampoline
 
 Tracing command intr pid 11 tid 100006 td 0xc7044250
 sched_switch(c7044250,0,109,191,6e898b90,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(109,0,c073bec7,52d,c70406f0,...) at 0xc0510680 = mi_switch+0x200
 ithread_loop(c70837e0,c6887d38,c073bc0b,343,c7042aa0,...) at
 0xc04e0f96 = ithread_loop+0x1f6
 fork_exit(c04e0da0,c70837e0,c6887d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6887d70, ebp = 0 ---
 
 Tracing command idle pid 10 tid 100005 td 0xc70444a0
 cpustop_handler(1,c6883c28,c06f4456,80008cce,369e99,...) at 0xc06e7712
 = cpustop_handler+0x32
 ipi_nmi_handler(80008cce,369e99,0,0,c7042d48,...) at 0xc06e779f =
 ipi_nmi_handler+0x2f
 trap(c6883c34) at 0xc06f4456 = trap+0x36
 calltrap() at 0xc06d6b0b = calltrap+0x6
 --- trap 0x13, eip = 0xc0c23ce5, esp = 0xc6883c74, ebp = 0xc6883c74 ---
 acpi_cpu_c1(0,ffffffff,0,c0529eb1,c07e1490,...) at 0xc0c23ce5 = acpi_cpu_c1+0x5
 acpi_cpu_idle(c6883cb4,c06e25bb,1,c6883cf8,c052a1ae,...) at 0xc0c1ca51
 = acpi_cpu_idle+0x1c1
 cpu_idle_acpi(1,c6883cf8,c052a1ae,1,c6883cd4,...) at 0xc06e36fb =
 cpu_idle_acpi+0x1b
 cpu_idle(1,c6883cd4,c0742824,9fa,c70444a0,...) at 0xc06e25bb = cpu_idle+0x1b
 sched_idletd(0,c6883d38,c073bc0b,343,c7042d48,...) at 0xc052a1ae =
 sched_idletd+0x23e
 fork_exit(c0529f70,0,c6883d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6883d70, ebp = 0 ---
 
 Tracing command idle pid 10 tid 100004 td 0xc70446f0
 sched_switch(c70446f0,0,60c,18c,776735b2,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(60c,0,c0742824,806,1,...) at 0xc0510680 = mi_switch+0x200
 sched_preempt(c70446f0,1,1f4,c6880c74,c06d722e,...) at 0xc05284af =
 sched_preempt+0x9f
 ipi_bitmap_handler(c07d0008,28,c0730028,c7081200,c7081218,...) at
 0xc06e63b4 = ipi_bitmap_handler+0x34
 Xipi_intr_bitmap_handler() at 0xc06d722e = Xipi_intr_bitmap_handler+0x2e
 --- interrupt, eip = 0xc0c23ce5, esp = 0xc6880c74, ebp = 0xc6880c74 ---
 acpi_cpu_c1(0,ffffffff,1,c0529eb1,c07e1490,...) at 0xc0c23ce5 = acpi_cpu_c1+0x5
 acpi_cpu_idle(c6880cb4,c06e25bb,0,c6880cf8,c052a1ae,...) at 0xc0c1ca51
 = acpi_cpu_idle+0x1c1
 cpu_idle_acpi(0,c6880cf8,c052a1ae,0,c6880cd4,...) at 0xc06e36fb =
 cpu_idle_acpi+0x1b
 cpu_idle(0,c6880cd4,c0742824,9fa,c70446f0,...) at 0xc06e25bb = cpu_idle+0x1b
 sched_idletd(0,c6880d38,c073bc0b,343,c7042d48,...) at 0xc052a1ae =
 sched_idletd+0x23e
 fork_exit(c0529f70,0,c6880d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6880d70, ebp = 0 ---
 
 Tracing command idle pid 10 tid 100003 td 0xc7044940
 cpustop_handler(4,c687dc28,c06f4456,c0742824,747,...) at 0xc06e7712 =
 cpustop_handler+0x32
 ipi_nmi_handler(c0742824,747,c687dbdc,2,c7042d48,...) at 0xc06e779f =
 ipi_nmi_handler+0x2f
 trap(c687dc34) at 0xc06f4456 = trap+0x36
 calltrap() at 0xc06d6b0b = calltrap+0x6
 --- trap 0x13, eip = 0xc0c23ce5, esp = 0xc687dc74, ebp = 0xc687dc74 ---
 acpi_cpu_c1(8,ffff0028,2,2,c07d3d00,...) at 0xc0c23ce5 = acpi_cpu_c1+0x5
 acpi_cpu_idle(c687dcb4,c06e25bb,1,c687dcf8,c052a1ae,...) at 0xc0c1ca51
 = acpi_cpu_idle+0x1c1
 cpu_idle_acpi(1,c687dcf8,c052a1ae,1,c687dcd4,...) at 0xc06e36fb =
 cpu_idle_acpi+0x1b
 cpu_idle(1,c687dcd4,c0742824,9fa,c7044940,...) at 0xc06e25bb = cpu_idle+0x1b
 sched_idletd(0,c687dd38,c073bc0b,343,c7042d48,...) at 0xc052a1ae =
 sched_idletd+0x23e
 fork_exit(c0529f70,0,c687dd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc687dd70, ebp = 0 ---
 
 Tracing command idle pid 10 tid 100002 td 0xc7044b90
 cpustop_handler(8,c687ac28,c06f4456,c0742824,747,...) at 0xc06e7712 =
 cpustop_handler+0x32
 ipi_nmi_handler(c0742824,747,c04f7cf4,3,c7042d48,...) at 0xc06e779f =
 ipi_nmi_handler+0x2f
 trap(c687ac34) at 0xc06f4456 = trap+0x36
 calltrap() at 0xc06d6b0b = calltrap+0x6
 --- trap 0x13, eip = 0xc0c23ce5, esp = 0xc687ac74, ebp = 0xc687ac74 ---
 acpi_cpu_c1(0,ffffffff,3,c0529eb1,c07e1490,...) at 0xc0c23ce5 = acpi_cpu_c1+0x5
 acpi_cpu_idle(c687acb4,c06e25bb,1,c687acf8,c052a1ae,...) at 0xc0c1ca51
 = acpi_cpu_idle+0x1c1
 cpu_idle_acpi(1,c687acf8,c052a1ae,1,c687acd4,...) at 0xc06e36fb =
 cpu_idle_acpi+0x1b
 cpu_idle(1,c687acd4,c0742824,9fa,c7044b90,...) at 0xc06e25bb = cpu_idle+0x1b
 sched_idletd(0,c687ad38,c073bc0b,343,c7042d48,...) at 0xc052a1ae =
 sched_idletd+0x23e
 fork_exit(c0529f70,0,c687ad38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc687ad70, ebp = 0 ---
 
 Tracing command init pid 1 tid 100001 td 0xc7046000
 sched_switch(c7046000,0,104,191,2c2544f6,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7046000,0,c0745097,1a0,5c,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 =
 sleepq_catch_signals+0xb7
 sleepq_wait_sig(c7043000,5c,c0747a51,100,0,...) at 0xc0540eb7 =
 sleepq_wait_sig+0x17
 _sleep(c7043000,c7043088,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354
 kern_wait(c7046000,ffffffff,c6876c78,0,0,...) at 0xc04dbde6 = kern_wait+0xb46
 wait4(c7046000,c6876cf8,c0770332,c0747868,282,...) at 0xc04dbe5b = wait4+0x3b
 syscall(c6876d38) at 0xc06f41e5 = syscall+0x1f5
 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054b2b, esp =
 0xbfbfe90c, ebp = 0xbfbfe928 ---
 
 Tracing command kernel pid 0 tid 100120 td 0xc781d940
 sched_switch(c781d940,0,104,191,78233d12,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c781d940,0,c0745097,260,c781d940,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c75991c0,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63
 msleep_spin(c75991c0,c75991d8,c0758b0d,0,c073f15b,...) at 0xc0510f4d =
 msleep_spin+0x21d
 taskqueue_thread_loop(c7f5dd20,e9630d38,c073bc0b,343,c07ccf00,...) at
 0xc0542864 = taskqueue_thread_loop+0x94
 fork_exit(c05427d0,c7f5dd20,e9630d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe9630d70, ebp = 0 ---
 
 Tracing command kernel pid 0 tid 100061 td 0xc7385000
 kdb_enter(c0740afa,c0740afa,c073a07e,c6bcbcd0,1,...) at 0xc053618a =
 kdb_enter+0x3a
 panic(c073a07e,c07166d9,c7c47940,3574e,c7c425d8,...) at 0xc05077f6 = panic+0x136
 deadlkres(0,c6bcbd38,c073bc0b,343,c07ccf00,...) at 0xc04c4a1a = deadlkres+0x27a
 fork_exit(c04c47a0,0,c6bcbd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6bcbd70, ebp = 0 ---
 
 Tracing command kernel pid 0 tid 100054 td 0xc727c000
 sched_switch(c727c000,0,104,191,717c5466,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c727c000,0,c0745097,260,c727c000,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c727ecc0,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63
 msleep_spin(c727ecc0,c727ecd8,c0758b0d,0,c073f15b,...) at 0xc0510f4d =
 msleep_spin+0x21d
 taskqueue_thread_loop(c72a449c,c6b9bd38,c073bc0b,343,c07ccf00,...) at
 0xc0542864 = taskqueue_thread_loop+0x94
 fork_exit(c05427d0,c72a449c,c6b9bd38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b9bd70, ebp = 0 ---
 
 Tracing command kernel pid 0 tid 100030 td 0xc71bc940
 sched_switch(c71bc940,0,104,191,4eb8cfaa,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71bc940,0,c0745097,260,c71bc940,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c7204300,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63
 msleep_spin(c7204300,c7204318,c0758b0d,0,c073f15b,...) at 0xc0510f4d =
 msleep_spin+0x21d
 taskqueue_thread_loop(c7209520,c6b49d38,c073bc0b,343,c07ccf00,...) at
 0xc0542864 = taskqueue_thread_loop+0x94
 fork_exit(c05427d0,c7209520,c6b49d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc6b49d70, ebp = 0 ---
 
 Tracing command kernel pid 0 tid 100026 td 0xc71bd4a0
 sched_switch(c71bd4a0,0,104,191,1d88da60,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71bd4a0,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_wait(c71a4ac0,0,c0758b0d,0,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _sleep(c71a4ac0,c71a4ad8,0,c0758b0d,0,...) at 0xc0510b9b = _sleep+0x36b
 taskqueue_thread_loop(c07cd7c4,c68c3d38,c073bc0b,343,c07ccf00,...) at
 0xc054288a = taskqueue_thread_loop+0xba
 fork_exit(c05427d0,c07cd7c4,c68c3d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68c3d70, ebp = 0 ---
 
 Tracing command kernel pid 0 tid 100023 td 0xc7084940
 sched_switch(c7084940,0,104,191,1d88c428,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7084940,0,c0745097,260,c7084940,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c71a5040,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63
 msleep_spin(c71a5040,c71a5058,c0758b0d,0,c073f15b,...) at 0xc0510f4d =
 msleep_spin+0x21d
 taskqueue_thread_loop(c0c3623c,c68bad38,c073bc0b,343,c07ccf00,...) at
 0xc0542864 = taskqueue_thread_loop+0x94
 fork_exit(c05427d0,c0c3623c,c68bad38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68bad70, ebp = 0 ---
 
 Tracing command kernel pid 0 tid 100022 td 0xc7084b90
 sched_switch(c7084b90,0,104,191,1d88afbb,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c7084b90,0,c0745097,260,c7084b90,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c71a5040,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63
 msleep_spin(c71a5040,c71a5058,c0758b0d,0,c073f15b,...) at 0xc0510f4d =
 msleep_spin+0x21d
 taskqueue_thread_loop(c0c3623c,c68b7d38,c073bc0b,343,c07ccf00,...) at
 0xc0542864 = taskqueue_thread_loop+0x94
 fork_exit(c05427d0,c0c3623c,c68b7d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68b7d70, ebp = 0 ---
 
 Tracing command kernel pid 0 tid 100021 td 0xc71b7000
 sched_switch(c71b7000,0,104,191,1d88982d,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71b7000,0,c0745097,260,c71b7000,...) at 0xc054028f =
 sleepq_switch+0x15f
 sleepq_wait(c71a5040,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63
 msleep_spin(c71a5040,c71a5058,c0758b0d,0,c073f15b,...) at 0xc0510f4d =
 msleep_spin+0x21d
 taskqueue_thread_loop(c0c3623c,c68b4d38,c073bc0b,343,c07ccf00,...) at
 0xc0542864 = taskqueue_thread_loop+0x94
 fork_exit(c05427d0,c0c3623c,c68b4d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68b4d70, ebp = 0 ---
 
 Tracing command kernel pid 0 tid 100020 td 0xc71b7250
 sched_switch(c71b7250,0,104,191,4427f86c,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c71b7250,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_wait(c71a5080,0,c0758b0d,0,0,...) at 0xc0540f63 = sleepq_wait+0x63
 _sleep(c71a5080,c71a5098,0,c0758b0d,0,...) at 0xc0510b9b = _sleep+0x36b
 taskqueue_thread_loop(c07e1688,c68b1d38,c073bc0b,343,c07ccf00,...) at
 0xc054288a = taskqueue_thread_loop+0xba
 fork_exit(c05427d0,c07e1688,c68b1d38) at 0xc04dddf8 = fork_exit+0xb8
 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xc68b1d70, ebp = 0 ---
 
 Tracing command kernel pid 0 tid 100000 td 0xc07cd1b0
 sched_switch(c07cd1b0,0,104,191,66059445,...) at 0xc05292c0 = sched_switch+0x330
 mi_switch(104,0,c0745097,1eb,44,...) at 0xc0510680 = mi_switch+0x200
 sleepq_switch(c07cd1b0,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f
 sleepq_timedwait(c07ccf00,44,c07430b5,0,0,...) at 0xc0540e6b =
 sleepq_timedwait+0x6b
 _sleep(c07ccf00,0,44,c07430b5,2710,...) at 0xc0510b69 = _sleep+0x339
 scheduler(0,101ec00,101ec00,101e000,1025000,...) at 0xc069e96e = scheduler+0x23e
 mi_startup() at 0xc04c2aa6 = mi_startup+0x96
 begin() at 0xc043adc5 = begin+0x2c
 
 
 
 -- 
 Sincerely, Dmitry
 nic-hdl: LYNX-RIPE
State-Changed-From-To: open->patched 
State-Changed-By: kib 
State-Changed-When: Sun Jun 20 13:36:07 UTC 2010 
State-Changed-Why:  
No reason not to grab the bug, since I already committed the patch. 


Responsible-Changed-From-To: freebsd-fs->kib 
Responsible-Changed-By: kib 
Responsible-Changed-When: Sun Jun 20 13:36:07 UTC 2010 
Responsible-Changed-Why:  
No reason not to grab the bug, since I already committed the patch. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=147890 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/147890: commit references a PR
Date: Sun, 20 Jun 2010 13:35:31 +0000 (UTC)

 Author: kib
 Date: Sun Jun 20 13:35:16 2010
 New Revision: 209367
 URL: http://svn.freebsd.org/changeset/base/209367
 
 Log:
   Ensure that VOP_ACCESSX is called with exclusively locked vnode for
   the kernel compiled with QUOTA option. ufs_accessx() upgrades the vdp
   vnode lock from shared to exclusive to assign the dquot structure to
   the vnode, and ufs_delete_denied() is called when tvp is locked. Since
   upgrade drops shared lock when non-blocked upgrade failed, LOR is there.
   
   Reported and tested by:	Dmitry Pryanishnikov <lynx.ripe gmail com>
   Tested by:	pho
   PR:	kern/147890
   MFC after:	1 week
 
 Modified:
   head/sys/ufs/ufs/ufs_lookup.c
 
 Modified: head/sys/ufs/ufs/ufs_lookup.c
 ==============================================================================
 --- head/sys/ufs/ufs/ufs_lookup.c	Sun Jun 20 12:52:33 2010	(r209366)
 +++ head/sys/ufs/ufs/ufs_lookup.c	Sun Jun 20 13:35:16 2010	(r209367)
 @@ -77,6 +77,32 @@ SYSCTL_INT(_debug, OID_AUTO, dircheck, C
  /* true if old FS format...*/
  #define OFSFMT(vp)	((vp)->v_mount->mnt_maxsymlinklen <= 0)
  
 +#ifdef QUOTA
 +static int
 +ufs_lookup_upgrade_lock(struct vnode *vp)
 +{
 +	int error;
 +
 +	ASSERT_VOP_LOCKED(vp, __FUNCTION__);
 +	if (VOP_ISLOCKED(vp) == LK_EXCLUSIVE)
 +		return (0);
 +
 +	error = 0;
 +
 +	/*
 +	 * Upgrade vnode lock, since getinoquota()
 +	 * requires exclusive lock to modify inode.
 +	 */
 +	vhold(vp);
 +	vn_lock(vp, LK_UPGRADE | LK_RETRY);
 +	VI_LOCK(vp);
 +	if (vp->v_iflag & VI_DOOMED)
 +		error = ENOENT;
 +	vdropl(vp);
 +	return (error);
 +}
 +#endif
 +
  static int
  ufs_delete_denied(struct vnode *vdp, struct vnode *tdp, struct ucred *cred,
      struct thread *td)
 @@ -232,6 +258,13 @@ ufs_lookup_ino(struct vnode *vdp, struct
  	vnode_create_vobject(vdp, DIP(dp, i_size), cnp->cn_thread);
  
  	bmask = VFSTOUFS(vdp->v_mount)->um_mountp->mnt_stat.f_iosize - 1;
 +#ifdef QUOTA
 +	if ((nameiop == DELETE || nameiop == RENAME) && (flags & ISLASTCN)) {
 +		error = ufs_lookup_upgrade_lock(vdp);
 +		if (error != 0)
 +			return (error);
 +	}
 +#endif
  
  restart:
  	bp = NULL;
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: kib 
State-Changed-When: Mon Jun 28 11:26:48 UTC 2010 
State-Changed-Why:  
Patch is in HEAD and RELENG_8. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=147890 
>Unformatted:
