From nobody@FreeBSD.org  Wed Nov 14 15:18:07 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 996D616A4C6
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 14 Nov 2007 15:18:07 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 84D7413C461
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 14 Nov 2007 15:18:07 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.1/8.14.1) with ESMTP id lAEFHaUB070185
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 14 Nov 2007 15:17:36 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.1/8.14.1/Submit) id lAEFHaqu070184;
	Wed, 14 Nov 2007 15:17:36 GMT
	(envelope-from nobody)
Message-Id: <200711141517.lAEFHaqu070184@www.freebsd.org>
Date: Wed, 14 Nov 2007 15:17:36 GMT
From: Per olof Ljungmark <peo@intersonic.se>
To: freebsd-gnats-submit@FreeBSD.org
Subject: panic: spin lock held too long
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         118044
>Category:       kern
>Synopsis:       [panic] [lor] spin lock held too long
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Nov 14 15:20:01 UTC 2007
>Closed-Date:    Sun Jul 03 07:43:11 UTC 2011
>Last-Modified:  Sun Jul 03 07:43:11 UTC 2011
>Originator:     Per olof Ljungmark
>Release:        RELENG_7
>Organization:
Intersonic AB
>Environment:
%uname -a
FreeBSD ruben.i.inter-sonic.com 7.0-BETA2 FreeBSD 7.0-BETA2 #0: Tue Nov 13 14:58:10 CET 2007     peo@ruben.i.inter-sonic.com:/usr/obj/usr/src/sys/SMP-DBG  i386
>Description:
Hi,
We have several all HP boxes running 7-STABLE (but the behaviour also 
showed in 7-CURRENT, it is not new) that can be reliadbly crashed by 
accessing them remotely from a remote network over a IPSec link running 
on our firewall. Today I've built a debug kernel and caught the attached 
information. Please let me know what more I can supply.

Among the affected units are ProLiant DL380G2, G3, G4, Workstations 
XW600, XW6200 running SMP kernels. Not affected are DL360G3 from what 
I've seen so far. Problem only shows with SMP kernels.

Kernels are GENERIC + SCHED_ULE

possibly connected to
http://www.freebsd.org/cgi/query-pr.cgi?pr=117913

================================

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-BETA2 #0: Tue Nov 13 14:58:10 CET 2007
    peo@ruben.i.inter-sonic.com:/usr/obj/usr/src/sys/SMP-DBG
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3189.72-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf25  Stepping = 5

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4400<CNXT-ID,xTPR>
  Logical CPUs per core: 2
real memory  = 2147418112 (2047 MB)
avail memory = 2091864064 (1994 MB)
ACPI APIC Table: <COMPAQ PLACER  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  6
 cpu3 (AP): APIC ID:  7
ioapic0: Changing APIC ID to 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <COMPAQ CPQ0054> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 7ff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0xf808-0xf80b on acpi0
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <Intel E7505 host to AGP bridge> on hostb0
pcib1: <PCI-PCI bridge> mem 0xf8000000-0xfbffffff at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> mem 
0xf1000000-0xf1ffffff,0xe8000000-0xefffffff irq 16 at device 0.0 on pci1
uhci0: <Intel 82801DB (ICH4) USB controller USB-A> port 0x2440-0x245f 
irq 16 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 82801DB (ICH4) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801DB (ICH4) USB controller USB-B> port 0x2460-0x247f 
irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 82801DB (ICH4) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
ehci0: <Intel 82801DB/L/M (ICH4) USB 2.0 controller> mem 
0xf0500000-0xf05003ff irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb2: EHCI version 1.0
usb2: wrong number of companions (3 != 2)
usb2: companion controllers, 2 ports each: usb0 usb1
usb2: <Intel 82801DB/L/M (ICH4) USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb2
uhub2: 6 ports with 6 removable, self powered
pcib2: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib2
pci0:5:2:0: bad VPD cksum, remain 72
bge0: <Broadcom Gigabit Ethernet Controller, ASIC rev. 0x1002> mem 
0xf0400000-0xf040ffff irq 21 at device 2.0 on pci5
miibus0: <MII bus> on bge0
brgphy0: <BCM5703 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
bge0: Ethernet address: 00:0d:9d:57:71:9a
bge0: [ITHREAD]
aac0: <Adaptec SCSI RAID 2120S> mem 0xe4000000-0xe7ffffff irq 18 at 
device 10.0 on pci5
aac0: New comm. interface enabled
aac0: [ITHREAD]
aac0: Adaptec Raid Controller 2.0.0-1
aacp0: <SCSI Passthrough Bus> on aac0
ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port 
0x1000-0x10ff,0x1400-0x14ff mem 0xf0410000-0xf0411fff irq 19 at device 
12.0 on pci5
ahd0: [ITHREAD]
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI 33 or 66Mhz, 512 SCBs
ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port 
0x1800-0x18ff,0x1c00-0x1cff mem 0xf0412000-0xf0413fff irq 22 at device 
12.1 on pci5
ahd1: [ITHREAD]
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI 33 or 66Mhz, 512 SCBs
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH4 UDMA100 controller> port 
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x24a0-0x24af irq 18 at device 31.1 
on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <multimedia, audio> at device 31.5 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model Generic PS/2 mouse, device ID 0
sio0: <Standard PC COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <Standard PC COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xea800-0xeffff pnpid ORM0000 on isa0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/13 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppi0: <Parallel I/O> on ppbus0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: CDROM <HL-DT-ST CD-ROM GCR-8482B/1.03> at ata1-master PIO4
Waiting 5 seconds for SCSI devices to settle
aacd0: <RAID 1 (Mirror)> on aac0
aacd0: 34970MB (71619584 sectors)
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/aacd0s1a
WARNING: / was not properly dismounted

==========================

rtfree: 0xc5874618 has 1 refs
ssppiinn lloocckk 00xxcc5507fc28aa0500 ((ttuurrnnssttiillee lloocckk)) 
hheelldd bbyy 00xxcc55111197868600 ((ttiidd 110000002154)) ttoooo lloonngg

panic: spin lock held too long
cpuid = 1
KDB: enter: panic
[thread pid 16 tid 100014 ]
Stopped at     kdb_enter+0x32: leave
db>
db> ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
22686 22684  7399     0  R+                          sed
22684 22651  7399     0  S+      wait     0xcb49eaa0 sh
22683 22205 22683     0  S+      ttywri   0xc5542c38 cat
22651 20101  7399     0  S+      piperd   0xc5bfb318 sh
22205 22198 22205     0  S+      pause    0xc6ef05b0 csh
22198 21992 22198  1001  S+      wait     0xc6eeb550 su
21992 21991 21992  1001  Ss+     pause    0xc5c9bb00 csh
21991 21982 21982  1001  L      *rtentry  0xc57c8a50 sshd
21982   705 21982     0  Ss      sbwait   0xc7600248 sshd
20101 20097  7399     0  S+      wait     0xcb4a0000 gmake
20097 20096  7399     0  S+      wait     0xc6eeb7f8 sh
20096 84323  7399     0  S+      wait     0xc559daa0 gmake
84323 84322  7399     0  S+      wait     0xcb4a0d48 sh
84322 84318  7399     0  S+      wait     0xca6ef2a8 gmake
84318 84317  7399     0  S+      wait     0xc5c20d48 sh
84317 84316  7399     0  S+      wait     0xc6eefaa0 gmake
84316 84315  7399     0  S+      wait     0xc5bc1d48 gmake
84315 20720  7399     0  S+      wait     0xc7224550 sh
20720 20719  7399     0  S+      wait     0xc5633000 make
--More--
20719 20703  7399     0  S+      wait     0xc5c97550 sh
20703 25073  7399     0  S+      wait     0xc5c87aa0 make
86175 85857 86175     0  S+      ttyin    0xc5528410 csh
85857 84932 85857  1001  S+      wait     0xc57ca7f8 su
84932 84911 84932  1001  Ss+     pause    0xc55d1308 csh
84911 83121 83121  1001  S       select   0xc0c03cbc sshd
83121   705 83121     0  Ss      sbwait   0xc57126ec sshd
25073  7399  7399     0  S+      wait     0xc5c97d48 sh
 7399   810  7399     0  S+      wait     0xc562aaa0 make
  810   809   810     0  S+      pause    0xc5633308 csh
  809   805   809  1001  S+      wait     0xc5a38550 su
  805   804   805  1001  Ss+     pause    0xc5a3c060 csh
  804   801   801  1001  S       select   0xc0c03cbc sshd
  801   705   801     0  Ss      sbwait   0xc5712a04 sshd
  793     1   793     0  Ss+     ttyin    0xc535d810 login
  792     1   792     0  Ss+     ttyin    0xc535f410 getty
  791     1   791     0  Ss+     ttyin    0xc538a010 getty
  790     1   790     0  Ss+     ttyin    0xc5389c10 getty
  789     1   789     0  Ss+     ttyin    0xc538ac10 getty
  788     1   788     0  Ss+     ttyin    0xc538a810 getty
787     1   787     0  Ss+     ttyin    0xc5388c10 getty
  786     1   786     0  Ss+     ttyin    0xc5388810 getty
  785     1   785     0  Ss+     ttyin    0xc5389810 getty
  764     1   764     0  Ss      select   0xc0c03cbc inetd
  737     1   737     0  Ss      select   0xc0c03cbc moused
  721     1   721     0  Ss      nanslp   0xc0bb2724 cron
  715     1   715    25  Ss      pause    0xc52d7da8 sendmail
  711     1   711     0  Ss      select   0xc0c03cbc sendmail
  705     1   705     0  Ss      select   0xc0c03cbc sshd
  587     1   587     0  Ss      select   0xc0c03cbc syslogd
  524     1   524     0  Ss      select   0xc0c03cbc devd
  328     1   328    65  Ss      select   0xc0c03cbc dhclient
  308     1    54     0  S+      select   0xc0c03cbc dhclient
   53     0     0     0  SL      sdflush  0xc0c0f9a4 [softdepflush]
   52     0     0     0  SL      syncer   0xc0bb254c [syncer]
   51     0     0     0  SL      vlruwt   0xc54f97f8 [vnlru]
   50     0     0     0  SL      psleep   0xc0c04164 [bufdaemon]
   49     0     0     0  SL      pgzero   0xc0c10560 [pagezero]
   48     0     0     0  SL      psleep   0xc0c10178 [vmdaemon]
   47     0     0     0  SL      psleep   0xc0c10140 [pagedaemon]
   46     0     0     0  SL      waiting_ 0xc0c05ecc [sctp_iterator]
   45     0     0     0  WL                          [irq7: ppc0]
   44     0     0     0  SL      -        0xc5324e3c [fdc0]
   43     0     0     0  WL                          [swi0: sio]
   42     0     0     0  WL                          [irq12: psm0]
   41     0     0     0  WL                          [irq1: atkbd0]
   40     0     0     0  WL                          [irq15: ata1]
   39     0     0     0  WL                          [irq14: ata0]
   38     0     0     0  SL      idle     0xc5326000 [aic_recovery1]
   37     0     0     0  WL                          [irq22: ahd1]
   36     0     0     0  SL      idle     0xc52db000 [aic_recovery0]
   35     0     0     0  SL      aifthd   0xc52d7550 [aac0aif]
   34     0     0     0  WL                          [irq18: aac0]
   33     0     0     0  LL     *rtentry  0xc50f2a00 [irq21: bge0]
   32     0     0     0  SL      usbevt   0xc520ba10 [usb2]
   31     0     0     0  WL                          [irq23: ehci0]
   30     0     0     0  SL      usbevt   0xc5243210 [usb1]
   29     0     0     0  WL                          [irq19: uhci1 ahd0]
   28     0     0     0  SL      usbtsk   0xc0bafe54 [usbtask-dr]
   27     0     0     0  SL      usbtsk   0xc0bafe40 [usbtask-hc]
   26     0     0     0  SL      usbevt   0xc5249210 [usb0]
   25     0     0     0  WL                          [irq16: uhci0]
   24     0     0     0  WL                          [irq9: acpi0]
   23     0     0     0  SL      -        0xc5213600 [kqueue taskq]
   22     0     0     0  WL                          [swi6: task queue]
   21     0     0     0  WL                          [swi6: Giant taskq]
    9     0     0     0  SL      -        0xc5213880 [thread taskq]
    8     0     0     0  SL      -        0xc5213900 [acpi_task_2]
    7     0     0     0  SL      -        0xc5213900 [acpi_task_1]
    6     0     0     0  SL      -        0xc5213900 [acpi_task_0]
   20     0     0     0  WL                          [swi5: +]
   19     0     0     0  WL                          [swi2: cambio]
    5     0     0     0  SL      ccb_scan 0xc0b81014 [xpt_thrd]
   18     0     0     0  SL      -        0xc0bb2554 [yarrow]
    4     0     0     0  SL      -        0xc0bb05cc [g_down]
    3     0     0     0  SL      -        0xc0bb05c8 [g_up]
    2     0     0     0  SL      -        0xc0bb05c0 [g_event]
   17     0     0     0  WL                          [swi3: vm]
   16     0     0     0  LL     *inp      0xc51165f0 [swi4: clock sio]
   15     0     0     0  WL                          [swi1: net]
   14     0     0     0  RL                          [idle: cpu0]
   13     0     0     0  RL                          [idle: cpu1]
   12     0     0     0  RL      CPU 2               [idle: cpu2]
   11     0     0     0  RL      CPU 3               [idle: cpu3]
    1     0     1     0  SLs     wait     0xc5114d48 [init]
   10     0     0     0  SL      audit_wo 0xc0c0f414 [audit]
    0     0     0     0  WLs                         [swapper]

db>          show pcpu
cpuid        = 1
curthread    = 0xc5117660: pid 16 "swi4: clock sio"
curpcb       = 0xe3cb7d90
fpcurthread  = none
idlethread   = 0xc5115aa0: pid 13 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:


db> show allpcpu
Current CPU: 1

cpuid        = 0
curthread    = 0xc5119880: pid 33 "irq21: bge0"
curpcb       = 0xe3ccfd90
fpcurthread  = none
idlethread   = 0xc5115cc0: pid 14 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:

cpuid        = 1
curthread    = 0xc5117660: pid 16 "swi4: clock sio"
curpcb       = 0xe3cb7d90
fpcurthread  = none
idlethread   = 0xc5115aa0: pid 13 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:

cpuid        = 2
curthread    = 0xc5115880: pid 12 "idle: cpu2"
curpcb       = 0xe3ca5d90
fpcurthread  = none
idlethread   = 0xc5115880: pid 12 "idle: cpu2"
APIC ID      = 6
currentldt   = 0x50
spin locks held:

cpuid        = 3
curthread    = 0xc5115660: pid 11 "idle: cpu3"
curpcb       = 0xe3ca2d90
fpcurthread  = none
idlethread   = 0xc5115660: pid 11 "idle: cpu3"
APIC ID      = 7
currentldt   = 0x50
spin locks held:


db> show locks
exclusive sleep mutex tcp r = 0 (0xc0c0664c) locked @ 
/usr/src/sys/netinet/tcp_timer.c:423


db> show alllocks
Process 22686 (sed) thread 0xc5c99440 (100188)
exclusive sleep mutex sleep mtxpool r = 0 (0xc50f1088) locked @ 
/usr/src/sys/kern/kern_descrip.c:2139
Process 21991 (sshd) thread 0xc57c6440 (100112)
exclusive sleep mutex rtentry r = 0 (0xc5874678) locked @ 
/usr/src/sys/net/route.c:1287
exclusive sleep mutex inp (tcpinp) r = 0 (0xc58a17c8) locked @ 
/usr/src/sys/netinet/tcp_usrreq.c:779
exclusive sx so_snd_sx r = 0 (0xc66cb748) locked @ 
/usr/src/sys/kern/uipc_sockbuf.c:145
Process 21982 (sshd) thread 0xc5a3bcc0 (100151)
exclusive sx so_rcv_sx r = 0 (0xc7600218) locked @ 
/usr/src/sys/kern/uipc_sockbuf.c:145
Process 83121 (sshd) thread 0xc5c1e000 (100154)
exclusive sx so_rcv_sx r = 0 (0xc57126bc) locked @ 
/usr/src/sys/kern/uipc_sockbuf.c:145
Process 801 (sshd) thread 0xc5a39aa0 (100127)
exclusive sx so_rcv_sx r = 0 (0xc57129d4) locked @ 
/usr/src/sys/kern/uipc_sockbuf.c:145
Process 33 (irq21: bge0) thread 0xc5119880 (100025)
exclusive sleep mutex rtentry r = 0 (0xc5719d80) locked @ 
/usr/src/sys/net/route.c:197
Process 16 (swi4: clock sio) thread 0xc5117660 (100014)
exclusive sleep mutex tcp r = 0 (0xc0c0664c) locked @ 
/usr/src/sys/netinet/tcp_timer.c:423


db> show lockedvnods
Locked vnodes


db> alltrace

Tracing command sed pid 22686 tid 100188 td 0xc5c99440
sched_switch(c5c99440,0,6,17b,3bcb1dac,...) at sched_switch+0x49b
mi_switch(6,0,c0a9e6f6,b6,c5c99440,...) at mi_switch+0x217
critical_exit(c0bba580,ebec8adc,c074280d,c0bba580,1,...) at 
critical_exit+0x99
spinlock_exit(c0bba580,1,c0a9b5fa,f4) at spinlock_exit+0x10
_mtx_unlock_spin_flags(c0bba580,0,c0a99497,2cc,c09f2677,...) at 
_mtx_unlock_spin_flags+0xfd
intr_event_schedule_thread(c515e500,0,5,c5c99440,c515e500,...) at 
intr_event_schedule_thread+0x114
intr_execute_handlers(c5107f34,ebec8b58,ebec8bb4,c09ec804,34,...) at 
intr_execute_handlers+0x12b
lapic_handle_intr(34,ebec8b58) at lapic_handle_intr+0x3f
Xapic_isr1() at Xapic_isr1+0x34
--- interrupt, eip = 0xc0742aa2, esp = 0xebec8b98, ebp = 0xebec8bb4 ---
_mtx_unlock_flags(c50f1088,0,c0a9881a,86c,fc000,...) at 
_mtx_unlock_flags+0x62
fdrop(c5c70e58,c5c99440,fc000,5,7,...) at fdrop+0x93
mmap(c5c99440,ebec8cfc,1c,c0a5612b,c0b4beb8,...) at mmap+0x3a7
syscall(ebec8d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (477, FreeBSD ELF32, mmap), eip = 0x28057fe3, esp = 
0xbfbfe00c, ebp = 0xbfbfe048 ---

Tracing command sh pid 22684 tid 100250 td 0xca6f1aa0
sched_switch(ca6f1aa0,0,1,180,3b60a3d8,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(ca6f1aa0,0,c0aa02cc,19b,cb49eaa0,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(cb49eaa0,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(cb49eaa0,cb49eb30,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(ca6f1aa0,ffffffff,ebfbdc2c,2,0,...) at kern_wait+0xa66
wait4(ca6f1aa0,ebfbdcfc,10,c0aa2214,c0b492a8,...) at wait4+0x3b
syscall(ebfbdd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149e7b, esp = 
0xbfbfcd3c, ebp = 0xbfbfcd58 ---

Tracing command cat pid 22683 tid 100077 td 0xc55d5440
sched_switch(c55d5440,0,1,180,3bd01c1c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c55d5440,0,c0aa02cc,19b,c5542c38,...) at sleepq_switch+0xf0
sleepq_catch_signals(ebcb1a70,c074280d,c0bba580,1,5a,...) at 
sleepq_catch_signals+0x21e
sleepq_timedwait_sig(c5542c38,0,c0a9d1f9,da,0,...) at 
sleepq_timedwait_sig+0x17
_sleep(c5542c38,0,15a,c0aa3ca0,493e0,...) at _sleep+0x331
ttysleep(c5542c00,c5542c38,15a,c0aa3ca0,493e0,...) at ttysleep+0x39
ttwrite(c5542c00,ebcb1c60,0,ebcb1bbc,c071e768,...) at ttwrite+0x5aa
ptswrite(c6b90c00,ebcb1c60,0,174,1000,...) at ptswrite+0x38
giant_write(c6b90c00,ebcb1c60,0,0,c0b562e0,...) at giant_write+0x48
devfs_write_f(c5ec4c18,ebcb1c60,c6abe900,0,c55d5440,...) at 
devfs_write_f+0x98
dofilewrite(ebcb1c60,ffffffff,ffffffff,0,c5ec4c18,...) at dofilewrite+0x95
kern_writev(c55d5440,1,ebcb1c60,282012bc,d44,...) at kern_writev+0x58
write(c55d5440,ebcb1cfc,c,c0a5612b,c0b49260,...) at write+0x4f
syscall(ebcb1d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (4, FreeBSD ELF32, write), eip = 0x28154483, esp = 
0xbfbfeb9c, ebp = 0xbfbfec48 ---

Tracing command sh pid 22651 tid 100202 td 0xc6eedcc0
sched_switch(c6eedcc0,0,1,180,3afa7a8c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c6eedcc0,0,c0aa02cc,19b,c5bfb318,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,4c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5bfb318,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5bfb318,c5bfb488,14c,c0aa2013,0,...) at _sleep+0x359
pipe_read(c58af0d8,ebf31c60,c587e700,0,c6eedcc0,...) at pipe_read+0x3f6
dofileread(ebf31c60,ffffffff,ffffffff,0,c58af0d8,...) at dofileread+0x96
kern_readv(c6eedcc0,3,ebf31c60,bfbfcf1c,80,...) at kern_readv+0x58
read(c6eedcc0,ebf31cfc,c,ebf31d38,c0b49248,...) at read+0x4f
syscall(ebf31d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281c54a3, esp = 
0xbfbfce6c, ebp = 0xbfbfcfb8 ---

Tracing command csh pid 22205 tid 100212 td 0xc6eec660
sched_switch(c6eec660,0,1,180,3a7253f8,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c6eec660,0,c0aa02cc,19b,c6ef05b0,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,68,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c6ef05b0,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c6ef05b0,c6ef05e0,168,c0a5f873,0,...) at _sleep+0x359
kern_sigsuspend(c6eec660,0,0,0,0,...) at kern_sigsuspend+0xe4
sigsuspend(c6eec660,ebf13cfc,4,c0aa22db,c0b4b1f8,...) at sigsuspend+0x4d
syscall(ebf13d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28166ffb, esp = 
0xbfbfe73c, ebp = 0xbfbfe7a8 ---

Tracing command su pid 22198 tid 100200 td 0xc55d2000
sched_switch(c55d2000,0,1,180,367f1a58,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c55d2000,0,c0aa02cc,19b,c6eeb550,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c6eeb550,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c6eeb550,c6eeb5e0,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c55d2000,56bd,ebc96c2c,2,0,...) at kern_wait+0xa66
wait4(c55d2000,ebc96cfc,10,ebc96d38,c0b492a8,...) at wait4+0x3b
syscall(ebc96d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2810ae7b, esp = 
0xbfbfe79c, ebp = 0xbfbfe7b8 ---

Tracing command csh pid 21992 tid 100196 td 0xc5c98220
sched_switch(c5c98220,0,1,180,2193d730,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5c98220,0,c0aa02cc,19b,c5c9bb00,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,68,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5c9bb00,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5c9bb00,c5c9bb30,168,c0a5f873,0,...) at _sleep+0x359
kern_sigsuspend(c5c98220,0,0,0,0,...) at kern_sigsuspend+0xe4
sigsuspend(c5c98220,ebeb0cfc,4,c0aa22db,c0b4b1f8,...) at sigsuspend+0x4d
syscall(ebeb0d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28166ffb, esp = 
0xbfbfe80c, ebp = 0xbfbfe878 ---

Tracing command sshd pid 21991 tid 100112 td 0xc57c6440
sched_switch(c57c6440,0,1,17b,917954c4,...) at sched_switch+0x49b
mi_switch(1,0,c0aa0a32,2e2,c57c8a50,...) at mi_switch+0x217
turnstile_wait(c57c8a50,c5119880,0,17a,c5719d80,...) at turnstile_wait+0x4cb
_mtx_lock_sleep(c5719d80,c57c6440,0,c0aaa355,517,...) at 
_mtx_lock_sleep+0x15a
_mtx_lock_flags(c5719d80,0,c0aaa355,517,ebd27964,...) at 
_mtx_lock_flags+0xef
rt_check(ebd2795c,ebd27978,cb4ee2d0,c0b4f564,c0aaa355,...) at rt_check+0x113
arpresolve(c526a400,c5874618,c550fb00,cb4ee2d0,ebd27992,...) at 
arpresolve+0xb0
ether_output(c526a400,c550fb00,cb4ee2d0,c5874618,c57c6440,...) at 
ether_output+0x7e
ip_output(c550fb00,0,ebd27a00,0,0,...) at ip_output+0xa04
tcp_output(c58a4378,c6148500,c0ab354d,30b,c58a17c8,...) at tcp_output+0x13f0
tcp_usr_send(c66cb630,0,c6148500,0,0,...) at tcp_usr_send+0x25a
sosend_generic(c66cb630,0,ebd27c60,0,0,...) at sosend_generic+0x50d
sosend(c66cb630,0,ebd27c60,0,0,...) at sosend+0x54
soo_write(c5be9d80,ebd27c60,c605ad00,0,c57c6440,...) at soo_write+0x4b
dofilewrite(ebd27c60,ffffffff,ffffffff,0,c5be9d80,...) at dofilewrite+0x95
kern_writev(c57c6440,3,ebd27c60,28535550,0,...) at kern_writev+0x58
write(c57c6440,ebd27cfc,c,c0a5612b,c0b49260,...) at write+0x4f
syscall(ebd27d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (4, FreeBSD ELF32, write), eip = 0x283e3483, esp = 
0xbfbfe5cc, ebp = 0xbfbfe5e8 ---

Tracing command sshd pid 21982 tid 100151 td 0xc5a3bcc0
sched_switch(c5a3bcc0,0,1,180,f317eb00,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5a3bcc0,0,c0aa02cc,19b,c7600248,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,58,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c7600248,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c7600248,c7600200,158,c0aa509f,0) at _sleep+0x359
sbwait(c76001dc,1,c0aa51b4,5ae,c7600200,...) at sbwait+0x76
soreceive_generic(c760018c,0,ebdf5c60,0,0,...) at soreceive_generic+0x42f
soreceive(c760018c,0,ebdf5c60,0,0,...) at soreceive+0x4d
soo_read(c6ee14c8,ebdf5c60,c99eed00,0,c5a3bcc0,...) at soo_read+0x3b
dofileread(ebdf5c60,ffffffff,ffffffff,0,c6ee14c8,...) at dofileread+0x96
kern_readv(c5a3bcc0,5,ebdf5c60,bfbfe64c,4,...) at kern_readv+0x58
read(c5a3bcc0,ebdf5cfc,c,c0ab76e1,c0b49248,...) at read+0x4f
syscall(ebdf5d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x283e34a3, esp = 
0xbfbfe60c, ebp = 0xbfbfe628 ---

Tracing command gmake pid 20101 tid 100252 td 0xca6f1660
sched_switch(ca6f1660,0,1,180,2da8e9cc,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(ca6f1660,0,c0aa02cc,19b,cb4a0000,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(cb4a0000,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(cb4a0000,cb4a0090,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(ca6f1660,ffffffff,ebfb7c2c,0,0,...) at kern_wait+0xa66
wait4(ca6f1660,ebfb7cfc,10,c0aa22db,c0b492a8,...) at wait4+0x3b
syscall(ebfb7d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x281f9e7b, esp = 
0xbfbfd49c, ebp = 0xbfbfd4b8 ---

Tracing command sh pid 20097 tid 100201 td 0xc6eee000
sched_switch(c6eee000,0,1,180,cca81b5c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c6eee000,0,c0aa02cc,19b,c6eeb7f8,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c6eeb7f8,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c6eeb7f8,c6eeb888,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c6eee000,ffffffff,ebf34c2c,2,0,...) at kern_wait+0xa66
wait4(c6eee000,ebf34cfc,10,c0aa23e0,c0b492a8,...) at wait4+0x3b
syscall(ebf34d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149e7b, esp = 
0xbfbfdf3c, ebp = 0xbfbfdf58 ---

Tracing command gmake pid 20096 tid 100064 td 0xc55a1440
sched_switch(c55a1440,0,1,180,cb25aefa,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c55a1440,0,c0aa02cc,19b,c559daa0,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c559daa0,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c559daa0,c559db30,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c55a1440,ffffffff,ebc76c2c,0,0,...) at kern_wait+0xa66
wait4(c55a1440,ebc76cfc,10,c0aa22db,c0b492a8,...) at wait4+0x3b
syscall(ebc76d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x281f9e7b, esp = 
0xbfbfd58c, ebp = 0xbfbfd5a8 ---

Tracing command sh pid 84323 tid 100257 td 0xca6edaa0
sched_switch(ca6edaa0,0,1,180,c8f4c658,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(ca6edaa0,0,c0aa02cc,19b,cb4a0d48,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(cb4a0d48,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(cb4a0d48,cb4a0dd8,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(ca6edaa0,ffffffff,ebfa8c2c,2,0,...) at kern_wait+0xa66
wait4(ca6edaa0,ebfa8cfc,10,c0aa23e0,c0b492a8,...) at wait4+0x3b
syscall(ebfa8d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149e7b, esp = 
0xbfbfdf1c, ebp = 0xbfbfdf38 ---

Tracing command gmake pid 84322 tid 100235 td 0xca6ecaa0
sched_switch(ca6ecaa0,0,1,180,1a565222,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(ca6ecaa0,0,c0aa02cc,19b,ca6ef2a8,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(ca6ef2a8,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(ca6ef2a8,ca6ef338,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(ca6ecaa0,ffffffff,ebf93c2c,0,0,...) at kern_wait+0xa66
wait4(ca6ecaa0,ebf93cfc,10,c0aa22db,c0b492a8,...) at wait4+0x3b
syscall(ebf93d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x281f9e7b, esp = 
0xbfbfd59c, ebp = 0xbfbfd5b8 ---

Tracing command sh pid 84318 tid 100167 td 0xc5c1c220
sched_switch(c5c1c220,0,1,180,18ffc18c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5c1c220,0,c0aa02cc,19b,c5c20d48,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5c20d48,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5c20d48,c5c20dd8,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c5c1c220,ffffffff,ebe25c2c,2,0,...) at kern_wait+0xa66
wait4(c5c1c220,ebe25cfc,10,c0aa23e0,c0b492a8,...) at wait4+0x3b
syscall(ebe25d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149e7b, esp = 
0xbfbfdecc, ebp = 0xbfbfdee8 ---

Tracing command gmake pid 84317 tid 100208 td 0xc6eed000
sched_switch(c6eed000,0,1,180,177a6518,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c6eed000,0,c0aa02cc,19b,c6eefaa0,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c6eefaa0,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c6eefaa0,c6eefb30,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c6eed000,ffffffff,ebf1fc2c,0,0,...) at kern_wait+0xa66
wait4(c6eed000,ebf1fcfc,10,c0aa22db,c0b492a8,...) at wait4+0x3b
syscall(ebf1fd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x281f9e7b, esp = 
0xbfbfd68c, ebp = 0xbfbfd6a8 ---

Tracing command gmake pid 84316 tid 100143 td 0xc5bbf000
sched_switch(c5bbf000,0,1,180,15579c9a,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5bbf000,0,c0aa02cc,19b,c5bc1d48,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5bc1d48,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5bc1d48,c5bc1dd8,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c5bbf000,ffffffff,ebe0dc2c,0,0,...) at kern_wait+0xa66
wait4(c5bbf000,ebe0dcfc,10,c0aa22db,c0b492a8,...) at wait4+0x3b
syscall(ebe0dd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x281f9e7b, esp = 
0xbfbfd76c, ebp = 0xbfbfd788 ---

Tracing command sh pid 84315 tid 100224 td 0xc7220880
sched_switch(c7220880,0,1,180,11e9dfac,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c7220880,0,c0aa02cc,19b,c7224550,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c7224550,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c7224550,c72245e0,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c7220880,ffffffff,ebf66c2c,2,0,...) at kern_wait+0xa66
wait4(c7220880,ebf66cfc,10,ebf66d38,c0b492a8,...) at wait4+0x3b
syscall(ebf66d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149e7b, esp = 
0xbfbfe41c, ebp = 0xbfbfe438 ---

Tracing command make pid 20720 tid 100096 td 0xc562b000
sched_switch(c562b000,0,1,180,115fc284,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c562b000,0,c0aa02cc,19b,c5633000,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5633000,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5633000,c5633090,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c562b000,ffffffff,ebce6c2c,0,0,...) at kern_wait+0xa66
wait4(c562b000,ebce6cfc,10,c0aa235c,c0b492a8,...) at wait4+0x3b
syscall(ebce6d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x80684cf, esp = 
0xbfbfdc9c, ebp = 0xbfbfdcb8 ---

Tracing command sh pid 20719 tid 100182 td 0xc5c1e880
sched_switch(c5c1e880,0,1,180,6a097586,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5c1e880,0,c0aa02cc,19b,c5c97550,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5c97550,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5c97550,c5c975e0,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c5c1e880,ffffffff,ebe58c2c,2,0,...) at kern_wait+0xa66
wait4(c5c1e880,ebe58cfc,10,ebe58d38,c0b492a8,...) at wait4+0x3b
syscall(ebe58d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149e7b, esp = 
0xbfbfe7fc, ebp = 0xbfbfe818 ---

Tracing command make pid 20703 tid 100178 td 0xc5c85220
sched_switch(c5c85220,0,1,180,696d66ec,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5c85220,0,c0aa02cc,19b,c5c87aa0,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5c87aa0,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5c87aa0,c5c87b30,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c5c85220,ffffffff,ebe86c2c,0,0,...) at kern_wait+0xa66
wait4(c5c85220,ebe86cfc,10,c0aa235c,c0b492a8,...) at wait4+0x3b
syscall(ebe86d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x80684cf, esp = 
0xbfbfdd9c, ebp = 0xbfbfddb8 ---

Tracing command csh pid 86175 tid 100093 td 0xc562b660
sched_switch(c562b660,0,1,180,ffd380c4,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c562b660,0,c0aa02cc,19b,c5528410,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5528410,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5528410,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c5528400,c5528410,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c5528400,ebcefc60,0,c843fd00,ebcefc60,...) at ttread+0x4e7
ptsread(c843fd00,ebcefc60,0,168,0,...) at ptsread+0x3e
giant_read(c843fd00,ebcefc60,0,0,1,...) at giant_read+0x48
devfs_read_f(c6ee1ca8,ebcefc60,c83ea300,0,c562b660,...) at devfs_read_f+0x71
dofileread(ebcefc60,ffffffff,ffffffff,0,c6ee1ca8,...) at dofileread+0x96
kern_readv(c562b660,10,ebcefc60,bfbfe836,1,...) at kern_readv+0x58
read(c562b660,ebcefcfc,c,c0a8d525,c0b49248,...) at read+0x4f
syscall(ebcefd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281f84a3, esp = 
0xbfbfe7ec, ebp = 0xbfbfe808 ---

Tracing command su pid 85857 tid 100111 td 0xc57c6660
sched_switch(c57c6660,0,1,180,35f498b4,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c57c6660,0,c0aa02cc,19b,c57ca7f8,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c57ca7f8,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c57ca7f8,c57ca888,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c57c6660,1509f,ebd2ac2c,2,0,...) at kern_wait+0xa66
wait4(c57c6660,ebd2acfc,10,ebd2ad38,c0b492a8,...) at wait4+0x3b
syscall(ebd2ad38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2810ae7b, esp = 
0xbfbfe79c, ebp = 0xbfbfe7b8 ---

Tracing command csh pid 84932 tid 100073 td 0xc55d5cc0
sched_switch(c55d5cc0,0,1,180,3faccc7c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c55d5cc0,0,c0aa02cc,19b,c55d1308,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,68,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c55d1308,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c55d1308,c55d1338,168,c0a5f873,0,...) at _sleep+0x359
kern_sigsuspend(c55d5cc0,0,0,0,0,...) at kern_sigsuspend+0xe4
sigsuspend(c55d5cc0,ebcbdcfc,4,c0aa22db,c0b4b1f8,...) at sigsuspend+0x4d
syscall(ebcbdd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28166ffb, esp = 
0xbfbfe80c, ebp = 0xbfbfe878 ---

Tracing command sshd pid 84911 tid 100068 td 0xc559eaa0
sched_switch(c559eaa0,0,1,180,7727e34,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c559eaa0,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(c0742baa,c0c03ca4,0,c0a9b5fa,c559eaa0,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c0c03cbc,c0c03ca4,c0ab9fa0,101,0,...) at 
sleepq_wait_sig+0x14
_cv_wait_sig(c0c03cbc,c0c03ca4,c0aa1e52,30f,c0a966e4,...) at 
_cv_wait_sig+0x1fc
kern_select(c559eaa0,9,285030e4,285030e8,0,0,3fd0,285030e4) at 
kern_select+0x6cc
select(c559eaa0,ebc6acfc,14,c0a5612b,c0b49ab8,...) at select+0x5e
syscall(ebc6ad38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283e3423, esp = 
0xbfbfe5fc, ebp = 0xbfbfe648 ---

Tracing command sshd pid 83121 tid 100154 td 0xc5c1e000
sched_switch(c5c1e000,0,1,180,6e89b21c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5c1e000,0,c0aa02cc,19b,c57126ec,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,58,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c57126ec,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c57126ec,c57126a4,158,c0aa509f,0) at _sleep+0x359
sbwait(c5712680,1,c0aa51b4,5ae,c57126a4,...) at sbwait+0x76
soreceive_generic(c5712630,0,ebe4cc60,0,0,...) at soreceive_generic+0x42f
soreceive(c5712630,0,ebe4cc60,0,0,...) at soreceive+0x4d
soo_read(c724d360,ebe4cc60,c8467e00,0,c5c1e000,...) at soo_read+0x3b
dofileread(ebe4cc60,ffffffff,ffffffff,0,c724d360,...) at dofileread+0x96
kern_readv(c5c1e000,5,ebe4cc60,bfbfe64c,4,...) at kern_readv+0x58
read(c5c1e000,ebe4ccfc,c,c0ab76e1,c0b49248,...) at read+0x4f
syscall(ebe4cd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x283e34a3, esp = 
0xbfbfe60c, ebp = 0xbfbfe628 ---

Tracing command sh pid 25073 tid 100185 td 0xc5c99aa0
sched_switch(c5c99aa0,0,1,180,55885b24,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5c99aa0,0,c0aa02cc,19b,c5c97d48,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5c97d48,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5c97d48,c5c97dd8,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c5c99aa0,ffffffff,ebed1c2c,2,0,...) at kern_wait+0xa66
wait4(c5c99aa0,ebed1cfc,10,c0aa23e0,c0b492a8,...) at wait4+0x3b
syscall(ebed1d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149e7b, esp = 
0xbfbfdb5c, ebp = 0xbfbfdb78 ---

Tracing command make pid 7399 tid 100088 td 0xc562c220
sched_switch(c562c220,0,1,180,f024bcc0,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c562c220,0,c0aa02cc,19b,c562aaa0,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c562aaa0,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c562aaa0,c562ab30,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c562c220,ffffffff,ebcfec2c,0,0,...) at kern_wait+0xa66
wait4(c562c220,ebcfecfc,10,c0aa235c,c0b492a8,...) at wait4+0x3b
syscall(ebcfed38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x80684cf, esp = 
0xbfbfde4c, ebp = 0xbfbfde68 ---

Tracing command csh pid 810 tid 100097 td 0xc55d6cc0
sched_switch(c55d6cc0,0,1,180,11513c44,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c55d6cc0,0,c0aa02cc,19b,c5633308,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,68,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5633308,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5633308,c5633338,168,c0a5f873,0,...) at _sleep+0x359
kern_sigsuspend(c55d6cc0,0,0,0,0,...) at kern_sigsuspend+0xe4
sigsuspend(c55d6cc0,ebcd2cfc,4,c0aa22db,c0b4b1f8,...) at sigsuspend+0x4d
syscall(ebcd2d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28166ffb, esp = 
0xbfbfe72c, ebp = 0xbfbfe798 ---

Tracing command su pid 809 tid 100122 td 0xc5a3b660
sched_switch(c5a3b660,0,1,180,5f8c36e4,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5a3b660,0,c0aa02cc,19b,c5a38550,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5a38550,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5a38550,c5a385e0,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c5a3b660,32a,ebdecc2c,2,0,...) at kern_wait+0xa66
wait4(c5a3b660,ebdeccfc,10,ebdecd38,c0b492a8,...) at wait4+0x3b
syscall(ebdecd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2810ae7b, esp = 
0xbfbfe78c, ebp = 0xbfbfe7a8 ---

Tracing command csh pid 805 tid 100126 td 0xc5a39cc0
sched_switch(c5a39cc0,0,1,180,8e4b9782,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5a39cc0,0,c0aa02cc,19b,c5a3c060,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,68,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5a3c060,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5a3c060,c5a3c090,168,c0a5f873,0,...) at _sleep+0x359
kern_sigsuspend(c5a39cc0,0,0,0,0,...) at kern_sigsuspend+0xe4
sigsuspend(c5a39cc0,ebde0cfc,4,c0aa22db,c0b4b1f8,...) at sigsuspend+0x4d
syscall(ebde0d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28166ffb, esp = 
0xbfbfe80c, ebp = 0xbfbfe878 ---

Tracing command sshd pid 804 tid 100070 td 0xc559e660
sched_switch(c559e660,0,1,180,2d9d0594,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c559e660,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(c0742baa,c0c03ca4,0,c0a9b5fa,c559e660,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c0c03cbc,c0c03ca4,c0ab9fa0,101,0,...) at 
sleepq_wait_sig+0x14
_cv_wait_sig(c0c03cbc,c0c03ca4,c0aa1e52,30f,c0ab354d,...) at 
_cv_wait_sig+0x1fc
kern_select(c559e660,9,285030e4,285030e8,0,0,0,280e05e8) at 
kern_select+0x6cc
select(c559e660,ebc64cfc,14,c0a8d525,c0b49ab8,...) at select+0x5e
syscall(ebc64d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283e3423, esp = 
0xbfbfe5fc, ebp = 0xbfbfe648 ---

Tracing command sshd pid 801 tid 100127 td 0xc5a39aa0
sched_switch(c5a39aa0,0,1,180,f0ecc0ae,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5a39aa0,0,c0aa02cc,19b,c5712a04,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,58,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5712a04,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5712a04,c57129bc,158,c0aa509f,0) at _sleep+0x359
sbwait(c5712998,1,c0aa51b4,5ae,c57129bc,...) at sbwait+0x76
soreceive_generic(c5712948,0,ebdddc60,0,0,...) at soreceive_generic+0x42f
soreceive(c5712948,0,ebdddc60,0,0,...) at soreceive+0x4d
soo_read(c55cbd80,ebdddc60,c5630000,0,c5a39aa0,...) at soo_read+0x3b
dofileread(ebdddc60,ffffffff,ffffffff,0,c55cbd80,...) at dofileread+0x96
kern_readv(c5a39aa0,5,ebdddc60,bfbfe64c,4,...) at kern_readv+0x58
read(c5a39aa0,ebdddcfc,c,c0ab76e1,c0b49248,...) at read+0x4f
syscall(ebdddd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x283e34a3, esp = 
0xbfbfe60c, ebp = 0xbfbfe628 ---

Tracing command login pid 793 tid 100082 td 0xc55d2880
sched_switch(c55d2880,0,1,180,7a9d1d84,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c55d2880,0,c0aa02cc,19b,c535d810,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c535d810,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c535d810,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c535d800,c535d810,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c535d800,ebca2c60,0,ebca2bb8,c071e7e8,...) at ttread+0x4e7
ttyread(c52d8300,ebca2c60,0,168,0,...) at ttyread+0x39
giant_read(c52d8300,ebca2c60,0,0,1,...) at giant_read+0x48
devfs_read_f(c5634ca8,ebca2c60,c511c800,0,c55d2880,...) at devfs_read_f+0x71
dofileread(ebca2c60,ffffffff,ffffffff,0,c5634ca8,...) at dofileread+0x96
kern_readv(c55d2880,0,ebca2c60,bfbfe9bb,1,...) at kern_readv+0x58
read(c55d2880,ebca2cfc,c,c0aa23f0,c0b49248,...) at read+0x4f
syscall(ebca2d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2817f4a3, esp = 
0xbfbfe72c, ebp = 0xbfbfe9c8 ---

Tracing command getty pid 792 tid 100069 td 0xc559e880
sched_switch(c559e880,0,1,180,397a7f30,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c559e880,0,c0aa02cc,19b,c535f410,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c535f410,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c535f410,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c535f400,c535f410,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c535f400,ebc67c60,0,ebc67b9c,c0679bfd,...) at ttread+0x4e7
ttyread(c5380800,ebc67c60,0,ebc67bb8,c071e7e8,...) at ttyread+0x39
scread(c5380800,ebc67c60,0,168,0,...) at scread+0x2d
giant_read(c5380800,ebc67c60,0,0,1,...) at giant_read+0x48
devfs_read_f(c58739d8,ebc67c60,c511c800,0,c559e880,...) at devfs_read_f+0x71
dofileread(ebc67c60,ffffffff,ffffffff,0,c58739d8,...) at dofileread+0x96
kern_readv(c559e880,0,ebc67c60,bfbfee4b,1,...) at kern_readv+0x58
read(c559e880,ebc67cfc,c,c0ace8dc,c0b49248,...) at read+0x4f
syscall(ebc67d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281644a3, esp = 
0xbfbfee2c, ebp = 0xbfbfee58 ---

Tracing command getty pid 791 tid 100092 td 0xc562b880
sched_switch(c562b880,0,1,180,391d0a04,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c562b880,0,c0aa02cc,19b,c538a010,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c538a010,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c538a010,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c538a000,c538a010,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c538a000,ebcf2c60,0,ebcf2b9c,c0679bfd,...) at ttread+0x4e7
ttyread(c5380700,ebcf2c60,0,ebcf2bb8,c071e7e8,...) at ttyread+0x39
scread(c5380700,ebcf2c60,0,168,0,...) at scread+0x2d
giant_read(c5380700,ebcf2c60,0,0,1,...) at giant_read+0x48
devfs_read_f(c55cbca8,ebcf2c60,c511c800,0,c562b880,...) at devfs_read_f+0x71
dofileread(ebcf2c60,ffffffff,ffffffff,0,c55cbca8,...) at dofileread+0x96
kern_readv(c562b880,0,ebcf2c60,bfbfee4b,1,...) at kern_readv+0x58
read(c562b880,ebcf2cfc,c,c0ace8dc,c0b49248,...) at read+0x4f
syscall(ebcf2d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281644a3, esp = 
0xbfbfee2c, ebp = 0xbfbfee58 ---

Tracing command getty pid 790 tid 100098 td 0xc55d6aa0
sched_switch(c55d6aa0,0,1,180,38e12210,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c55d6aa0,0,c0aa02cc,19b,c5389c10,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5389c10,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5389c10,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c5389c00,c5389c10,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c5389c00,ebccfc60,0,ebccfb9c,c0679bfd,...) at ttread+0x4e7
ttyread(c5380600,ebccfc60,0,ebccfbb8,c071e7e8,...) at ttyread+0x39
scread(c5380600,ebccfc60,0,168,0,...) at scread+0x2d
giant_read(c5380600,ebccfc60,0,0,1,...) at giant_read+0x48
devfs_read_f(c55a6bd0,ebccfc60,c511c800,0,c55d6aa0,...) at devfs_read_f+0x71
dofileread(ebccfc60,ffffffff,ffffffff,0,c55a6bd0,...) at dofileread+0x96
kern_readv(c55d6aa0,0,ebccfc60,bfbfee4b,1,...) at kern_readv+0x58
read(c55d6aa0,ebccfcfc,c,c0ace8dc,c0b49248,...) at read+0x4f
syscall(ebccfd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281644a3, esp = 
0xbfbfee2c, ebp = 0xbfbfee58 ---

Tracing command getty pid 789 tid 100130 td 0xc5a39440
sched_switch(c5a39440,0,1,180,3940407c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5a39440,0,c0aa02cc,19b,c538ac10,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c538ac10,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c538ac10,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c538ac00,c538ac10,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c538ac00,ebdd4c60,0,ebdd4b9c,c0679bfd,...) at ttread+0x4e7
ttyread(c5380500,ebdd4c60,0,ebdd4bb8,c071e7e8,...) at ttyread+0x39
scread(c5380500,ebdd4c60,0,168,0,...) at scread+0x2d
giant_read(c5380500,ebdd4c60,0,0,1,...) at giant_read+0x48
devfs_read_f(c5873c18,ebdd4c60,c511c800,0,c5a39440,...) at devfs_read_f+0x71
dofileread(ebdd4c60,ffffffff,ffffffff,0,c5873c18,...) at dofileread+0x96
kern_readv(c5a39440,0,ebdd4c60,bfbfee4b,1,...) at kern_readv+0x58
read(c5a39440,ebdd4cfc,c,c0ace8dc,c0b49248,...) at read+0x4f
syscall(ebdd4d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281644a3, esp = 
0xbfbfee2c, ebp = 0xbfbfee58 ---

Tracing command getty pid 788 tid 100075 td 0xc55d5880
sched_switch(c55d5880,0,1,180,39b9a5dc,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c55d5880,0,c0aa02cc,19b,c538a810,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c538a810,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c538a810,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c538a800,c538a810,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c538a800,ebcb7c60,0,ebcb7b9c,c0679bfd,...) at ttread+0x4e7
ttyread(c5380400,ebcb7c60,0,ebcb7bb8,c071e7e8,...) at ttyread+0x39
scread(c5380400,ebcb7c60,0,168,0,...) at scread+0x2d
giant_read(c5380400,ebcb7c60,0,0,1,...) at giant_read+0x48
devfs_read_f(c55a6510,ebcb7c60,c511c800,0,c55d5880,...) at devfs_read_f+0x71
dofileread(ebcb7c60,ffffffff,ffffffff,0,c55a6510,...) at dofileread+0x96
kern_readv(c55d5880,0,ebcb7c60,bfbfee4b,1,...) at kern_readv+0x58
read(c55d5880,ebcb7cfc,c,c0ace8dc,c0b49248,...) at read+0x4f
syscall(ebcb7d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281644a3, esp = 
0xbfbfee2c, ebp = 0xbfbfee58 ---

Tracing command getty pid 787 tid 100087 td 0xc562c440
sched_switch(c562c440,0,1,180,396e591c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c562c440,0,c0aa02cc,19b,c5388c10,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5388c10,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5388c10,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c5388c00,c5388c10,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c5388c00,ebd01c60,0,ebd01b9c,c0679bfd,...) at ttread+0x4e7
ttyread(c5380300,ebd01c60,0,ebd01bb8,c071e7e8,...) at ttyread+0x39
scread(c5380300,ebd01c60,0,168,0,...) at scread+0x2d
giant_read(c5380300,ebd01c60,0,0,1,...) at giant_read+0x48
devfs_read_f(c5873cf0,ebd01c60,c511c800,0,c562c440,...) at devfs_read_f+0x71
dofileread(ebd01c60,ffffffff,ffffffff,0,c5873cf0,...) at dofileread+0x96
kern_readv(c562c440,0,ebd01c60,bfbfee4b,1,...) at kern_readv+0x58
read(c562c440,ebd01cfc,c,c0ace8dc,c0b49248,...) at read+0x4f
syscall(ebd01d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281644a3, esp = 
0xbfbfee2c, ebp = 0xbfbfee58 ---

Tracing command getty pid 786 tid 100091 td 0xc562baa0
sched_switch(c562baa0,0,1,180,394c38d0,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c562baa0,0,c0aa02cc,19b,c5388810,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5388810,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5388810,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c5388800,c5388810,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c5388800,ebcf5c60,0,ebcf5b9c,c0679bfd,...) at ttread+0x4e7
ttyread(c5380200,ebcf5c60,0,ebcf5bb8,c071e7e8,...) at ttyread+0x39
scread(c5380200,ebcf5c60,0,168,0,...) at scread+0x2d
giant_read(c5380200,ebcf5c60,0,0,1,...) at giant_read+0x48
devfs_read_f(c55cb630,ebcf5c60,c511c800,0,c562baa0,...) at devfs_read_f+0x71
dofileread(ebcf5c60,ffffffff,ffffffff,0,c55cb630,...) at dofileread+0x96
kern_readv(c562baa0,0,ebcf5c60,bfbfee4b,1,...) at kern_readv+0x58
read(c562baa0,ebcf5cfc,c,c0ace8dc,c0b49248,...) at read+0x4f
syscall(ebcf5d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281644a3, esp = 
0xbfbfee2c, ebp = 0xbfbfee58 ---

Tracing command getty pid 785 tid 100085 td 0xc52d5440
sched_switch(c52d5440,0,1,180,f4cd50d8,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c52d5440,0,c0aa02cc,19b,c5389810,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba580,1,c0a9b5fa,59,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5389810,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5389810,0,159,c0aa3c6f,0,...) at _sleep+0x359
ttysleep(c5389800,c5389810,159,c0aa3c6f,0,...) at ttysleep+0x39
ttread(c5389800,e99b4c60,0,e99b4b9c,c0679bfd,...) at ttread+0x4e7
ttyread(c5380100,e99b4c60,0,e99b4bb8,c071e7e8,...) at ttyread+0x39
scread(c5380100,e99b4c60,0,168,0,...) at scread+0x2d
giant_read(c5380100,e99b4c60,0,0,1,...) at giant_read+0x48
devfs_read_f(c5634510,e99b4c60,c511c800,0,c52d5440,...) at devfs_read_f+0x71
dofileread(e99b4c60,ffffffff,ffffffff,0,c5634510,...) at dofileread+0x96
kern_readv(c52d5440,0,e99b4c60,bfbfee4b,1,...) at kern_readv+0x58
read(c52d5440,e99b4cfc,c,c0ace8dc,c0b49248,...) at read+0x4f
syscall(e99b4d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281644a3, esp = 
0xbfbfee2c, ebp = 0xbfbfee58 ---

Tracing command inetd pid 764 tid 100094 td 0xc562b440
sched_switch(c562b440,0,1,180,1f622752,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c562b440,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(c0742baa,c0c03ca4,0,c0a9b5fa,c562b440,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c0c03cbc,c0c03ca4,c0ab9fa0,101,0,...) at 
sleepq_wait_sig+0x14
_cv_wait_sig(c0c03cbc,c0c03ca4,c0aa1e52,30f,ebcecb18,...) at 
_cv_wait_sig+0x1fc
kern_select(c562b440,6,bfbfec8c,0,0,0,c58af288,bfbfee48) at 
kern_select+0x6cc
select(c562b440,ebceccfc,14,c0aa2454,c0b49ab8,...) at select+0x5e
syscall(ebcecd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x28177423, esp = 
0xbfbfe08c, ebp = 0xbfbfee78 ---

Tracing command moused pid 737 tid 100089 td 0xc562c000
sched_switch(c562c000,0,1,180,b377436,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c562c000,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(c0742baa,c0c03ca4,0,c0a9b5fa,c562c000,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c0c03cbc,c0c03ca4,c0ab9fa0,101,0,...) at 
sleepq_wait_sig+0x14
_cv_wait_sig(c0c03cbc,c0c03ca4,c0aa1e52,30f,c065563f,...) at 
_cv_wait_sig+0x1fc
kern_select(c562c000,400,bfbfeaac,0,0,0,4739b5e9,0) at kern_select+0x6cc
select(c562c000,ebcfbcfc,14,c0ace8dc,c0b49ab8,...) at select+0x5e
syscall(ebcfbd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2817c423, esp = 
0xbfbfea2c, ebp = 0xbfbfebc8 ---

Tracing command cron pid 721 tid 100086 td 0xc52d5220
sched_switch(c52d5220,0,1,180,49e8e632,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c52d5220,0,c0aa02cc,19b,c0bb2724,...) at sleepq_switch+0xf0
sleepq_catch_signals(e99b1bc4,c074280d,c0bba580,1,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_timedwait_sig(c0bb2724,0,c0a9d1f9,da,0,...) at 
sleepq_timedwait_sig+0x17
_sleep(c0bb2724,0,15c,c0a9d9d5,ea61,...) at _sleep+0x331
kern_nanosleep(c52d5220,e99b1c64,e99b1c6c,3c,0,...) at kern_nanosleep+0xc1
nanosleep(c52d5220,e99b1cfc,8,c0aa2545,c0b4a880,...) at nanosleep+0x6f
syscall(e99b1d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28150f7f, esp = 
0xbfbfed0c, ebp = 0xbfbfed38 ---

Tracing command sendmail pid 715 tid 100047 td 0xc5255aa0
sched_switch(c5255aa0,0,1,180,3d3fe238,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5255aa0,0,c0aa02cc,19b,c52d7da8,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,68,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c52d7da8,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c52d7da8,c52d7dd8,168,c0a5f873,0,...) at _sleep+0x359
kern_sigsuspend(c5255aa0,0,0,0,0,...) at kern_sigsuspend+0xe4
sigsuspend(c5255aa0,e5757cfc,4,c0aa22db,c0b4b1f8,...) at sigsuspend+0x4d
syscall(e5757d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2830bffb, esp = 
0xbfbfdb1c, ebp = 0xbfbfdb48 ---

Tracing command sendmail pid 711 tid 100071 td 0xc559e440
sched_switch(c559e440,0,1,180,7f893e1c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c559e440,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(ebc61a84,c0742baa,c0c03ca4,0,c559e440,...) at 
sleepq_catch_signals+0x21e
sleepq_timedwait_sig(c0c03cbc,1389,c0ab9fa0,101,0,...) at 
sleepq_timedwait_sig+0x17
_cv_timedwait_sig(c0c03cbc,c0c03ca4,1389,30f,c073fd6d,...) at 
_cv_timedwait_sig+0x20b
kern_select(c559e440,5,bfbfd92c,0,0,ebc61c70,5,0) at kern_select+0x6b4
select(c559e440,ebc61cfc,14,c0aa27fa,c0b49ab8,...) at select+0x5e
syscall(ebc61d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2839d423, esp = 
0xbfbfd09c, ebp = 0xbfbfdb38 ---

Tracing command sshd pid 705 tid 100081 td 0xc55d2aa0
sched_switch(c55d2aa0,0,1,180,e160d12c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c55d2aa0,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(c0742baa,c0c03ca4,0,c0a9b5fa,c55d2aa0,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c0c03cbc,c0c03ca4,c0ab9fa0,101,0,...) at 
sleepq_wait_sig+0x14
_cv_wait_sig(c0c03cbc,c0c03ca4,c0aa1e52,30f,ebca5afc,...) at 
_cv_wait_sig+0x1fc
kern_select(c55d2aa0,7,2850b0b8,0,0,0,c,ebca5c98) at kern_select+0x6cc
select(c55d2aa0,ebca5cfc,14,ebca5d38,c0b49ab8,...) at select+0x5e
syscall(ebca5d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283e3423, esp = 
0xh(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5255cc0,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(c0742baa,c0c03ca4,0,c0a9b5fa,c5255cc0,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c0c03cbc,c0c03ca4,c0ab9fa0,101,0,...) at 
sleepq_wait_sig+0x14
_cv_wait_sig(c0c03cbc,c0c03ca4,c0aa1e52,30f,0,...) at _cv_wait_sig+0x1fc
kern_select(c5255cc0,9,282200b8,0,0,0,396c5,28177818) at kern_select+0x6cc
select(c5255cc0,e575acfc,14,c0aa23f0,c0b49ab8,...) at select+0x5e
syscall(e575ad38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x28167423, esp = 
0xbfbfdf3c, ebp = 0xbfbfee88 ---

Tracing command devd pid 524 tid 100048 td 0xc5255880
sched_switch(c5255880,0,1,180,88841474,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5255880,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(c0742baa,c0c03ca4,0,c0a9b5fa,c5255880,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c0c03cbc,c0c03ca4,c0ab9fa0,101,0,...) at 
sleepq_wait_sig+0x14
_cv_wait_sig(c0c03cbc,c0c03ca4,c0aa1e52,30f,e5754afc,...) at 
_cv_wait_sig+0x1fc
kern_select(c5255880,5,bfbfedf8,0,0,0,0,bfbfe99c) at kern_select+0x6cc
select(c5255880,e5754cfc,14,c0aa2748,c0b49ab8,...) at select+0x5e
syscall(e5754d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x807f95f, esp = 
0xbfbfe9bc, ebp = 0xbfbfee98 ---

Tracing command dhclient pid 328 tid 100084 td 0xc52d5660
sched_switch(c52d5660,0,1,180,3bd05a58,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c52d5660,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(e99b7ad8,c0742baa,c0c03ca4,0,c52d5660,...) at 
sleepq_catch_signals+0x21e
sleepq_timedwait_sig(c0c03cbc,14ff1,c0ab9fa0,101,0,...) at 
sleepq_timedwait_sig+0x17
_cv_timedwait_sig(c0c03cbc,c0c03ca4,14ff1,3bb,e99b7b48,...) at 
_cv_timedwait_sig+0x20b
poll(c52d5660,e99b7cfc,c,c0a5612b,c0b4a598,...) at poll+0x554
syscall(e99b7d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x2810f9af, esp = 
0xbfbfee1c, ebp = 0xbfbfee48 ---

Tracing command dhclient pid 308 tid 100095 td 0xc562b220
sched_switch(c562b220,0,1,180,50643cea,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c562b220,0,c0aa02cc,19b,c0c03cbc,...) at sleepq_switch+0xf0
sleepq_catch_signals(c0742baa,c0c03ca4,0,c0a9b5fa,c562b220,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c0c03cbc,c0c03ca4,c0ab9fa0,101,0,...) at 
sleepq_wait_sig+0x14
_cv_wait_sig(c0c03cbc,c0c03ca4,c0aa1e52,3bb,c0aa1eef,...) at 
_cv_wait_sig+0x1fc
poll(c562b220,ebce9cfc,c,c0a8d525,c0b4a598,...) at poll+0x56a
syscall(ebce9d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x2810f9af, esp = 
0xbfbfee1c, ebp = 0xbfbfee48 ---

Tracing command softdepflush pid 53 tid 100049 td 0xc5255660
sched_switch(c5255660,0,1,180,9dbda24,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,44,...) at mi_switch+0x217
sleepq_switch(c5255660,0,c0aa02cc,240,44,...) at sleepq_switch+0xf0
sleepq_timedwait(c0c0f9a4,3e8,c0abbaca,0,0,...) at sleepq_timedwait+0x68
_sleep(c0c0f9a4,c0c0f97c,44,c0abbaca,3e8,...) at _sleep+0x345
softdep_flush(0,e5751d38,c0a9920b,2ea,c54f92a8,...) at softdep_flush+0x2c8
fork_exit(c09354b0,0,e5751d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe5751d70, ebp = 0 ---

Tracing command syncer pid 52 tid 100050 td 0xc54facc0
sched_switch(c54facc0,0,1,180,a09f520,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c54facc0,...) at mi_switch+0x217
sleepq_switch(c54facc0,0,c0aa02cc,21d,68,...) at sleepq_switch+0xf0
sleepq_wait(c0bb254c,0,c0aa8698,0,0,...) at sleepq_wait+0x60
_sleep(c0bb254c,c0c04444,68,c0aa8698,0,...) at _sleep+0x369
sched_sync(0,e9a69d38,c0a9920b,2ea,c54f9550,...) at sched_sync+0x56e
fork_exit(c07cb690,0,e9a69d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe9a69d70, ebp = 0 ---

Tracing command vnlru pid 51 tid 100051 td 0xc54faaa0
sched_switch(c54faaa0,0,1,180,49fd376c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,50,...) at mi_switch+0x217
sleepq_switch(c54faaa0,0,c0aa02cc,240,50,...) at sleepq_switch+0xf0
sleepq_timedwait(c54f97f8,3e8,c0aa8742,0,0,...) at sleepq_timedwait+0x68
_sleep(c54f97f8,c0c04418,250,c0aa8742,3e8,...) at _sleep+0x345
vnlru_proc(0,e9a66d38,c0a9920b,2ea,c54f97f8,...) at vnlru_proc+0x114
fork_exit(c07cc450,0,e9a66d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe9a66d70, ebp = 0 ---

Tracing command bufdaemon pid 50 tid 100052 td 0xc54fa880
sched_switch(c54fa880,0,1,180,da7be8b8,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,44,...) at mi_switch+0x217
sleepq_switch(c54fa880,0,c0aa02cc,240,44,...) at sleepq_switch+0xf0
sleepq_timedwait(c0c04164,3e8,c0aa6585,0,0,...) at sleepq_timedwait+0x68
_sleep(c0c04164,c0c04168,44,c0aa6585,3e8,...) at _sleep+0x345
buf_daemon(0,e9a63d38,c0a9920b,2ea,c54f9aa0,...) at buf_daemon+0x21e
fork_exit(c07b8280,0,e9a63d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe9a63d70, ebp = 0 ---

Tracing command pagezero pid 49 tid 100053 td 0xc54fa660
sched_switch(c54fa660,0,1,180,6413cea6,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c54fa660,0,c0aa02cc,240,0,...) at sleepq_switch+0xf0
sleepq_timedwait(c0c10560,493e0,c0ac062b,0,0,...) at sleepq_timedwait+0x68
_sleep(c0c10560,c0c10104,0,c0ac062b,493e0,...) at _sleep+0x345
vm_pagezero(0,e9a60d38,c0a9920b,2ea,c54f9d48,...) at vm_pagezero+0xdc
fork_exit(c096ea00,0,e9a60d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe9a60d70, ebp = 0 ---

Tracing command vmdaemon pid 48 tid 100054 td 0xc54fa440
sched_switch(c54fa440,0,1,180,4968ab00,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c54fa440,...) at mi_switch+0x217
sleepq_switch(c54fa440,0,c0aa02cc,21d,68,...) at sleepq_switch+0xf0
sleepq_wait(c0c10178,0,c0a9d1f9,da,0,...) at sleepq_wait+0x60
_sleep(c0c10178,c0c1017c,68,c0aa6585,0,...) at _sleep+0x369
vm_daemon(0,e9a5dd38,c0a9920b,2ea,c54fb000,...) at vm_daemon+0x59
fork_exit(c0969e40,0,e9a5dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe9a5dd70, ebp = 0 ---

Tracing command pagedaemon pid 47 tid 100055 td 0xc54fa220
sched_switch(c54fa220,0,1,180,502fd750,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,44,...) at mi_switch+0x217
sleepq_switch(c54fa220,0,c0aa02cc,240,44,...) at sleepq_switch+0xf0
sleepq_timedwait(c0c10140,1388,c0aa6585,0,0,...) at sleepq_timedwait+0x68
_sleep(c0c10140,c0c10104,44,c0aa6585,1388,...) at _sleep+0x345
vm_pageout(0,e9a5ad38,c0a9920b,2ea,c54fb2a8,...) at vm_pageout+0x2bb
fork_exit(c096a9f0,0,e9a5ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe9a5ad70, ebp = 0 ---

Tracing command sctp_iterator pid 46 tid 100056 td 0xc54fa000
sched_switch(c54fa000,0,1,180,16b33bc0,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c54fa000,...) at mi_switch+0x217
sleepq_switch(c54fa000,0,c0aa02cc,21d,0,...) at sleepq_switch+0xf0
sleepq_wait(c0c05ecc,c0c05dd4,c0aaf4d9,0,0,...) at sleepq_wait+0x60
_sleep(c0c05ecc,c0c05dd4,0,c0aaf4d9,0,...) at _sleep+0x369
sctp_iterator_thread(0,e9a6cd38,c0a9920b,2ea,c54fb550,...) at 
sctp_iterator_thread+0x60
fork_exit(c0833730,0,e9a6cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe9a6cd70, ebp = 0 ---

Tracing command irq7: ppc0 pid 45 tid 100034 td 0xc5254440
fork_trampoline() at fork_trampoline

Tracing command fdc0 pid 44 tid 100035 td 0xc5254220
sched_switch(c5254220,0,1,180,9d4623c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,4c,...) at mi_switch+0x217
sleepq_switch(c5254220,0,c0aa02cc,240,4c,...) at sleepq_switch+0xf0
sleepq_timedwait(c5324e3c,3e8,c0a94196,0,0,...) at sleepq_timedwait+0x68
_sleep(c5324e3c,c5324ef0,4c,c0a94196,3e8,...) at _sleep+0x345
fdc_thread(c5324e00,e5736d38,c0a9920b,2ea,c5256d48,...) at fdc_thread+0x2b8
fork_exit(c09c69b0,c5324e00,e5736d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe5736d70, ebp = 0 ---

Tracing command swi0: sio pid 43 tid 100036 td 0xc5254000
sched_switch(c5254000,0,1,180,37bd6d44,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c53629e4,...) at mi_switch+0x217
ithread_loop(c5360b50,e5733d38,c0a9920b,2ea,c52d3000,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c5360b50,e5733d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe5733d70, ebp = 0 ---

Tracing command irq12: psm0 pid 42 tid 100037 td 0xc5160cc0
fork_trampoline() at fork_trampoline

Tracing command irq1: atkbd0 pid 41 tid 100038 td 0xc52d5000
sched_switch(c52d5000,0,1,180,4f6f164,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c5111464,...) at mi_switch+0x217
ithread_loop(c5379340,e99aed38,c0a9920b,2ea,c52d3550,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c5379340,e99aed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe99aed70, ebp = 0 ---

Tracing command irq15: ata1 pid 40 tid 100039 td 0xc52d4cc0
sched_switch(c52d4cc0,0,1,180,4a2ac8a4,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c5110c64,...) at mi_switch+0x217
ithread_loop(c535bab0,e99abd38,c0a9920b,2ea,c52d37f8,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c535bab0,e99abd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe99abd70, ebp = 0 ---

Tracing command irq14: ata0 pid 39 tid 100040 td 0xc52d4aa0
fork_trampoline() at fork_trampoline

Tracing command aic_recovery1 pid 38 tid 100041 td 0xc52d4880
sched_switch(c52d4880,0,1,180,8167da80,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c52d4880,...) at mi_switch+0x217
sleepq_switch(c52d4880,0,c0aa02cc,21d,a0,...) at sleepq_switch+0xf0
sleepq_wait(c5326000,0,c0a9d1f9,da,0,...) at sleepq_wait+0x60
_sleep(c5326000,c53212b4,a0,c0a5f5b8,0,...) at _sleep+0x369
ahd_recovery_thread(c5326000,e99a5d38,c0a9920b,2ea,c52d3d48,...) at 
ahd_recovery_thread+0x72
fork_exit(c04c3d60,c5326000,e99a5d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe99a5d70, ebp = 0 ---

Tracing command irq22: ahd1 pid 37 tid 100042 td 0xc52d4660
sched_switch(c52d4660,0,1,180,45c9b3ec,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c515e4e4,...) at mi_switch+0x217
ithread_loop(c535b470,e99a2d38,c0a9920b,2ea,c52d7000,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c535b470,e99a2d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe99a2d70, ebp = 0 ---

Tracing command aic_recovery0 pid 36 tid 100043 td 0xc52d4440
sched_switch(c52d4440,0,1,180,8167a830,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c52d4440,...) at mi_switch+0x217
sleepq_switch(c52d4440,0,c0aa02cc,21d,a0,...) at sleepq_switch+0xf0
sleepq_wait(c52db000,0,c0a9d1f9,da,0,...) at sleepq_wait+0x60
_sleep(c52db000,c5265bb4,a0,c0a5f5b8,0,...) at _sleep+0x369
ahd_recovery_thread(c52db000,e999fd38,c0a9920b,2ea,c52d72a8,...) at 
ahd_recovery_thread+0x72
fork_exit(c04c3d60,c52db000,e999fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe999fd70, ebp = 0 ---

Tracing command aac0aif pid 35 tid 100044 td 0xc52d4220
sched_switch(c52d4220,0,1,180,deb41c52,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,4c,...) at mi_switch+0x217
sleepq_switch(c52d4220,0,c0aa02cc,240,4c,...) at sleepq_switch+0xf0
sleepq_timedwait(c52d7550,4e20,c0a5acdf,0,0,...) at sleepq_timedwait+0x68
_sleep(c52d7550,c5276128,4c,c0a5acdf,4e20,...) at _sleep+0x345
aac_command_thread(c5276000,e999cd38,c0a9920b,2ea,c52d7550,...) at 
aac_command_thread+0x8d
fork_exit(c0496140,c5276000,e999cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe999cd70, ebp = 0 ---

Tracing command irq18: aac0 pid 34 tid 100024 td 0xc5119aa0
sched_switch(c5119aa0,0,1,180,fc64d9ac,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c515e6e4,...) at mi_switch+0x217
ithread_loop(c5271a50,e3cd2d38,c0a9920b,2ea,c5251000,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c5271a50,e3cd2d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cd2d70, ebp = 0 ---

Tracing command irq21: bge0 pid 33 tid 100025 td 0xc5119880
cpustop_handler(1,e3ccf958,c0a05b10,1,0,...) at cpustop_handler+0x32
ipi_nmi_handler(1,0,0,0,13,...) at ipi_nmi_handler+0x2f
trap(e3ccf964) at trap+0x30
calltrap() at calltrap+0x6
--- trap 0x13, eip = 0xc074e586, esp = 0xe3ccf9a4, ebp = 0xe3ccf9c8 ---
panic(c0a9b762,c5117660,c0aa0a06,c5117660,186ae,...) at panic+0x26
_mtx_lock_spin_failed(1,19,c0aa0a32,cb,19,...) at _mtx_lock_spin_failed+0x51
_thread_lock_flags(c57c6440,10,c0aa0a32,cb,1,...) at _thread_lock_flags+0xc7
propagate_priority(c0bbceec,0,c0aa0a32,2e2,c50f2a00,...) at 
propagate_priority+0xe0
turnstile_wait(c50f2a00,c57c6440,0,17a,c5874678,...) at turnstile_wait+0x48c
_mtx_lock_sleep(c5874678,c5119880,0,c0aaa355,41c,...) at 
_mtx_lock_sleep+0x15a
_mtx_lock_flags(c5874678,0,c0aaa355,41c,e3ccfaf4,...) at 
_mtx_lock_flags+0xef
rt_setgate(c5874618,cb4ee2c0,e3ccfbac,c5119880,e3ccfb18,...) at 
rt_setgate+0x196
rtredirect(e3ccfbbc,e3ccfbac,0,26,e3ccfb9c,...) at rtredirect+0x18e
icmp_input(c6fb6500,14,246,c0b9c7c0,e3ccfc0c,...) at icmp_input+0x50f
ip_input(c6fb6500,14e,800,c526a400,800,...) at ip_input+0x650
netisr_dispatch(2,c6fb6500,10,3,0,...) at netisr_dispatch+0x73
ether_demux(c526a400,c6fb6500,3,0,3,...) at ether_demux+0x1f1
ether_input(c526a400,c6fb6500,c0a6af05,bc4,c526a400,...) at 
ether_input+0x37f
bge_intr(c526d000,0,c0a99497,471,c515e564,...) at bge_intr+0x7da
ithread_loop(c5271160,e3ccfd38,c0a9920b,2ea,c52512a8,...) at 
ithread_loop+0x1b5
fork_exit(c0732ea0,c5271160,e3ccfd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3ccfd70, ebp = 0 ---

Tracing command usb2 pid 32 tid 100026 td 0xc5119660
sched_switch(c5119660,0,1,180,5e0d4486,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,5c,...) at mi_switch+0x217
sleepq_switch(c5119660,0,c0aa02cc,240,5c,...) at sleepq_switch+0xf0
sleepq_timedwait(c520ba10,ea60,c0a9111c,0,0,...) at sleepq_timedwait+0x68
_sleep(c520ba10,0,5c,c0a9111c,ea60,...) at _sleep+0x345
usb_event_thread(c52685c0,e3cccd38,c0a9920b,2ea,c5251550,...) at 
usb_event_thread+0xca
fork_exit(c06cbb40,c52685c0,e3cccd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cccd70, ebp = 0 ---

Tracing command irq23: ehci0 pid 31 tid 100027 td 0xc5255440
fork_trampoline() at fork_trampoline

Tracing command usb1 pid 30 tid 100028 td 0xc5255220
sched_switch(c5255220,0,1,180,5e0e59ae,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,5c,...) at mi_switch+0x217
sleepq_switch(c5255220,0,c0aa02cc,240,5c,...) at sleepq_switch+0xf0
sleepq_timedwait(c5243210,ea60,c0a9111c,0,0,...) at sleepq_timedwait+0x68
_sleep(c5243210,0,5c,c0a9111c,ea60,...) at _sleep+0x345
usb_event_thread(c523cc00,e574bd38,c0a9920b,2ea,c5251aa0,...) at 
usb_event_thread+0xca
fork_exit(c06cbb40,c523cc00,e574bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe574bd70, ebp = 0 ---

Tracing command irq19: uhci1 ahd0 pid 29 tid 100029 td 0xc5255000
sched_switch(c5255000,0,1,180,2f1617b8,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c515e664,...) at mi_switch+0x217
ithread_loop(c5242a20,e5748d38,c0a9920b,2ea,c5251d48,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c5242a20,e5748d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe5748d70, ebp = 0 ---

Tracing command usbtask-dr pid 28 tid 100030 td 0xc5254cc0
sched_switch(c5254cc0,0,1,180,81669054,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c5254cc0,...) at mi_switch+0x217
sleepq_switch(c5254cc0,0,c0aa02cc,21d,5c,...) at sleepq_switch+0xf0
sleepq_wait(c0bafe54,0,c0a9d1f9,da,0,...) at sleepq_wait+0x60
_sleep(c0bafe54,0,5c,c0a91159,0,...) at _sleep+0x369
usb_task_thread(c0bafe54,e5745d38,c0a9920b,2ea,c5256000,...) at 
usb_task_thread+0x62
fork_exit(c06cc2c0,c0bafe54,e5745d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe5745d70, ebp = 0 ---

Tracing command usbtask-hc pid 27 tid 100031 td 0xc5254aa0
sched_switch(c5254aa0,0,1,180,816663d8,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c5254aa0,...) at mi_switch+0x217
sleepq_switch(c5254aa0,0,c0aa02cc,21d,5c,...) at sleepq_switch+0xf0
sleepq_wait(c0bafe40,0,c0a9d1f9,da,0,...) at sleepq_wait+0x60
_sleep(c0bafe40,0,5c,c0a91159,0,...) at _sleep+0x369
usb_task_thread(c0bafe40,e5742d38,c0a9920b,2ea,c52562a8,...) at 
usb_task_thread+0x62
fork_exit(c06cc2c0,c0bafe40,e5742d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe5742d70, ebp = 0 ---

Tracing command usb0 pid 26 tid 100032 td 0xc5254880
sched_switch(c5254880,0,1,180,5e0ecc9a,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,5c,...) at mi_switch+0x217
sleepq_switch(c5254880,0,c0aa02cc,240,5c,...) at sleepq_switch+0xf0
sleepq_timedwait(c5249210,ea60,c0a9111c,0,0,...) at sleepq_timedwait+0x68
_sleep(c5249210,0,5c,c0a9111c,ea60,...) at _sleep+0x345
usb_event_thread(c524d680,e573fd38,c0a9920b,2ea,c5256550,...) at 
usb_event_thread+0xca
fork_exit(c06cbb40,c524d680,e573fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe573fd70, ebp = 0 ---

Tracing command irq16: uhci0 pid 25 tid 100033 td 0xc5254660
fork_trampoline() at fork_trampoline

Tracing command irq9: acpi0 pid 24 tid 100015 td 0xc5117440
fork_trampoline() at fork_trampoline

Tracing command kqueue taskq pid 23 tid 100016 td 0xc5117220
sched_switch(c5117220,0,1,180,81b1c448,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c5117220,...) at mi_switch+0x217
sleepq_switch(c5117220,0,c0aa02cc,21d,0,...) at sleepq_switch+0xf0
sleepq_wait(c5213600,c521361c,c0a94196,0,0,...) at sleepq_wait+0x60
_sleep(c5213600,c521361c,0,c0a94196,0,...) at _sleep+0x369
taskqueue_thread_loop(c0bb0dc4,e3cb1d38,c0a9920b,2ea,c515faa0,...) at 
taskqueue_thread_loop+0xb4
fork_exit(c0780830,c0bb0dc4,e3cb1d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cb1d70, ebp = 0 ---

Tracing command swi6: task queue pid 22 tid 100017 td 0xc5160aa0
sched_switch(c5160aa0,0,1,180,1fb23b70,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c52136e4,...) at mi_switch+0x217
ithread_loop(c520a0e0,e570ad38,c0a9920b,2ea,c515fd48,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c520a0e0,e570ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe570ad70, ebp = 0 ---

Tracing command swi6: Giant taskq pid 21 tid 100018 td 0xc5160880
sched_switch(c5160880,0,1,180,4f7572c,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c52137e4,...) at mi_switch+0x217
ithread_loop(c520a0f0,e5707d38,c0a9920b,2ea,c5214000,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c520a0f0,e5707d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe5707d70, ebp = 0 ---

Tracing command thread taskq pid 9 tid 100019 td 0xc5160660
sched_switch(c5160660,0,1,180,45ed1704,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c5160660,...) at mi_switch+0x217
sleepq_switch(c5160660,0,c0aa02cc,21d,0,...) at sleepq_switch+0xf0
sleepq_wait(c5213880,c521389c,c0a94196,0,0,...) at sleepq_wait+0x60
_sleep(c5213880,c521389c,0,c0a94196,0,...) at _sleep+0x369
taskqueue_thread_loop(c0bbc6d8,e5704d38,c0a9920b,2ea,c52142a8,...) at 
taskqueue_thread_loop+0xb4
fork_exit(c0780830,c0bbc6d8,e5704d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe5704d70, ebp = 0 ---

Tracing command acpi_task_2 pid 8 tid 100020 td 0xc5160440
sched_switch(c5160440,0,1,180,817489b8,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c5160440,...) at mi_switch+0x217
sleepq_switch(c5160440,0,c0aa02cc,21d,0,...) at sleepq_switch+0xf0
sleepq_wait(c5213900,c521391c,c0a94196,0,0,...) at sleepq_wait+0x60
_sleep(c5213900,c521391c,0,c0a94196,0,...) at _sleep+0x369
taskqueue_thread_loop(c0d82ebc,e5701d38,c0a9920b,2ea,c5214550,...) at 
taskqueue_thread_loop+0xb4
fork_exit(c0780830,c0d82ebc,e5701d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe5701d70, ebp = 0 ---

Tracing command acpi_task_1 pid 7 tid 100021 td 0xc5160220
sched_switch(c5160220,0,1,180,81745958,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c5160220,...) at mi_switch+0x217
sleepq_switch(c5160220,0,c0aa02cc,21d,0,...) at sleepq_switch+0xf0
sleepq_wait(c5213900,c521391c,c0a94196,0,0,...) at sleepq_wait+0x60
_sleep(c5213900,c521391c,0,c0a94196,0,...) at _sleep+0x369
taskqueue_thread_loop(c0d82ebc,e56fed38,c0a9920b,2ea,c52147f8,...) at 
taskqueue_thread_loop+0xb4
fork_exit(c0780830,c0d82ebc,e56fed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe56fed70, ebp = 0 ---

Tracing command acpi_task_0 pid 6 tid 100022 td 0xc5160000
sched_switch(c5160000,0,1,180,817425a0,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c5160000,...) at mi_switch+0x217
sleepq_switch(c5160000,0,c0aa02cc,21d,0,...) at sleepq_switch+0xf0
sleepq_wait(c5213900,c521391c,c0a94196,0,0,...) at sleepq_wait+0x60
_sleep(c5213900,c521391c,0,c0a94196,0,...) at _sleep+0x369
taskqueue_thread_loop(c0d82ebc,e56fbd38,c0a9920b,2ea,c5214aa0,...) at 
taskqueue_thread_loop+0xb4
fork_exit(c0780830,c0d82ebc,e56fbd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe56fbd70, ebp = 0 ---

Tracing command swi5: + pid 20 tid 100023 td 0xc5119cc0
fork_trampoline() at fork_trampoline

Tracing command swi2: cambio pid 19 tid 100007 td 0xc5115000
sched_switch(c5115000,0,1,180,45cc7cd4,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c5213ae4,...) at mi_switch+0x217
ithread_loop(c520a130,e3c99d38,c0a9920b,2ea,c51182a8,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c520a130,e3c99d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3c99d70, ebp = 0 ---

Tracing command xpt_thrd pid 5 tid 100008 td 0xc5119440
sched_switch(c5119440,0,1,180,8165f1e0,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c5119440,...) at mi_switch+0x217
sleepq_switch(c5119440,0,c0aa02cc,21d,4c,...) at sleepq_switch+0xf0
sleepq_wait(c0b81014,0,c0a9d1f9,da,0,...) at sleepq_wait+0x60
_sleep(c0b81014,c0b8102c,4c,c0a533ca,0,...) at _sleep+0x369
xpt_scanner_thread(0,e3cc9d38,c0a9920b,2ea,c5118550,...) at 
xpt_scanner_thread+0x41
fork_exit(c04683a0,0,e3cc9d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cc9d70, ebp = 0 ---

Tracing command yarrow pid 18 tid 100009 td 0xc5119220
sched_switch(c5119220,0,1,180,9d6bbb0,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5119220,0,c0aa02cc,240,0,...) at sleepq_switch+0xf0
sleepq_timedwait(c0bb2554,64,c0a94196,2,0,...) at sleepq_timedwait+0x68
_sleep(c0bb2554,0,0,c0a94196,64,...) at _sleep+0x345
pause(c0a94196,64,c0a88aa0,112,0,...) at pause+0x47
random_kthread(0,e3cc6d38,c0a9920b,2ea,c51187f8,...) at random_kthread+0x1d9
fork_exit(c0656330,0,e3cc6d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cc6d70, ebp = 0 ---

Tracing command g_down pid 4 tid 100010 td 0xc5119000
sched_switch(c5119000,0,1,180,9d58524,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,4c,...) at mi_switch+0x217
sleepq_switch(c5119000,0,c0aa02cc,240,4c,...) at sleepq_switch+0xf0
sleepq_timedwait(c0bb05cc,64,c0a94196,0,0,...) at sleepq_timedwait+0x68
_sleep(c0bb05cc,c0bb04e8,24c,c0a94196,64,...) at _sleep+0x345
g_io_schedule_down(c5119000,0,c0a95891,74,0,...) at g_io_schedule_down+0x6b
g_down_procbody(0,e3cc3d38,c0a9920b,2ea,c5118aa0,...) at 
g_down_procbody+0x8d
fork_exit(c0702520,0,e3cc3d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cc3d70, ebp = 0 ---

Tracing command g_up pid 3 tid 100011 td 0xc5117cc0
sched_switch(c5117cc0,0,1,180,da7c29c0,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,4c,...) at mi_switch+0x217
sleepq_switch(c5117cc0,0,c0aa02cc,240,4c,...) at sleepq_switch+0xf0
sleepq_timedwait(c0bb05c8,64,c0a94196,0,0,...) at sleepq_timedwait+0x68
_sleep(c0bb05c8,c0bb0528,24c,c0a94196,64,...) at _sleep+0x345
g_io_schedule_up(c5117cc0,0,c0a95891,5d,0,...) at g_io_schedule_up+0x133
g_up_procbody(0,e3cc0d38,c0a9920b,2ea,c5118d48,...) at g_up_procbody+0x8d
fork_exit(c07025b0,0,e3cc0d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cc0d70, ebp = 0 ---

Tracing command g_event pid 2 tid 100012 td 0xc5117aa0
sched_switch(c5117aa0,0,1,180,da7c8d90,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,4c,...) at mi_switch+0x217
sleepq_switch(c5117aa0,0,c0aa02cc,240,4c,...) at sleepq_switch+0xf0
sleepq_timedwait(c0bb05c0,64,c0a94196,0,0,...) at sleepq_timedwait+0x68
_sleep(c0bb05c0,0,4c,c0a94196,64,...) at _sleep+0x345
g_event_procbody(0,e3cbdd38,c0a9920b,2ea,c515f000,...) at 
g_event_procbody+0xcb
fork_exit(c0702640,0,e3cbdd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cbdd70, ebp = 0 ---

Tracing command swi3: vm pid 17 tid 100013 td 0xc5117880
fork_trampoline() at fork_trampoline

Tracing command swi4: clock sio pid 16 tid 100014 td 0xc5117660
kdb_enter(c0a9c88e,1,c0a9b762,e3cb7b7c,1,...) at kdb_enter+0x32
panic(c0a9b762,c5119880,c0aa0a06,c5119880,186b9,...) at panic+0x124
_mtx_lock_spin_failed(1,19,c0aa0a32,cb,19,...) at _mtx_lock_spin_failed+0x51
_thread_lock_flags(c5119880,10,c0aa0a32,cb,1,...) at _thread_lock_flags+0xc7
propagate_priority(c0bbc9c8,0,c0aa0a32,2e2,c51165f0,...) at 
propagate_priority+0xe0
turnstile_wait(c51165f0,c57c6440,0,17a,c58a17c8,...) at turnstile_wait+0x48c
_mtx_lock_sleep(c58a17c8,c5117660,0,c0ab2bb5,1b6,...) at 
_mtx_lock_sleep+0x15a
_mtx_lock_flags(c58a17c8,0,c0ab2bb5,1b6,412b9e7,...) at _mtx_lock_flags+0xef
tcp_timer_rexmt(c58a4378,0,c0a9daa7,ef,24,...) at tcp_timer_rexmt+0x96
softclock(0,0,c0a99497,471,c515e364,...) at softclock+0x266
ithread_loop(c5113280,e3cb7d38,c0a9920b,2ea,c515f550,...) at 
ithread_loop+0x1b5
fork_exit(c0732ea0,c5113280,e3cb7d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cb7d70, ebp = 0 ---

Tracing command swi1: net pid 15 tid 100000 td 0xc5117000
sched_switch(c5117000,0,1,180,3bca4e84,...) at sched_switch+0x49b
mi_switch(1,0,c0a99497,46b,c515e3e4,...) at mi_switch+0x217
ithread_loop(c5113290,e3caed38,c0a9920b,2ea,c5114000,...) at 
ithread_loop+0x34c
fork_exit(c0732ea0,c5113290,e3caed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3caed70, ebp = 0 ---

Tracing command idle: cpu0 pid 14 tid 100001 td 0xc5115cc0
sched_switch(c5115cc0,0,6,17b,3b3b9d38,...) at sched_switch+0x49b
mi_switch(6,0,c0ac91ab,495,0,...) at mi_switch+0x217
ipi_bitmap_handler(c5110008,e3ca0028,c5110028,c515c018,c515c000,...) at 
ipi_bitmap_handler+0x72
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x2e
--- interrupt, eip = 0x20, esp = 0xc0d6b83c, ebp = 0xe3cabc7c ---
uart_z8530_class(c0d724a5,20,246,e3cabcc0,c0d6b83c,...) at 0x20

Tracing command idle: cpu1 pid 13 tid 100002 td 0xc5115aa0
sched_switch(c5115aa0,0,6,17b,9d3229c,...) at sched_switch+0x49b
mi_switch(6,0,c0ac91ab,495,1,...) at mi_switch+0x217
ipi_bitmap_handler(8,c0bb0028,28,0,f,...) at ipi_bitmap_handler+0x72
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x2e
--- interrupt, eip = 0x20, esp = 0xe3ca8cc8, ebp = 0xe3ca8ca8 ---
uart_z8530_class(c09f9bb2,20,202,2,e3ca8cc8,...) at 0x20

Tracing command idle: cpu2 pid 12 tid 100003 td 0xc5115880
cpustop_handler(4,e3ca5c3c,c0a05b10,c5115880,e3ca5bbc,...) at 
cpustop_handler+0x32
ipi_nmi_handler(c5115880,e3ca5bbc,c076b378,e3ca5bf4,13,...) at 
ipi_nmi_handler+0x2f
trap(e3ca5c48) at trap+0x30
calltrap() at calltrap+0x6
--- trap 0x13, eip = 0xc07431e5, esp = 0xe3ca5c88, ebp = 0xe3ca5ca8 ---
_mtx_lock_spin_flags(c0bba580,0,c0a9e25c,26d,c0bb47c0,...) at 
_mtx_lock_spin_flags+0xd5
tdq_lock_pair(c0bba600,0,c0a9e25c,322,c0bb4e00,...) at tdq_lock_pair+0x82
sched_idletd(0,e3ca5d38,c0a9920b,2ea,c51147f8,...) at sched_idletd+0x19d
fork_exit(c076c300,0,e3ca5d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3ca5d70, ebp = 0 ---

Tracing command idle: cpu3 pid 11 tid 100004 td 0xc5115660
cpustop_handler(8,e3ca2c70,c0a05b10,c076b378,e3ca2c20,...) at 
cpustop_handler+0x32
ipi_nmi_handler(c076b378,e3ca2c20,c076bd94,c0bba600,13,...) at 
ipi_nmi_handler+0x2f
trap(e3ca2c7c) at trap+0x30
calltrap() at calltrap+0x6
--- trap 0x13, eip = 0xc09f9bb2, esp = 0xe3ca2cbc, ebp = 0xe3ca2cc0 ---
mp_grab_cpu_hlt(e3ca2cf8,c076c549,c0bba600,0,c0a9e25c,...) at 
mp_grab_cpu_hlt+0x22
cpu_idle(c0bba600,0,c0a9e25c,322,c0bb5440,...) at cpu_idle+0x8
sched_idletd(0,e3ca2d38,c0a9920b,2ea,c5114aa0,...) at sched_idletd+0x249
fork_exit(c076c300,0,e3ca2d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3ca2d70, ebp = 0 ---

Tracing command init pid 1 tid 100005 td 0xc5115440
sched_switch(c5115440,0,1,180,e5706560,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,0,...) at mi_switch+0x217
sleepq_switch(c5115440,0,c0aa02cc,19b,c5114d48,...) at sleepq_switch+0xf0
sleepq_catch_signals(c074280d,c0bba600,1,c0a9b5fa,5c,...) at 
sleepq_catch_signals+0x21e
sleepq_wait_sig(c5114d48,0,c0a9d1f9,da,0,...) at sleepq_wait_sig+0x14
_sleep(c5114d48,c5114dd8,15c,c0aa2401,0,...) at _sleep+0x359
kern_wait(c5115440,ffffffff,e3c9fc2c,0,0,...) at kern_wait+0xa66
wait4(c5115440,e3c9fcfc,10,c0aa2214,c0b492a8,...) at wait4+0x3b
syscall(e3c9fd38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x805452f, esp = 
0xbfbfe97c, ebp = 0xbfbfe998 ---

Tracing command audit pid 10 tid 100006 td 0xc5115220
sched_switch(c5115220,0,1,180,815aaf9c,...) at sched_switch+0x49b
mi_switch(1,0,c0aa02cc,1bf,c5115220,...) at mi_switch+0x217
sleepq_switch(c5115220,0,c0aa02cc,21d,c5115220,...) at sleepq_switch+0xf0
sleepq_wait(c0c0f414,c0c0f3f4,c0ab96d6,1,0,...) at sleepq_wait+0x60
_cv_wait(c0c0f414,c0c0f3f4,c0aba083,1c2,0,...) at _cv_wait+0x1fc
audit_worker(0,e3c9cd38,c0a9920b,2ea,c5118000,...) at audit_worker+0x99
fork_exit(c091b170,0,e3c9cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3c9cd70, ebp = 0 ---

Tracing command swapper pid 0 tid 0 td 0xc0bb0930
sched_switch(c0bb0930,0,1,180,4acff9e0,...) at sched_switch+0x49b
mi_switch(1,0,c0abe20a,2d9,c0ac912b,...) at mi_switch+0x217
scheduler(0,101ec00,101ec00,101e000,1028000,...) at scheduler+0x2f2
mi_startup() at mi_startup+0x96
begin() at begin+0x2c
db>
>How-To-Repeat:
Install RELENG_7 on i386/SMP
Let the box do something (compile KDE in my case)
Access the box remotely and do something, "ps ax" or cat /var/log/messages was it that triggered the faultr in this particular case.
In our case the IPSec link is over our firewall and I believe this is a key point.
>Fix:


>Release-Note:
>Audit-Trail:

From: Per olof Ljungmark <peo@intersonic.se>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/118044: panic: spin lock held too long
Date: Wed, 05 Dec 2007 06:31:15 +0100

 Correction: The type of CPU is not important - the DL360's are on 
 another network segment and does not see the ICMP-REDIRECT.
 With UP kernel, only the rtfree msgs are displayed, no freeze
 With SMP kernel, freezes occur regardless of machine type.

From: Maxime Henrion <mux@FreeBSD.org>
To: bug-followup@FreeBSD.org, peo@intersonic.se
Cc:  
Subject: Re: kern/118044: panic: spin lock held too long
Date: Thu, 27 Dec 2007 11:51:47 +0100

 	Hi Per,
 
 
 There is a suspicion that the problem you are experiencing might
 be fixed by a patch that has just been committed to HEAD.  Could
 you please give it a try?  It's a fairly small patch, you just need
 to apply revision 1.125 of sys/net/route.c and revision 1.68 of
 sys/net/route.h.
 
 Let me know if you need some help in doing this testing.
 
 Cheers,
 Maxime

From: Per olof Ljungmark <peo@intersonic.se>
To: Maxime Henrion <mux@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/118044: panic: spin lock held too long
Date: Thu, 27 Dec 2007 14:16:16 +0100

 Maxime Henrion wrote:
 > 	Hi Per,
 > 
 > 
 > There is a suspicion that the problem you are experiencing might
 > be fixed by a patch that has just been committed to HEAD.  Could
 > you please give it a try?  It's a fairly small patch, you just need
 > to apply revision 1.125 of sys/net/route.c and revision 1.68 of
 > sys/net/route.h.
 > 
 > Let me know if you need some help in doing this testing.
 
 Hi Maxime,
 
 Thank you for your kind attantion to this problem.
 
 Did as per your advise, result below. System was rebuilt from sources 
 today + patched route.c/h.
 
 savecore: no dumps found
 Initial i386 initialization:.
 Additional ABI support:.
 Starting named.
 Setting date via ntp.
 lock order reversal:
   1st 0xc58370d8 rtentry (rtentry) @ /usr/src/sys/net/route.c:196
   2nd 0xc538d07c radix node head (radix node head) @ 
 /usr/src/sys/net/route.c:692
 KDB: stack backtrace:
 db_trace_self_wrapper(c0a9fc45,e5797a70,c078792e,c0aa2117,c538d07c,...) 
 at db_trace_self_wrapper+0x26
 kdb_backtrace(c0aa2117,c538d07c,c0aa2178,c0aa2178,c0aaaa4b,...) at 
 kdb_backtrace+0x29
 witness_checkorder(c538d07c,9,c0aaaa4b,2b4,c5214d58,...) at 
 witness_checkorder+0x6de
 _mtx_lock_flags(c538d07c,0,c0aaaa4b,2b4,c0a9bb4b,...) at 
 _mtx_lock_flags+0xbc
 rtrequest1(1,e5797b0c,e5797b3c,e5797b48,c08911ac,...) at rtrequest1+0x4f
 rtredirect(e5797bbc,e5797bac,0,16,e5797b9c,...) at rtredirect+0x126
 icmp_input(c5738000,14,246,c0b9aca0,e5797c0c,...) at icmp_input+0x50f
 ip_input(c5738000,14e,800,c5250400,800,...) at ip_input+0x650
 netisr_dispatch(2,c5738000,10,3,0,...) at netisr_dispatch+0x73
 ether_demux(c5250400,c5738000,3,0,3,...) at ether_demux+0x1f1
 ether_input(c5250400,c5738000,c0a6b43a,bc4,c5250400,...) at 
 ether_input+0x37f
 bge_intr(c526d000,0,c0a999ef,471,c515e564,...) at bge_intr+0x7da
 ithread_loop(c5268640,e5797d38,c0a99763,2ea,c5263558,...) at 
 ithread_loop+0x1b5
 fork_exit(c07328a0,c5268640,e5797d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe5797d70, ebp = 0 ---
 27 Dec 14:09:38 ntpdate[648]: adjust time server 192.168.1.2 offset 
 -0.408824 sec
 Clearing /tmp (X related).
 Starting local daemons:.
 Updating motd.
 Mounting late file systems:.
 Starting ntpd.
 postfix/postfix-script: starting the Postfix mail system
 Starting mysql.
 Performing sanity check on apache22 configuration:
 Syntax OK
 Starting apache22.
 Configuring syscons: keymap keyrate font8x16 font8x14 font8x8 blanktime.
 Starting sshd.
 Starting cron.
 Local package initialization:.
 Starting default moused:.
 Starting inetd.
 Starting background file system checks in 60 seconds.
 
 Thu Dec 27 14:09:43 CET 2007
 
 FreeBSD/i386 (ruben.i.inter-sonic.com) (ttyd0)
 
 login: rtfree: 0xc5722bb8 has 1 refs
 Dec 27 14:10:19 ruben su: peo to root on /dev/ttyp0
 rtfree: 0xc5722bb8 has 1 refs
 rtfree: 0xc5722bb8 has 1 refs
 rtfree: 0xc5722bb8 has 1 refs
 KDB: enter: Line break on console
 [thread pid 33 tid 100037 ]
 Stopped at      kdb_enter+0x32: leave
 db> wh
 Tracing pid 33 tid 100037 td 0xc5214cc0
 kdb_enter(c0a8e6e4,c536f8f8,c0c1b0e8,8,c536f88c,...) at kdb_enter+0x32
 siointr1(c0c1b0e8,0,c0ac563d,56f,e57979dc,...) at siointr1+0x11c
 siointr(c536f800,c0bb0844,2,c5214cc0,c5111280,...) at siointr+0x32
 intr_execute_handlers(c5107cd0,e57979fc,e5797a58,c09eb844,3b,...) at 
 intr_execute_handlers+0xec
 lapic_handle_intr(3b,e57979fc) at lapic_handle_intr+0x3f
 Xapic_isr1() at Xapic_isr1+0x34
 --- interrupt, eip = 0xc07429c2, esp = 0xe5797a3c, ebp = 0xe5797a58 ---
 _mtx_lock_sleep(c538d07c,c5214cc0,0,c0aaaa4b,93,...) at _mtx_lock_sleep+0xa2
 _mtx_lock_flags(c538d07c,0,c0aaaa4b,93,c0a9bb4b,...) at _mtx_lock_flags+0xef
 rtalloc1(e5797bac,1,0,e5797b20,c07426fc,...) at rtalloc1+0x63
 rtredirect(e5797bbc,e5797bac,0,26,e5797b9c,...) at rtredirect+0x35e
 icmp_input(c57d1700,14,246,c0b9aca0,e5797c0c,...) at icmp_input+0x50f
 ip_input(c57d1700,14e,800,c5250400,800,...) at ip_input+0x650
 netisr_dispatch(2,c57d1700,10,3,0,...) at netisr_dispatch+0x73
 ether_demux(c5250400,c57d1700,3,0,3,...) at ether_demux+0x1f1
 ether_input(c5250400,c57d1700,c0a6b43a,bc4,c5250400,...) at 
 ether_input+0x37f
 bge_intr(c526d000,0,c0a999ef,471,c515e564,...) at bge_intr+0x7da
 ithread_loop(c5268640,e5797d38,c0a99763,2ea,c5263558,...) at 
 ithread_loop+0x1b5
 fork_exit(c07328a0,c5268640,e5797d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe5797d70, ebp = 0 ---
 db> ps
    pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
    954   949   954     0  S+      ttywri   0xc5355838 dmesg
    949   948   949     0  S+      pause    0xc56ef30c csh
    948   944   948  1001  S+      wait     0xc5af0804 su
    944   943   944  1001  Ss+     pause    0xc5540b10 csh
    943   940   940  1001  R       CPU 1               sshd
    940   852   940     0  Ss      sbwait   0xc58a8560 sshd
    936   835   835    80  S       accept   0xc58a866a httpd
    935   835   835    80  S       accept   0xc58a866a httpd
    934   835   835    80  S       accept   0xc58a866a httpd
    933   835   835    80  S       accept   0xc58a866a httpd
    932   835   835    80  S       accept   0xc58a866a httpd
    931     1   931     0  Ss+     ttyin    0xc5357c10 getty
    930     1   930     0  Ss+     ttyin    0xc538a810 getty
    929     1   929     0  Ss+     ttyin    0xc538ac10 getty
    928     1   928     0  Ss+     ttyin    0xc5389810 getty
    927     1   927     0  Ss+     ttyin    0xc538b410 getty
    926     1   926     0  Ss+     ttyin    0xc538c410 getty
    925     1   925     0  Ss+     ttyin    0xc538c010 getty
    924     1   924     0  Ss+     ttyin    0xc537fc10 getty
    923     1   923     0  Ss+     ttyin    0xc5380410 getty
    921   919    54     0  S+      nanslp   0xc0bb0824 sleep
    920     1    54     0  S+      piperd   0xc586d948 logger
    919   918    54     0  S+      wait     0xc559f2ac sh
    918     1    54     0  S+      wait     0xc5598558 sh
    902     1   902     0  Ss      select   0xc0c01ddc inetd
    875     1   875     0  Ss      select   0xc0c01ddc moused
    859     1   859     0  Ss      nanslp   0xc0bb0824 cron
    852     1   852     0  Ss      select   0xc0c01ddc sshd
    835     1   835     0  Ss      select   0xc0c01ddc httpd
    833   808   805    88  S+      (threaded)          mysqld
 100139                   S       ucond    0xc561f2c0 mysqld
 100138                   S       sigwait  0xebe8cbe0 mysqld
 100135                   S       ucond    0xc561f240 mysqld
 100137                   S       select   0xc0c01ddc mysqld
 100136                   S       select   0xc0c01ddc mysqld
 100134                   S       ucond    0xc561f140 mysqld
 100133                   S       ucond    0xc561f100 mysqld
 100132                   S       ucond    0xc55ae740 mysqld
 100131                   S       ucond    0xc55ae700 mysqld
 100086                   S       select   0xc0c01ddc initial thread
    808     1   805    88  S+      wait     0xc585fab0 sh
    803   795   795   125  S       kqread   0xc587da00 qmgr
    802   795   795   125  S       kqread   0xc55aab80 pickup
    795     1   795     0  Ss      kqread   0xc5564080 master
    728     1   728     0  Ss      select   0xc0c01ddc ntpd
    643     1   643    53  Ss      (threaded)          named
 100130                   S       select   0xc0c01ddc named
 100129                   S       ucond    0xc555d240 named
 100128                   S       ucond    0xc555d200 named
 100127                   S       ucond    0xc555d1c0 named
 100126                   S       ucond    0xc555d180 named
 100125                   S       ucond    0xc555d140 named
 100107                   S       sigwait  0xe9b14be0 named
    578     1   578     0  Ss      select   0xc0c01ddc syslogd
    517     1   517     0  Ss      select   0xc0c01ddc devd
    321     1   321    65  Ss      select   0xc0c01ddc dhclient
    301     1    54     0  S+      select   0xc0c01ddc dhclient
     53     0     0     0  SL      sdflush  0xc0c0db44 [softdepflush]
     52     0     0     0  SL      syncer   0xc0bb064c [syncer]
     51     0     0     0  SL      vlruwt   0xc5374000 [vnlru]
     50     0     0     0  SL      psleep   0xc0c02244 [bufdaemon]
     49     0     0     0  SL      pgzero   0xc0c0e700 [pagezero]
     48     0     0     0  SL      psleep   0xc0c0e318 [vmdaemon]
     47     0     0     0  SL      psleep   0xc0c0e2e0 [pagedaemon]
     46     0     0     0  SL      waiting_ 0xc0c03fac [sctp_iterator]
     45     0     0     0  WL                          [irq7: ppc0]
     44     0     0     0  SL      -        0xc531ce3c [fdc0]
     43     0     0     0  WL                          [swi0: sio]
     42     0     0     0  WL                          [irq12: psm0]
     41     0     0     0  WL                          [irq1: atkbd0]
     40     0     0     0  WL                          [irq15: ata1]
     39     0     0     0  WL                          [irq14: ata0]
     38     0     0     0  SL      idle     0xc531e000 [aic_recovery1]
     37     0     0     0  WL                          [irq22: ahd1]
     36     0     0     0  SL      idle     0xc52d5000 [aic_recovery0]
     35     0     0     0  SL      aifthd   0xc5263000 [aac0aif]
     34     0     0     0  WL                          [irq18: aac0]
     33     0     0     0  RL      CPU 0               [irq21: bge0]
     32     0     0     0  SL      usbevt   0xc5210210 [usb2]
     31     0     0     0  WL                          [irq23: ehci0]
     30     0     0     0  SL      usbevt   0xc5245210 [usb1]
     29     0     0     0  WL                          [irq19: uhci1 ahd0]
     28     0     0     0  SL      usbtsk   0xc0bae2f4 [usbtask-dr]
     27     0     0     0  SL      usbtsk   0xc0bae2e0 [usbtask-hc]
     26     0     0     0  SL      usbevt   0xc524b210 [usb0]
     25     0     0     0  WL                          [irq16: uhci0]
     24     0     0     0  WL                          [irq9: acpi0]
     23     0     0     0  SL      -        0xc5212600 [kqueue taskq]
     22     0     0     0  WL                          [swi6: task queue]
     21     0     0     0  WL                          [swi6: Giant taskq]
      9     0     0     0  SL      -        0xc5212880 [thread taskq]
      8     0     0     0  SL      -        0xc5212900 [acpi_task_2]
      7     0     0     0  SL      -        0xc5212900 [acpi_task_1]
      6     0     0     0  SL      -        0xc5212900 [acpi_task_0]
     20     0     0     0  WL                          [swi5: +]
     19     0     0     0  WL                          [swi2: cambio]
      5     0     0     0  SL      ccb_scan 0xc0b7f514 [xpt_thrd]
     18     0     0     0  SL      -        0xc0bb0654 [yarrow]
      4     0     0     0  SL      -        0xc0bae6cc [g_down]
      3     0     0     0  SL      -        0xc0bae6c8 [g_up]
      2     0     0     0  SL      -        0xc0bae6c0 [g_event]
     17     0     0     0  WL                          [swi3: vm]
     16     0     0     0  RL      CPU 3               [swi4: clock sio]
     15     0     0     0  WL                          [swi1: net]
     14     0     0     0  RL                          [idle: cpu0]
     13     0     0     0  RL                          [idle: cpu1]
     12     0     0     0  RL      CPU 2               [idle: cpu2]
     11     0     0     0  RL                          [idle: cpu3]
      1     0     1     0  SLs     wait     0xc5114ab0 [init]
     10     0     0     0  SL      audit_wo 0xc0c0d5b4 [audit]
      0     0     0     0  WLs                         [swapper]
 db> show allpcpu
 Current CPU: 0
 
 cpuid        = 0
 curthread    = 0xc5214cc0: pid 33 "irq21: bge0"
 curpcb       = 0xe5797d90
 fpcurthread  = none
 idlethread   = 0xc5117000: pid 14 "idle: cpu0"
 APIC ID      = 0
 currentldt   = 0x50
 spin locks held:
 exclusive spin mutex sio r = 0 (0xc0c1b0e8) locked @ 
 /usr/src/sys/dev/sio/sio.c:1391
 
 cpuid        = 1
 curthread    = 0xc5861440: pid 943 "sshd"
 curpcb       = 0xebe1fd90
 fpcurthread  = none
 idlethread   = 0xc5115cc0: pid 13 "idle: cpu1"
 APIC ID      = 1
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 2
 curthread    = 0xc5115aa0: pid 12 "idle: cpu2"
 curpcb       = 0xe3d32d90
 fpcurthread  = none
 idlethread   = 0xc5115aa0: pid 12 "idle: cpu2"
 APIC ID      = 6
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 3
 curthread    = 0xc5117cc0: pid 16 "swi4: clock sio"
 curpcb       = 0xe3d4ad90
 fpcurthread  = none
 idlethread   = 0xc5115880: pid 11 "idle: cpu3"
 APIC ID      = 7
 currentldt   = 0x50
 spin locks held:
 
 db> alltrace
 
 Tracing command dmesg pid 954 tid 100095 td 0xc56ee000
 sched_switch(c56ee000,0,1,180,3869c968,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c56ee000,0,c0aa0842,19b,c5355838,...) at sleepq_switch+0xf0
 sleepq_catch_signals(ebdafa70,c07423dd,c0bb8700,4,5a,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c5355838,0,c0a9d757,da,0,...) at 
 sleepq_timedwait_sig+0x17
 _sleep(c5355838,0,15a,c0aa420e,493e0,...) at _sleep+0x331
 ttysleep(c5355800,c5355838,15a,c0aa420e,493e0,...) at ttysleep+0x39
 ttwrite(c5355800,ebdafc60,0,ebdafbbc,c071e148,...) at ttwrite+0x5aa
 ptswrite(c5afb000,ebdafc60,0,174,41,...) at ptswrite+0x38
 giant_write(c5afb000,ebdafc60,0,0,c0b56b60,...) at giant_write+0x48
 devfs_write_f(c55a4438,ebdafc60,c5af9000,0,c56ee000,...) at 
 devfs_write_f+0x98
 dofilewrite(ebdafc60,ffffffff,ffffffff,0,c55a4438,...) at dofilewrite+0x95
 kern_writev(c56ee000,1,ebdafc60,28201041,0,...) at kern_writev+0x58
 write(c56ee000,ebdafcfc,c,c0a8d9fa,c0b49ae0,...) at write+0x4f
 syscall(ebdafd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (4, FreeBSD ELF32, write), eip = 0x2815b423, esp = 
 0xbfbfe40c, ebp = 0xbfbfe428 ---
 
 Tracing command csh pid 949 tid 100096 td 0xc56e9cc0
 sched_switch(c56e9cc0,0,1,180,371d3b46,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c56e9cc0,0,c0aa0842,19b,c56ef30c,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,68,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c56ef30c,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c56ef30c,c56ef33c,168,c0a5fd27,0,...) at _sleep+0x359
 kern_sigsuspend(c56e9cc0,0,0,0,0,...) at kern_sigsuspend+0xe4
 sigsuspend(c56e9cc0,ebdaccfc,4,c0aa2849,c0b4ba78,...) at sigsuspend+0x4d
 syscall(ebdacd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28166efb, esp = 
 0xbfbfe73c, ebp = 0xbfbfe7a8 ---
 
 Tracing command su pid 948 tid 100143 td 0xc5861660
 sched_switch(c5861660,0,1,180,d92ee264,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5861660,0,c0aa0842,19b,c5af0804,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,5c,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c5af0804,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c5af0804,c5af0894,15c,c0aa296f,0,...) at _sleep+0x359
 kern_wait(c5861660,3b5,ebe22c2c,2,0,...) at kern_wait+0xa66
 wait4(c5861660,ebe22cfc,10,ebe22d38,c0b49b28,...) at wait4+0x3b
 syscall(ebe22d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2810ad7b, esp = 
 0xbfbfe79c, ebp = 0xbfbfe7b8 ---
 
 Tracing command csh pid 944 tid 100059 td 0xc5265880
 sched_switch(c5265880,0,1,180,77538768,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5265880,0,c0aa0842,19b,c5540b10,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,68,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c5540b10,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c5540b10,c5540b40,168,c0a5fd27,0,...) at _sleep+0x359
 kern_sigsuspend(c5265880,0,0,0,0,...) at kern_sigsuspend+0xe4
 sigsuspend(c5265880,e580bcfc,4,c0aa2849,c0b4ba78,...) at sigsuspend+0x4d
 syscall(e580bd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28166efb, esp = 
 0xbfbfe80c, ebp = 0xbfbfe878 ---
 
 Tracing command sshd pid 943 tid 100144 td 0xc5861440
 cpustop_handler(2,ebe1f890,c0a04a50,38,c0aa17fa,...) at cpustop_handler+0x32
 ipi_nmi_handler(38,c0aa17fa,6ce,c0c01c00,13,...) at ipi_nmi_handler+0x2f
 trap(ebe1f89c) at trap+0x30
 calltrap() at calltrap+0x6
 --- trap 0x13, eip = 0xc07429c2, esp = 0xebe1f8dc, ebp = 0xebe1f8f8 ---
 _mtx_lock_sleep(c5722c18,c5861440,0,c0aaaa4b,c4,...) at _mtx_lock_sleep+0xa2
 _mtx_lock_flags(c5722c18,0,c0aaaa4b,c4,c0786d22,...) at _mtx_lock_flags+0xef
 rtalloc1(ebe1fa04,1,0,100,ebe1fa00,...) at rtalloc1+0x225
 rtalloc_ign(ebe1fa00,0,c0aaf933,7f,c5861440,...) at rtalloc_ign+0xe6
 ip_output(c5893500,0,ebe1fa00,0,0,...) at ip_output+0x355
 tcp_output(c588a570,c5bb3c00,c0ab3c29,30b,c58887a4,...) at tcp_output+0x13f0
 tcp_usr_send(c5903318,0,c5bb3c00,0,0,...) at tcp_usr_send+0x25a
 sosend_generic(c5903318,0,ebe1fc60,0,0,...) at sosend_generic+0x50d
 sosend(c5903318,0,ebe1fc60,0,0,...) at sosend+0x54
 soo_write(c5853240,ebe1fc60,c5976800,0,c5861440,...) at soo_write+0x4b
 dofilewrite(ebe1fc60,ffffffff,ffffffff,0,c5853240,...) at dofilewrite+0x95
 kern_writev(c5861440,3,ebe1fc60,28533110,0,...) at kern_writev+0x58
 write(c5861440,ebe1fcfc,c,c0a5633b,c0b49ae0,...) at write+0x4f
 syscall(ebe1fd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (4, FreeBSD ELF32, write), eip = 0x283e3423, esp = 
 0xbfbfe5cc, ebp = 0xbfbfe5e8 ---
 
 Tracing command sshd pid 940 tid 100145 td 0xc5861220
 sched_switch(c5861220,0,1,180,94a3b5cc,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5861220,0,c0aa0842,19b,c58a8560,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,58,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c58a8560,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c58a8560,c58a8518,158,c0aa5515,0) at _sleep+0x359
 sbwait(c58a84f4,4,c0aa562a,5ae,c58a8518,...) at sbwait+0x76
 soreceive_generic(c58a84a4,0,ebe1cc60,0,0,...) at soreceive_generic+0x42f
 soreceive(c58a84a4,0,ebe1cc60,0,0,...) at soreceive+0x4d
 soo_read(c55e76c0,ebe1cc60,c5976300,0,c5861220,...) at soo_read+0x3b
 dofileread(ebe1cc60,ffffffff,ffffffff,0,c55e76c0,...) at dofileread+0x96
 kern_readv(c5861220,5,ebe1cc60,bfbfe64c,4,...) at kern_readv+0x58
 read(c5861220,ebe1ccfc,c,c0ab7dcb,c0b49ac8,...) at read+0x4f
 syscall(ebe1cd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x283e3443, esp = 
 0xbfbfe60c, ebp = 0xbfbfe628 ---
 
 Tracing command httpd pid 936 tid 100105 td 0xc5543880
 sched_switch(c5543880,0,1,180,21cc628c,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5543880,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,58,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359
 kern_accept(c5543880,3,e9b1ac70,e9b1ac6c,e9b1ac68,...) at kern_accept+0x12b
 accept(c5543880,e9b1acfc,c,e9b1ad38,c0b49d50,...) at accept+0x8f
 syscall(e9b1ad38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 
 0xbfbfeb7c, ebp = 0xbfbfeb98 ---
 
 Tracing command httpd pid 935 tid 100147 td 0xc56eecc0
 sched_switch(c56eecc0,0,1,180,216c709c,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c56eecc0,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,58,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359
 kern_accept(c56eecc0,3,ebdc1c70,ebdc1c6c,ebdc1c68,...) at kern_accept+0x12b
 accept(c56eecc0,ebdc1cfc,c,ebdc1d38,c0b49d50,...) at accept+0x8f
 syscall(ebdc1d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 
 0xbfbfeb7c, ebp = 0xbfbfeb98 ---
 
 Tracing command httpd pid 934 tid 100148 td 0xc5af5000
 sched_switch(c5af5000,0,1,180,207a8436,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5af5000,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,58,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359
 kern_accept(c5af5000,3,ebecec70,ebecec6c,ebecec68,...) at kern_accept+0x12b
 accept(c5af5000,ebececfc,c,ebeced38,c0b49d50,...) at accept+0x8f
 syscall(ebeced38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 
 0xbfbfeb7c, ebp = 0xbfbfeb98 ---
 
 Tracing command httpd pid 933 tid 100149 td 0xc5af4cc0
 sched_switch(c5af4cc0,0,1,180,2035600c,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5af4cc0,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,58,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359
 kern_accept(c5af4cc0,3,ebecbc70,ebecbc6c,ebecbc68,...) at kern_accept+0x12b
 accept(c5af4cc0,ebecbcfc,c,ebecbd38,c0b49d50,...) at accept+0x8f
 syscall(ebecbd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 
 0xbfbfeb7c, ebp = 0xbfbfeb98 ---
 
 Tracing command httpd pid 932 tid 100150 td 0xc5af4aa0
 sched_switch(c5af4aa0,0,1,180,1f5dfefc,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5af4aa0,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,58,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359
 kern_accept(c5af4aa0,3,ebec8c70,ebec8c6c,ebec8c68,...) at kern_accept+0x12b
 accept(c5af4aa0,ebec8cfc,c,ebec8d38,c0b49d50,...) at accept+0x8f
 syscall(ebec8d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 
 0xbfbfeb7c, ebp = 0xbfbfeb98 ---
 
 Tracing command getty pid 931 tid 100064 td 0xc5543aa0
 sched_switch(c5543aa0,0,1,180,aa94c4ec,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5543aa0,0,c0aa0842,19b,c5357c10,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,59,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c5357c10,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c5357c10,0,159,c0aa41dd,0,...) at _sleep+0x359
 ttysleep(c5357c00,c5357c10,159,c0aa41dd,0,...) at ttysleep+0x39
 ttread(c5357c00,e9b1dc60,0,e9b1dbb8,c071e1c8,...) at ttread+0x4e7
 ttyread(c5372100,e9b1dc60,0,168,0,...) at ttyread+0x39
 giant_read(c5372100,e9b1dc60,0,0,1,...) at giant_read+0x48
 devfs_read_f(c56563f0,e9b1dc60,c50f4300,0,c5543aa0,...) at devfs_read_f+0x71
 dofileread(e9b1dc60,ffffffff,ffffffff,0,c56563f0,...) at dofileread+0x96
 kern_readv(c5543aa0,0,e9b1dc60,bfbfee4b,1,...) at kern_readv+0x58
 read(c5543aa0,e9b1dcfc,c,c0acf182,c0b49ac8,...) at read+0x4f
 syscall(e9b1dd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 
 0xbfbfee2c, ebp = 0xbfbfee58 ---
 
 Tracing command getty pid 930 tid 100065 td 0xc559d000
 sched_switch(c559d000,0,1,180,ab249a30,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559d000,0,c0aa0842,19b,c538a810,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c538a810,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c538a810,0,159,c0aa41dd,0,...) at _sleep+0x359
 ttysleep(c538a800,c538a810,159,c0aa41dd,0,...) at ttysleep+0x39
 ttread(c538a800,ebd39c60,0,ebd39b9c,c067a61d,...) at ttread+0x4e7
 ttyread(c538e100,ebd39c60,0,ebd39bb8,c071e1c8,...) at ttyread+0x39
 scread(c538e100,ebd39c60,0,168,0,...) at scread+0x2d
 giant_read(c538e100,ebd39c60,0,0,1,...) at giant_read+0x48
 devfs_read_f(c55a41b0,ebd39c60,c50f4300,0,c559d000,...) at devfs_read_f+0x71
 dofileread(ebd39c60,ffffffff,ffffffff,0,c55a41b0,...) at dofileread+0x96
 kern_readv(c559d000,0,ebd39c60,bfbfee4b,1,...) at kern_readv+0x58
 read(c559d000,ebd39cfc,c,c0acf182,c0b49ac8,...) at read+0x4f
 syscall(ebd39d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 
 0xbfbfee2c, ebp = 0xbfbfee58 ---
 
 Tracing command getty pid 929 tid 100151 td 0xc5af4880
 sched_switch(c5af4880,0,1,180,aa9a3df4,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5af4880,0,c0aa0842,19b,c538ac10,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c538ac10,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c538ac10,0,159,c0aa41dd,0,...) at _sleep+0x359
 ttysleep(c538ac00,c538ac10,159,c0aa41dd,0,...) at ttysleep+0x39
 ttread(c538ac00,ebec5c60,0,ebec5b9c,c067a61d,...) at ttread+0x4e7
 ttyread(c5373900,ebec5c60,0,ebec5bb8,c071e1c8,...) at ttyread+0x39
 scread(c5373900,ebec5c60,0,168,0,...) at scread+0x2d
 giant_read(c5373900,ebec5c60,0,0,1,...) at giant_read+0x48
 devfs_read_f(c55a31f8,ebec5c60,c50f4300,0,c5af4880,...) at devfs_read_f+0x71
 dofileread(ebec5c60,ffffffff,ffffffff,0,c55a31f8,...) at dofileread+0x96
 kern_readv(c5af4880,0,ebec5c60,bfbfee4b,1,...) at kern_readv+0x58
 read(c5af4880,ebec5cfc,c,c0acf182,c0b49ac8,...) at read+0x4f
 syscall(ebec5d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 
 0xbfbfee2c, ebp = 0xbfbfee58 ---
 
 Tracing command getty pid 928 tid 100077 td 0xc5599440
 sched_switch(c5599440,0,1,180,ab04c1b4,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5599440,0,c0aa0842,19b,c5389810,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,59,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c5389810,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c5389810,0,159,c0aa41dd,0,...) at _sleep+0x359
 ttysleep(c5389800,c5389810,159,c0aa41dd,0,...) at ttysleep+0x39
 ttread(c5389800,ebd15c60,0,ebd15b9c,c067a61d,...) at ttread+0x4e7
 ttyread(c5373a00,ebd15c60,0,ebd15bb8,c071e1c8,...) at ttyread+0x39
 scread(c5373a00,ebd15c60,0,168,0,...) at scread+0x2d
 giant_read(c5373a00,ebd15c60,0,0,1,...) at giant_read+0x48
 devfs_read_f(c55a4990,ebd15c60,c50f4300,0,c5599440,...) at devfs_read_f+0x71
 dofileread(ebd15c60,ffffffff,ffffffff,0,c55a4990,...) at dofileread+0x96
 kern_readv(c5599440,0,ebd15c60,bfbfee4b,1,...) at kern_readv+0x58
 read(c5599440,ebd15cfc,c,c0acf182,c0b49ac8,...) at read+0x4f
 syscall(ebd15d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 
 0xbfbfee2c, ebp = 0xbfbfee58 ---
 
 Tracing command getty pid 927 tid 100055 td 0xc5377220
 sched_switch(c5377220,0,1,180,aae49548,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5377220,0,c0aa0842,19b,c538b410,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,59,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c538b410,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c538b410,0,159,c0aa41dd,0,...) at _sleep+0x359
 ttysleep(c538b400,c538b410,159,c0aa41dd,0,...) at ttysleep+0x39
 ttread(c538b400,e9adac60,0,e9adab9c,c067a61d,...) at ttread+0x4e7
 ttyread(c5373b00,e9adac60,0,e9adabb8,c071e1c8,...) at ttyread+0x39
 scread(c5373b00,e9adac60,0,168,0,...) at scread+0x2d
 giant_read(c5373b00,e9adac60,0,0,1,...) at giant_read+0x48
 devfs_read_f(c55a3480,e9adac60,c50f4300,0,c5377220,...) at devfs_read_f+0x71
 dofileread(e9adac60,ffffffff,ffffffff,0,c55a3480,...) at dofileread+0x96
 kern_readv(c5377220,0,e9adac60,bfbfee4b,1,...) at kern_readv+0x58
 read(c5377220,e9adacfc,c,c0acf182,c0b49ac8,...) at read+0x4f
 syscall(e9adad38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 
 0xbfbfee2c, ebp = 0xbfbfee58 ---
 
 Tracing command getty pid 926 tid 100108 td 0xc5543220
 sched_switch(c5543220,0,1,180,aaf9cf5c,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5543220,0,c0aa0842,19b,c538c410,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c538c410,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c538c410,0,159,c0aa41dd,0,...) at _sleep+0x359
 ttysleep(c538c400,c538c410,159,c0aa41dd,0,...) at ttysleep+0x39
 ttread(c538c400,e9b11c60,0,e9b11b9c,c067a61d,...) at ttread+0x4e7
 ttyread(c537d700,e9b11c60,0,e9b11bb8,c071e1c8,...) at ttyread+0x39
 scread(c537d700,e9b11c60,0,168,0,...) at scread+0x2d
 giant_read(c537d700,e9b11c60,0,0,1,...) at giant_read+0x48
 devfs_read_f(c5617d38,e9b11c60,c50f4300,0,c5543220,...) at devfs_read_f+0x71
 dofileread(e9b11c60,ffffffff,ffffffff,0,c5617d38,...) at dofileread+0x96
 kern_readv(c5543220,0,e9b11c60,bfbfee4b,1,...) at kern_readv+0x58
 read(c5543220,e9b11cfc,c,c0acf182,c0b49ac8,...) at read+0x4f
 syscall(e9b11d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 
 0xbfbfee2c, ebp = 0xbfbfee58 ---
 
 Tracing command getty pid 925 tid 100120 td 0xc5863880
 sched_switch(c5863880,0,1,180,ab30623c,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5863880,0,c0aa0842,19b,c538c010,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,59,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c538c010,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c538c010,0,159,c0aa41dd,0,...) at _sleep+0x359
 ttysleep(c538c000,c538c010,159,c0aa41dd,0,...) at ttysleep+0x39
 ttread(c538c000,ebe3ac60,0,ebe3ab9c,c067a61d,...) at ttread+0x4e7
 ttyread(c537d600,ebe3ac60,0,ebe3abb8,c071e1c8,...) at ttyread+0x39
 scread(c537d600,ebe3ac60,0,168,0,...) at scread+0x2d
 giant_read(c537d600,ebe3ac60,0,0,1,...) at giant_read+0x48
 devfs_read_f(c5853c60,ebe3ac60,c50f4300,0,c5863880,...) at devfs_read_f+0x71
 dofileread(ebe3ac60,ffffffff,ffffffff,0,c5853c60,...) at dofileread+0x96
 kern_readv(c5863880,0,ebe3ac60,bfbfee4b,1,...) at kern_readv+0x58
 read(c5863880,ebe3acfc,c,c0acf182,c0b49ac8,...) at read+0x4f
 syscall(ebe3ad38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 
 0xbfbfee2c, ebp = 0xbfbfee58 ---
 
 Tracing command getty pid 924 tid 100054 td 0xc5377440
 sched_switch(c5377440,0,1,180,ab0373e0,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5377440,0,c0aa0842,19b,c537fc10,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c537fc10,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c537fc10,0,159,c0aa41dd,0,...) at _sleep+0x359
 ttysleep(c537fc00,c537fc10,159,c0aa41dd,0,...) at ttysleep+0x39
 ttread(c537fc00,e9addc60,0,e9addb9c,c067a61d,...) at ttread+0x4e7
 ttyread(c537d500,e9addc60,0,e9addbb8,c071e1c8,...) at ttyread+0x39
 scread(c537d500,e9addc60,0,168,0,...) at scread+0x2d
 giant_read(c537d500,e9addc60,0,0,1,...) at giant_read+0x48
 devfs_read_f(c55e7cf0,e9addc60,c50f4300,0,c5377440,...) at devfs_read_f+0x71
 dofileread(e9addc60,ffffffff,ffffffff,0,c55e7cf0,...) at dofileread+0x96
 kern_readv(c5377440,0,e9addc60,bfbfee4b,1,...) at kern_readv+0x58
 read(c5377440,e9addcfc,c,c0acf182,c0b49ac8,...) at read+0x4f
 syscall(e9addd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 
 0xbfbfee2c, ebp = 0xbfbfee58 ---
 
 Tracing command getty pid 923 tid 100089 td 0xc55c4000
 sched_switch(c55c4000,0,1,180,ab35ec44,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c55c4000,0,c0aa0842,19b,c5380410,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c5380410,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c5380410,0,159,c0aa41dd,0,...) at _sleep+0x359
 ttysleep(c5380400,c5380410,159,c0aa41dd,0,...) at ttysleep+0x39
 ttread(c5380400,ebd5fc60,0,ebd5fb9c,c067a61d,...) at ttread+0x4e7
 ttyread(c537d400,ebd5fc60,0,ebd5fbb8,c071e1c8,...) at ttyread+0x39
 scread(c537d400,ebd5fc60,0,168,0,...) at scread+0x2d
 giant_read(c537d400,ebd5fc60,0,0,1,...) at giant_read+0x48
 devfs_read_f(c55a3678,ebd5fc60,c50f4300,0,c55c4000,...) at devfs_read_f+0x71
 dofileread(ebd5fc60,ffffffff,ffffffff,0,c55a3678,...) at dofileread+0x96
 kern_readv(c55c4000,0,ebd5fc60,bfbfee4b,1,...) at kern_readv+0x58
 read(c55c4000,ebd5fcfc,c,c0acf182,c0b49ac8,...) at read+0x4f
 syscall(ebd5fd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 
 0xbfbfee2c, ebp = 0xbfbfee58 ---
 
 Tracing command sleep pid 921 tid 100100 td 0xc56e9440
 sched_switch(c56e9440,0,1,180,87825700,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c56e9440,0,c0aa0842,19b,c0bb0824,...) at sleepq_switch+0xf0
 sleepq_catch_signals(ebda0bc4,c07423dd,c0bb8680,4,5c,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c0bb0824,0,c0a9d757,da,0,...) at 
 sleepq_timedwait_sig+0x17
 _sleep(c0bb0824,0,15c,c0a9df4b,ea61,...) at _sleep+0x331
 kern_nanosleep(c56e9440,ebda0c64,ebda0c6c,3c,0,...) at kern_nanosleep+0xc1
 nanosleep(c56e9440,ebda0cfc,8,ebda0d38,c0b4b100,...) at nanosleep+0x6f
 syscall(ebda0d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28135f2f, esp = 
 0xbfbfee5c, ebp = 0xbfbfee98 ---
 
 Tracing command logger pid 920 tid 100066 td 0xc559ccc0
 sched_switch(c559ccc0,0,1,180,8553358a,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559ccc0,0,c0aa0842,19b,c586d948,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,4c,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c586d948,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c586d948,c586dab8,14c,c0aa2569,0,...) at _sleep+0x359
 pipe_read(c58a2dc8,ebd36c60,c50f4300,0,c559ccc0,...) at pipe_read+0x3f6
 dofileread(ebd36c60,ffffffff,ffffffff,0,c58a2dc8,...) at dofileread+0x96
 kern_readv(c559ccc0,0,ebd36c60,28206000,1000,...) at kern_readv+0x58
 read(c559ccc0,ebd36cfc,c,c0aa289e,c0b49ac8,...) at read+0x4f
 syscall(ebd36d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28154443, esp = 
 0xbfbfe99c, ebp = 0xbfbfe9b8 ---
 
 Tracing command sh pid 919 tid 100076 td 0xc5599660
 sched_switch(c5599660,0,1,180,848ec352,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5599660,0,c0aa0842,19b,c559f2ac,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,5c,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c559f2ac,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c559f2ac,c559f33c,15c,c0aa296f,0,...) at _sleep+0x359
 kern_wait(c5599660,ffffffff,ebd18c2c,2,0,...) at kern_wait+0xa66
 wait4(c5599660,ebd18cfc,10,c0aa294e,c0b49b28,...) at wait4+0x3b
 syscall(ebd18d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149d7b, esp = 
 0xbfbfd8fc, ebp = 0xbfbfd918 ---
 
 Tracing command sh pid 918 tid 100067 td 0xc559caa0
 sched_switch(c559caa0,0,1,180,842dbd5e,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559caa0,0,c0aa0842,19b,c5598558,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,5c,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c5598558,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c5598558,c55985e8,15c,c0aa296f,0,...) at _sleep+0x359
 kern_wait(c559caa0,ffffffff,ebd33c2c,2,0,...) at kern_wait+0xa66
 wait4(c559caa0,ebd33cfc,10,c0aa294e,c0b49b28,...) at wait4+0x3b
 syscall(ebd33d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149d7b, esp = 
 0xbfbfda9c, ebp = 0xbfbfdab8 ---
 
 Tracing command inetd pid 902 tid 100081 td 0xc55c7220
 sched_switch(c55c7220,0,1,180,7daee5f4,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c55c7220,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c55c7220,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at 
 sleepq_wait_sig+0x14
 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,ebd77b18,...) at 
 _cv_wait_sig+0x1fc
 kern_select(c55c7220,6,bfbfec8c,0,0,0,c55a44c8,bfbfee48) at 
 kern_select+0x6cc
 select(c55c7220,ebd77cfc,14,c0aa29c2,c0b4a338,...) at select+0x5e
 syscall(ebd77d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x281773c3, esp = 
 0xbfbfe08c, ebp = 0xbfbfee78 ---
 
 Tracing command moused pid 875 tid 100068 td 0xc559c880
 sched_switch(c559c880,0,1,180,609d1204,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559c880,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c559c880,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at 
 sleepq_wait_sig+0x14
 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,c065602f,...) at 
 _cv_wait_sig+0x1fc
 kern_select(c559c880,400,bfbfeaac,0,0,0,4773a417,0) at kern_select+0x6cc
 select(c559c880,ebd30cfc,14,c0acf182,c0b4a338,...) at select+0x5e
 syscall(ebd30d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2817c3c3, esp = 
 0xbfbfea2c, ebp = 0xbfbfebc8 ---
 
 Tracing command cron pid 859 tid 100118 td 0xc5863cc0
 sched_switch(c5863cc0,0,1,180,b0b05d04,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5863cc0,0,c0aa0842,19b,c0bb0824,...) at sleepq_switch+0xf0
 sleepq_catch_signals(ebe40bc4,c07423dd,c0bb8700,4,5c,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c0bb0824,0,c0a9d757,da,0,...) at 
 sleepq_timedwait_sig+0x17
 _sleep(c0bb0824,0,15c,c0a9df4b,ea61,...) at _sleep+0x331
 kern_nanosleep(c5863cc0,ebe40c64,ebe40c6c,3c,0,...) at kern_nanosleep+0xc1
 nanosleep(c5863cc0,ebe40cfc,8,c0aa2ab3,c0b4b100,...) at nanosleep+0x6f
 syscall(ebe40d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28150f2f, esp = 
 0xbfbfed0c, ebp = 0xbfbfed38 ---
 
 Tracing command sshd pid 852 tid 100082 td 0xc55c7000
 sched_switch(c55c7000,0,1,180,86031ef8,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c55c7000,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c55c7000,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at 
 sleepq_wait_sig+0x14
 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,ebd74afc,...) at 
 _cv_wait_sig+0x1fc
 kern_select(c55c7000,7,2850b0b8,0,0,0,c,ebd74c98) at kern_select+0x6cc
 select(c55c7000,ebd74cfc,14,ebd74d38,c0b4a338,...) at select+0x5e
 syscall(ebd74d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x283e33c3, esp = 
 0xbfbfe6cc, ebp = 0xbfbfee98 ---
 
 Tracing command httpd pid 835 tid 100109 td 0xc5543000
 sched_switch(c5543000,0,1,180,e4d583d8,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5543000,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(e9b0ea84,c074278a,c0c01dc4,0,c5543000,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c0c01ddc,3e9,c0aba6ad,101,0,...) at 
 sleepq_timedwait_sig+0x17
 _cv_timedwait_sig(c0c01ddc,c0c01dc4,3e9,30f,e9b0eafc,...) at 
 _cv_timedwait_sig+0x20b
 kern_select(c5543000,0,0,0,0,e9b0ec70,1,0) at kern_select+0x6b4
 select(c5543000,e9b0ecfc,14,c0aa278a,c0b4a338,...) at select+0x5e
 syscall(e9b0ed38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x283333c3, esp = 
 0xbfbfec1c, ebp = 0xbfbfec48 ---
 
 Tracing command mysqld pid 833 tid 100139 td 0xc591eaa0
 sched_switch(c591eaa0,0,1,180,9e87e438,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c591eaa0,0,c0aa0842,19b,c561f2c0,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c561f2c0,c0bb14d0,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c561f2c0,c0bb14d0,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c591eaa0,ebe89cfc,ebe89d2c,c0a04743,c591eaa0,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c591eaa0,ebe89cfc,14,ebe89d38,c0b4c510,...) at _umtx_op+0x27
 syscall(ebe89d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 
 0xbf1c5edc, ebp = 0xbf1c5ef8 ---
 
 Tracing command mysqld pid 833 tid 100138 td 0xc591ecc0
 sched_switch(c591ecc0,0,1,180,6dc2a8fc,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c591ecc0,0,c0aa0842,19b,ebe8cbe0,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,68,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(ebe8cbe0,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(ebe8cbe0,c55c833c,168,c0a99fde,0,...) at _sleep+0x359
 kern_sigtimedwait(c591ecc0,26005,0,0,0,...) at kern_sigtimedwait+0x2af
 sigwait(c591ecc0,ebe8ccfc,8,c0aa2849,c0b4c2b8,...) at sigwait+0x77
 syscall(ebe8cd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (429, FreeBSD ELF32, sigwait), eip = 0x2864e6bb, esp = 
 0xbf1f6f1c, ebp = 0xbf1f6f48 ---
 
 Tracing command mysqld pid 833 tid 100135 td 0xc591f000
 sched_switch(c591f000,0,1,180,1453f504,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c591f000,0,c0aa0842,19b,c561f240,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c561f240,c0bb1440,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c561f240,c0bb1440,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c591f000,ebe8fcfc,ebe8fd2c,c0a04743,c591f000,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c591f000,ebe8fcfc,14,ebe8fd38,c0b4c510,...) at _umtx_op+0x27
 syscall(ebe8fd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 
 0xbf2f7eac, ebp = 0xbf2f7ec8 ---
 
 Tracing command mysqld pid 833 tid 100137 td 0xc591f220
 sched_switch(c591f220,0,1,180,801e49e8,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c591f220,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(ebe92a84,c074278a,c0c01dc4,0,c591f220,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c0c01ddc,7d1,c0aba6ad,101,0,...) at 
 sleepq_timedwait_sig+0x17
 _cv_timedwait_sig(c0c01ddc,c0c01dc4,7d1,30f,0,...) at 
 _cv_timedwait_sig+0x20b
 kern_select(c591f220,0,0,0,0,ebe92c70,2,0) at kern_select+0x6b4
 select(c591f220,ebe92cfc,14,16,c0b4a338,...) at select+0x5e
 syscall(ebe92d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x286e03c3, esp = 
 0xbf3f8f0c, ebp = 0xbf3f8f38 ---
 
 Tracing command mysqld pid 833 tid 100136 td 0xc591f440
 sched_switch(c591f440,0,1,180,4272d1ae,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c591f440,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(ebe95a84,c074278a,c0c01dc4,0,c591f440,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c0c01ddc,3e9,c0aba6ad,101,0,...) at 
 sleepq_timedwait_sig+0x17
 _cv_timedwait_sig(c0c01ddc,c0c01dc4,3e9,30f,0,...) at 
 _cv_timedwait_sig+0x20b
 kern_select(c591f440,0,0,0,0,ebe95c70,1,0) at kern_select+0x6b4
 select(c591f440,ebe95cfc,14,16,c0b4a338,...) at select+0x5e
 syscall(ebe95d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x286e03c3, esp = 
 0xbf4f9f0c, ebp = 0xbf4f9f38 ---
 
 Tracing command mysqld pid 833 tid 100134 td 0xc591f880
 sched_switch(c591f880,0,1,180,992f4b8,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c591f880,0,c0aa0842,19b,c561f140,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c561f140,c0bb19b0,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c561f140,c0bb19b0,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c591f880,ebe9bcfc,ebe9bd2c,c0a04743,c591f880,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c591f880,ebe9bcfc,14,c07871f8,c0b4c510,...) at _umtx_op+0x27
 syscall(ebe9bd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 
 0xbf6fbcfc, ebp = 0xbf6fbd18 ---
 
 Tracing command mysqld pid 833 tid 100133 td 0xc591faa0
 sched_switch(c591faa0,0,1,180,46f7d224,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c591faa0,0,c0aa0842,19b,c561f100,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c561f100,c0bb1710,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c561f100,c0bb1710,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c591faa0,ebe9ecfc,ebe9ed2c,c0a04743,c591faa0,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c591faa0,ebe9ecfc,14,ebe9ed38,c0b4c510,...) at _umtx_op+0x27
 syscall(ebe9ed38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 
 0xbf7fccfc, ebp = 0xbf7fcd18 ---
 
 Tracing command mysqld pid 833 tid 100132 td 0xc55c7660
 sched_switch(c55c7660,0,1,180,40ae3094,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c55c7660,0,c0aa0842,19b,c55ae740,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c55ae740,c0bb1470,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c55ae740,c0bb1470,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c55c7660,ebd7dcfc,ebd7dd2c,c0a04743,c55c7660,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c55c7660,ebd7dcfc,14,c0aa363a,c0b4c510,...) at _umtx_op+0x27
 syscall(ebd7dd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 
 0xbf8fdcfc, ebp = 0xbf8fdd18 ---
 
 Tracing command mysqld pid 833 tid 100131 td 0xc55c7880
 sched_switch(c55c7880,0,1,180,9830e02,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c55c7880,0,c0aa0842,19b,c55ae700,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c55ae700,c0bb1200,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c55ae700,c0bb1200,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c55c7880,ebd80cfc,ebd80d2c,c0a04743,c55c7880,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c55c7880,ebd80cfc,14,c0aa363a,c0b4c510,...) at _umtx_op+0x27
 syscall(ebd80d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 
 0xbf9fecfc, ebp = 0xbf9fed18 ---
 
 Tracing command mysqld pid 833 tid 100086 td 0xc55c4660
 sched_switch(c55c4660,0,1,180,9e866198,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c55c4660,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c55c4660,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at 
 sleepq_wait_sig+0x14
 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,246,...) at _cv_wait_sig+0x1fc
 kern_select(c55c4660,d,bfbfead8,0,0,0,0,2860772c) at kern_select+0x6cc
 select(c55c4660,ebd68cfc,14,c0aa29c2,c0b4a338,...) at select+0x5e
 syscall(ebd68d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x286e03c3, esp = 
 0xbfbfe6ac, ebp = 0xbfbfe6d8 ---
 
 Tracing command sh pid 808 tid 100114 td 0xc5542440
 sched_switch(c5542440,0,1,180,88d8c3f6,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5542440,0,c0aa0842,19b,c585fab0,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,5c,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c585fab0,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c585fab0,c585fb40,15c,c0aa296f,0,...) at _sleep+0x359
 kern_wait(c5542440,ffffffff,e9affc2c,2,0,...) at kern_wait+0xa66
 wait4(c5542440,e9affcfc,10,c0aa294e,c0b49b28,...) at wait4+0x3b
 syscall(e9affd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149d7b, esp = 
 0xbfbfe7fc, ebp = 0xbfbfe818 ---
 
 Tracing command qmgr pid 803 tid 100056 td 0xc5377000
 sched_switch(c5377000,0,1,180,f72875dc,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5377000,0,c0aa0842,19b,c587da00,...) at sleepq_switch+0xf0
 sleepq_catch_signals(e9ad7aac,c07423dd,c0bb8700,4,58,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c587da00,0,c0a9d757,da,0,...) at 
 sleepq_timedwait_sig+0x17
 _sleep(c587da00,c587da00,158,c0a99450,493e1,...) at _sleep+0x331
 kern_kevent(c5377000,8,0,64,e9ad7c60,...) at kern_kevent+0x3c3
 kevent(c5377000,e9ad7cfc,18,c0aa2ab3,c0b4bc88,...) at kevent+0x19b
 syscall(e9ad7d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2839c14b, esp = 
 0xbfbfe16c, ebp = 0xbfbfe978 ---
 
 Tracing command pickup pid 802 tid 100074 td 0xc5599aa0
 sched_switch(c5599aa0,0,1,180,efe64654,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5599aa0,0,c0aa0842,19b,c55aab80,...) at sleepq_switch+0xf0
 sleepq_catch_signals(ebd1eaac,c07423dd,c0bb8700,4,58,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c55aab80,0,c0a9d757,da,0,...) at 
 sleepq_timedwait_sig+0x17
 _sleep(c55aab80,c55aab80,158,c0a99450,186a1,...) at _sleep+0x331
 kern_kevent(c5599aa0,8,0,64,ebd1ec60,...) at kern_kevent+0x3c3
 kevent(c5599aa0,ebd1ecfc,18,c0aa2ab3,c0b4bc88,...) at kevent+0x19b
 syscall(ebd1ed38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2839014b, esp = 
 0xbfbfe19c, ebp = 0xbfbfe9a8 ---
 
 Tracing command master pid 795 tid 100079 td 0xc5599000
 sched_switch(c5599000,0,1,180,f7267f80,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5599000,0,c0aa0842,19b,c5564080,...) at sleepq_switch+0xf0
 sleepq_catch_signals(ebd0faac,c07423dd,c0bb8680,4,58,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c5564080,0,c0a9d757,da,0,...) at 
 sleepq_timedwait_sig+0x17
 _sleep(c5564080,c5564080,158,c0a99450,ea61,...) at _sleep+0x331
 kern_kevent(c5599000,c,0,64,ebd0fc60,...) at kern_kevent+0x3c3
 kevent(c5599000,ebd0fcfc,18,c0aa2ab3,c0b4bc88,...) at kevent+0x19b
 syscall(ebd0fd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2838314b, esp = 
 0xbfbfe3ec, ebp = 0xbfbfebf8 ---
 
 Tracing command ntpd pid 728 tid 100112 td 0xc5542880
 sched_switch(c5542880,0,1,180,3ff48692,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5542880,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c5542880,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at 
 sleepq_wait_sig+0x14
 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,0,...) at _cv_wait_sig+0x1fc
 kern_select(c5542880,a,bfbfed7c,0,0,0,246,0) at kern_select+0x6cc
 select(c5542880,e9b05cfc,14,16,c0b4a338,...) at select+0x5e
 syscall(e9b05d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2830d3c3, esp = 
 0xbfbfed5c, ebp = 0xbfbfee18 ---
 
 Tracing command named pid 643 tid 100130 td 0xc559d220
 sched_switch(c559d220,0,1,180,149d8050,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559d220,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c559d220,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at 
 sleepq_wait_sig+0x14
 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,0,...) at _cv_wait_sig+0x1fc
 kern_select(c559d220,1a,bf4f9f10,bf4f9e90,0,0,8,2836572c) at 
 kern_select+0x6cc
 select(c559d220,ebd3ccfc,14,c0a5633b,c0b4a338,...) at select+0x5e
 syscall(ebd3cd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2843e3c3, esp = 
 0xbf4f9cfc, ebp = 0xbf4f9d28 ---
 
 Tracing command named pid 643 tid 100129 td 0xc559d440
 sched_switch(c559d440,0,1,180,532312a8,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559d440,0,c0aa0842,19b,c555d240,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c077f359,c559d5d4,7530,c077e7d0,0,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c555d240,7530,c0a9e334,100,0,...) at 
 sleepq_timedwait_sig+0x17
 _sleep(c555d240,c0bb0900,100,c0a9e334,7530,...) at _sleep+0x331
 __umtx_op_cv_wait(c559d440,ebd3fcfc,ebd3fd2c,c0a04743,c559d440,...) at 
 __umtx_op_cv_wait+0x430
 _umtx_op(c559d440,ebd3fcfc,14,c0aa2df1,c0b4c510,...) at _umtx_op+0x27
 syscall(ebd3fd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 
 0xbf5fae0c, ebp = 0xbf5fae28 ---
 
 Tracing command named pid 643 tid 100128 td 0xc559d660
 sched_switch(c559d660,0,1,180,52c4f6a4,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559d660,0,c0aa0842,19b,c555d200,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c555d200,c0bb0ea0,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c555d200,c0bb0ea0,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c559d660,ebd42cfc,ebd42d2c,c0a04743,c559d660,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c559d660,ebd42cfc,14,c0aa2ab3,c0b4c510,...) at _umtx_op+0x27
 syscall(ebd42d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 
 0xbf6fbecc, ebp = 0xbf6fbee8 ---
 
 Tracing command named pid 643 tid 100127 td 0xc559d880
 sched_switch(c559d880,0,1,180,532425b8,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559d880,0,c0aa0842,19b,c555d1c0,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c555d1c0,c0bb0ea0,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c555d1c0,c0bb0ea0,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c559d880,ebd45cfc,ebd45d2c,c0a04743,c559d880,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c559d880,ebd45cfc,14,c0aa2ab3,c0b4c510,...) at _umtx_op+0x27
 syscall(ebd45d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 
 0xbf7fcecc, ebp = 0xbf7fcee8 ---
 
 Tracing command named pid 643 tid 100126 td 0xc559daa0
 sched_switch(c559daa0,0,1,180,14a409e8,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559daa0,0,c0aa0842,19b,c555d180,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c555d180,c0bb0ea0,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c555d180,c0bb0ea0,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c559daa0,ebd48cfc,ebd48d2c,c0a04743,c559daa0,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c559daa0,ebd48cfc,14,c0aa2ab3,c0b4c510,...) at _umtx_op+0x27
 syscall(ebd48d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 
 0xbf8fdecc, ebp = 0xbf8fdee8 ---
 
 Tracing command named pid 643 tid 100125 td 0xc559dcc0
 sched_switch(c559dcc0,0,1,180,1497db68,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c559dcc0,0,c0aa0842,19b,c555d140,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e
 sleepq_wait_sig(c555d140,c0bb0ea0,c0a9e334,100,0,...) at 
 sleepq_wait_sig+0x14
 _sleep(c555d140,c0bb0ea0,100,c0a9e334,0,...) at _sleep+0x359
 __umtx_op_cv_wait(c559dcc0,ebd4bcfc,ebd4bd2c,c0a04743,c559dcc0,...) at 
 __umtx_op_cv_wait+0x354
 _umtx_op(c559dcc0,ebd4bcfc,14,c0aa2ab3,c0b4c510,...) at _umtx_op+0x27
 syscall(ebd4bd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 
 0xbf9feecc, ebp = 0xbf9feee8 ---
 
 Tracing command named pid 643 tid 100107 td 0xc5543440
 sched_switch(c5543440,0,1,180,ff77ecb0,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5543440,0,c0aa0842,19b,e9b14be0,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,68,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(e9b14be0,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(e9b14be0,c585e5e8,168,c0a99fde,0,...) at _sleep+0x359
 kern_sigtimedwait(c5543440,4003,0,0,0,...) at kern_sigtimedwait+0x2af
 sigwait(c5543440,e9b14cfc,8,c0aa363a,c0b4c2b8,...) at sigwait+0x77
 syscall(e9b14d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (429, FreeBSD ELF32, sigwait), eip = 0x283ac6bb, esp = 
 0xbfbfecdc, ebp = 0xbfbfed08 ---
 
 Tracing command syslogd pid 578 tid 100116 td 0xc5542000
 sched_switch(c5542000,0,1,180,b40d7874,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5542000,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c5542000,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at 
 sleepq_wait_sig+0x14
 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,d9191108,...) at 
 _cv_wait_sig+0x1fc
 kern_select(c5542000,b,282200b8,0,0,0,c56f2000,28204000) at 
 kern_select+0x6cc
 select(c5542000,e9af9cfc,14,c0abbe3e,c0b4a338,...) at select+0x5e
 syscall(e9af9d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x281673c3, esp = 
 0xbfbfdefc, ebp = 0xbfbfee48 ---
 
 Tracing command devd pid 517 tid 100099 td 0xc56e9660
 sched_switch(c56e9660,0,1,180,ff905800,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c56e9660,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c56e9660,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at 
 sleepq_wait_sig+0x14
 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,ebda3afc,...) at 
 _cv_wait_sig+0x1fc
 kern_select(c56e9660,5,bfbfedf8,0,0,0,0,bfbfe99c) at kern_select+0x6cc
 select(c56e9660,ebda3cfc,14,c0aa2cb6,c0b4a338,...) at select+0x5e
 syscall(ebda3d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (93, FreeBSD ELF32, select), eip = 0x807f96f, esp = 
 0xbfbfe9bc, ebp = 0xbfbfee98 ---
 
 Tracing command dhclient pid 321 tid 100058 td 0xc5265aa0
 sched_switch(c5265aa0,0,1,180,386c9404,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5265aa0,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(e580ead8,c074278a,c0c01dc4,0,c5265aa0,...) at 
 sleepq_catch_signals+0x21e
 sleepq_timedwait_sig(c0c01ddc,362f19,c0aba6ad,101,0,...) at 
 sleepq_timedwait_sig+0x17
 _cv_timedwait_sig(c0c01ddc,c0c01dc4,362f19,3bb,e580eb48,...) at 
 _cv_timedwait_sig+0x20b
 poll(c5265aa0,e580ecfc,c,c0a5633b,c0b4ae18,...) at poll+0x554
 syscall(e580ed38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2810f8df, esp = 
 0xbfbfee1c, ebp = 0xbfbfee48 ---
 
 Tracing command dhclient pid 301 tid 100088 td 0xc55c4220
 sched_switch(c55c4220,0,1,180,83d8d1d0,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c55c4220,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c55c4220,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at 
 sleepq_wait_sig+0x14
 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,3bb,c0aa2465,...) at 
 _cv_wait_sig+0x1fc
 poll(c55c4220,ebd62cfc,c,c0a8d9fa,c0b4ae18,...) at poll+0x56a
 syscall(ebd62d38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2810f8df, esp = 
 0xbfbfee1c, ebp = 0xbfbfee48 ---
 
 Tracing command softdepflush pid 53 tid 100061 td 0xc5265440
 sched_switch(c5265440,0,1,180,dd9baf10,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,44,...) at mi_switch+0x217
 sleepq_switch(c5265440,0,c0aa0842,240,44,...) at sleepq_switch+0xf0
 sleepq_timedwait(c0c0db44,3e8,c0abc2bb,0,0,...) at sleepq_timedwait+0x68
 _sleep(c0c0db44,c0c0db1c,44,c0abc2bb,3e8,...) at _sleep+0x345
 softdep_flush(0,e5805d38,c0a99763,2ea,c55412ac,...) at softdep_flush+0x2d2
 fork_exit(c0935c40,0,e5805d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe5805d70, ebp = 0 ---
 
 Tracing command syncer pid 52 tid 100062 td 0xc5265220
 sched_switch(c5265220,0,1,180,7e75a8c6,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5265220,...) at mi_switch+0x217
 sleepq_switch(c5265220,0,c0aa0842,21d,68,...) at sleepq_switch+0xf0
 sleepq_wait(c0bb064c,0,c0aa8d55,0,0,...) at sleepq_wait+0x60
 _sleep(c0bb064c,c0c02524,68,c0aa8d55,0,...) at _sleep+0x369
 sched_sync(0,e5802d38,c0a99763,2ea,c5541558,...) at sched_sync+0x742
 fork_exit(c07cba20,0,e5802d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe5802d70, ebp = 0 ---
 
 Tracing command vnlru pid 51 tid 100040 td 0xc5264aa0
 sched_switch(c5264aa0,0,1,180,dd6b4bf0,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,50,...) at mi_switch+0x217
 sleepq_switch(c5264aa0,0,c0aa0842,240,50,...) at sleepq_switch+0xf0
 sleepq_timedwait(c5374000,3e8,c0aa8e62,0,0,...) at sleepq_timedwait+0x68
 _sleep(c5374000,c0c024f8,250,c0aa8e62,3e8,...) at _sleep+0x345
 vnlru_proc(0,e57f9d38,c0a99763,2ea,c5374000,...) at vnlru_proc+0x115
 fork_exit(c07cd020,0,e57f9d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57f9d70, ebp = 0 ---
 
 Tracing command bufdaemon pid 50 tid 100041 td 0xc5264880
 sched_switch(c5264880,0,1,180,dd6ac18a,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,44,...) at mi_switch+0x217
 sleepq_switch(c5264880,0,c0aa0842,240,44,...) at sleepq_switch+0xf0
 sleepq_timedwait(c0c02244,3e8,c0aa6a11,0,0,...) at sleepq_timedwait+0x68
 _sleep(c0c02244,c0c02248,44,c0aa6a11,3e8,...) at _sleep+0x345
 buf_daemon(0,e57f6d38,c0a99763,2ea,c53742ac,...) at buf_daemon+0x21e
 fork_exit(c07b7510,0,e57f6d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57f6d70, ebp = 0 ---
 
 Tracing command pagezero pid 49 tid 100042 td 0xc5264660
 sched_switch(c5264660,0,1,180,38314fb8,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5264660,0,c0aa0842,240,0,...) at sleepq_switch+0xf0
 sleepq_timedwait(c0c0e700,493e0,c0ac0e41,0,0,...) at sleepq_timedwait+0x68
 _sleep(c0c0e700,c0c0e2a4,0,c0ac0e41,493e0,...) at _sleep+0x345
 vm_pagezero(0,e57f3d38,c0a99763,2ea,c5374558,...) at vm_pagezero+0xdc
 fork_exit(c096f430,0,e57f3d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57f3d70, ebp = 0 ---
 
 Tracing command vmdaemon pid 48 tid 100043 td 0xc5264440
 sched_switch(c5264440,0,1,180,3821bd24,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5264440,...) at mi_switch+0x217
 sleepq_switch(c5264440,0,c0aa0842,21d,68,...) at sleepq_switch+0xf0
 sleepq_wait(c0c0e318,0,c0a9d757,da,0,...) at sleepq_wait+0x60
 _sleep(c0c0e318,c0c0e31c,68,c0aa6a11,0,...) at _sleep+0x369
 vm_daemon(0,e57f0d38,c0a99763,2ea,c5374804,...) at vm_daemon+0x59
 fork_exit(c096a870,0,e57f0d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57f0d70, ebp = 0 ---
 
 Tracing command pagedaemon pid 47 tid 100044 td 0xc5264220
 sched_switch(c5264220,0,1,180,1f4cbb44,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,44,...) at mi_switch+0x217
 sleepq_switch(c5264220,0,c0aa0842,240,44,...) at sleepq_switch+0xf0
 sleepq_timedwait(c0c0e2e0,1388,c0aa6a11,0,0,...) at sleepq_timedwait+0x68
 _sleep(c0c0e2e0,c0c0e2a4,44,c0aa6a11,1388,...) at _sleep+0x345
 vm_pageout(0,e57edd38,c0a99763,2ea,c5374ab0,...) at vm_pageout+0x2bb
 fork_exit(c096b420,0,e57edd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57edd70, ebp = 0 ---
 
 Tracing command sctp_iterator pid 46 tid 100045 td 0xc5264000
 sched_switch(c5264000,0,1,180,770887a0,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5264000,...) at mi_switch+0x217
 sleepq_switch(c5264000,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0
 sleepq_wait(c0c03fac,c0c03eb4,c0aafbe2,0,0,...) at sleepq_wait+0x60
 _sleep(c0c03fac,c0c03eb4,0,c0aafbe2,0,...) at _sleep+0x369
 sctp_iterator_thread(0,e9af6d38,c0a99763,2ea,c5375000,...) at 
 sctp_iterator_thread+0x60
 fork_exit(c0833a90,0,e9af6d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe9af6d70, ebp = 0 ---
 
 Tracing command irq7: ppc0 pid 45 tid 100046 td 0xc5222cc0
 fork_trampoline() at fork_trampoline
 
 Tracing command fdc0 pid 44 tid 100047 td 0xc5222aa0
 sched_switch(c5222aa0,0,1,180,c9a4620e,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217
 sleepq_switch(c5222aa0,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0
 sleepq_timedwait(c531ce3c,3e8,c0a94778,0,0,...) at sleepq_timedwait+0x68
 _sleep(c531ce3c,c531cef0,4c,c0a94778,3e8,...) at _sleep+0x345
 fdc_thread(c531ce00,e57bed38,c0a99763,2ea,c5375558,...) at fdc_thread+0x2b8
 fork_exit(c09c82c0,c531ce00,e57bed38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57bed70, ebp = 0 ---
 
 Tracing command swi0: sio pid 43 tid 100048 td 0xc5222880
 sched_switch(c5222880,0,1,180,da52d0f0,...) at sched_switch+0x49b
 mi_switch(1,0,c0a999ef,46b,c535cd64,...) at mi_switch+0x217
 ithread_loop(c536d440,e57bbd38,c0a99763,2ea,c5375804,...) at 
 ithread_loop+0x34c
 fork_exit(c07328a0,c536d440,e57bbd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57bbd70, ebp = 0 ---
 
 Tracing command irq12: psm0 pid 42 tid 100049 td 0xc5222660
 fork_trampoline() at fork_trampoline
 
 Tracing command irq1: atkbd0 pid 41 tid 100050 td 0xc5377cc0
 sched_switch(c5377cc0,0,1,180,c36742f4,...) at sched_switch+0x49b
 mi_switch(1,0,c0a999ef,46b,c5111464,...) at mi_switch+0x217
 ithread_loop(c536d820,e9ae9d38,c0a99763,2ea,c5376000,...) at 
 ithread_loop+0x34c
 fork_exit(c07328a0,c536d820,e9ae9d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe9ae9d70, ebp = 0 ---
 
 Tracing command irq15: ata1 pid 40 tid 100030 td 0xc5221cc0
 sched_switch(c5221cc0,0,1,180,3a187a8c,...) at sched_switch+0x49b
 mi_switch(1,0,c0a999ef,46b,c5110c64,...) at mi_switch+0x217
 ithread_loop(c53582e0,e57acd38,c0a99763,2ea,c5262000,...) at 
 ithread_loop+0x34c
 fork_exit(c07328a0,c53582e0,e57acd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57acd70, ebp = 0 ---
 
 Tracing command irq14: ata0 pid 39 tid 100031 td 0xc5221aa0
 fork_trampoline() at fork_trampoline
 
 Tracing command aic_recovery1 pid 38 tid 100032 td 0xc5221880
 sched_switch(c5221880,0,1,180,6ffb8158,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5221880,...) at mi_switch+0x217
 sleepq_switch(c5221880,0,c0aa0842,21d,a0,...) at sleepq_switch+0xf0
 sleepq_wait(c531e000,0,c0a9d757,da,0,...) at sleepq_wait+0x60
 _sleep(c531e000,c531b634,a0,c0a5fa6c,0,...) at _sleep+0x369
 ahd_recovery_thread(c531e000,e57a6d38,c0a99763,2ea,c5262558,...) at 
 ahd_recovery_thread+0x72
 fork_exit(c04c4080,c531e000,e57a6d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57a6d70, ebp = 0 ---
 
 Tracing command irq22: ahd1 pid 37 tid 100033 td 0xc5221660
 sched_switch(c5221660,0,1,180,34633be0,...) at sched_switch+0x49b
 mi_switch(1,0,c0a999ef,46b,c515e4e4,...) at mi_switch+0x217
 ithread_loop(c534e950,e57a3d38,c0a99763,2ea,c5262804,...) at 
 ithread_loop+0x34c
 fork_exit(c07328a0,c534e950,e57a3d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57a3d70, ebp = 0 ---
 
 Tracing command aic_recovery0 pid 36 tid 100034 td 0xc5221440
 sched_switch(c5221440,0,1,180,6ffb359c,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5221440,...) at mi_switch+0x217
 sleepq_switch(c5221440,0,c0aa0842,21d,a0,...) at sleepq_switch+0xf0
 sleepq_wait(c52d5000,0,c0a9d757,da,0,...) at sleepq_wait+0x60
 _sleep(c52d5000,c52696b4,a0,c0a5fa6c,0,...) at _sleep+0x369
 ahd_recovery_thread(c52d5000,e57a0d38,c0a99763,2ea,c5262ab0,...) at 
 ahd_recovery_thread+0x72
 fork_exit(c04c4080,c52d5000,e57a0d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57a0d70, ebp = 0 ---
 
 Tracing command aac0aif pid 35 tid 100035 td 0xc5221220
 sched_switch(c5221220,0,1,180,405b856,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217
 sleepq_switch(c5221220,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0
 sleepq_timedwait(c5263000,4e20,c0a5af1f,0,0,...) at sleepq_timedwait+0x68
 _sleep(c5263000,c5277124,4c,c0a5af1f,4e20,...) at _sleep+0x345
 aac_command_thread(c5277000,e579dd38,c0a99763,2ea,c5263000,...) at 
 aac_command_thread+0x8d
 fork_exit(c0496410,c5277000,e579dd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe579dd70, ebp = 0 ---
 
 Tracing command irq18: aac0 pid 34 tid 100036 td 0xc5221000
 sched_switch(c5221000,0,1,180,b40a4750,...) at sched_switch+0x49b
 mi_switch(1,0,c0a999ef,46b,c515e6e4,...) at mi_switch+0x217
 ithread_loop(c5276230,e579ad38,c0a99763,2ea,c52632ac,...) at 
 ithread_loop+0x34c
 fork_exit(c07328a0,c5276230,e579ad38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe579ad70, ebp = 0 ---
 
 Tracing command irq21: bge0 pid 33 tid 100037 td 0xc5214cc0
 kdb_enter(c0a8e6e4,c536f8f8,c0c1b0e8,8,c536f88c,...) at kdb_enter+0x32
 siointr1(c0c1b0e8,0,c0ac563d,56f,e57979dc,...) at siointr1+0x11c
 siointr(c536f800,c0bb0844,2,c5214cc0,c5111280,...) at siointr+0x32
 intr_execute_handlers(c5107cd0,e57979fc,e5797a58,c09eb844,3b,...) at 
 intr_execute_handlers+0xec
 lapic_handle_intr(3b,e57979fc) at lapic_handle_intr+0x3f
 Xapic_isr1() at Xapic_isr1+0x34
 --- interrupt, eip = 0xc07429c2, esp = 0xe5797a3c, ebp = 0xe5797a58 ---
 _mtx_lock_sleep(c538d07c,c5214cc0,0,c0aaaa4b,93,...) at _mtx_lock_sleep+0xa2
 _mtx_lock_flags(c538d07c,0,c0aaaa4b,93,c0a9bb4b,...) at _mtx_lock_flags+0xef
 rtalloc1(e5797bac,1,0,e5797b20,c07426fc,...) at rtalloc1+0x63
 rtredirect(e5797bbc,e5797bac,0,26,e5797b9c,...) at rtredirect+0x35e
 icmp_input(c57d1700,14,246,c0b9aca0,e5797c0c,...) at icmp_input+0x50f
 ip_input(c57d1700,14e,800,c5250400,800,...) at ip_input+0x650
 netisr_dispatch(2,c57d1700,10,3,0,...) at netisr_dispatch+0x73
 ether_demux(c5250400,c57d1700,3,0,3,...) at ether_demux+0x1f1
 ether_input(c5250400,c57d1700,c0a6b43a,bc4,c5250400,...) at 
 ether_input+0x37f
 bge_intr(c526d000,0,c0a999ef,471,c515e564,...) at bge_intr+0x7da
 ithread_loop(c5268640,e5797d38,c0a99763,2ea,c5263558,...) at 
 ithread_loop+0x1b5
 fork_exit(c07328a0,c5268640,e5797d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe5797d70, ebp = 0 ---
 
 Tracing command usb2 pid 32 tid 100038 td 0xc5265000
 sched_switch(c5265000,0,1,180,204964e,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,5c,...) at mi_switch+0x217
 sleepq_switch(c5265000,0,c0aa0842,240,5c,...) at sleepq_switch+0xf0
 sleepq_timedwait(c5210210,ea60,c0a916e6,0,0,...) at sleepq_timedwait+0x68
 _sleep(c5210210,0,5c,c0a916e6,ea60,...) at _sleep+0x345
 usb_event_thread(c523f800,e57ffd38,c0a99763,2ea,c5263804,...) at 
 usb_event_thread+0xca
 fork_exit(c06cc3b0,c523f800,e57ffd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57ffd70, ebp = 0 ---
 
 Tracing command irq23: ehci0 pid 31 tid 100039 td 0xc5264cc0
 fork_trampoline() at fork_trampoline
 
 Tracing command usb1 pid 30 tid 100021 td 0xc5214220
 sched_switch(c5214220,0,1,180,235540a,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,5c,...) at mi_switch+0x217
 sleepq_switch(c5214220,0,c0aa0842,240,5c,...) at sleepq_switch+0xf0
 sleepq_timedwait(c5245210,ea60,c0a916e6,0,0,...) at sleepq_timedwait+0x68
 _sleep(c5245210,0,5c,c0a916e6,ea60,...) at _sleep+0x345
 usb_event_thread(c5253800,e5788d38,c0a99763,2ea,c52152ac,...) at 
 usb_event_thread+0xca
 fork_exit(c06cc3b0,c5253800,e5788d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe5788d70, ebp = 0 ---
 
 Tracing command irq19: uhci1 ahd0 pid 29 tid 100022 td 0xc5214000
 sched_switch(c5214000,0,1,180,1d89dcfc,...) at sched_switch+0x49b
 mi_switch(1,0,c0a999ef,46b,c515e664,...) at mi_switch+0x217
 ithread_loop(c524f1b0,e5785d38,c0a99763,2ea,c5215558,...) at 
 ithread_loop+0x34c
 fork_exit(c07328a0,c524f1b0,e5785d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe5785d70, ebp = 0 ---
 
 Tracing command usbtask-dr pid 28 tid 100023 td 0xc5119cc0
 sched_switch(c5119cc0,0,1,180,6ff99adc,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5119cc0,...) at mi_switch+0x217
 sleepq_switch(c5119cc0,0,c0aa0842,21d,5c,...) at sleepq_switch+0xf0
 sleepq_wait(c0bae2f4,0,c0a9d757,da,0,...) at sleepq_wait+0x60
 _sleep(c0bae2f4,0,5c,c0a91723,0,...) at _sleep+0x369
 usb_task_thread(c0bae2f4,e3d5fd38,c0a99763,2ea,c5215804,...) at 
 usb_task_thread+0x62
 fork_exit(c06ccb30,c0bae2f4,e3d5fd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d5fd70, ebp = 0 ---
 
 Tracing command usbtask-hc pid 27 tid 100024 td 0xc5119aa0
 sched_switch(c5119aa0,0,1,180,6ff957ac,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5119aa0,...) at mi_switch+0x217
 sleepq_switch(c5119aa0,0,c0aa0842,21d,5c,...) at sleepq_switch+0xf0
 sleepq_wait(c0bae2e0,0,c0a9d757,da,0,...) at sleepq_wait+0x60
 _sleep(c0bae2e0,0,5c,c0a91723,0,...) at _sleep+0x369
 usb_task_thread(c0bae2e0,e3d5cd38,c0a99763,2ea,c5215ab0,...) at 
 usb_task_thread+0x62
 fork_exit(c06ccb30,c0bae2e0,e3d5cd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d5cd70, ebp = 0 ---
 
 Tracing command usb0 pid 26 tid 100025 td 0xc5119880
 sched_switch(c5119880,0,1,180,235937a,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,5c,...) at mi_switch+0x217
 sleepq_switch(c5119880,0,c0aa0842,240,5c,...) at sleepq_switch+0xf0
 sleepq_timedwait(c524b210,ea60,c0a916e6,0,0,...) at sleepq_timedwait+0x68
 _sleep(c524b210,0,5c,c0a916e6,ea60,...) at _sleep+0x345
 usb_event_thread(c5254280,e3d59d38,c0a99763,2ea,c5220000,...) at 
 usb_event_thread+0xca
 fork_exit(c06cc3b0,c5254280,e3d59d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d59d70, ebp = 0 ---
 
 Tracing command irq16: uhci0 pid 25 tid 100026 td 0xc5119660
 fork_trampoline() at fork_trampoline
 
 Tracing command irq9: acpi0 pid 24 tid 100027 td 0xc5222440
 fork_trampoline() at fork_trampoline
 
 Tracing command kqueue taskq pid 23 tid 100028 td 0xc5222220
 sched_switch(c5222220,0,1,180,70473070,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5222220,...) at mi_switch+0x217
 sleepq_switch(c5222220,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0
 sleepq_wait(c5212600,c521261c,c0a94778,0,0,...) at sleepq_wait+0x60
 _sleep(c5212600,c521261c,0,c0a94778,0,...) at _sleep+0x369
 taskqueue_thread_loop(c0baeec4,e57b2d38,c0a99763,2ea,c5220804,...) at 
 taskqueue_thread_loop+0xb4
 fork_exit(c0780c30,c0baeec4,e57b2d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57b2d70, ebp = 0 ---
 
 Tracing command swi6: task queue pid 22 tid 100029 td 0xc5222000
 sched_switch(c5222000,0,1,180,cc82e284,...) at sched_switch+0x49b
 mi_switch(1,0,c0a999ef,46b,c52126e4,...) at mi_switch+0x217
 ithread_loop(c5206120,e57afd38,c0a99763,2ea,c5220ab0,...) at 
 ithread_loop+0x34c
 fork_exit(c07328a0,c5206120,e57afd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe57afd70, ebp = 0 ---
 
 Tracing command swi6: Giant taskq pid 21 tid 100013 td 0xc5117880
 fork_trampoline() at fork_trampoline
 
 Tracing command thread taskq pid 9 tid 100014 td 0xc5117660
 sched_switch(c5117660,0,1,180,348742c8,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5117660,...) at mi_switch+0x217
 sleepq_switch(c5117660,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0
 sleepq_wait(c5212880,c521289c,c0a94778,0,0,...) at sleepq_wait+0x60
 _sleep(c5212880,c521289c,0,c0a94778,0,...) at _sleep+0x369
 taskqueue_thread_loop(c0bba7d8,e3d41d38,c0a99763,2ea,c515fab0,...) at 
 taskqueue_thread_loop+0xb4
 fork_exit(c0780c30,c0bba7d8,e3d41d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d41d70, ebp = 0 ---
 
 Tracing command acpi_task_2 pid 8 tid 100015 td 0xc5117440
 sched_switch(c5117440,0,1,180,700a22a0,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5117440,...) at mi_switch+0x217
 sleepq_switch(c5117440,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0
 sleepq_wait(c5212900,c521291c,c0a94778,0,0,...) at sleepq_wait+0x60
 _sleep(c5212900,c521291c,0,c0a94778,0,...) at _sleep+0x369
 taskqueue_thread_loop(c0d7eebc,e3d3ed38,c0a99763,2ea,c5213000,...) at 
 taskqueue_thread_loop+0xb4
 fork_exit(c0780c30,c0d7eebc,e3d3ed38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d3ed70, ebp = 0 ---
 
 Tracing command acpi_task_1 pid 7 tid 100016 td 0xc5117220
 sched_switch(c5117220,0,1,180,7009e0a0,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5117220,...) at mi_switch+0x217
 sleepq_switch(c5117220,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0
 sleepq_wait(c5212900,c521291c,c0a94778,0,0,...) at sleepq_wait+0x60
 _sleep(c5212900,c521291c,0,c0a94778,0,...) at _sleep+0x369
 taskqueue_thread_loop(c0d7eebc,e3d3bd38,c0a99763,2ea,c52132ac,...) at 
 taskqueue_thread_loop+0xb4
 fork_exit(c0780c30,c0d7eebc,e3d3bd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d3bd70, ebp = 0 ---
 
 Tracing command acpi_task_0 pid 6 tid 100017 td 0xc5214aa0
 sched_switch(c5214aa0,0,1,180,70099d0c,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5214aa0,...) at mi_switch+0x217
 sleepq_switch(c5214aa0,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0
 sleepq_wait(c5212900,c521291c,c0a94778,0,0,...) at sleepq_wait+0x60
 _sleep(c5212900,c521291c,0,c0a94778,0,...) at _sleep+0x369
 taskqueue_thread_loop(c0d7eebc,e5794d38,c0a99763,2ea,c5213558,...) at 
 taskqueue_thread_loop+0xb4
 fork_exit(c0780c30,c0d7eebc,e5794d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe5794d70, ebp = 0 ---
 
 Tracing command swi5: + pid 20 tid 100018 td 0xc5214880
 fork_trampoline() at fork_trampoline
 
 Tracing command swi2: cambio pid 19 tid 100019 td 0xc5214660
 sched_switch(c5214660,0,1,180,34663108,...) at sched_switch+0x49b
 mi_switch(1,0,c0a999ef,46b,c5212ae4,...) at mi_switch+0x217
 ithread_loop(c5206170,e578ed38,c0a99763,2ea,c5213ab0,...) at 
 ithread_loop+0x34c
 fork_exit(c07328a0,c5206170,e578ed38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe578ed70, ebp = 0 ---
 
 Tracing command xpt_thrd pid 5 tid 100020 td 0xc5214440
 sched_switch(c5214440,0,1,180,6ff8aad4,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5214440,...) at mi_switch+0x217
 sleepq_switch(c5214440,0,c0aa0842,21d,4c,...) at sleepq_switch+0xf0
 sleepq_wait(c0b7f514,0,c0a9d757,da,0,...) at sleepq_wait+0x60
 _sleep(c0b7f514,c0b7f52c,4c,c0a535da,0,...) at _sleep+0x369
 xpt_scanner_thread(0,e578bd38,c0a99763,2ea,c5215000,...) at 
 xpt_scanner_thread+0x41
 fork_exit(c0468360,0,e578bd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe578bd70, ebp = 0 ---
 
 Tracing command yarrow pid 18 tid 100006 td 0xc5115220
 sched_switch(c5115220,0,1,180,79be0e56,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5115220,0,c0aa0842,240,0,...) at sleepq_switch+0xf0
 sleepq_timedwait(c0bb0654,64,c0a94778,2,0,...) at sleepq_timedwait+0x68
 _sleep(c0bb0654,0,0,c0a94778,64,...) at _sleep+0x345
 pause(c0a94778,64,c0a88f43,112,0,...) at pause+0x47
 random_kthread(0,e3d26d38,c0a99763,2ea,c51182ac,...) at random_kthread+0x1d9
 fork_exit(c0656d20,0,e3d26d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d26d70, ebp = 0 ---
 
 Tracing command g_down pid 4 tid 100007 td 0xc5115000
 sched_switch(c5115000,0,1,180,7c36f0fe,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217
 sleepq_switch(c5115000,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0
 sleepq_timedwait(c0bae6cc,64,c0a94778,0,0,...) at sleepq_timedwait+0x68
 _sleep(c0bae6cc,c0bae5e8,24c,c0a94778,64,...) at _sleep+0x345
 g_io_schedule_down(c5115000,0,c0a95db7,74,0,...) at g_io_schedule_down+0x6b
 g_down_procbody(0,e3d23d38,c0a99763,2ea,c5118558,...) at 
 g_down_procbody+0x8d
 fork_exit(c07019c0,0,e3d23d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d23d70, ebp = 0 ---
 
 Tracing command g_up pid 3 tid 100008 td 0xc5119440
 sched_switch(c5119440,0,1,180,7c372a56,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217
 sleepq_switch(c5119440,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0
 sleepq_timedwait(c0bae6c8,64,c0a94778,0,0,...) at sleepq_timedwait+0x68
 _sleep(c0bae6c8,c0bae628,24c,c0a94778,64,...) at _sleep+0x345
 g_io_schedule_up(c5119440,0,c0a95db7,5d,0,...) at g_io_schedule_up+0x133
 g_up_procbody(0,e3d53d38,c0a99763,2ea,c5118804,...) at g_up_procbody+0x8d
 fork_exit(c0701a50,0,e3d53d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d53d70, ebp = 0 ---
 
 Tracing command g_event pid 2 tid 100009 td 0xc5119220
 sched_switch(c5119220,0,1,180,79eea34a,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217
 sleepq_switch(c5119220,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0
 sleepq_timedwait(c0bae6c0,64,c0a94778,0,0,...) at sleepq_timedwait+0x68
 _sleep(c0bae6c0,0,4c,c0a94778,64,...) at _sleep+0x345
 g_event_procbody(0,e3d50d38,c0a99763,2ea,c5118ab0,...) at 
 g_event_procbody+0xcb
 fork_exit(c0701ae0,0,e3d50d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d50d70, ebp = 0 ---
 
 Tracing command swi3: vm pid 17 tid 100010 td 0xc5119000
 fork_trampoline() at fork_trampoline
 
 Tracing command swi4: clock sio pid 16 tid 100011 td 0xc5117cc0
 cpustop_handler(8,e3d4abd0,c0a04a50,38,c0aa17fa,...) at cpustop_handler+0x32
 ipi_nmi_handler(38,c0aa17fa,6ce,c0c01c00,13,...) at ipi_nmi_handler+0x2f
 trap(e3d4abdc) at trap+0x30
 calltrap() at calltrap+0x6
 --- trap 0x13, eip = 0xc07429df, esp = 0xe3d4ac1c, ebp = 0xe3d4ac38 ---
 _mtx_lock_sleep(c58887a4,c5117cc0,0,c0ab3291,1b6,...) at 
 _mtx_lock_sleep+0xbf
 _mtx_lock_flags(c58887a4,0,c0ab3291,1b6,10207,...) at _mtx_lock_flags+0xef
 tcp_timer_rexmt(c588a570,0,c0a9e01d,ef,0,...) at tcp_timer_rexmt+0x96
 softclock(0,0,c0a999ef,471,c515e364,...) at softclock+0x266
 ithread_loop(c5113260,e3d4ad38,c0a99763,2ea,c515f2ac,...) at 
 ithread_loop+0x1b5
 fork_exit(c07328a0,c5113260,e3d4ad38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d4ad70, ebp = 0 ---
 
 Tracing command swi1: net pid 15 tid 100012 td 0xc5117aa0
 sched_switch(c5117aa0,0,1,180,386ad10c,...) at sched_switch+0x49b
 mi_switch(1,0,c0a999ef,46b,c515e3e4,...) at mi_switch+0x217
 ithread_loop(c5113270,e3d47d38,c0a99763,2ea,c515f558,...) at 
 ithread_loop+0x34c
 fork_exit(c07328a0,c5113270,e3d47d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d47d70, ebp = 0 ---
 
 Tracing command idle: cpu0 pid 14 tid 100000 td 0xc5117000
 sched_switch(c5117000,0,6,17b,385804f8,...) at sched_switch+0x49b
 mi_switch(6,0,c0a9ec6c,b6,0,...) at mi_switch+0x217
 critical_exit(c515e500,0,2,c5117000,c515e500,...) at critical_exit+0x99
 intr_execute_handlers(c5107f34,e3d38c50,e3d38c90,c09eb844,34,...) at 
 intr_execute_handlers+0x14b
 lapic_handle_intr(34,e3d38c50) at lapic_handle_intr+0x3f
 Xapic_isr1() at Xapic_isr1+0x34
 --- interrupt, eip = 0xc0d6e4a5, esp = 0xe3d38c90, ebp = 0xe3d38c90 ---
 acpi_cpu_c1(c0b4fdf8,c5117000,e3d38cac,0,c0bb8680,...) at acpi_cpu_c1+0x5
 acpi_cpu_idle(e3d38cf8,c076c949,c0bb8680,0,c0a9e7d2,...) at 
 acpi_cpu_idle+0x18c
 cpu_idle(c0bb8680,0,c0a9e7d2,322,c0bb2280,...) at cpu_idle+0x28
 sched_idletd(0,e3d38d38,c0a99763,2ea,c5114000,...) at sched_idletd+0x249
 fork_exit(c076c700,0,e3d38d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d38d70, ebp = 0 ---
 
 Tracing command idle: cpu1 pid 13 tid 100001 td 0xc5115cc0
 sched_switch(c5115cc0,0,1,17b,37c397f4,...) at sched_switch+0x49b
 mi_switch(1,0,c0a9e7d2,320,c0bb28c0,...) at mi_switch+0x217
 sched_idletd(0,e3d35d38,c0a99763,2ea,c51142ac,...) at sched_idletd+0x217
 fork_exit(c076c700,0,e3d35d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d35d70, ebp = 0 ---
 
 Tracing command idle: cpu2 pid 12 tid 100002 td 0xc5115aa0
 cpustop_handler(4,e3d32c44,c0a04a50,38,c0aa17fa,...) at cpustop_handler+0x32
 ipi_nmi_handler(38,c0aa17fa,6ce,c0c01c00,13,...) at ipi_nmi_handler+0x2f
 trap(e3d32c50) at trap+0x30
 calltrap() at calltrap+0x6
 --- trap 0x13, eip = 0xc0d6e4a5, esp = 0xe3d32c90, ebp = 0xe3d32c90 ---
 acpi_cpu_c1(c0b4fdf8,c5115aa0,e3d32cac,2,c0bb8700,...) at acpi_cpu_c1+0x5
 acpi_cpu_idle(e3d32cf8,c076c949,c0bb8700,0,c0a9e7d2,...) at 
 acpi_cpu_idle+0x18c
 cpu_idle(c0bb8700,0,c0a9e7d2,322,c0bb2f00,...) at cpu_idle+0x28
 sched_idletd(0,e3d32d38,c0a99763,2ea,c5114558,...) at sched_idletd+0x249
 fork_exit(c076c700,0,e3d32d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d32d70, ebp = 0 ---
 
 Tracing command idle: cpu3 pid 11 tid 100003 td 0xc5115880
 sched_switch(c5115880,0,1,17b,879e466e,...) at sched_switch+0x49b
 mi_switch(1,0,c0a9e7d2,320,c0bb3540,...) at mi_switch+0x217
 sched_idletd(0,e3d2fd38,c0a99763,2ea,c5114804,...) at sched_idletd+0x217
 fork_exit(c076c700,0,e3d2fd38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d2fd70, ebp = 0 ---
 
 Tracing command init pid 1 tid 100004 td 0xc5115660
 sched_switch(c5115660,0,1,180,93649798,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217
 sleepq_switch(c5115660,0,c0aa0842,19b,c5114ab0,...) at sleepq_switch+0xf0
 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,5c,...) at 
 sleepq_catch_signals+0x21e
 sleepq_wait_sig(c5114ab0,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14
 _sleep(c5114ab0,c5114b40,15c,c0aa296f,0,...) at _sleep+0x359
 kern_wait(c5115660,ffffffff,e3d2cc2c,0,0,...) at kern_wait+0xa66
 wait4(c5115660,e3d2ccfc,10,e3d2cd38,c0b49b28,...) at wait4+0x3b
 syscall(e3d2cd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x805455f, esp = 
 0xbfbfe97c, ebp = 0xbfbfe998 ---
 
 Tracing command audit pid 10 tid 100005 td 0xc5115440
 sched_switch(c5115440,0,1,180,6fef90bc,...) at sched_switch+0x49b
 mi_switch(1,0,c0aa0842,1bf,c5115440,...) at mi_switch+0x217
 sleepq_switch(c5115440,0,c0aa0842,21d,c5115440,...) at sleepq_switch+0xf0
 sleepq_wait(c0c0d5b4,c0c0d594,c0ab9dd3,1,0,...) at sleepq_wait+0x60
 _cv_wait(c0c0d5b4,c0c0d594,c0aba790,1c2,0,...) at _cv_wait+0x1fc
 audit_worker(0,e3d29d38,c0a99763,2ea,c5118000,...) at audit_worker+0x99
 fork_exit(c091b750,0,e3d29d38) at fork_exit+0xb8
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe3d29d70, ebp = 0 ---
 
 Tracing command swapper pid 0 tid 0 td 0xc0baea30
 sched_switch(c0baea30,0,1,180,3991d04c,...) at sched_switch+0x49b
 mi_switch(1,0,c0abea20,2d9,c0ac973d,...) at mi_switch+0x217
 scheduler(0,101ec00,101ec00,101e000,1028000,...) at scheduler+0x2f2
 mi_startup() at mi_startup+0x96
 begin() at begin+0x2c
 db>                                           show allocks
 No such command
 db> show alllocks
 Process 943 (sshd) thread 0xc5861440 (100144)
 exclusive sleep mutex radix node head r = 0 (0xc538d07c) locked @ 
 /usr/src/sys/net/route.c:147
 exclusive sleep mutex inp (tcpinp) r = 0 (0xc58887a4) locked @ 
 /usr/src/sys/netinet/tcp_usrreq.c:779
 exclusive sx so_snd_sx r = 0 (0xc5903430) locked @ 
 /usr/src/sys/kern/uipc_sockbuf.c:145
 Process 940 (sshd) thread 0xc5861220 (100145)
 exclusive sx so_rcv_sx r = 0 (0xc58a8530) locked @ 
 /usr/src/sys/kern/uipc_sockbuf.c:145
 Process 33 (irq21: bge0) thread 0xc5214cc0 (100037)
 exclusive sleep mutex rtentry r = 0 (0xc5722c18) locked @ 
 /usr/src/sys/net/route.c:1065
 exclusive spin mutex sio r = 0 (0xc0c1b0e8) locked @ 
 /usr/src/sys/dev/sio/sio.c:1391
 Process 16 (swi4: clock sio) thread 0xc5117cc0 (100011)
 exclusive sleep mutex tcp r = 0 (0xc0c0472c) locked @ 
 /usr/src/sys/netinet/tcp_timer.c:423
 db> show lockedvnods
 Locked vnodes
 db> call boot(0)
 panic: blockable sleep lock (sleep mutex) eventhandler @ 
 /usr/src/sys/kern/subr_eventhandler.c:212
 cpuid = 0
 KDB: enter: panic
 [thread pid 33 tid 100037 ]
 Stopped at      kdb_enter+0x32: leave
 db> reboot

From: Per olof Ljungmark <peo@intersonic.se>
To: Maxime Henrion <mux@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/118044: panic: spin lock held too long
Date: Thu, 27 Dec 2007 16:49:33 +0100

 Maxime Henrion wrote:
 > 	Hi Per,
 > 
 > 
 > There is a suspicion that the problem you are experiencing might
 > be fixed by a patch that has just been committed to HEAD.  Could
 > you please give it a try?  It's a fairly small patch, you just need
 > to apply revision 1.125 of sys/net/route.c and revision 1.68 of
 > sys/net/route.h.
 > 
 > Let me know if you need some help in doing this testing.
 > 
 
 ...and here is the type of packet I *think* triggers the bug.
 
 
 16:48:24.680815 IP (tos 0x0, ttl 255, id 61287, offset 0, flags [none], 
 proto ICMP (1), length 56) aiko.i.inter-sonic.com > 
 ruben.i.inter-sonic.com: ICMP redirect kreutzman.h.inter-sonic.com to 
 host aiko.i.inter-sonic.com, length 36
          IP (tos 0x10, ttl 64, id 36864, offset 0, flags [none], proto 
 TCP (6), length 100) ruben.i.inter-sonic.com.ssh > 
 kreutzman.h.inter-sonic.com.60239: [|tcp]
          0x0000:  000d 9d57 719a 0008 0252 f9e6 0800 4500  ...Wq....R....E.
          0x0010:  0038 ef67 0000 ff01 476c c0a8 01fe c0a8  .8.g....Gl......
          0x0020:  01a2 0501 a8df c0a8 01fe 4510 0064 9000  ..........E..d..
          0x0030:  0000 4006 7a18 c0a8 01a2 ac10 0211 0016  ..@.z...........
          0x0040:  eb4f 1dc5 864d                           .O...M
 

From: Bryan Drewery <bryan@xzibition.com>
To: bug-followup@FreeBSD.org,  peo@intersonic.se
Cc:  
Subject: Re: kern/118044: [panic] spin lock held too long
Date: Tue, 17 Jun 2008 08:06:30 -0500

 I get this on reboot. SMP + ULE here as well.
State-Changed-From-To: open->feedback 
State-Changed-By: jh 
State-Changed-When: Mon May 23 15:23:57 UTC 2011 
State-Changed-Why:  
Can you still reproduce this on a supported release? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=118044 
State-Changed-From-To: feedback->closed 
State-Changed-By: jh 
State-Changed-When: Sun Jul 3 07:43:10 UTC 2011 
State-Changed-Why:  
Feedback timeout. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=118044 
>Unformatted:
