From doublef-ctm@yandex.ru  Sat Oct 14 12:52:00 2006
Return-Path: <doublef-ctm@yandex.ru>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 2AFCF16A40F
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 14 Oct 2006 12:52:00 +0000 (UTC)
	(envelope-from doublef-ctm@yandex.ru)
Received: from smtp1.yandex.ru (smtp1.yandex.ru [213.180.223.87])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 4770A43D45
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 14 Oct 2006 12:51:59 +0000 (GMT)
	(envelope-from doublef-ctm@yandex.ru)
Received: from [83.239.189.140] ([83.239.189.140]:47318 "EHLO shark" smtp-auth:
	"doublef-ctm" TLS-CIPHER: <none> TLS-PEER-CN1: <none>)
	by mail.yandex.ru with ESMTP id S2077601AbWJNMvp (ORCPT
	<rfc822;FreeBSD-gnats-submit@freebsd.org>);
	Sat, 14 Oct 2006 16:51:45 +0400
Received: by shark (Postfix, from userid 1000)
	id C71931743B; Sat, 14 Oct 2006 16:51:42 +0400 (MSD)
Message-Id: <20061014125142.C71931743B@shark>
Date: Sat, 14 Oct 2006 16:51:42 +0400 (MSD)
From: Sergey Zaharchenko <doublef-ctm@yandex.ru>
Reply-To: Sergey Zaharchenko <doublef-ctm@yandex.ru>
To: FreeBSD-gnats-submit@freebsd.org
Subject: Processes get stuck in "ufs" state under persistent CPU load
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         104406
>Category:       kern
>Synopsis:       [ufs] Processes get stuck in "ufs" state under persistent CPU load
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Oct 14 13:00:30 GMT 2006
>Closed-Date:    
>Last-Modified:  Mon May 18 03:02:36 UTC 2009
>Originator:     Sergey Zaharchenko
>Release:        FreeBSD 7.0-CURRENT i386
>Organization:
Volgograd State Technical University
>Environment:

System: FreeBSD shark.localdomain 7.0-CURRENT FreeBSD 7.0-CURRENT #5: Fri Oct 13 22:03:33 MSD 2006 root@shark.localdomain:/var/obj/src/usr.src/sys/GENERIC i386
The problem has also been observed on 7.0-CURRENT of August 2006.
FWIW 4.8-RELEASE didn't have the problem.
CPU: AMD Sempron(tm)   2500+ (1753.99-MHz 686-class CPU)
A UP system, GENERIC kernel, no RAID, etc.

>Description:

When a single process loads the CPU for a long(*) time, other processes
which want to access to the filesystem get stuck in the "ufs" state when
trying to do that. Other processes which don't need to access the
filesystem (like top, etc.) proceed normally.

The owner UID, nice- and idprio- status of the offending (or offended)
process do not matter.

It seems essential that a single process is working all the time (e.g.
two hours of compilation don't show up any errors, because there are
many processes).

Example top outputs for this situation:

last pid:  9798;  load averages:  2.00,  1.96,  1.72    up 0+02:29:59  11:02:17
113 processes: 3 running, 109 sleeping, 1 zombie
CPU states:  0.0% user, 96.2% nice,  0.4% system,  3.4% interrupt,  0.0% idle
Mem: 134M Active, 193M Inact, 88M Wired, 316K Cache, 59M Buf, 70M Free
Swap: 4097M Total, 64M Used, 4033M Free, 1% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 2661 df            1 139   20 63532K 58468K RUN    107:21 97.46% generic_slave
 8035 df            1  96    0  7608K  2752K select   0:00  0.05% mc
  702 root          1  96    0  5200K   792K select   0:41  0.00% syslogd
 1245 root          1  -4    0  5152K   572K ufs      0:35  0.00% tail
  912 squid         1  -4    0 13928K  4288K ufs      0:31  0.00% squid
 1163 df            1  -4    0  9716K  2512K ufs      0:27  0.00% fetchmail
 2600 root          1 -32    0  5516K  2052K RUN      0:13  0.00% top
 1042 mysql         6  20    0 59128K  1956K kserel   0:06  0.00% mysqld
 9338 df            1  -4    0 15988K  9812K ufs      0:06  0.00% links
 1179 root          1  96    0  5200K   112K select   0:02  0.00% moused

last pid:  2739;  load averages:  2.00,  1.95,  1.57    up 0+00:31:39  16:46:48
91 processes:  3 running, 87 sleeping, 1 zombie
CPU states: 97.8% user,  0.0% nice,  0.4% system,  1.9% interrupt,  0.0% idle
Mem: 120M Active, 29M Inact, 32M Wired, 17M Cache, 15M Buf, 287M Free
Swap: 4097M Total, 4097M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 1352 df            1 132    0  2248K   896K RUN     25:58 97.41% testcase
 1580 df            1  96    0  7608K  2696K select   0:09  0.00% mc
  970 squid         1  -4    0 13928K  8108K ufs      0:03  0.00% squid
 1316 root          1 -32    0  5516K  1608K RUN      0:03  0.00% top
 1238 df            1  -4    0  7668K  1884K ufs      0:02  0.00% fetchmail
 1110 mysql         6  20    0 59128K 51600K kserel   0:01  0.00% mysqld
  754 root          1  96    0  5200K  1088K select   0:01  0.00% syslogd
 1322 root          1  96    0 22656K 16968K select   0:01  0.00% Xorg
  868 root          1   8    0  9232K  4748K nanslp   0:00  0.00% httpd
 1126 news          1   8    0  5484K  1308K wait     0:00  0.00% sh
 1317 root          1  -4    0  5152K   684K ufs      0:00  0.00% tail
 1144 news          1   4    4  7816K  3396K sbwait   0:00  0.00% perl5.8.8
 1120 news          1  -4    0   150M 12828K ufs      0:00  0.00% innd
 1351 df            1  20    0  4032K  1784K pause    0:00  0.00% csh
  993 squid         1  -4    0  5640K   824K msgwai   0:00  0.00% diskd
 1699 df            1   8    0  5244K  3104K ppwait   0:00  0.00% csh
 1182 root          1   8    0  5200K  1100K nanslp   0:00  0.00% cron

(*) for values of `long' from 10 minutes to 2 hours for me.

>How-To-Repeat:

A program to generate the necessary load can be quite simple, like

int
main(void)
{
	/* Crunch some numbers (really meaningless) */
	unsigned u=1;
	while (1)
	{
		u*=0x8088405;
	}
}

Compile and run it, run `top', and wait for a long (see above) time.
Browse directories with `ls' from time to time on a different terminal.
See `ls' hang at some time. View the `top' terminal.

>Fix:

I don't know the fix, but an offending process can be stopped with 'kill
-STOP' and continued with 'kill -CONT', which allows other processes to
access the filesystem (until another such failure occurs). Periodic
stopping and starting processes might count as a lousy workaround.
>Release-Note:
>Audit-Trail:

From: "Oleg Derevenetz" <oleg@vsi.ru>
To: <bug-followup@FreeBSD.org>
Cc:  
Subject: Re: kern/104406: [ufs] Processes get stuck in "ufs" state under persistent CPU load
Date: Wed, 7 Mar 2007 05:10:59 +0300

 I have the problem with the same symptoms on SMP FreeBSD 6.2-STABLE with 
 dual AMD Opteron(tm) Processor 850. Sometimes (apparently when CPU load 
 suddenly goes up) all processes that interacts with disk gets stuck in "ufs" 
 state, but in my case SIGSTOP/SIGCONT seemingly does not help.
 
 uname -a output:
 
 FreeBSD serv2.vsi.ru 6.2-STABLE FreeBSD 6.2-STABLE #2: Sat Mar  3 01:59:08 
 MSK 2007     oleg@serv2.vsi.ru:/usr/obj/usr/src/sys/serv2  i386
 
 dmesg.boot:
 
 Copyright (c) 1992-2007 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 6.2-STABLE #2: Sat Mar  3 01:59:08 MSK 2007
     oleg@serv2.vsi.ru:/usr/obj/usr/src/sys/serv2
 Timecounter "i8254" frequency 1193182 Hz quality 0
 CPU: AMD Opteron(tm) Processor 850 (2389.26-MHz 686-class CPU)
   Origin = "AuthenticAMD"  Id = 0x20f51  Stepping = 1
   Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
   Features2=0x1<SSE3>
   AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
   AMD Features2=0x1<LAHF>
 real memory  = 8589934592 (8192 MB)
 avail memory = 8350457856 (7963 MB)
 ACPI APIC Table: <PTLTD          APIC  >
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
 MADT: Forcing active-low polarity and level trigger for SCI
 ioapic0 <Version 1.1> irqs 0-23 on motherboard
 ioapic1 <Version 1.1> irqs 24-27 on motherboard
 ioapic2 <Version 1.1> irqs 28-31 on motherboard
 ioapic3 <Version 1.1> irqs 32-35 on motherboard
 ioapic4 <Version 1.1> irqs 36-39 on motherboard
 ioapic5 <Version 1.1> irqs 40-43 on motherboard
 ioapic6 <Version 1.1> irqs 44-47 on motherboard
 kbd1 at kbdmux0
 acpi0: <PTLTD    XSDT> on motherboard
 acpi0: Power Button (fixed)
 unknown: I/O range not supported
 unknown: I/O range not supported
 Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xf008-0xf00b on acpi0
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 acpi_button0: <Power Button> on acpi0
 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0xf000-0xf07f,0xf080-0xf0ff 
 iomem 0xd8000-0xdbfff on acpi0
 pci0: <ACPI PCI bus> on pcib0
 pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0
 pci1: <ACPI PCI bus> on pcib1
 ohci0: <OHCI (generic) USB controller> mem 0xfc900000-0xfc900fff irq 19 at 
 device 0.0 on pci1
 ohci0: [GIANT-LOCKED]
 usb0: OHCI version 1.0, legacy support
 usb0: SMM does not respond, resetting
 usb0: <OHCI (generic) USB controller> on ohci0
 usb0: USB revision 1.0
 uhub0: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
 uhub0: 3 ports with 3 removable, self powered
 ohci1: <OHCI (generic) USB controller> mem 0xfc901000-0xfc901fff irq 19 at 
 device 0.1 on pci1
 ohci1: [GIANT-LOCKED]
 usb1: OHCI version 1.0, legacy support
 usb1: SMM does not respond, resetting
 usb1: <OHCI (generic) USB controller> on ohci1
 usb1: USB revision 1.0
 uhub1: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
 uhub1: 3 ports with 3 removable, self powered
 pci1: <display, VGA> at device 5.0 (no driver attached)
 isab0: <PCI-ISA bridge> at device 7.0 on pci0
 isa0: <ISA bus> on isab0
 atapci0: <AMD 8111 UDMA133 controller> port 
 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1000-0x100f at device 7.1 on pci0
 ata0: <ATA channel 0> on atapci0
 ata1: <ATA channel 1> on atapci0
 pci0: <bridge> at device 7.3 (no driver attached)
 pcib2: <ACPI PCI-PCI bridge> at device 10.0 on pci0
 pci2: <ACPI PCI bus> on pcib2
 bge0: <Broadcom BCM5704 A3, ASIC rev. 0x2003> mem 
 0xfe010000-0xfe01ffff,0xfe000000-0xfe00ffff irq 25 at device 2.0 on pci2
 miibus0: <MII bus> on bge0
 brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus0
 brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 
 1000baseTX-FDX, auto
 bge0: Ethernet address: 00:09:3d:13:fd:00
 bge1: <Broadcom BCM5704 A3, ASIC rev. 0x2003> mem 
 0xfe030000-0xfe03ffff,0xfe020000-0xfe02ffff irq 26 at device 2.1 on pci2
 miibus1: <MII bus> on bge1
 brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus1
 brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 
 1000baseTX-FDX, auto
 bge1: Ethernet address: 00:09:3d:13:fd:01
 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 
 0xfe050000-0xfe05ffff,0xfe040000-0xfe04ffff irq 27 at device 4.0 on pci2
 mpt0: [GIANT-LOCKED]
 mpt0: MPI Version=1.2.15.0
 mpt0: Capabilities: ( RAID-1E RAID-1 SAFTE )
 mpt0: 0 Active Volumes (1 Max)
 mpt0: 0 Hidden Drive Members (6 Max)
 pci0: <base peripheral, interrupt controller> at device 10.1 (no driver 
 attached)
 pcib3: <ACPI PCI-PCI bridge> at device 11.0 on pci0
 pci3: <ACPI PCI bus> on pcib3
 pci0: <base peripheral, interrupt controller> at device 11.1 (no driver 
 attached)
 pcib4: <ACPI Host-PCI bridge> iomem 
 0xfe301000-0xfe301fff,0xfe303000-0xfe303fff,0xfe305000-0xfe305fff,0xfe307000-0xfe307fff 
 on acpi0
 pci32: <ACPI PCI bus> on pcib4
 pcib5: <ACPI PCI-PCI bridge> mem 0xfe300000-0xfe300fff irq 32 at device 1.0 
 on pci32
 pci33: <ACPI PCI bus> on pcib5
 pci32: <base peripheral, interrupt controller> at device 1.1 (no driver 
 attached)
 pcib6: <ACPI PCI-PCI bridge> mem 0xfe302000-0xfe302fff irq 36 at device 2.0 
 on pci32
 pci34: <ACPI PCI bus> on pcib6
 pci32: <base peripheral, interrupt controller> at device 2.1 (no driver 
 attached)
 pcib7: <ACPI PCI-PCI bridge> mem 0xfe304000-0xfe304fff irq 40 at device 3.0 
 on pci32
 pci35: <ACPI PCI bus> on pcib7
 pci32: <base peripheral, interrupt controller> at device 3.1 (no driver 
 attached)
 pcib8: <ACPI PCI-PCI bridge> mem 0xfe306000-0xfe306fff irq 44 at device 4.0 
 on pci32
 pci36: <ACPI PCI bus> on pcib8
 pci32: <base peripheral, interrupt controller> at device 4.1 (no driver 
 attached)
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on 
 acpi0
 sio0: type 16550A
 fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
 fdc0: [FAST]
 fd0: <1440-KB 3.5" drive> on fdc0 drive 0
 pmtimer0 on isa0
 orm0: <ISA Option ROMs> at iomem 
 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcafff,0xcb000-0xcefff on isa0
 ppc0: parallel port not found.
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=0x300>
 sio1 at port 0x2f8-0x2ff irq 3 on isa0
 sio1: type 16550A
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
 Timecounters tick every 1.000 msec
 IP Filter: v4.1.13 initialized.  Default = block all, Logging = enabled
 Waiting 5 seconds for SCSI devices to settle
 acd0: DMA limited to UDMA33, controller found non-ATA66 cable
 acd0: DVDROM <MATSHITADVD-ROM SR-8178/PZ21> at ata1-master UDMA33
 ses0 at mpt0 bus 0 target 6 lun 0
 ses0: <SDR GEM318P 1> Fixed Processor SCSI-2 device
 ses0: 3.300MB/s transfers
 ses0: SAF-TE Compliant Device
 SMP: AP CPU #1 Launched!
 da0 at mpt0 bus 0 target 0 lun 0
 da0: <SEAGATE ST373207LC 0003> Fixed Direct Access SCSI-3 device
 da0: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing 
 Enabled
 da0: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C)
 da1 at mpt0 bus 0 target 2 lun 0
 da1: <SEAGATE ST336807LC 0C01> Fixed Direct Access SCSI-3 device
 da1: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing 
 Enabled
 da1: 35003MB (71687372 512 byte sectors: 255H 63S/T 4462C)
 Trying to mount root from ufs:/dev/da0s1a
 Accounting enabled
 
 Any ideas on how to debug this ?
 
 --
 Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
 Phone: +7 4732 539880
 Fax:   +7 4732 531415         http://www.vsi.ru
 CenterTelecom Voronezh ISP    http://isp.vsi.ru
 

From: "Jeffrey D. Wheelhouse" <jdw@wheelhouse.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/104406: [ufs] Processes get stuck in "ufs" state under persistent
 CPU load
Date: Tue, 05 Jun 2007 16:26:26 -0400

 I believe we have also experienced this bug (or a very similar one) on 
 our 8-core amd64 systems under 6.2-RELEASE-p4.
 
 In our case, "top" shows that the system is 100% CPU utilized, with the 
 vast majority of it as "system" time.  (Ordinarily the system
 
 In the last case, we ended up with about 200 Apache processes that 
 looked like this in ps:
 
    UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
 25000 27121 26860 1977  -4  5 146324 33732 ufs    DN    ??    0:03.75 httpd
 25000 27147 37257 1994  -4  5 153748 29280 ufs    DN    ??    0:03.72 httpd
 25000 27157 36912 1805  -4  5 150756 26592 ufs    DN    ??    0:02.91 httpd
 25000 27224 27030 1845  -4  5 137536 24804 ufs    DN    ??    0:01.25 httpd
 25000 27274 26794 1829  -4  5 148140 35416 ufs    DN    ??    0:02.90 httpd
 
 Once a process gets "stuck" in WCHAN ufs, it's blocked indefinitely, as 
 described here, or at least so slow as to be indistinguishable from 
 stuck.  (Typical wait channels for our httpds are accept or kqread, as 
 one would expect.)
 
 Each process in this state counts against the load average, so we often 
 see load averages north of 200 when this is occurring.  (Typical load 
 average is below 2.)
 
 Kill enough processes (or possibly enough to hit the "right" process) 
 and everything picks up again right where it left off.
 
 I also have no idea how to debug this.
 
 Thanks,
 Jeff
 
 -- 
 Jeff Wheelhouse
 jdw@wheelhouse.org

From: "Oleg Derevenetz" <oleg@vsi.ru>
To: <bug-followup@FreeBSD.org>, <doublef-ctm@yandex.ru>
Cc:  
Subject: Re: kern/104406: [ufs] Processes get stuck in "ufs" state under persistent CPU load
Date: Fri, 19 Oct 2007 17:36:27 +0400

 This problem experiences on our another AMD64 machine (also with 6-STABLE). When we copy large amount of small files using mc from 
 FTP to local filesystem, after some time mc hangs in "wdrain" state, and all other processes that need to access filesystem are 
 hangs in "ufs" state. There are some debug stuff:
 
 uname -a:
 
 FreeBSD serv13.vsi.ru 6.2-STABLE FreeBSD 6.2-STABLE #1: Fri Oct 19 16:28:07 MSD 2007 
 oleg@serv13.vsi.ru:/usr/obj/usr/src/sys/serv13  i386
 
 dmesg:
 
 Copyright (c) 1992-2007 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 6.2-STABLE #1: Fri Oct 19 16:28:07 MSD 2007
     oleg@serv13.vsi.ru:/usr/obj/usr/src/sys/serv13
 WARNING: WITNESS option enabled, expect reduced performance.
 WARNING: DIAGNOSTIC option enabled, expect reduced performance.
 Timecounter "i8254" frequency 1193182 Hz quality 0
 CPU: Dual-Core AMD Opteron(tm) Processor 2212 (2010.31-MHz 686-class CPU)
   Origin = "AuthenticAMD"  Id = 0x40f12  Stepping = 2
   Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
   Features2=0x2001<SSE3,CX16>
   AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
   AMD Features2=0x1f<LAHF,CMP,SVM,ExtAPIC,CR8>
   Cores per package: 2
 real memory  = 3220176896 (3071 MB)
 avail memory = 3149598720 (3003 MB)
 ACPI APIC Table: <PTLTD          APIC  >
 FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
 ioapic0 <Version 1.1> irqs 0-23 on motherboard
 ioapic1 <Version 1.1> irqs 24-47 on motherboard
 kbd1 at kbdmux0
 acpi0: <PTLTD   RSDT> on motherboard
 acpi0: Power Button (fixed)
 Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 cpu2: <ACPI CPU> on acpi0
 cpu3: <ACPI CPU> on acpi0
 acpi_button0: <Power Button> on acpi0
 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 pci0: <ACPI PCI bus> on pcib0
 pci0: <memory, RAM> at device 0.0 (no driver attached)
 isab0: <PCI-ISA bridge> port 0x1c00-0x1c7f at device 1.0 on pci0
 isa0: <ISA bus> on isab0
 pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
 ohci0: <OHCI (generic) USB controller> mem 0xc0040000-0xc0040fff irq 16 at device 2.0 on pci0
 ohci0: [GIANT-LOCKED]
 usb0: OHCI version 1.0, legacy support
 usb0: SMM does not respond, resetting
 usb0: <OHCI (generic) USB controller> on ohci0
 usb0: USB revision 1.0
 uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
 uhub0: 10 ports with 10 removable, self powered
 ehci0: <EHCI (generic) USB 2.0 controller> mem 0xc0041000-0xc00410ff irq 17 at device 2.1 on pci0
 ehci0: [GIANT-LOCKED]
 usb1: EHCI version 1.0
 usb1: companion controller, 10 ports each: usb0
 usb1: <EHCI (generic) USB 2.0 controller> on ehci0
 usb1: USB revision 2.0
 uhub1: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
 uhub1: 10 ports with 10 removable, self powered
 atapci0: <nVidia nForce MCP55 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3480-0x348f at device 4.0 on pci0
 ata0: <ATA channel 0> on atapci0
 ata1: <ATA channel 1> on atapci0
 pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0
 pci1: <ACPI PCI bus> on pcib1
 pcib2: <PCI-PCI bridge> at device 4.0 on pci1
 pci2: <PCI bus> on pcib2
 mly0: <Mylex AcceleRAID 170> mem 0xc0600000-0xc0601fff irq 20 at device 4.1 on pci1
 mly0: [GIANT-LOCKED]
 mly0: AcceleRAID 170  , 1 channel, firmware 6.00-7-00 (20001214), 32MB RAM
 pci0: <bridge> at device 8.0 (no driver attached)
 pci0: <bridge> at device 9.0 (no driver attached)
 pcib3: <ACPI PCI-PCI bridge> at device 13.0 on pci0
 pci3: <ACPI PCI bus> on pcib3
 pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
 pci4: <ACPI PCI bus> on pcib4
 fxp0: <Intel 82559 Pro/100 Ethernet> port 0x4000-0x403f mem 0xc0300000-0xc0300fff,0xc0200000-0xc02fffff irq 22 at device 9.0 on pci4
 miibus0: <MII bus> on fxp0
 inphy0: <i82555 10/100 media interface> on miibus0
 inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 fxp0: Ethernet address: 00:d0:b7:1c:80:7e
 pcib5: <ACPI PCI-PCI bridge> mem 0xc0100000-0xc010007f irq 21 at device 0.1 on pci3
 pci5: <ACPI PCI bus> on pcib5
 pcib6: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 pci128: <ACPI PCI bus> on pcib6
 pci128: <memory, RAM> at device 0.0 (no driver attached)
 pci128: <memory, RAM> at device 1.0 (no driver attached)
 pci128: <serial bus, SMBus> at device 1.1 (no driver attached)
 pcib7: <ACPI PCI-PCI bridge> at device 13.0 on pci128
 pci129: <ACPI PCI bus> on pcib7
 pcib8: <ACPI PCI-PCI bridge> at device 15.0 on pci128
 pci130: <ACPI PCI bus> on pcib8
 pci130: <display, VGA> at device 0.0 (no driver attached)
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
 sio0: type 16550A
 fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
 fdc0: [FAST]
 pmtimer0 on isa0
 orm0: <ISA Option ROMs> at iomem 0xc0000-0xcefff,0xcf000-0xd07ff on isa0
 ppc0: <Parallel port> at port 0x278-0x27f irq 7 on isa0
 ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode
 ppbus0: <Parallel port bus> on ppc0
 plip0: <PLIP network interface> on ppbus0
 lpt0: <Printer> on ppbus0
 lpt0: Interrupt-driven port
 ppi0: <Parallel I/O> on ppbus0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=0x300>
 sio1: configured irq 3 not in bitmap of probed irqs 0
 sio1: port may not be enabled
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
 Timecounters tick every 1.000 msec
 IP Filter: v4.1.13 initialized.  Default = block all, Logging = enabled
 ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, default to accept, logging disabled
 acd0: CDROM <NEC CD-ROM CD-3002A/C000> at ata0-master UDMA33
 da0 at mly0 bus 1 target 0 lun 0
 da0: <RAID 7 online > Fixed Direct Access SCSI-3 device
 da0: 135.168MB/s transfers
 da0: 34712MB (71090176 512 byte sectors: 255H 63S/T 4425C)
 da1 at mly0 bus 1 target 1 lun 0
 da1: <RAID 7 online > Fixed Direct Access SCSI-3 device
 da1: 135.168MB/s transfers
 da1: 34712MB (71090176 512 byte sectors: 255H 63S/T 4425C)
 SMP: AP CPU #3 Launched!
 SMP: AP CPU #2 Launched!
 SMP: AP CPU #1 Launched!
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da1:mly0:1:1:0): Sense Error Code 0x0
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da0:mly0:1:0:0): Sense Error Code 0x0
 Trying to mount root from ufs:/dev/da0s1a
 WARNING: / was not properly dismounted
 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da1:mly0:1:1:0): Sense Error Code 0x0
 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da1:mly0:1:1:0): Sense Error Code 0x0
 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da1:mly0:1:1:0): Sense Error Code 0x0
 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da1:mly0:1:1:0): Sense Error Code 0x0
 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
 (da1:mly0:1:1:0): Sense Error Code 0x0
 acquiring duplicate lock of same type: "vnode interlock"
  1st vnode interlock @ /usr/src/sys/kern/vfs_vnops.c:806
  2nd vnode interlock @ /usr/src/sys/kern/vfs_subr.c:2039
 KDB: stack backtrace:
 kdb_backtrace(3,c80e6780,c07a89f0,c07a89f0,c07678a4,...) at kdb_backtrace+0x29
 witness_checkorder(c7eb0d04,9,c072db32,7f7) at witness_checkorder+0x578
 _mtx_lock_flags(c7eb0d04,0,c072db32,7f7,c8055980,...) at _mtx_lock_flags+0x78
 vrefcnt(c7eb0c3c) at vrefcnt+0x20
 null_checkvp(c81dac3c,c071c808,215) at null_checkvp+0x56
 null_lock(ea872a68) at null_lock+0x66
 VOP_LOCK_APV(c0760b20,ea872a68) at VOP_LOCK_APV+0x87
 vn_lock(c81dac3c,1002,c80e6780,c81dac3c,c81dae60,...) at vn_lock+0xac
 nullfs_root(c8209000,2,ea872ae0,c80e6780,0,8,0,c07e5fe0,0,c072d3b0,407) at nullfs_root+0x26
 vfs_domount(c80e6780,c8055100,c8055700,d,c8055030,c0797ae0,0,c072d3b0,2bf) at vfs_domount+0x975
 vfs_donmount(c80e6780,d,c80ede80,c80ede80,0,...) at vfs_donmount+0x3f9
 nmount(c80e6780,ea872d04) at nmount+0x8b
 syscall(3b,3b,3b,bfbfe5f5,bfbfeea0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (378, FreeBSD ELF32, nmount), eip = 0x280bc11b, esp = 0xbfbfe5bc, ebp = 0xbfbfee38 ---
 Accounting enabled
 
 show pcpu:
 
 cpuid = 0
 curthread = 0xc7ce3300: pid 20 "swi6: Giant tasq"
 curpcb = 0xe68dbd90
 fpcurthread = none
 idlethread = 0xc7ce3a80: pid 13 "idle: cpu0"
 APIC ID = 0
 currentldt = 0x50
 spin locks held:
 
 show locks:
 
 exclusive sleep mutex Giant r = 0 (0xc0797ae0) locked @ /usr/src/sys/kern/kern_intr.c: 681
 
 show alllocks:
 
 Process 20 (swi6: Giant tasq) thread 0xc7ce3300 (100008)
 exclusive sleep mutex Giant r = 0 (0xc0797ae0) locked @ /usr/src/sys/kern/kern_intr.c: 681
 
 After getting a kernel dump I can obtain more information.
 
 --
 Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
 Phone: +7 4732 539880
 Fax:   +7 4732 531415         http://www.vsi.ru
 CenterTelecom Voronezh ISP    http://isp.vsi.ru
 

From: "Oleg Derevenetz" <oleg@vsi.ru>
To: <bug-followup@FreeBSD.org>, <doublef-ctm@yandex.ru>
Cc:  
Subject: Re: kern/104406: [ufs] Processes get stuck in "ufs" state under persistent CPU load
Date: Mon, 22 Oct 2007 16:24:53 +0400

 There are all required information from "Debugging Deadlocks" chapter of FreeBSD Developer's Handbook:
 
 db> show pcpu
 cpuid        = 1
 curthread    = 0xc7ce3900: pid 12 "idle: cpu1"
 curpcb       = 0xe68e7d90
 fpcurthread  = none
 idlethread   = 0xc7ce3900: pid 12 "idle: cpu1"
 APIC ID      = 1
 currentldt   = 0x50
 spin locks held:
 exclusive spin mutex sio r = 0 (0xc07f61a0) locked @ /usr/src/sys/dev/sio/sio.c:1390
 
 db> show allpcpu
 Current CPU: 1
 
 cpuid        = 0
 curthread    = 0xc7ce3a80: pid 13 "idle: cpu0"
 curpcb       = 0xe68ead90
 fpcurthread  = none
 idlethread   = 0xc7ce3a80: pid 13 "idle: cpu0"
 APIC ID      = 0
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 1
 curthread    = 0xc7ce3900: pid 12 "idle: cpu1"
 curpcb       = 0xe68e7d90
 fpcurthread  = none
 idlethread   = 0xc7ce3900: pid 12 "idle: cpu1"
 APIC ID      = 1
 currentldt   = 0x50
 spin locks held:
 exclusive spin mutex sio r = 0 (0xc07f61a0) locked @ /usr/src/sys/dev/sio/sio.c:1390
 
 cpuid        = 2
 curthread    = 0xc7ce3780: pid 11 "idle: cpu2"
 curpcb       = 0xe68e4d90
 fpcurthread  = none
 idlethread   = 0xc7ce3780: pid 11 "idle: cpu2"
 APIC ID      = 2
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 3
 curthread    = 0xc7ce3600: pid 10 "idle: cpu3"
 curpcb       = 0xe68e1d90
 fpcurthread  = none
 idlethread   = 0xc7ce3600: pid 10 "idle: cpu3"
 APIC ID      = 3
 currentldt   = 0x50
 spin locks held:
 
 db> show locks
 exclusive spin mutex sio r = 0 (0xc07f61a0) locked @ /usr/src/sys/dev/sio/sio.c:1390
 
 db> show alllocks
 Process 938 (sendmail) thread 0xc82e8300 (100115)
 shared sx acct_sx r = 0 (0xc0795f00) locked @ /usr/src/sys/kern/kern_acct.c:308
 Process 937 (sendmail) thread 0xc83c4780 (100154)
 shared sx acct_sx r = 0 (0xc0795f00) locked @ /usr/src/sys/kern/kern_acct.c:308
 
 db> show lockedvnods
 Locked vnodes
 
 0xc7eb1ae0: tag ufs, type VREG
     usecount 3, writecount 0, refcount 4 mountedhere 0
     flags (VV_TEXT)
     v_object 0xc106d318 ref 1 pages 25
      lock type ufs: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc0559c15 at lockmgr+0x4ed
 #1 0xc0673b06 at ffs_lock+0x76
 #2 0xc06e9797 at VOP_LOCK_APV+0x87
 #3 0xc05c9444 at vn_lock+0xac
 #4 0xc05bd2ea at vget+0xbe
 #5 0xc06725b1 at ffs_sync+0x1c1
 #6 0xc05bee5c at sync_fsync+0x164
 #7 0xc06e8b27 at VOP_FSYNC_APV+0x9b
 #8 0xc05bcace at sync_vnode+0x106
 #9 0xc05bcd49 at sched_sync+0x1ed
 #10 0xc0550cc4 at fork_exit+0xa0
 #11 0xc06c23bc at fork_trampoline+0x8
 
         ino 70723, on dev da0s1a
 
 0xc817a570: tag syncer, type VNON
     usecount 1, writecount 0, refcount 2 mountedhere 0
     flags ()
      lock type syncer: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc0559c15 at lockmgr+0x4ed
 #1 0xc05b4189 at vop_stdlock+0x21
 #2 0xc06e9797 at VOP_LOCK_APV+0x87
 #3 0xc05c9444 at vn_lock+0xac
 #4 0xc05bcaab at sync_vnode+0xe3
 #5 0xc05bcd49 at sched_sync+0x1ed
 #6 0xc0550cc4 at fork_exit+0xa0
 #7 0xc06c23bc at fork_trampoline+0x8
 
 
 0xc8228000: tag ufs, type VDIR
     usecount 1, writecount 0, refcount 2 mountedhere 0
     flags ()
     v_object 0xc814e4a4 ref 0 pages 0
      lock type ufs: EXCL (count 1) by thread 0xc8105d80 (pid 939)#0 0xc0559c15 at lockmgr+0x4ed
 #1 0xc0673b06 at ffs_lock+0x76
 #2 0xc06e9797 at VOP_LOCK_APV+0x87
 #3 0xc05c9444 at vn_lock+0xac
 #4 0xc05bd2ea at vget+0xbe
 #5 0xc05b1a66 at cache_lookup+0x34a
 #6 0xc05b1f9a at vfs_cache_lookup+0x92
 #7 0xc06e7857 at VOP_LOOKUP_APV+0x87
 #8 0xc05b61d0 at lookup+0x4a4
 #9 0xc05b5a8a at namei+0x39a
 #10 0xc05c8443 at vn_open_cred+0x26f
 #11 0xc05c81d2 at vn_open+0x1e
 #12 0xc05c1879 at kern_open+0x231
 #13 0xc05c1612 at open+0x1a
 #14 0xc06d6d4b at syscall+0x25b
 #15 0xc06c23af at Xint0x80_syscall+0x1f
 
         ino 282624, on dev da0s1e
 
 0xc8302ae0: tag ufs, type VREG
     usecount 1, writecount 1, refcount 3 mountedhere 0
     flags ()
     v_object 0xc8233840 ref 0 pages 10
      lock type ufs: EXCL (count 1) by thread 0xc8087a80 (pid 548)#0 0xc0559c15 at lockmgr+0x4ed
 #1 0xc0673b06 at ffs_lock+0x76
 #2 0xc06e9797 at VOP_LOCK_APV+0x87
 #3 0xc05c9444 at vn_lock+0xac
 #4 0xc05c8de0 at vn_write+0x138
 #5 0xc058ba28 at dofilewrite+0x7c
 #6 0xc058b8c7 at kern_writev+0x3b
 #7 0xc058b870 at writev+0x30
 #8 0xc06d6d4b at syscall+0x25b
 #9 0xc06c23af at Xint0x80_syscall+0x1f
 
         ino 259099, on dev da0s1e
 
 0xc830bc3c: tag ufs, type VREG
     usecount 1, writecount 1, refcount 4 mountedhere 0
     flags ()
     v_object 0xc81578c4 ref 0 pages 12
      lock type ufs: EXCL (count 1) by thread 0xc83c4780 (pid 937) with 1 pending#0 0xc0559c15 at lockmgr+0x4ed
 #1 0xc0673b06 at ffs_lock+0x76
 #2 0xc06e9797 at VOP_LOCK_APV+0x87
 #3 0xc05c9444 at vn_lock+0xac
 #4 0xc05c87dd at vn_rdwr+0x85
 #5 0xc053d3bf at acct_process+0x31f
 #6 0xc054eb82 at exit1+0x716
 #7 0xc054e46c at exit1+0
 #8 0xc06d6d4b at syscall+0x25b
 #9 0xc06c23af at Xint0x80_syscall+0x1f
 
         ino 23555, on dev da0s1e
 
 0xc837915c: tag ufs, type VREG
     usecount 0, writecount 0, refcount 4 mountedhere 0
     flags ()
     v_object 0xc8282210 ref 0 pages 1
      lock type ufs: EXCL (count 1) by thread 0xc83c4480 (pid 936) with 1 pending#0 0xc0559c15 at lockmgr+0x4ed
 #1 0xc0673b06 at ffs_lock+0x76
 #2 0xc06e9797 at VOP_LOCK_APV+0x87
 #3 0xc05c9444 at vn_lock+0xac
 #4 0xc05c8708 at vn_close+0x5c
 #5 0xc05c9582 at vn_closefile+0xca
 #6 0xc05488ac at fdrop_locked+0xb0
 #7 0xc05487f4 at fdrop+0x24
 #8 0xc05472e3 at closef+0x367
 #9 0xc0544dd2 at kern_close+0x1aa
 #10 0xc0544c24 at close+0x10
 #11 0xc06d6d4b at syscall+0x25b
 #12 0xc06c23af at Xint0x80_syscall+0x1f
 
         ino 282642, on dev da0s1e
 
 0xc9feb984: tag ufs, type VREG
     usecount 1, writecount 1, refcount 19 mountedhere 0
     flags ()
     v_object 0xc9fee210 ref 0 pages 68
      lock type ufs: EXCL (count 1) by thread 0xc8193c00 (pid 844)#0 0xc0559c15 at lockmgr+0x4ed
 #1 0xc0673b06 at ffs_lock+0x76
 #2 0xc06e9797 at VOP_LOCK_APV+0x87
 #3 0xc05c9444 at vn_lock+0xac
 #4 0xc05c8de0 at vn_write+0x138
 #5 0xc058ba28 at dofilewrite+0x7c
 #6 0xc058b8c7 at kern_writev+0x3b
 #7 0xc058b7ed at write+0x45
 #8 0xc06d6d4b at syscall+0x25b
 #9 0xc06c23af at Xint0x80_syscall+0x1f
 
         ino 4098799, on dev da1s1d
 
 db> alltrace
 
 Tracing command sshd pid 939 tid 100071 td 0xc8105d80
 sched_switch(c8105d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c83791b4,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c83791b4,0,c8379224,b7,c072d198,...) at sleepq_wait+0x46
 msleep(c83791b4,c07969b0,50,c072b884,0,...) at msleep+0x27d
 acquire(ea87e820,40,60000,c8105d80,0,...) at acquire+0x76
 lockmgr(c83791b4,2002,c8379224,c8105d80) at lockmgr+0x44e
 ffs_lock(ea87e878) at ffs_lock+0x76
 VOP_LOCK_APV(c077a640,ea87e878) at VOP_LOCK_APV+0x87
 vn_lock(c837915c,2002,c8105d80,c837915c) at vn_lock+0xac
 vget(c837915c,2002,c8105d80) at vget+0xbe
 cache_lookup(c8228000,ea87ebe0,ea87ebf4) at cache_lookup+0x34a
 vfs_cache_lookup(ea87e9bc) at vfs_cache_lookup+0x92
 VOP_LOOKUP_APV(c077a640,ea87e9bc) at VOP_LOOKUP_APV+0x87
 lookup(ea87ebcc) at lookup+0x4a4
 namei(ea87ebcc) at namei+0x39a
 vn_open_cred(ea87ebcc,ea87eccc,0,c82fe380,3,...) at vn_open_cred+0x26f
 vn_open(ea87ebcc,ea87eccc,0,3,c0797aa0,...) at vn_open+0x1e
 kern_open(c8105d80,280915c0,0,1,0,...) at kern_open+0x231
 open(c8105d80,ea87ed04) at open+0x1a
 syscall(3b,3b,3b,2809d000,804aa41,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (5, FreeBSD ELF32, open), eip = 0x2807ca7f, esp = 0xbfbfec1c, ebp = 0xbfbfecc8 ---
 
 Tracing command sendmail pid 938 tid 100115 td 0xc82e8300
 sched_switch(c82e8300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c830bc94,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c830bc94,0,c830bd04,b7,c071f3eb,...) at sleepq_wait+0x46
 msleep(c830bc94,c07975ec,50,c072b884,0,...) at msleep+0x27d
 acquire(ea951ac0,40,60000,c82e8300,0,...) at acquire+0x76
 lockmgr(c830bc94,3002,c830bd04,c82e8300) at lockmgr+0x44e
 ffs_lock(ea951b18) at ffs_lock+0x76
 VOP_LOCK_APV(c077a640,ea951b18) at VOP_LOCK_APV+0x87
 vn_lock(c830bc3c,1002,c82e8300) at vn_lock+0xac
 vn_rdwr(1,c830bc3c,ea951c34,30,0,0,1,3,c7ce0780,0,0,c82e8300) at vn_rdwr+0x85
 acct_process(c82e8300,c0797c20,c07209a0,163,c8364648,...) at acct_process+0x31f
 exit1(c82e8300,0,ea951d30,c06d6d4b,c82e8300,...) at exit1+0x716
 exit1(c82e8300,ea951d04) at exit1
 syscall(3b,3b,3b,0,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x282fc0fb, esp = 0xbfbfcb1c, ebp = 0xbfbfcb38 ---
 
 Tracing command sendmail pid 937 tid 100154 td 0xc83c4780
 sched_switch(c83c4780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(dc011d14,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(dc011d14,0,c830bd04,b7,c071f3eb,...) at sleepq_wait+0x46
 msleep(dc011d14,c0796dc4,50,c072c4b4,0,...) at msleep+0x27d
 acquire(ea9db860,120,60000,c83c4780,0,...) at acquire+0x76
 lockmgr(dc011d14,202122,c830bd04,c83c4780) at lockmgr+0x44e
 getblk(c830bc3c,10,0,4000,0,...) at getblk+0x12f
 breadn(c830bc3c,10,0,4000,0,...) at breadn+0x31
 bread(c830bc3c,10,0,4000,0,ea9db970) at bread+0x20
 ffs_balloc_ufs2(c830bc3c,41a60,0,30,c7ce0780,...) at ffs_balloc_ufs2+0x1849
 ffs_write(ea9dbb54) at ffs_write+0x2c0
 VOP_WRITE_APV(c077a640,ea9dbb54) at VOP_WRITE_APV+0x132
 vn_rdwr(1,c830bc3c,ea9dbc34,30,0,0,1,3,c7ce0780,0,0,c83c4780) at vn_rdwr+0x142
 acct_process(c83c4780,c0797c20,c07209a0,163,c865b860,...) at acct_process+0x31f
 exit1(c83c4780,0,ea9dbd30,c06d6d4b,c83c4780,...) at exit1+0x716
 exit1(c83c4780,ea9dbd04) at exit1
 syscall(3b,3b,3b,0,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x282fc0fb, esp = 0xbfbfbf4c, ebp = 0xbfbfbf68 ---
 
 Tracing command sh pid 936 tid 100156 td 0xc83c4480
 sched_switch(c83c4480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(dbfd0b84,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(dbfd0b84,0,c817a8f0,b7,c072e794,...) at sleepq_wait+0x46
 msleep(dbfd0b84,c0796af4,50,c072c4b4,0,...) at msleep+0x27d
 acquire(ea9d59bc,120,60000,c83c4480,0,...) at acquire+0x76
 lockmgr(dbfd0b84,202122,c817a8f0,c83c4480) at lockmgr+0x44e
 getblk(c817a828,44ea60,0,4000,0,...) at getblk+0x12f
 breadn(c817a828,44ea60,0,4000,0,...) at breadn+0x31
 bread(c817a828,44ea60,0,4000,0,ea9d5aa8) at bread+0x20
 ffs_update(c837915c,0) at ffs_update+0x22c
 ufs_inactive(ea9d5b04) at ufs_inactive+0x2a6
 VOP_INACTIVE_APV(c077a640,ea9d5b04) at VOP_INACTIVE_APV+0x9b
 vinactive(c837915c,c83c4480) at vinactive+0x8d
 vput(c837915c,c8227000,c078b740,c837915c,1,...) at vput+0x1d3
 vn_close(c837915c,1,c8593780,c83c4480) at vn_close+0x97
 vn_closefile(c812e2d0,c83c4480) at vn_closefile+0xca
 fdrop_locked(c812e2d0,c83c4480,c7ccba0c,0,c072016e,...) at fdrop_locked+0xb0
 fdrop(c812e2d0,c83c4480,6ac,c079efa0,0,...) at fdrop+0x24
 closef(c812e2d0,c83c4480,0,c83c4480,0,...) at closef+0x367
 kern_close(c83c4480,3,ea9d5d30,c06d6d4b,c83c4480,...) at kern_close+0x1aa
 close(c83c4480,ea9d5d04) at close+0x10
 syscall(3b,3b,3b,2808e000,3,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (6, FreeBSD ELF32, close), eip = 0x2806ba9f, esp = 0xbfbfebcc, ebp = 0xbfbfec78 ---
 
 Tracing command cron pid 935 tid 100160 td 0xc8366d80
 sched_switch(c8366d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8121330) at sleepq_switch+0xc1
 sleepq_wait_sig(c8121330) at sleepq_wait_sig+0x1d
 msleep(c8121330,c81214a0,14c,c0728d39,0) at msleep+0x26e
 pipe_read(c8750ca8,ea9c9cbc,c810e400,0,c8366d80) at pipe_read+0x377
 dofileread(c8366d80,5,c8750ca8,ea9c9cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8366d80,5,ea9c9cbc,805a000,1000,...) at kern_readv+0x36
 read(c8366d80,ea9c9d04) at read+0x45
 syscall(3b,3b,3b,28171040,28171040,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2814b953, esp = 0xbfbfe65c, ebp = 0xbfbfe678 ---
 
 Tracing command mc pid 844 tid 100148 td 0xc8193c00
 sched_switch(c8193c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5d04,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c07e5d04,0,c072be5a,180,c072e794,...) at sleepq_wait+0x46
 msleep(c07e5d04,c07e5d20,44,c072be76,0) at msleep+0x27d
 waitrunningbufspace(dbd8a778,10,0,ea90d9a4,c05acc09,...) at waitrunningbufspace+0x4a
 bufwrite(dbd502cc) at bufwrite+0x191
 bawrite(dbd502cc,c9feba4c,c1e75650,c9feba4c,c9feba4c,...) at bawrite+0x51
 cluster_wbuild(c9feb984,4000,8,0,8) at cluster_wbuild+0x61c
 cluster_write(c9feb984,dbd8a778,40000,0,50) at cluster_write+0x503
 ffs_write(ea90dbf4) at ffs_write+0x523
 VOP_WRITE_APV(c077a640,ea90dbf4) at VOP_WRITE_APV+0x132
 vn_write(c8738af8,ea90dcbc,c810b200,0,c8193c00) at vn_write+0x1ea
 dofilewrite(c8193c00,7,c8738af8,ea90dcbc,ffffffff,...) at dofilewrite+0x7c
 kern_writev(c8193c00,7,ea90dcbc,86c4130,570,...) at kern_writev+0x3b
 write(c8193c00,ea90dd04) at write+0x45
 syscall(3b,3b,3b,16a0,7,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (4, FreeBSD ELF32, write), eip = 0x2836d933, esp = 0xbfbfde6c, ebp = 0xbfbfde98 ---
 
 Tracing command csh pid 841 tid 100143 td 0xc83c5a80
 sched_switch(c83c5a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c852f034) at sleepq_switch+0xc1
 sleepq_wait_sig(c852f034) at sleepq_wait_sig+0x1d
 msleep(c852f034,c852f068,168,c0724047,0) at msleep+0x26e
 kern_sigsuspend(c83c5a80,2,0,0,0,...) at kern_sigsuspend+0xa3
 sigsuspend(c83c5a80,ea9ffd04) at sigsuspend+0x33
 syscall(3b,3b,3b,8299008,bfbf1ad0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1abc, ebp = 0xbfbf1ae8 ---
 
 Tracing command su pid 840 tid 100146 td 0xc8193d80
 sched_switch(c8193d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c852f648) at sleepq_switch+0xc1
 sleepq_wait_sig(c852f648) at sleepq_wait_sig+0x1d
 msleep(c852f648,c852f6b0,15c,c0729071,0) at msleep+0x26e
 kern_wait(c8193d80,349,ea910c84,2,0) at kern_wait+0x6bb
 wait4(c8193d80,ea910d04) at wait4+0x2a
 syscall(3b,3b,3b,bfbfe794,bfbfe810,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x280f451f, esp = 0xbfbfe70c, ebp = 0xbfbfe728 ---
 
 Tracing command csh pid 835 tid 100098 td 0xc8102480
 sched_switch(c8102480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c82e6034) at sleepq_switch+0xc1
 sleepq_wait_sig(c82e6034) at sleepq_wait_sig+0x1d
 msleep(c82e6034,c82e6068,168,c0724047,0) at msleep+0x26e
 kern_sigsuspend(c8102480,2,0,0,0,...) at kern_sigsuspend+0xa3
 sigsuspend(c8102480,ea84ed04) at sigsuspend+0x33
 syscall(3b,3b,3b,81c1308,bfbf1bd0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1bbc, ebp = 0xbfbf1be8 ---
 
 Tracing command sshd pid 834 tid 100042 td 0xc7e05480
 sched_switch(c7e05480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
 kern_select(c7e05480,9,807bb60,807bb70,0,...) at kern_select+0x4bf
 select(c7e05480,e8386d04) at select+0x44
 syscall(3b,3b,3b,0,bfbfe11c,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe0ac, ebp = 0xbfbfe0f8 ---
 
 Tracing command sshd pid 831 tid 100074 td 0xc8105900
 sched_switch(c8105900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8275370) at sleepq_switch+0xc1
 sleepq_wait_sig(c8275370) at sleepq_wait_sig+0x1d
 msleep(c8275370,c827533c,158,c072b38c,0,...) at msleep+0x26e
 sbwait(c8275318,ea875c0c,c058ae7c,4,0,...) at sbwait+0x4b
 soreceive(c82752c8,0,ea875cbc,0,0,0) at soreceive+0x2ee
 soo_read(c8111c18,ea875cbc,c82e1380,0,c8105900) at soo_read+0x41
 dofileread(c8105900,5,c8111c18,ea875cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8105900,5,ea875cbc,bfbfe108,4,...) at kern_readv+0x36
 read(c8105900,ea875d04) at read+0x45
 syscall(3b,3b,3b,4,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2834d953, esp = 0xbfbfe0ac, ebp = 0xbfbfe0d8 ---
 
 Tracing command httpd pid 830 tid 100087 td 0xc818a780
 sched_switch(c818a780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b0900) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b0900) at sleepq_wait_sig+0x1d
 msleep(c80b0900,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b0900,16,ffffffff,ffffffff,ea8c8cb0,...) at lf_setlock+0x181
 lf_advlock(ea8c8c90,c84cfd9c,0,0,ea8c8c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8c8c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8c8c90) at VOP_ADVLOCK_APV+0xa9
 flock(c818a780,ea8c8d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 829 tid 100084 td 0xc818ac00
 sched_switch(c818ac00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b0440) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b0440) at sleepq_wait_sig+0x1d
 msleep(c80b0440,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b0440,16,ffffffff,ffffffff,ea8d1cb0,...) at lf_setlock+0x181
 lf_advlock(ea8d1c90,c84cfd9c,0,0,ea8d1c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8d1c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8d1c90) at VOP_ADVLOCK_APV+0xa9
 flock(c818ac00,ea8d1d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 828 tid 100063 td 0xc815ea80
 sched_switch(c815ea80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c815b3c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c815b3c0) at sleepq_wait_sig+0x1d
 msleep(c815b3c0,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c815b3c0,16,ffffffff,ffffffff,ea8a3cb0,...) at lf_setlock+0x181
 lf_advlock(ea8a3c90,c84cfd9c,0,0,ea8a3c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8a3c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8a3c90) at VOP_ADVLOCK_APV+0xa9
 flock(c815ea80,ea8a3d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 827 tid 100080 td 0xc818e300
 sched_switch(c818e300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8067540) at sleepq_switch+0xc1
 sleepq_wait_sig(c8067540) at sleepq_wait_sig+0x1d
 msleep(c8067540,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c8067540,16,ffffffff,ffffffff,ea8ddcb0,...) at lf_setlock+0x181
 lf_advlock(ea8ddc90,c84cfd9c,0,0,ea8ddc7c,...) at lf_advlock+0x203
 ufs_advlock(ea8ddc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8ddc90) at VOP_ADVLOCK_APV+0xa9
 flock(c818e300,ea8ddd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 826 tid 100043 td 0xc7e05300
 sched_switch(c7e05300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8067c80) at sleepq_switch+0xc1
 sleepq_wait_sig(c8067c80) at sleepq_wait_sig+0x1d
 msleep(c8067c80,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c8067c80,16,ffffffff,ffffffff,e8383cb0,...) at lf_setlock+0x181
 lf_advlock(e8383c90,c84cfd9c,0,0,e8383c7c,...) at lf_advlock+0x203
 ufs_advlock(e8383c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,e8383c90) at VOP_ADVLOCK_APV+0xa9
 flock(c7e05300,e8383d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 825 tid 100076 td 0xc8105600
 sched_switch(c8105600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b0940) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b0940) at sleepq_wait_sig+0x1d
 msleep(c80b0940,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b0940,16,ffffffff,ffffffff,ea86fcb0,...) at lf_setlock+0x181
 lf_advlock(ea86fc90,c84cfd9c,0,0,ea86fc7c,...) at lf_advlock+0x203
 ufs_advlock(ea86fc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea86fc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8105600,ea86fd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 824 tid 100064 td 0xc815e900
 sched_switch(c815e900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b0980) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b0980) at sleepq_wait_sig+0x1d
 msleep(c80b0980,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b0980,16,ffffffff,ffffffff,ea8a0cb0,...) at lf_setlock+0x181
 lf_advlock(ea8a0c90,c84cfd9c,0,0,ea8a0c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8a0c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8a0c90) at VOP_ADVLOCK_APV+0xa9
 flock(c815e900,ea8a0d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 823 tid 100110 td 0xc82e8a80
 sched_switch(c82e8a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c812a540) at sleepq_switch+0xc1
 sleepq_wait_sig(c812a540) at sleepq_wait_sig+0x1d
 msleep(c812a540,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c812a540,16,ffffffff,ffffffff,ea960cb0,...) at lf_setlock+0x181
 lf_advlock(ea960c90,c84cfd9c,0,0,ea960c7c,...) at lf_advlock+0x203
 ufs_advlock(ea960c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea960c90) at VOP_ADVLOCK_APV+0xa9
 flock(c82e8a80,ea960d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 822 tid 100105 td 0xc8087900
 sched_switch(c8087900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c812bd40) at sleepq_switch+0xc1
 sleepq_wait_sig(c812bd40) at sleepq_wait_sig+0x1d
 msleep(c812bd40,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c812bd40,16,ffffffff,ffffffff,e864dcb0,...) at lf_setlock+0x181
 lf_advlock(e864dc90,c84cfd9c,0,0,e864dc7c,...) at lf_advlock+0x203
 ufs_advlock(e864dc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,e864dc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8087900,e864dd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 821 tid 100039 td 0xc7e05900
 sched_switch(c7e05900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b09c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b09c0) at sleepq_wait_sig+0x1d
 msleep(c80b09c0,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b09c0,16,ffffffff,ffffffff,e838fcb0,...) at lf_setlock+0x181
 lf_advlock(e838fc90,c84cfd9c,0,0,e838fc7c,...) at lf_advlock+0x203
 ufs_advlock(e838fc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,e838fc90) at VOP_ADVLOCK_APV+0xa9
 flock(c7e05900,e838fd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 820 tid 100089 td 0xc818a480
 sched_switch(c818a480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80af540) at sleepq_switch+0xc1
 sleepq_wait_sig(c80af540) at sleepq_wait_sig+0x1d
 msleep(c80af540,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80af540,16,ffffffff,ffffffff,ea8c2cb0,...) at lf_setlock+0x181
 lf_advlock(ea8c2c90,c84cfd9c,0,0,ea8c2c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8c2c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8c2c90) at VOP_ADVLOCK_APV+0xa9
 flock(c818a480,ea8c2d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 819 tid 100092 td 0xc818a000
 sched_switch(c818a000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8067480) at sleepq_switch+0xc1
 sleepq_wait_sig(c8067480) at sleepq_wait_sig+0x1d
 msleep(c8067480,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c8067480,16,ffffffff,ffffffff,ea8b9cb0,...) at lf_setlock+0x181
 lf_advlock(ea8b9c90,c84cfd9c,0,0,ea8b9c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8b9c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8b9c90) at VOP_ADVLOCK_APV+0xa9
 flock(c818a000,ea8b9d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 818 tid 100106 td 0xc8087780
 sched_switch(c8087780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c815b4c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c815b4c0) at sleepq_wait_sig+0x1d
 msleep(c815b4c0,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c815b4c0,16,ffffffff,ffffffff,e864acb0,...) at lf_setlock+0x181
 lf_advlock(e864ac90,c84cfd9c,0,0,e864ac7c,...) at lf_advlock+0x203
 ufs_advlock(e864ac90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,e864ac90) at VOP_ADVLOCK_APV+0xa9
 flock(c8087780,e864ad04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 817 tid 100079 td 0xc818e480
 sched_switch(c818e480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8067240) at sleepq_switch+0xc1
 sleepq_wait_sig(c8067240) at sleepq_wait_sig+0x1d
 msleep(c8067240,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c8067240,16,ffffffff,ffffffff,ea8e0cb0,...) at lf_setlock+0x181
 lf_advlock(ea8e0c90,c84cfd9c,0,0,ea8e0c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8e0c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8e0c90) at VOP_ADVLOCK_APV+0xa9
 flock(c818e480,ea8e0d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 816 tid 100041 td 0xc7e05600
 sched_switch(c7e05600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b0a00) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b0a00) at sleepq_wait_sig+0x1d
 msleep(c80b0a00,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b0a00,16,ffffffff,ffffffff,e8389cb0,...) at lf_setlock+0x181
 lf_advlock(e8389c90,c84cfd9c,0,0,e8389c7c,...) at lf_advlock+0x203
 ufs_advlock(e8389c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,e8389c90) at VOP_ADVLOCK_APV+0xa9
 flock(c7e05600,e8389d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 815 tid 100075 td 0xc8105780
 sched_switch(c8105780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c815b580) at sleepq_switch+0xc1
 sleepq_wait_sig(c815b580) at sleepq_wait_sig+0x1d
 msleep(c815b580,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c815b580,16,ffffffff,ffffffff,ea872cb0,...) at lf_setlock+0x181
 lf_advlock(ea872c90,c84cfd9c,0,0,ea872c7c,...) at lf_advlock+0x203
 ufs_advlock(ea872c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea872c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8105780,ea872d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 814 tid 100078 td 0xc818e600
 sched_switch(c818e600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8067180) at sleepq_switch+0xc1
 sleepq_wait_sig(c8067180) at sleepq_wait_sig+0x1d
 msleep(c8067180,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c8067180,16,ffffffff,ffffffff,ea8e3cb0,...) at lf_setlock+0x181
 lf_advlock(ea8e3c90,c84cfd9c,0,0,ea8e3c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8e3c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8e3c90) at VOP_ADVLOCK_APV+0xa9
 flock(c818e600,ea8e3d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 813 tid 100095 td 0xc8195600
 sched_switch(c8195600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b0a40) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b0a40) at sleepq_wait_sig+0x1d
 msleep(c80b0a40,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b0a40,16,ffffffff,ffffffff,ea91fcb0,...) at lf_setlock+0x181
 lf_advlock(ea91fc90,c84cfd9c,0,0,ea91fc7c,...) at lf_advlock+0x203
 ufs_advlock(ea91fc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea91fc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8195600,ea91fd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 812 tid 100120 td 0xc8195a80
 sched_switch(c8195a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c815b6c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c815b6c0) at sleepq_wait_sig+0x1d
 msleep(c815b6c0,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c815b6c0,16,ffffffff,ffffffff,ea928cb0,...) at lf_setlock+0x181
 lf_advlock(ea928c90,c84cfd9c,0,0,ea928c7c,...) at lf_advlock+0x203
 ufs_advlock(ea928c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea928c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8195a80,ea928d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 811 tid 100051 td 0xc8087480
 sched_switch(c8087480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8066e00) at sleepq_switch+0xc1
 sleepq_wait_sig(c8066e00) at sleepq_wait_sig+0x1d
 msleep(c8066e00,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c8066e00,16,ffffffff,ffffffff,e8644cb0,...) at lf_setlock+0x181
 lf_advlock(e8644c90,c84cfd9c,0,0,e8644c7c,...) at lf_advlock+0x203
 ufs_advlock(e8644c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,e8644c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8087480,e8644d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 810 tid 100117 td 0xc82e8000
 sched_switch(c82e8000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b0a80) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b0a80) at sleepq_wait_sig+0x1d
 msleep(c80b0a80,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b0a80,16,ffffffff,ffffffff,ea94bcb0,...) at lf_setlock+0x181
 lf_advlock(ea94bc90,c84cfd9c,0,0,ea94bc7c,...) at lf_advlock+0x203
 ufs_advlock(ea94bc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea94bc90) at VOP_ADVLOCK_APV+0xa9
 flock(c82e8000,ea94bd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 809 tid 100052 td 0xc8087300
 sched_switch(c8087300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c815b8c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c815b8c0) at sleepq_wait_sig+0x1d
 msleep(c815b8c0,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c815b8c0,16,ffffffff,ffffffff,e8641cb0,...) at lf_setlock+0x181
 lf_advlock(e8641c90,c84cfd9c,0,0,e8641c7c,...) at lf_advlock+0x203
 ufs_advlock(e8641c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,e8641c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8087300,e8641d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 808 tid 100038 td 0xc7e05a80
 sched_switch(c7e05a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8066c80) at sleepq_switch+0xc1
 sleepq_wait_sig(c8066c80) at sleepq_wait_sig+0x1d
 msleep(c8066c80,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c8066c80,16,ffffffff,ffffffff,e8392cb0,...) at lf_setlock+0x181
 lf_advlock(e8392c90,c84cfd9c,0,0,e8392c7c,...) at lf_advlock+0x203
 ufs_advlock(e8392c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,e8392c90) at VOP_ADVLOCK_APV+0xa9
 flock(c7e05a80,e8392d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 807 tid 100070 td 0xc815e000
 sched_switch(c815e000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b0ac0) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b0ac0) at sleepq_wait_sig+0x1d
 msleep(c80b0ac0,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b0ac0,16,ffffffff,ffffffff,ea88ecb0,...) at lf_setlock+0x181
 lf_advlock(ea88ec90,c84cfd9c,0,0,ea88ec7c,...) at lf_advlock+0x203
 ufs_advlock(ea88ec90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea88ec90) at VOP_ADVLOCK_APV+0xa9
 flock(c815e000,ea88ed04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 806 tid 100077 td 0xc818e780
 sched_switch(c818e780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c815a440) at sleepq_switch+0xc1
 sleepq_wait_sig(c815a440) at sleepq_wait_sig+0x1d
 msleep(c815a440,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c815a440,16,ffffffff,ffffffff,ea8e6cb0,...) at lf_setlock+0x181
 lf_advlock(ea8e6c90,c84cfd9c,0,0,ea8e6c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8e6c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8e6c90) at VOP_ADVLOCK_APV+0xa9
 flock(c818e780,ea8e6d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 805 tid 100083 td 0xc818ad80
 sched_switch(c818ad80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c815a000) at sleepq_switch+0xc1
 sleepq_wait_sig(c815a000) at sleepq_wait_sig+0x1d
 msleep(c815a000,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c815a000,16,ffffffff,ffffffff,ea8d4cb0,...) at lf_setlock+0x181
 lf_advlock(ea8d4c90,c84cfd9c,0,0,ea8d4c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8d4c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8d4c90) at VOP_ADVLOCK_APV+0xa9
 flock(c818ad80,ea8d4d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 804 tid 100067 td 0xc815e480
 sched_switch(c815e480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80b0b00) at sleepq_switch+0xc1
 sleepq_wait_sig(c80b0b00) at sleepq_wait_sig+0x1d
 msleep(c80b0b00,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c80b0b00,16,ffffffff,ffffffff,ea897cb0,...) at lf_setlock+0x181
 lf_advlock(ea897c90,c84cfd9c,0,0,ea897c7c,...) at lf_advlock+0x203
 ufs_advlock(ea897c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea897c90) at VOP_ADVLOCK_APV+0xa9
 flock(c815e480,ea897d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 803 tid 100112 td 0xc82e8780
 sched_switch(c82e8780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8066c40) at sleepq_switch+0xc1
 sleepq_wait_sig(c8066c40) at sleepq_wait_sig+0x1d
 msleep(c8066c40,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c8066c40,16,ffffffff,ffffffff,ea95acb0,...) at lf_setlock+0x181
 lf_advlock(ea95ac90,c84cfd9c,0,0,ea95ac7c,...) at lf_advlock+0x203
 ufs_advlock(ea95ac90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea95ac90) at VOP_ADVLOCK_APV+0xa9
 flock(c82e8780,ea95ad04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 802 tid 100088 td 0xc818a600
 sched_switch(c818a600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c815bbc0) at sleepq_switch+0xc1
 sleepq_wait_sig(c815bbc0) at sleepq_wait_sig+0x1d
 msleep(c815bbc0,0,168,c0765c84,0) at msleep+0x26e
 lf_setlock(c815bbc0,16,ffffffff,ffffffff,ea8c5cb0,...) at lf_setlock+0x181
 lf_advlock(ea8c5c90,c84cfd9c,0,0,ea8c5c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8c5c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c077a640,ea8c5c90) at VOP_ADVLOCK_APV+0xa9
 flock(c818a600,ea8c5d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 801 tid 100082 td 0xc818e000
 sched_switch(c818e000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,3b7,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
 poll(c818e000,ea8d7d04) at poll+0x3b9
 syscall(3b,3b,3b,2,28,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x282c3a87, esp = 0xbfbfebcc, ebp = 0xbfbfec18 ---
 
 Tracing command getty pid 800 tid 100093 td 0xc815ed80
 sched_switch(c815ed80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e95410) at sleepq_switch+0xc1
 sleepq_wait_sig(c7e95410) at sleepq_wait_sig+0x1d
 msleep(c7e95410,0,159,c072a5af,0) at msleep+0x26e
 ttysleep(c7e95400,c7e95410,159,c072a5af,0,...) at ttysleep+0x21
 ttread(c7e95400,ea8a9cbc,0) at ttread+0x48f
 ttyread(c7e8b800,ea8a9cbc,0,c0797ae0,0,...) at ttyread+0x2f
 giant_read(c7e8b800,ea8a9cbc,0) at giant_read+0x2d
 devfs_read_f(c83d6048,ea8a9cbc,c7ce0780,0,c815ed80) at devfs_read_f+0x62
 dofileread(c815ed80,0,c83d6048,ea8a9cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c815ed80,0,ea8a9cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c815ed80,ea8a9d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 799 tid 100123 td 0xc8366900
 sched_switch(c8366900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea7c10) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea7c10) at sleepq_wait_sig+0x1d
 msleep(c7ea7c10,0,159,c072a5af,0) at msleep+0x26e
 ttysleep(c7ea7c00,c7ea7c10,159,c072a5af,0,...) at ttysleep+0x21
 ttread(c7ea7c00,ea9c0cbc,0) at ttread+0x48f
 ttyread(c7e8c800,ea9c0cbc,0,ea9c0c14,c054002d,...) at ttyread+0x2f
 scread(c7e8c800,ea9c0cbc,0,c0797ae0,0,...) at scread+0x22
 giant_read(c7e8c800,ea9c0cbc,0) at giant_read+0x2d
 devfs_read_f(c811c2d0,ea9c0cbc,c7ce0780,0,c8366900) at devfs_read_f+0x62
 dofileread(c8366900,0,c811c2d0,ea9c0cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8366900,0,ea9c0cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c8366900,ea9c0d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 798 tid 100103 td 0xc8087c00
 sched_switch(c8087c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea6c10) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea6c10) at sleepq_wait_sig+0x1d
 msleep(c7ea6c10,0,159,c072a5af,0) at msleep+0x26e
 ttysleep(c7ea6c00,c7ea6c10,159,c072a5af,0,...) at ttysleep+0x21
 ttread(c7ea6c00,e8653cbc,0) at ttread+0x48f
 ttyread(c7e8c700,e8653cbc,0,e8653c14,c054002d,...) at ttyread+0x2f
 scread(c7e8c700,e8653cbc,0,c0797ae0,0,...) at scread+0x22
 giant_read(c7e8c700,e8653cbc,0) at giant_read+0x2d
 devfs_read_f(c8111e10,e8653cbc,c7ce0780,0,c8087c00) at devfs_read_f+0x62
 dofileread(c8087c00,0,c8111e10,e8653cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8087c00,0,e8653cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c8087c00,e8653d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 797 tid 100113 td 0xc82e8600
 sched_switch(c82e8600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea5810) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea5810) at sleepq_wait_sig+0x1d
 msleep(c7ea5810,0,159,c072a5af,0) at msleep+0x26e
 ttysleep(c7ea5800,c7ea5810,159,c072a5af,0,...) at ttysleep+0x21
 ttread(c7ea5800,ea957cbc,0) at ttread+0x48f
 ttyread(c7e8c600,ea957cbc,0,ea957c14,c054002d,...) at ttyread+0x2f
 scread(c7e8c600,ea957cbc,0,c0797ae0,0,...) at scread+0x22
 giant_read(c7e8c600,ea957cbc,0) at giant_read+0x2d
 devfs_read_f(c811c558,ea957cbc,c7ce0780,0,c82e8600) at devfs_read_f+0x62
 dofileread(c82e8600,0,c811c558,ea957cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c82e8600,0,ea957cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c82e8600,ea957d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 796 tid 100081 td 0xc818e180
 sched_switch(c818e180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea5c10) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea5c10) at sleepq_wait_sig+0x1d
 msleep(c7ea5c10,0,159,c072a5af,0) at msleep+0x26e
 ttysleep(c7ea5c00,c7ea5c10,159,c072a5af,0,...) at ttysleep+0x21
 ttread(c7ea5c00,ea8dacbc,0) at ttread+0x48f
 ttyread(c7e8c500,ea8dacbc,0,ea8dac14,c054002d,...) at ttyread+0x2f
 scread(c7e8c500,ea8dacbc,0,c0797ae0,0,...) at scread+0x22
 giant_read(c7e8c500,ea8dacbc,0) at giant_read+0x2d
 devfs_read_f(c8108ca8,ea8dacbc,c7ce0780,0,c818e180) at devfs_read_f+0x62
 dofileread(c818e180,0,c8108ca8,ea8dacbc,ffffffff,...) at dofileread+0x85
 kern_readv(c818e180,0,ea8dacbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c818e180,ea8dad04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 795 tid 100116 td 0xc82e8180
 sched_switch(c82e8180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea8410) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea8410) at sleepq_wait_sig+0x1d
 msleep(c7ea8410,0,159,c072a5af,0) at msleep+0x26e
 ttysleep(c7ea8400,c7ea8410,159,c072a5af,0,...) at ttysleep+0x21
 ttread(c7ea8400,ea94ecbc,0) at ttread+0x48f
 ttyread(c7e8c400,ea94ecbc,0,ea94ec14,c054002d,...) at ttyread+0x2f
 scread(c7e8c400,ea94ecbc,0,c0797ae0,0,...) at scread+0x22
 giant_read(c7e8c400,ea94ecbc,0) at giant_read+0x2d
 devfs_read_f(c811c480,ea94ecbc,c7ce0780,0,c82e8180) at devfs_read_f+0x62
 dofileread(c82e8180,0,c811c480,ea94ecbc,ffffffff,...) at dofileread+0x85
 kern_readv(c82e8180,0,ea94ecbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c82e8180,ea94ed04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 794 tid 100072 td 0xc8105c00
 sched_switch(c8105c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e9f410) at sleepq_switch+0xc1
 sleepq_wait_sig(c7e9f410) at sleepq_wait_sig+0x1d
 msleep(c7e9f410,0,159,c072a5af,0) at msleep+0x26e
 ttysleep(c7e9f400,c7e9f410,159,c072a5af,0,...) at ttysleep+0x21
 ttread(c7e9f400,ea87bcbc,0) at ttread+0x48f
 ttyread(c7e8c300,ea87bcbc,0,ea87bc14,c054002d,...) at ttyread+0x2f
 scread(c7e8c300,ea87bcbc,0,c0797ae0,0,...) at scread+0x22
 giant_read(c7e8c300,ea87bcbc,0) at giant_read+0x2d
 devfs_read_f(c8111c60,ea87bcbc,c7ce0780,0,c8105c00) at devfs_read_f+0x62
 dofileread(c8105c00,0,c8111c60,ea87bcbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8105c00,0,ea87bcbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c8105c00,ea87bd04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 793 tid 100122 td 0xc8195780
 sched_switch(c8195780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea7010) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea7010) at sleepq_wait_sig+0x1d
 msleep(c7ea7010,0,159,c072a5af,0) at msleep+0x26e
 ttysleep(c7ea7000,c7ea7010,159,c072a5af,0,...) at ttysleep+0x21
 ttread(c7ea7000,ea922cbc,0) at ttread+0x48f
 ttyread(c7e8c200,ea922cbc,0,ea922c14,c054002d,...) at ttyread+0x2f
 scread(c7e8c200,ea922cbc,0,c0797ae0,0,...) at scread+0x22
 giant_read(c7e8c200,ea922cbc,0) at giant_read+0x2d
 devfs_read_f(c811cd80,ea922cbc,c7ce0780,0,c8195780) at devfs_read_f+0x62
 dofileread(c8195780,0,c811cd80,ea922cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8195780,0,ea922cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c8195780,ea922d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 792 tid 100066 td 0xc815e600
 sched_switch(c815e600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e9bc10) at sleepq_switch+0xc1
 sleepq_wait_sig(c7e9bc10) at sleepq_wait_sig+0x1d
 msleep(c7e9bc10,0,159,c072a5af,0) at msleep+0x26e
 ttysleep(c7e9bc00,c7e9bc10,159,c072a5af,0,...) at ttysleep+0x21
 ttread(c7e9bc00,ea89acbc,0) at ttread+0x48f
 ttyread(c7e8c100,ea89acbc,0,ea89ac14,c054002d,...) at ttyread+0x2f
 scread(c7e8c100,ea89acbc,0,c0797ae0,0,...) at scread+0x22
 giant_read(c7e8c100,ea89acbc,0) at giant_read+0x2d
 devfs_read_f(c8108240,ea89acbc,c7ce0780,0,c815e600) at devfs_read_f+0x62
 dofileread(c815e600,0,c8108240,ea89acbc,ffffffff,...) at dofileread+0x85
 kern_readv(c815e600,0,ea89acbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c815e600,ea89ad04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command inetd pid 778 tid 100057 td 0xc8102d80
 sched_switch(c8102d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
 kern_select(c8102d80,7,bfbfe8a0,0,0,...) at kern_select+0x4bf
 select(c8102d80,ea860d04) at select+0x44
 syscall(3b,3b,3b,bfbfe8a0,bfbfee30,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2815a8d3, esp = 0xbfbfdb7c, ebp = 0xbfbfee68 ---
 
 Tracing command httpd pid 758 tid 100054 td 0xc8087000
 sched_switch(c8087000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07e5744,c07e5744,3e9,e863bb40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07e5744,c07e5720,3e9) at cv_timedwait_sig+0x194
 kern_select(c8087000,0,0,0,0,...) at kern_select+0x4a9
 select(c8087000,e863bd04) at select+0x44
 syscall(3b,3b,3b,0,f4240,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2830c8d3, esp = 0xbfbfec3c, ebp = 0xbfbfec78 ---
 
 Tracing command mysqld pid 757 tid 100128 td 0xc82e9480
 sched_switch(c82e9480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e066f4) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e
 msleep(c7e066f4,c8364cf8,168,c0721153,290,...) at msleep+0x244
 kse_release(c82e9480,ea972d04) at kse_release+0x1df
 syscall(3b,3b,3b,8530900,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c76f28, ebp = 0x13c76f64 ---
 
 Tracing command mysqld pid 757 tid 100129 td 0xc818ed80
 sched_switch(c818ed80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e066f4) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e
 msleep(c7e066f4,c8364cf8,168,c0721153,291,...) at msleep+0x244
 kse_release(c818ed80,ea8f2d04) at kse_release+0x1df
 syscall(3b,3b,3b,8530a00,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7af28, ebp = 0x13c7af64 ---
 
 Tracing command mysqld pid 757 tid 100132 td 0xc82e9a80
 sched_switch(c82e9a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e066f4) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e
 msleep(c7e066f4,c8364cf8,168,c0721153,290,...) at msleep+0x244
 kse_release(c82e9a80,ea97ed04) at kse_release+0x1df
 syscall(3b,3b,3b,8530b00,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7ef28, ebp = 0x13c7ef64 ---
 
 Tracing command mysqld pid 757 tid 100131 td 0xc8193a80
 sched_switch(c8193a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e066f4) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e
 msleep(c7e066f4,c8364cf8,168,c0721153,290,...) at msleep+0x244
 kse_release(c8193a80,ea90ad04) at kse_release+0x1df
 syscall(3b,3b,3b,8530000,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x8534f28, ebp = 0x8534f64 ---
 
 Tracing command mysqld pid 757 tid 100127 td 0xc83c5180
 sched_switch(c83c5180,c82e9480,1) at sched_switch+0x177
 mi_switch(1,c82e9480) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
 kern_select(c83c5180,f,bfbfebf0,0,0,...) at kern_select+0x4bf
 select(c83c5180,ea9edd04) at select+0x44
 syscall(3b,3b,3b,8536000,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2861b8d3, esp = 0xbfbfeacc, ebp = 0xbfbfeb08 ---
 
 Tracing command mysqld pid 757 tid 100134 td 0xc83c5780
 sched_switch(c83c5780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(ea9f9c14) at sleepq_switch+0xc1
 sleepq_wait_sig(ea9f9c14) at sleepq_wait_sig+0x1d
 msleep(ea9f9c14,c8364cf8,168,c072114b,0,...) at msleep+0x26e
 kern_sigtimedwait(c83c5780,26005,0,0,0,ea9f9c80,0) at kern_sigtimedwait+0x30b
 sigtimedwait(c83c5780,ea9f9d04) at sigtimedwait+0x6b
 syscall(3b,3b,3b,13c83e00,bf0f5f37,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (345, FreeBSD ELF32, sigtimedwait), eip = 0x284735b7, esp = 0xbf0f5e2c, ebp = 0xbf0f5ee8 ---
 
 Tracing command mysqld pid 757 tid 100126 td 0xc8105480
 sched_switch(c8105480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8364dc8) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c8364dc8) at sleepq_timedwait_sig+0x1e
 msleep(c8364dc8,c8364cf8,168,c0721148,7531,...) at msleep+0x244
 kse_release(c8105480,ea86cd04) at kse_release+0x121
 syscall(3b,3b,3b,8530800,81,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0xbf9fef2c, ebp = 0xbf9fef88 ---
 
 Tracing command sh pid 727 tid 100111 td 0xc82e8900
 sched_switch(c82e8900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c82e7c90) at sleepq_switch+0xc1
 sleepq_wait_sig(c82e7c90) at sleepq_wait_sig+0x1d
 msleep(c82e7c90,c82e7cf8,15c,c0729071,0) at msleep+0x26e
 kern_wait(c82e8900,ffffffff,ea95dc84,2,0) at kern_wait+0x6bb
 wait4(c82e8900,ea95dd04) at wait4+0x2a
 syscall(3b,3b,3b,bfbfe678,2d,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2813d51f, esp = 0xbfbfe62c, ebp = 0xbfbfe648 ---
 
 Tracing command cron pid 724 tid 100108 td 0xc82e8d80
 sched_switch(c82e8d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07983cc) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07983cc) at sleepq_timedwait_sig+0x1e
 msleep(c07983cc,0,15c,c0724ee0,ea61,...) at msleep+0x244
 kern_nanosleep(c82e8d80,ea966ccc,ea966cc4) at kern_nanosleep+0xab
 nanosleep(c82e8d80,ea966d04) at nanosleep+0x4f
 syscall(3b,3b,3b,8050e6c,3c,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 ---
 
 Tracing command snmpd pid 716 tid 100101 td 0xc8102000
 sched_switch(c8102000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07e5744,c07e5744,1389,ea845b40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07e5744,c07e5720,1389) at cv_timedwait_sig+0x194
 kern_select(c8102000,9,bfbfed50,bfbfecd0,bfbfec50,...) at kern_select+0x4a9
 select(c8102000,ea845d04) at select+0x44
 syscall(3b,3b,3b,bfbfecd0,bfbfec38,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x284598d3, esp = 0xbfbfea4c, ebp = 0xbfbfee68 ---
 
 Tracing command cron pid 705 tid 100119 td 0xc8195c00
 sched_switch(c8195c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07983cc) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07983cc) at sleepq_timedwait_sig+0x1e
 msleep(c07983cc,0,15c,c0724ee0,ea61,...) at msleep+0x244
 kern_nanosleep(c8195c00,ea92bccc,ea92bcc4) at kern_nanosleep+0xab
 nanosleep(c8195c00,ea92bd04) at nanosleep+0x4f
 syscall(3b,3b,3b,805008c,3c,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 ---
 
 Tracing command sendmail pid 699 tid 100121 td 0xc8195900
 sched_switch(c8195900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8365464) at sleepq_switch+0xc1
 sleepq_wait_sig(c8365464) at sleepq_wait_sig+0x1d
 msleep(c8365464,c8365498,168,c0724047,0) at msleep+0x26e
 kern_sigsuspend(c8195900,0,0,0,0,...) at kern_sigsuspend+0xa3
 sigsuspend(c8195900,ea925d04) at sigsuspend+0x33
 syscall(3b,3b,3b,0,bfbfd000,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2827e3fb, esp = 0xbfbfcfec, ebp = 0xbfbfd018 ---
 
 Tracing command sendmail pid 693 tid 100056 td 0xc8105000
 sched_switch(c8105000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07e5744,c07e5744,1389,ea863b40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07e5744,c07e5720,1389) at cv_timedwait_sig+0x194
 kern_select(c8105000,6,bfbfc530,0,0,...) at kern_select+0x4a9
 select(c8105000,ea863d04) at select+0x44
 syscall(3b,3b,3b,2,5,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x282fc8d3, esp = 0xbfbfc44c, ebp = 0xbfbfd038 ---
 
 Tracing command sshd pid 688 tid 100059 td 0xc8102a80
 sched_switch(c8102a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
 kern_select(c8102a80,7,807b920,0,0,...) at kern_select+0x4bf
 select(c8102a80,ea85ad04) at select+0x44
 syscall(3b,3b,3b,1,a,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe16c, ebp = 0xbfbfee78 ---
 
 Tracing command ntpd pid 663 tid 100069 td 0xc815e180
 sched_switch(c815e180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
 kern_select(c815e180,a,bfbfed60,0,0,...) at kern_select+0x4bf
 select(c815e180,ea891d04) at select+0x44
 syscall(3b,3b,3b,bfbfed60,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2828a8d3, esp = 0xbfbfed1c, ebp = 0xbfbfedf8 ---
 
 Tracing command usbd pid 632 tid 100085 td 0xc818aa80
 sched_switch(c818aa80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07e5744,c07e5744,7531,ea8ceb40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07e5744,c07e5720,7531) at cv_timedwait_sig+0x194
 kern_select(c818aa80,6,bfbfeda0,bfbfed20,0,...) at kern_select+0x4a9
 select(c818aa80,ea8ced04) at select+0x44
 syscall(3b,3b,3b,bfbfed20,1e,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2813c8d3, esp = 0xbfbfeccc, ebp = 0xbfbfee78 ---
 
 Tracing command accounting pid 585 tid 100053 td 0xc8087180
 sched_switch(c8087180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c0795f00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c0795f00) at sleepq_wait+0x46
 _sx_xlock_hard(c0795f00,c8087180,0,c071f3eb,215) at _sx_xlock_hard+0x195
 _sx_xlock(c0795f00,0,c071f3eb,215) at _sx_xlock+0x85
 acct_thread(0,e863ed38,0,c053d6c4,0,...) at acct_thread+0xee
 fork_exit(c053d6c4,0,e863ed38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe863ed6c, ebp = 0 ---
 
 Tracing command named pid 568 tid 100086 td 0xc818a900
 sched_switch(c818a900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07e5744,c07e5744,7531,ea8cbb40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07e5744,c07e5720,7531) at cv_timedwait_sig+0x194
 kern_select(c818a900,1a,bfbfed60,bfbfece0,0,...) at kern_select+0x4a9
 select(c818a900,ea8cbd04) at select+0x44
 syscall(3b,3b,3b,0,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2835e8d3, esp = 0xbfbfec9c, ebp = 0xbfbfedf8 ---
 
 Tracing command syslogd pid 548 tid 100104 td 0xc8087a80
 sched_switch(c8087a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(dbfc2388,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(dbfc2388,0,c8302ba8,b7,c072e794,...) at sleepq_wait+0x46
 msleep(dbfc2388,c0797a48,50,c072c4b4,0,...) at msleep+0x27d
 acquire(e865090c,120,60000,c8087a80,0,...) at acquire+0x76
 lockmgr(dbfc2388,202122,c8302ba8,c8087a80) at lockmgr+0x44e
 getblk(c8302ae0,4,0,1800,0,...) at getblk+0x12f
 breadn(c8302ae0,4,0,1800,0,...) at breadn+0x31
 bread(c8302ae0,4,0,1800,0,e8650a2c) at bread+0x20
 ffs_balloc_ufs2(c8302ae0,1173b,0,7d,c7ce0780,...) at ffs_balloc_ufs2+0xcd3
 ffs_write(e8650c10) at ffs_write+0x2c0
 VOP_WRITE_APV(c077a640,e8650c10) at VOP_WRITE_APV+0x132
 vn_write(c8111ab0,c8593980,c7ce0780,0,c8087a80) at vn_write+0x1ea
 dofilewrite(c8087a80,c,c8111ab0,c8593980,ffffffff,...) at dofilewrite+0x7c
 kern_writev(c8087a80,c,c8593980,c8593980,0,...) at kern_writev+0x3b
 writev(c8087a80,e8650d04) at writev+0x30
 syscall(3b,3b,3b,0,bfbfd050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (121, FreeBSD ELF32, writev), eip = 0x28137f1f, esp = 0xbfbfc96c, ebp = 0xbfbfd078 ---
 
 Tracing command ipmon pid 531 tid 100050 td 0xc8087600
 sched_switch(c8087600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07983cc) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07983cc) at sleepq_timedwait_sig+0x1e
 msleep(c07983cc,0,15c,c0724ee0,3e9,...) at msleep+0x244
 kern_nanosleep(c8087600,e8647ccc,e8647cc4) at kern_nanosleep+0xab
 nanosleep(c8087600,e8647d04) at nanosleep+0x4f
 syscall(3b,3b,3b,2,1,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2812f513, esp = 0xbfbfcd3c, ebp = 0xbfbfcd68 ---
 
 Tracing command devd pid 435 tid 100107 td 0xc82e9000
 sched_switch(c82e9000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5744) at sleepq_switch+0xc1
 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
 kern_select(c82e9000,5,bfbfe9d0,0,0,...) at kern_select+0x4bf
 select(c82e9000,ea969d04) at select+0x44
 syscall(3b,3b,3b,0,8,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x8075917, esp = 0xbfbfe96c, ebp = 0xbfbfee78 ---
 
 Tracing command adjkerntz pid 164 tid 100060 td 0xc8102900
 sched_switch(c8102900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8101894) at sleepq_switch+0xc1
 sleepq_wait_sig(c8101894) at sleepq_wait_sig+0x1d
 msleep(c8101894,c81018c8,168,c0724047,0) at msleep+0x26e
 kern_sigsuspend(c8102900,0,0,0,0,...) at kern_sigsuspend+0xa3
 sigsuspend(c8102900,ea857d04) at sigsuspend+0x33
 syscall(3b,3b,3b,0,ffffc7c0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x280bc3fb, esp = 0xbfbfed9c, ebp = 0xbfbfee78 ---
 
 Tracing command schedcpu pid 44 tid 100044 td 0xc7e05180
 sched_switch(c7e05180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(e8380d04,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(e8380d04) at sleepq_timedwait+0x4a
 msleep(e8380d04,0,0,c071cc2c,3e8) at msleep+0x259
 schedcpu_thread(0,e8380d38,0,c05768b4,0,...) at schedcpu_thread+0x26
 fork_exit(c05768b4,0,e8380d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8380d6c, ebp = 0 ---
 
 Tracing command softdepflush pid 43 tid 100045 td 0xc7e05000
 sched_switch(c7e05000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07f3254,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c07f3254) at sleepq_timedwait+0x4a
 msleep(c07f3254,c07f3220,44,c0738a08,3e8) at msleep+0x259
 softdep_flush(0,e837dd38,0,c06669b0,0,...) at softdep_flush+0x227
 fork_exit(c06669b0,0,e837dd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe837dd6c, ebp = 0 ---
 
 Tracing command vnlru pid 42 tid 100046 td 0xc7dbdd80
 sched_switch(c7dbdd80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8084860,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c8084860) at sleepq_timedwait+0x4a
 msleep(c8084860,c07e60e0,250,c072dc00,3e8,c07e6160) at msleep+0x259
 vnlru_proc(0,e836dd38,0,c05bb048,0,...) at vnlru_proc+0xdf
 fork_exit(c05bb048,0,e836dd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe836dd6c, ebp = 0 ---
 
 Tracing command syncer pid 41 tid 100047 td 0xc7dbdc00
 sched_switch(c7dbdc00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(dbf1b09c,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(dbf1b09c,0,c7eb0ba8,b7,c072db32,...) at sleepq_wait+0x46
 msleep(dbf1b09c,c07975ec,50,c072c4b4,0,...) at msleep+0x27d
 acquire(e836aacc,120,60000,c7dbdc00,0,...) at acquire+0x76
 lockmgr(dbf1b09c,202122,c7eb0ba8,c7dbdc00) at lockmgr+0x44e
 getblk(c7eb0ae0,113b60,0,4000,0,...) at getblk+0x12f
 breadn(c7eb0ae0,113b60,0,4000,0,...) at breadn+0x31
 bread(c7eb0ae0,113b60,0,4000,0,e836abb8) at bread+0x20
 ffs_update(c7eb1ae0,0,7,0,0,...) at ffs_update+0x22c
 ffs_syncvnode(c7eb1ae0,3) at ffs_syncvnode+0x3be
 ffs_sync(c80cbb90,3,c7dbdc00,c80cbb90,2,...) at ffs_sync+0x209
 sync_fsync(e836acbc) at sync_fsync+0x164
 VOP_FSYNC_APV(c0770380,e836acbc) at VOP_FSYNC_APV+0x9b
 sync_vnode(c817a67c,c7dbdc00) at sync_vnode+0x106
 sched_sync(0,e836ad38,0,c05bcb5c,0,...) at sched_sync+0x1ed
 fork_exit(c05bcb5c,0,e836ad38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe836ad6c, ebp = 0 ---
 
 Tracing command bufdaemon pid 40 tid 100048 td 0xc7dbda80
 sched_switch(c7dbda80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07e5cc0,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c07e5cc0) at sleepq_timedwait+0x4a
 msleep(c07e5cc0,c07e5ce0,44,c072c3ea,3e8) at msleep+0x259
 buf_daemon(0,e8367d38,0,c05ae534,0,...) at buf_daemon+0x108
 fork_exit(c05ae534,0,e8367d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8367d6c, ebp = 0 ---
 
 Tracing command pagezero pid 39 tid 100049 td 0xc7dbd900
 sched_switch(c7dbd900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07f4224,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c07f4224) at sleepq_timedwait+0x4a
 msleep(c07f4224,c07f3ca0,200,c073dabc,493e0,c07f3ca0,0,c073da9e,9d) at msleep+0x259
 vm_pagezero(0,e8364d38,0,c0697cbc,0,...) at vm_pagezero+0x6c
 fork_exit(c0697cbc,0,e8364d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8364d6c, ebp = 0 ---
 
 Tracing command vmdaemon pid 38 tid 100027 td 0xc7dbd180
 sched_switch(c7dbd180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07f3d2c,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c07f3d2c,0,c073d73f,63e,0,...) at sleepq_wait+0x46
 msleep(c07f3d2c,c07f3d40,68,c072c3ea,0,...) at msleep+0x27d
 vm_daemon(0,e8355d38,0,c0696cd0,0,...) at vm_daemon+0x36
 fork_exit(c0696cd0,0,e8355d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8355d6c, ebp = 0 ---
 
 Tracing command pagedaemon pid 37 tid 100028 td 0xc7dbd000
 sched_switch(c7dbd000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07f3ce8,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c07f3ce8) at sleepq_timedwait+0x4a
 msleep(c07f3ce8,c07f3ca0,44,c072c3ea,1388) at msleep+0x259
 vm_pageout(0,e8352d38,0,c0696958,0,...) at vm_pageout+0x260
 fork_exit(c0696958,0,e8352d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8352d6c, ebp = 0 ---
 
 Tracing command irq7: ppc0 pid 36 tid 100029 td 0xc7ce9d80
 fork_trampoline() at fork_trampoline
 
 Tracing command fdc0 pid 35 tid 100030 td 0xc7ce9c00
 sched_switch(c7ce9c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e1f03c,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c7e1f03c) at sleepq_timedwait+0x4a
 msleep(c7e1f03c,c7e1f0f0,4c,c071cc2c,3e8) at msleep+0x259
 fdc_worker(c7e1f000,c7e1f0f0,0,c073f3ff,4a1) at fdc_worker+0x1fe
 fdc_thread(c7e1f000,e6929d38,c7e1f000,c06a57f4,0,...) at fdc_thread+0x54
 fork_exit(c06a57f4,c7e1f000,e6929d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6929d6c, ebp = 0 ---
 
 Tracing command swi0: sio pid 34 tid 100031 td 0xc7ce9a80
 sched_switch(c7ce9a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7e89540,e6926d38,c7e89540,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7e89540,e6926d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6926d6c, ebp = 0 ---
 
 Tracing command irq1: atkbd0 pid 33 tid 100032 td 0xc7ce9900
 sched_switch(c7ce9900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7e846c0,e6923d38,c7e846c0,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7e846c0,e6923d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6923d6c, ebp = 0 ---
 
 Tracing command irq22: fxp0 pid 32 tid 100033 td 0xc7ce9780
 sched_switch(c7ce9780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7e25010,e6920d38,c7e25010,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7e25010,e6920d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6920d6c, ebp = 0 ---
 
 Tracing command irq20: mly0 pid 31 tid 100034 td 0xc7ce9600
 sched_switch(c7ce9600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7dfea40,e691dd38,c7dfea40,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7dfea40,e691dd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe691dd6c, ebp = 0 ---
 
 Tracing command irq15: ata1 pid 30 tid 100035 td 0xc7ce9480
 fork_trampoline() at fork_trampoline
 
 Tracing command irq14: ata0 pid 29 tid 100036 td 0xc7e05d80
 sched_switch(c7e05d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7e072e0,e8398d38,c7e072e0,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7e072e0,e8398d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8398d6c, ebp = 0 ---
 
 Tracing command usb1 pid 28 tid 100037 td 0xc7e05c00
 sched_switch(c7e05c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7dada10,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c7dada10) at sleepq_timedwait+0x4a
 msleep(c7dada10,0,5c,c071b821,ea60,c7debd00) at msleep+0x259
 usb_event_thread(c7debd00,e8395d38,c7debd00,c051db30,0,...) at usb_event_thread+0x70
 fork_exit(c051db30,c7debd00,e8395d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8395d6c, ebp = 0 ---
 
 Tracing command irq17: ehci0 pid 27 tid 100017 td 0xc7ce5900
 fork_trampoline() at fork_trampoline
 
 Tracing command usbtask pid 26 tid 100018 td 0xc7ce5780
 sched_switch(c7ce5780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07953e4,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c07953e4,0,c055d0cc,c0797ae0,c071b806,...) at sleepq_wait+0x46
 msleep(c07953e4,0,5c,c071b828,0,...) at msleep+0x27d
 usb_task_thread(0,e6902d38,0,c051dbc0,0,...) at usb_task_thread+0x47
 fork_exit(c051dbc0,0,e6902d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6902d6c, ebp = 0 ---
 
 Tracing command usb0 pid 25 tid 100019 td 0xc7ce5600
 sched_switch(c7ce5600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7db4210,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c7db4210) at sleepq_timedwait+0x4a
 msleep(c7db4210,0,5c,c071b821,ea60,c7dea640) at msleep+0x259
 usb_event_thread(c7dea640,e68ffd38,c7dea640,c051db30,0,...) at usb_event_thread+0x70
 fork_exit(c051db30,c7dea640,e68ffd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68ffd6c, ebp = 0 ---
 
 Tracing command irq16: ohci0 pid 24 tid 100020 td 0xc7ce5480
 fork_trampoline() at fork_trampoline
 
 Tracing command irq9: acpi0 pid 23 tid 100021 td 0xc7ce5300
 fork_trampoline() at fork_trampoline
 
 Tracing command acpi_task_2 pid 22 tid 100022 td 0xc7ce5180
 sched_switch(c7ce5180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d43c00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d43c00,0,c07273e4,49,8,...) at sleepq_wait+0x46
 msleep(c7d43c00,c7d43c1c,0,c071cc2c,0) at msleep+0x27d
 taskqueue_thread_loop(c0791fc8,e68f6d38,c0791fc8,c0585fd4,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c0585fd4,c0791fc8,e68f6d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68f6d6c, ebp = 0 ---
 
 Tracing command acpi_task_1 pid 9 tid 100023 td 0xc7dbd780
 sched_switch(c7dbd780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d43c00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d43c00,0,c07273e4,49,8,...) at sleepq_wait+0x46
 msleep(c7d43c00,c7d43c1c,0,c071cc2c,0) at msleep+0x27d
 taskqueue_thread_loop(c0791fc8,e8361d38,c0791fc8,c0585fd4,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c0585fd4,c0791fc8,e8361d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8361d6c, ebp = 0 ---
 
 Tracing command acpi_task_0 pid 8 tid 100024 td 0xc7dbd600
 sched_switch(c7dbd600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d43c00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d43c00,0,c07273e4,49,c7ddf260,...) at sleepq_wait+0x46
 msleep(c7d43c00,c7d43c1c,0,c071cc2c,0) at msleep+0x27d
 taskqueue_thread_loop(c0791fc8,e835ed38,c0791fc8,c0585fd4,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c0585fd4,c0791fc8,e835ed38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe835ed6c, ebp = 0 ---
 
 Tracing command swi5: + pid 21 tid 100025 td 0xc7dbd480
 fork_trampoline() at fork_trampoline
 
 Tracing command thread taskq pid 7 tid 100026 td 0xc7dbd300
 sched_switch(c7dbd300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d43d80,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d43d80,0,c07273e4,49,c806aeb0,...) at sleepq_wait+0x46
 msleep(c7d43d80,c7d43d9c,0,c071cc2c,0) at msleep+0x27d
 taskqueue_thread_loop(c079da88,e8358d38,c079da88,c0585fd4,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c0585fd4,c079da88,e8358d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8358d6c, ebp = 0 ---
 
 Tracing command swi6: Giant taskq pid 20 tid 100008 td 0xc7ce3300
 sched_switch(c7ce3300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8b70,e68dbd38,c7cc8b70,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7cc8b70,e68dbd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68dbd6c, ebp = 0 ---
 
 Tracing command swi6: task queue pid 19 tid 100009 td 0xc7ce3180
 sched_switch(c7ce3180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8b80,e68d8d38,c7cc8b80,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7cc8b80,e68d8d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68d8d6c, ebp = 0 ---
 
 Tracing command kqueue taskq pid 6 tid 100010 td 0xc7ce3000
 sched_switch(c7ce3000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d72100,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d72100,0,c07273e4,49,8,...) at sleepq_wait+0x46
 msleep(c7d72100,c7d7211c,0,c071cc2c,0) at msleep+0x27d
 taskqueue_thread_loop(c0796120,e68d5d38,c0796120,c0585fd4,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c0585fd4,c0796120,e68d5d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68d5d6c, ebp = 0 ---
 
 Tracing command swi2: cambio pid 18 tid 100011 td 0xc7ce9300
 sched_switch(c7ce9300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8ba0,e6917d38,c7cc8ba0,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7cc8ba0,e6917d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6917d6c, ebp = 0 ---
 
 Tracing command xpt_thrd pid 5 tid 100012 td 0xc7ce9180
 sched_switch(c7ce9180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c078c4c4,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
 sleepq_wait(c078c4c4,0,c055d0cc,c0797ae0,c06fdf9f,...) at sleepq_wait+0x46
 msleep(c078c4c4,0,4c,c06fdfba,0) at msleep+0x27d
 xpt_scanner_thread(0,e6914d38,0,c04441f4,0,...) at xpt_scanner_thread+0x35
 fork_exit(c04441f4,0,e6914d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6914d6c, ebp = 0 ---
 
 Tracing command yarrow pid 17 tid 100013 td 0xc7ce9000
 sched_switch(c7ce9000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07933e0,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c07933e0) at sleepq_timedwait+0x4a
 msleep(c07933e0,0,0,c071cc2c,64) at msleep+0x259
 random_kthread(0,e6911d38,0,c0506b28,0,...) at random_kthread+0x180
 fork_exit(c0506b28,0,e6911d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6911d6c, ebp = 0 ---
 
 Tracing command g_down pid 4 tid 100014 td 0xc7ce5d80
 sched_switch(c7ce5d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c0795888,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c0795888) at sleepq_timedwait+0x4a
 msleep(c0795888,c0795768,24c,c071cc2c,64) at msleep+0x259
 g_io_schedule_down(c7ce5d80) at g_io_schedule_down+0x56
 g_down_procbody(0,e690ed38,0,c052fab0,0,...) at g_down_procbody+0x5a
 fork_exit(c052fab0,0,e690ed38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe690ed6c, ebp = 0 ---
 
 Tracing command g_up pid 3 tid 100015 td 0xc7ce5c00
 sched_switch(c7ce5c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c0795884,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c0795884) at sleepq_timedwait+0x4a
 msleep(c0795884,c07957a8,24c,c071cc2c,64) at msleep+0x259
 g_io_schedule_up(c7ce5c00) at g_io_schedule_up+0x127
 g_up_procbody(0,e690bd38,0,c052fa50,0,...) at g_up_procbody+0x5a
 fork_exit(c052fa50,0,e690bd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe690bd6c, ebp = 0 ---
 
 Tracing command g_event pid 2 tid 100016 td 0xc7ce5a80
 sched_switch(c7ce5a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c079587c,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c079587c) at sleepq_timedwait+0x4a
 msleep(c079587c,0,4c,c071cc2c,64) at msleep+0x259
 g_event_procbody(0,e6908d38,0,c052fb10,0,...) at g_event_procbody+0x9e
 fork_exit(c052fb10,0,e6908d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6908d6c, ebp = 0 ---
 
 Tracing command swi3: vm pid 16 tid 100000 td 0xc7ce5000
 fork_trampoline() at fork_trampoline
 
 Tracing command swi4: clock sio pid 15 tid 100001 td 0xc7ce3d80
 sched_switch(c7ce3d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8870,e68f0d38,c7cc8870,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7cc8870,e68f0d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68f0d6c, ebp = 0 ---
 
 Tracing command swi1: net pid 14 tid 100002 td 0xc7ce3c00
 sched_switch(c7ce3c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8880,e68edd38,c7cc8880,c0551b84,0,...) at ithread_loop+0xda
 fork_exit(c0551b84,c7cc8880,e68edd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68edd6c, ebp = 0 ---
 
 Tracing command idle: cpu0 pid 13 tid 100003 td 0xc7ce3a80
 sched_switch(c072811b,6a9,c07e5594,e68eac40,c058a4e6,...) at sched_switch+0x177
 (null)(c072813c,50000,c7ccb010,c07981b0,0,...) at 0
 __func__.0(c055d930,0,2001,1,94378,...) at 0xc072813c
 (null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa
 
 Tracing command idle: cpu1 pid 12 tid 100004 td 0xc7ce3900
 kdb_enter(c074002a) at kdb_enter+0x2b
 siointr1(c7e95800,c07f61a0,0,c073fe30,56e,...) at siointr1+0xce
 siointr(c7e95800) at siointr+0x21
 intr_execute_handlers(c7cd74c8,e68e7c94,4,e68e7cd8,c06c2713,...) at intr_execute_handlers+0xe1
 lapic_handle_intr(38) at lapic_handle_intr+0x2e
 Xapic_isr1() at Xapic_isr1+0x33
 --- interrupt, eip = 0xc06bb645, esp = 0xe68e7cd8, ebp = 0xe68e7cd8 ---
 acpi_cpu_c1(c0797aa0,2,e68e7cf8,2,c7ce2860,...) at acpi_cpu_c1+0x5
 acpi_cpu_idle(e68e7d0c,c0550ee1,c0550e84,e68e7d24,c0550cc4,...) at acpi_cpu_idle+0x15d
 cpu_idle(c0550e84,e68e7d24,c0550cc4,0,e68e7d38,...) at cpu_idle+0x28
 idle_proc(0,e68e7d38,0,c0550e84,0,...) at idle_proc+0x5d
 fork_exit(c0550e84,0,e68e7d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68e7d6c, ebp = 0 ---
 
 Tracing command idle: cpu2 pid 11 tid 100005 td 0xc7ce3780
 sched_switch(c079efa0,0,ffc06034) at sched_switch+0x177
 mi_switch(c055d321,c07e5590,e68e4c1c,c058ae7c,c079efa0,...) at mi_switch+0x270
 end(c072813c,50000,c7ccb010,c07981b0,0,...) at 0xe68e4bf0
 __func__.0(c055d930,0,2001,1,94378,...) at 0xc072813c
 (null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa
 
 Tracing command idle: cpu3 pid 10 tid 100006 td 0xc7ce3600
 sched_switch(28,12,c06c2aa1,c080fd90,8,...) at sched_switch+0x177
 
 Tracing command init pid 1 tid 100007 td 0xc7ce3480
 sched_switch(c7ce3480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ce8000) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ce8000) at sleepq_wait_sig+0x1d
 msleep(c7ce8000,c7ce8068,15c,c0729071,0) at msleep+0x26e
 kern_wait(c7ce3480,ffffffff,e68dec84,0,0) at kern_wait+0x6bb
 wait4(c7ce3480,e68ded04) at wait4+0x2a
 syscall(3b,3b,3b,0,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8052f47, esp = 0xbfbfed7c, ebp = 0xbfbfed98 ---
 
 Tracing command swapper pid 0 tid 0 td 0xc0795ba0
 sched_switch(c0795ba0,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 scheduler(0,c1ec00,c1e000,0,c0440d25,...) at scheduler+0x199
 mi_startup() at mi_startup+0x96
 begin() at begin+0x2c
 
 db> call doadump
 Dumping 3070 MB (2 chunks)
 Aborting dump due to I/O error.
 status == 0xb, scsi status == 0x0
 
 ** DUMP FAILED (ERROR 5) **
 = 0x1d
 
 --
 Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
 Phone: +7 4732 539880
 Fax:   +7 4732 531415         http://www.vsi.ru
 CenterTelecom Voronezh ISP    http://isp.vsi.ru
 

From: Kostik Belousov <kostikbel@gmail.com>
To: bug-followup@FreeBSD.org
Cc: Oleg Derevenetz <oleg@vsi.ru>
Subject: Re: kern/104406: [ufs] Processes get stuck in "ufs" stateunderpersistent CPU load
Date: Tue, 23 Oct 2007 12:54:15 +0300

 --wIc/V6YLA2QdyfT4
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 On Mon, Oct 22, 2007 at 04:29:32PM +0400, Oleg Derevenetz wrote:
 > I posted a followup to kern/104406 that includes all information listed i=
 n=20
 > "Debugging Deadlocks" chapter of FreeBSD Developer's Handbook. Can anyone=
 =20
 > take a look on it and say - is this certainly a hardware problem or some=
 =20
 > sort of software problem ?
 
 The ps output from ddb would simplify the analysis.
 
 The buffer space on the machine seems to be contaminated with a critical
 amount of the dirty buffers that are sent for the write. The pid
 844 (that is the only one producing the writes) is blocked in
 waitrunningbufspace(), waiting for the write queue to be drained. There
 are other processes, for instance, 937 etc, that wait for the buffer
 lock. Most likely, these buffers are scheduled for write and cause the
 processes to be uninterruptibly blocked.
 
 If this state of the system is static, then the write queue is not
 processed. Most plausible cause is the controller (mly0) or the controller
 driver fault.
 
 --wIc/V6YLA2QdyfT4
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.4 (FreeBSD)
 
 iD8DBQFHHcTGC3+MBN1Mb4gRAodzAKDLMJIiPnjWNRejDrnj/6R/Kic1LwCbBxK+
 9GLcje2brT6bTZA1dzB17Yg=
 =2chV
 -----END PGP SIGNATURE-----
 
 --wIc/V6YLA2QdyfT4--

From: "Oleg Derevenetz" <oleg@vsi.ru>
To: <bug-followup@FreeBSD.org>, "Kostik Belousov" <kostikbel@gmail.com>
Cc:  
Subject: Re: kern/104406: [ufs] Processes get stuck in "ufs" state under persistent CPU load
Date: Tue, 23 Oct 2007 16:19:57 +0400

 OK, here is information grabbed from ddb for the another "hang" case including ps output:
 
 db> show pcpu
 cpuid        = 1
 curthread    = 0xc7ce3900: pid 12 "idle: cpu1"
 curpcb       = 0xe68e7d90
 fpcurthread  = none
 idlethread   = 0xc7ce3900: pid 12 "idle: cpu1"
 APIC ID      = 1
 currentldt   = 0x50
 spin locks held:
 exclusive spin mutex sio r = 0 (0xc07df9c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
 db> show allpcpu
 Current CPU: 1
 
 cpuid        = 0
 curthread    = 0xc7ce3a80: pid 13 "idle: cpu0"
 curpcb       = 0xe68ead90
 fpcurthread  = none
 idlethread   = 0xc7ce3a80: pid 13 "idle: cpu0"
 APIC ID      = 0
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 1
 curthread    = 0xc7ce3900: pid 12 "idle: cpu1"
 curpcb       = 0xe68e7d90
 fpcurthread  = none
 idlethread   = 0xc7ce3900: pid 12 "idle: cpu1"
 APIC ID      = 1
 currentldt   = 0x50
 spin locks held:
 exclusive spin mutex sio r = 0 (0xc07df9c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
 
 cpuid        = 2
 curthread    = 0xc7ce3780: pid 11 "idle: cpu2"
 curpcb       = 0xe68e4d90
 fpcurthread  = none
 idlethread   = 0xc7ce3780: pid 11 "idle: cpu2"
 APIC ID      = 2
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 3
 curthread    = 0xc7ce3600: pid 10 "idle: cpu3"
 curpcb       = 0xe68e1d90
 fpcurthread  = none
 idlethread   = 0xc7ce3600: pid 10 "idle: cpu3"
 APIC ID      = 3
 currentldt   = 0x50
 spin locks held:
 
 db> show locks
 exclusive spin mutex sio r = 0 (0xc07df9c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
 db> show alllocks
 db> show lockedvnods
 Locked vnodes
 
 0xc820f414: tag ufs, type VDIR
     usecount 1, writecount 0, refcount 3 mountedhere 0
     flags (VV_ROOT)
      lock type ufs: EXCL (count 1) by thread 0xc7e05600 (pid 894) with 1 pending#0 0xc054901d at lockmgr+0x4ed
 #1 0xc0662f0e at ffs_lock+0x76
 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87
 #3 0xc05b884c at vn_lock+0xac
 #4 0xc05ac6f2 at vget+0xbe
 #5 0xc05a43d9 at vfs_hash_get+0x8d
 #6 0xc0661c77 at ffs_vget+0x27
 #7 0xc066b065 at ufs_root+0x19
 #8 0xc05a58fc at lookup+0x7c8
 #9 0xc05a4e92 at namei+0x39a
 #10 0xc053c321 at do_execve+0x159
 #11 0xc053c168 at kern_execve+0x74
 #12 0xc053c0ce at execve+0x32
 #13 0xc06c615b at syscall+0x25b
 #14 0xc06b17bf at Xint0x80_syscall+0x1f
 
         ino 2, on dev da0s1d
 
 0xc824815c: tag ufs, type VDIR
     usecount 1, writecount 0, refcount 4 mountedhere 0
     flags ()
      lock type ufs: EXCL (count 1) by thread 0xc84e8180 (pid 891) with 1 pending#0 0xc054901d at lockmgr+0x4ed
 #1 0xc0662f0e at ffs_lock+0x76
 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87
 #3 0xc05b884c at vn_lock+0xac
 #4 0xc05ac6f2 at vget+0xbe
 #5 0xc05a0e6e at cache_lookup+0x34a
 #6 0xc05a13a2 at vfs_cache_lookup+0x92
 #7 0xc06d6c67 at VOP_LOOKUP_APV+0x87
 #8 0xc05a55d8 at lookup+0x4a4
 #9 0xc05a4e92 at namei+0x39a
 #10 0xc053c321 at do_execve+0x159
 #11 0xc053c168 at kern_execve+0x74
 #12 0xc053c0ce at execve+0x32
 #13 0xc06c615b at syscall+0x25b
 #14 0xc06b17bf at Xint0x80_syscall+0x1f
 
         ino 518144, on dev da0s1d
 
 0xc98dc414: tag ufs, type VREG
     usecount 2, writecount 0, refcount 3 mountedhere 0
     flags (VV_TEXT)
     v_object 0xc9752294 ref 1 pages 4
      lock type ufs: EXCL (count 1) by thread 0xc8150480 (pid 806)#0 0xc054901d at lockmgr+0x4ed
 #1 0xc05a44b2 at vfs_hash_insert+0x26
 #2 0xc0661dc1 at ffs_vget+0x171
 #3 0xc066861a at ufs_lookup+0xab6
 #4 0xc06d6d3b at VOP_CACHEDLOOKUP_APV+0x9b
 #5 0xc05a13c5 at vfs_cache_lookup+0xb5
 #6 0xc06d6c67 at VOP_LOOKUP_APV+0x87
 #7 0xc05a55d8 at lookup+0x4a4
 #8 0xc05a4e92 at namei+0x39a
 #9 0xc053c321 at do_execve+0x159
 #10 0xc053c168 at kern_execve+0x74
 #11 0xc053c0ce at execve+0x32
 #12 0xc06c615b at syscall+0x25b
 #13 0xc06b17bf at Xint0x80_syscall+0x1f
 
         ino 683213, on dev da0s1d
 
 0xc822e6cc: tag syncer, type VNON
     usecount 1, writecount 0, refcount 2 mountedhere 0
     flags ()
      lock type syncer: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc054901d at lockmgr+0x4ed
 #1 0xc05a3591 at vop_stdlock+0x21
 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87
 #3 0xc05b884c at vn_lock+0xac
 #4 0xc05abeb3 at sync_vnode+0xe3
 #5 0xc05ac151 at sched_sync+0x1ed
 #6 0xc05400cc at fork_exit+0xa0
 #7 0xc06b17cc at fork_trampoline+0x8
 
 
 0xc9956570: tag ufs, type VDIR
     usecount 1, writecount 0, refcount 3 mountedhere 0
     flags ()
      lock type ufs: EXCL (count 1) by thread 0xc8332d80 (pid 851)#0 0xc054901d at lockmgr+0x4ed
 #1 0xc0662f0e at ffs_lock+0x76
 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87
 #3 0xc05b884c at vn_lock+0xac
 #4 0xc05ac6f2 at vget+0xbe
 #5 0xc05a0e6e at cache_lookup+0x34a
 #6 0xc05a13a2 at vfs_cache_lookup+0x92
 #7 0xc06d6c67 at VOP_LOOKUP_APV+0x87
 #8 0xc05a55d8 at lookup+0x4a4
 #9 0xc05a4e92 at namei+0x39a
 #10 0xc05b7637 at vn_open_cred+0x5b
 #11 0xc05b75da at vn_open+0x1e
 #12 0xc05b0c81 at kern_open+0x231
 #13 0xc05b0a1a at open+0x1a
 #14 0xc06c615b at syscall+0x25b
 #15 0xc06b17bf at Xint0x80_syscall+0x1f
 
         ino 94845, on dev da1s1d
 
 0xc99a16cc: tag ufs, type VREG
     usecount 1, writecount 0, refcount 7 mountedhere 0
     flags ()
     v_object 0xc99b8a50 ref 0 pages 19
      lock type ufs: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc054901d at lockmgr+0x4ed
 #1 0xc0662f0e at ffs_lock+0x76
 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87
 #3 0xc05b884c at vn_lock+0xac
 #4 0xc05ac6f2 at vget+0xbe
 #5 0xc06619b9 at ffs_sync+0x1c1
 #6 0xc05ae264 at sync_fsync+0x164
 #7 0xc06d7f37 at VOP_FSYNC_APV+0x9b
 #8 0xc05abed6 at sync_vnode+0x106
 #9 0xc05ac151 at sched_sync+0x1ed
 #10 0xc05400cc at fork_exit+0xa0
 #11 0xc06b17cc at fork_trampoline+0x8
 
         ino 94916, on dev da1s1d
 db> alltrace
 
 Tracing command inetd pid 897 tid 100176 td 0xc8966780
 sched_switch(c8966780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c820f46c,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c820f46c,0,c820f4dc,b7,c071716d,...) at sleepq_wait+0x46
 msleep(c820f46c,c0780164,50,c0715859,0,...) at msleep+0x27d
 acquire(eaa608c8,40,60000,c8966780,0,...) at acquire+0x76
 lockmgr(c820f46c,2002,c820f4dc,c8966780) at lockmgr+0x44e
 ffs_lock(eaa60920) at ffs_lock+0x76
 VOP_LOCK_APV(c0764180,eaa60920) at VOP_LOCK_APV+0x87
 vn_lock(c820f414,2002,c8966780,c820f414) at vn_lock+0xac
 vget(c820f414,2002,c8966780) at vget+0xbe
 vfs_hash_get(c822d5c8,2,2,c8966780,eaa60a38,0,0) at vfs_hash_get+0x8d
 ffs_vget(c822d5c8,2,2,eaa60a38) at ffs_vget+0x27
 ufs_root(c822d5c8,2,eaa60a84,c8966780,c7eb0000,...) at ufs_root+0x19
 lookup(eaa60c24) at lookup+0x7c8
 namei(eaa60c24) at namei+0x39a
 do_execve(c8966780,eaa60cb4,0) at do_execve+0x159
 kern_execve(c8966780,eaa60cb4,0) at kern_execve+0x74
 execve(c8966780,eaa60d04) at execve+0x32
 syscall(3b,3b,3b,bfbfedb0,805f200,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (59, FreeBSD ELF32, execve), eip = 0x280f14df, esp = 0xbfbfdb6c, ebp = 0xbfbfdb88 ---
 
 Tracing command sh pid 894 tid 100041 td 0xc7e05600
 sched_switch(c7e05600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c82481b4,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c82481b4,0,c8248224,b7,c071716d,...) at sleepq_wait+0x46
 msleep(c82481b4,c07806e0,50,c0715859,0,...) at msleep+0x27d
 acquire(e83898fc,40,60000,c7e05600,0,...) at acquire+0x76
 lockmgr(c82481b4,2002,c8248224,c7e05600) at lockmgr+0x44e
 ffs_lock(e8389954) at ffs_lock+0x76
 VOP_LOCK_APV(c0764180,e8389954) at VOP_LOCK_APV+0x87
 vn_lock(c824815c,2002,c7e05600,c824815c) at vn_lock+0xac
 vget(c824815c,2002,c7e05600) at vget+0xbe
 cache_lookup(c820f414,e8389c38,e8389c4c) at cache_lookup+0x34a
 vfs_cache_lookup(e8389a98) at vfs_cache_lookup+0x92
 VOP_LOOKUP_APV(c0764180,e8389a98) at VOP_LOOKUP_APV+0x87
 lookup(e8389c24) at lookup+0x4a4
 namei(e8389c24) at namei+0x39a
 do_execve(c7e05600,e8389cb4,0) at do_execve+0x159
 kern_execve(c7e05600,e8389cb4,0) at kern_execve+0x74
 execve(c7e05600,e8389d04) at execve+0x32
 syscall(3b,3b,3b,0,8067038,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (59, FreeBSD ELF32, execve), eip = 0x2813d4df, esp = 0xbfbfeb9c, ebp = 0xbfbfebc8 ---
 
 Tracing command sh pid 893 tid 100177 td 0xc8966600
 sched_switch(c8966600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8964c90) at sleepq_switch+0xc1
 sleepq_wait_sig(c8964c90) at sleepq_wait_sig+0x1d
 msleep(c8964c90,c8964cf8,15c,c0713046,0) at msleep+0x26e
 kern_wait(c8966600,ffffffff,eaa5dc84,2,0) at kern_wait+0x6bb
 wait4(c8966600,eaa5dd04) at wait4+0x2a
 syscall(3b,3b,3b,bfbfebc8,37d,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2813d51f, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 ---
 
 Tracing command cron pid 892 tid 100059 td 0xc80e3a80
 sched_switch(c80e3a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8107990) at sleepq_switch+0xc1
 sleepq_wait_sig(c8107990) at sleepq_wait_sig+0x1d
 msleep(c8107990,c8107b00,14c,c0712d0e,0) at msleep+0x26e
 pipe_read(c8859948,ea85acbc,c8052080,0,c80e3a80) at pipe_read+0x377
 dofileread(c80e3a80,5,c8859948,ea85acbc,ffffffff,...) at dofileread+0x85
 kern_readv(c80e3a80,5,ea85acbc,805a000,1000,...) at kern_readv+0x36
 read(c80e3a80,ea85ad04) at read+0x45
 syscall(3b,3b,3b,28171040,28171040,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2814b953, esp = 0xbfbfe65c, ebp = 0xbfbfe678 ---
 
 Tracing command inetd pid 891 tid 100159 td 0xc84e8180
 sched_switch(c84e8180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(dbdb3fdc,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(dbdb3fdc,0,c0715e2f,e78,c071716d,...) at sleepq_wait+0x46
 msleep(dbdb3fdc,c07cf5c0,4c,c07165fb,0) at msleep+0x27d
 bwait(dbdb3fdc,4c,c07165fb) at bwait+0x47
 bufwait(dbdb3fdc,1,0,0,0,...) at bufwait+0x1a
 breadn(c824815c,0,0,800,0,...) at breadn+0x266
 bread(c824815c,0,0,800,0,ea9b18e8) at bread+0x20
 ffs_blkatoff(c824815c,0,0,0,ea9b1954) at ffs_blkatoff+0x9e
 ufs_lookup(ea9b19f8) at ufs_lookup+0x2cd
 VOP_CACHEDLOOKUP_APV(c0764180,ea9b19f8) at VOP_CACHEDLOOKUP_APV+0x9b
 vfs_cache_lookup(ea9b1a98) at vfs_cache_lookup+0xb5
 VOP_LOOKUP_APV(c0764180,ea9b1a98) at VOP_LOOKUP_APV+0x87
 lookup(ea9b1c24) at lookup+0x4a4
 namei(ea9b1c24) at namei+0x39a
 do_execve(c84e8180,ea9b1cb4,0) at do_execve+0x159
 kern_execve(c84e8180,ea9b1cb4,0) at kern_execve+0x74
 execve(c84e8180,ea9b1d04) at execve+0x32
 syscall(3b,3b,3b,bfbfedb0,805f1f0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (59, FreeBSD ELF32, execve), eip = 0x280f14df, esp = 0xbfbfdb6c, ebp = 0xbfbfdb88 ---
 
 Tracing command mc pid 851 tid 100161 td 0xc8332d80
 sched_switch(c8332d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(dbfdafa4,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(dbfdafa4,0,c9956638,b7,c071716d,...) at sleepq_wait+0x46
 msleep(dbfdafa4,c07811b4,50,c0716489,0,...) at msleep+0x27d
 acquire(ea9ab718,120,60000,c8332d80,0,...) at acquire+0x76
 lockmgr(dbfdafa4,202122,c9956638,c8332d80) at lockmgr+0x44e
 getblk(c9956570,0,0,1000,0,...) at getblk+0x12f
 breadn(c9956570,0,0,1000,0,...) at breadn+0x31
 bread(c9956570,0,0,1000,0,ea9ab7d4) at bread+0x20
 ffs_blkatoff(c9956570,800,0,ea9ab810,ea9ab814,c8064600,0,c07243ac,20f) at ffs_blkatoff+0x9e
 ufsdirhash_findfree(c973b000,18,ea9ab880,c8332d80,1,...) at ufsdirhash_findfree+0x11a
 ufs_lookup(ea9ab91c) at ufs_lookup+0x126
 VOP_CACHEDLOOKUP_APV(c0764180,ea9ab91c) at VOP_CACHEDLOOKUP_APV+0x9b
 vfs_cache_lookup(ea9ab9bc) at vfs_cache_lookup+0xb5
 VOP_LOOKUP_APV(c0764180,ea9ab9bc) at VOP_LOOKUP_APV+0x87
 lookup(ea9abbcc) at lookup+0x4a4
 namei(ea9abbcc) at namei+0x39a
 vn_open_cred(ea9abbcc,ea9abccc,180,c8591880,6,...) at vn_open_cred+0x5b
 vn_open(ea9abbcc,ea9abccc,180,6,c057a284,...) at vn_open+0x1e
 kern_open(c8332d80,8128900,0,602,180,...) at kern_open+0x231
 open(c8332d80,ea9abd04) at open+0x1a
 syscall(3b,3b,3b,80cfc20,8128900,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (5, FreeBSD ELF32, open), eip = 0x28360ab7, esp = 0xbfbfde7c, ebp = 0xbfbfde98 ---
 
 Tracing command csh pid 848 tid 100163 td 0xc8332a80
 sched_switch(c8332a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c851dcc4) at sleepq_switch+0xc1
 sleepq_wait_sig(c851dcc4) at sleepq_wait_sig+0x1d
 msleep(c851dcc4,c851dcf8,168,c070e01c,0) at msleep+0x26e
 kern_sigsuspend(c8332a80,2,0,0,0,...) at kern_sigsuspend+0xa3
 sigsuspend(c8332a80,ea9a5d04) at sigsuspend+0x33
 syscall(3b,3b,3b,8299008,bfbf1ad0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1abc, ebp = 0xbfbf1ae8 ---
 
 Tracing command su pid 847 tid 100162 td 0xc8332c00
 sched_switch(c8332c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c851da78) at sleepq_switch+0xc1
 sleepq_wait_sig(c851da78) at sleepq_wait_sig+0x1d
 msleep(c851da78,c851dae0,15c,c0713046,0) at msleep+0x26e
 kern_wait(c8332c00,350,ea9a8c84,2,0) at kern_wait+0x6bb
 wait4(c8332c00,ea9a8d04) at wait4+0x2a
 syscall(3b,3b,3b,bfbfe794,bfbfe810,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x280f451f, esp = 0xbfbfe70c, ebp = 0xbfbfe728 ---
 
 Tracing command csh pid 842 tid 100120 td 0xc8157480
 sched_switch(c8157480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c835224c) at sleepq_switch+0xc1
 sleepq_wait_sig(c835224c) at sleepq_wait_sig+0x1d
 msleep(c835224c,c8352280,168,c070e01c,0) at msleep+0x26e
 kern_sigsuspend(c8157480,2,0,0,0,...) at kern_sigsuspend+0xa3
 sigsuspend(c8157480,ea8fed04) at sigsuspend+0x33
 syscall(3b,3b,3b,81c1308,bfbf1bd0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1bbc, ebp = 0xbfbf1be8 ---
 
 Tracing command sshd pid 841 tid 100055 td 0xc8086d80
 sched_switch(c8086d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
 kern_select(c8086d80,9,807bb70,807bb80,0,...) at kern_select+0x4bf
 select(c8086d80,e8638d04) at select+0x44
 syscall(3b,3b,3b,0,bfbfe11c,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe0ac, ebp = 0xbfbfe0f8 ---
 
 Tracing command sshd pid 837 tid 100165 td 0xc8332780
 sched_switch(c8332780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c827720c) at sleepq_switch+0xc1
 sleepq_wait_sig(c827720c) at sleepq_wait_sig+0x1d
 msleep(c827720c,c82771d8,158,c0715361,0,...) at msleep+0x26e
 sbwait(c82771b4,ea99fc0c,c057a284,4,0,...) at sbwait+0x4b
 soreceive(c8277164,0,ea99fcbc,0,0,0) at soreceive+0x2ee
 soo_read(c83c2168,ea99fcbc,c849a980,0,c8332780) at soo_read+0x41
 dofileread(c8332780,5,c83c2168,ea99fcbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8332780,5,ea99fcbc,bfbfe108,4,...) at kern_readv+0x36
 read(c8332780,ea99fd04) at read+0x45
 syscall(3b,3b,3b,4,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x2834d953, esp = 0xbfbfe0ac, ebp = 0xbfbfe0d8 ---
 
 Tracing command httpd pid 836 tid 100124 td 0xc8153d80
 sched_switch(c8153d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80bba80) at sleepq_switch+0xc1
 sleepq_wait_sig(c80bba80) at sleepq_wait_sig+0x1d
 msleep(c80bba80,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80bba80,16,ffffffff,ffffffff,ea8f2cb0,...) at lf_setlock+0x181
 lf_advlock(ea8f2c90,c83c90b8,0,0,ea8f2c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8f2c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea8f2c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8153d80,ea8f2d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 835 tid 100125 td 0xc8153c00
 sched_switch(c8153c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8113680) at sleepq_switch+0xc1
 sleepq_wait_sig(c8113680) at sleepq_wait_sig+0x1d
 msleep(c8113680,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c8113680,16,ffffffff,ffffffff,ea8efcb0,...) at lf_setlock+0x181
 lf_advlock(ea8efc90,c83c90b8,0,0,ea8efc7c,...) at lf_advlock+0x203
 ufs_advlock(ea8efc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea8efc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8153c00,ea8efd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 834 tid 100136 td 0xc8158d80
 sched_switch(c8158d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80c0840) at sleepq_switch+0xc1
 sleepq_wait_sig(c80c0840) at sleepq_wait_sig+0x1d
 msleep(c80c0840,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80c0840,16,ffffffff,ffffffff,ea92ecb0,...) at lf_setlock+0x181
 lf_advlock(ea92ec90,c83c90b8,0,0,ea92ec7c,...) at lf_advlock+0x203
 ufs_advlock(ea92ec90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea92ec90) at VOP_ADVLOCK_APV+0xa9
 flock(c8158d80,ea92ed04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 833 tid 100137 td 0xc8158c00
 sched_switch(c8158c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8112880) at sleepq_switch+0xc1
 sleepq_wait_sig(c8112880) at sleepq_wait_sig+0x1d
 msleep(c8112880,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c8112880,16,ffffffff,ffffffff,ea92bcb0,...) at lf_setlock+0x181
 lf_advlock(ea92bc90,c83c90b8,0,0,ea92bc7c,...) at lf_advlock+0x203
 ufs_advlock(ea92bc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea92bc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8158c00,ea92bd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 832 tid 100138 td 0xc8158a80
 sched_switch(c8158a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea6080) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea6080) at sleepq_wait_sig+0x1d
 msleep(c7ea6080,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c7ea6080,16,ffffffff,ffffffff,ea928cb0,...) at lf_setlock+0x181
 lf_advlock(ea928c90,c83c90b8,0,0,ea928c7c,...) at lf_advlock+0x203
 ufs_advlock(ea928c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea928c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8158a80,ea928d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 831 tid 100139 td 0xc8158900
 sched_switch(c8158900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80c08c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c80c08c0) at sleepq_wait_sig+0x1d
 msleep(c80c08c0,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80c08c0,16,ffffffff,ffffffff,ea925cb0,...) at lf_setlock+0x181
 lf_advlock(ea925c90,c83c90b8,0,0,ea925c7c,...) at lf_advlock+0x203
 ufs_advlock(ea925c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea925c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8158900,ea925d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 830 tid 100140 td 0xc8158780
 sched_switch(c8158780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c81124c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c81124c0) at sleepq_wait_sig+0x1d
 msleep(c81124c0,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c81124c0,16,ffffffff,ffffffff,ea922cb0,...) at lf_setlock+0x181
 lf_advlock(ea922c90,c83c90b8,0,0,ea922c7c,...) at lf_advlock+0x203
 ufs_advlock(ea922c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea922c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8158780,ea922d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 829 tid 100141 td 0xc8086480
 sched_switch(c8086480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80c0800) at sleepq_switch+0xc1
 sleepq_wait_sig(c80c0800) at sleepq_wait_sig+0x1d
 msleep(c80c0800,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80c0800,16,ffffffff,ffffffff,e8626cb0,...) at lf_setlock+0x181
 lf_advlock(e8626c90,c83c90b8,0,0,e8626c7c,...) at lf_advlock+0x203
 ufs_advlock(e8626c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,e8626c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8086480,e8626d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 828 tid 100142 td 0xc8087c00
 sched_switch(c8087c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80bc840) at sleepq_switch+0xc1
 sleepq_wait_sig(c80bc840) at sleepq_wait_sig+0x1d
 msleep(c80bc840,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80bc840,16,ffffffff,ffffffff,e8653cb0,...) at lf_setlock+0x181
 lf_advlock(e8653c90,c83c90b8,0,0,e8653c7c,...) at lf_advlock+0x203
 ufs_advlock(e8653c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,e8653c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8087c00,e8653d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 827 tid 100143 td 0xc8331480
 sched_switch(c8331480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80bc600) at sleepq_switch+0xc1
 sleepq_wait_sig(c80bc600) at sleepq_wait_sig+0x1d
 msleep(c80bc600,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80bc600,16,ffffffff,ffffffff,ea97bcb0,...) at lf_setlock+0x181
 lf_advlock(ea97bc90,c83c90b8,0,0,ea97bc7c,...) at lf_advlock+0x203
 ufs_advlock(ea97bc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea97bc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8331480,ea97bd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 826 tid 100144 td 0xc8087900
 sched_switch(c8087900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80c0980) at sleepq_switch+0xc1
 sleepq_wait_sig(c80c0980) at sleepq_wait_sig+0x1d
 msleep(c80c0980,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80c0980,16,ffffffff,ffffffff,e864dcb0,...) at lf_setlock+0x181
 lf_advlock(e864dc90,c83c90b8,0,0,e864dc7c,...) at lf_advlock+0x203
 ufs_advlock(e864dc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,e864dc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8087900,e864dd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 825 tid 100145 td 0xc84e9780
 sched_switch(c84e9780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea8cc0) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea8cc0) at sleepq_wait_sig+0x1d
 msleep(c7ea8cc0,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c7ea8cc0,16,ffffffff,ffffffff,ea9dbcb0,...) at lf_setlock+0x181
 lf_advlock(ea9dbc90,c83c90b8,0,0,ea9dbc7c,...) at lf_advlock+0x203
 ufs_advlock(ea9dbc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea9dbc90) at VOP_ADVLOCK_APV+0xa9
 flock(c84e9780,ea9dbd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 824 tid 100146 td 0xc84e9600
 sched_switch(c84e9600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80bbb00) at sleepq_switch+0xc1
 sleepq_wait_sig(c80bbb00) at sleepq_wait_sig+0x1d
 msleep(c80bbb00,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80bbb00,16,ffffffff,ffffffff,ea9d8cb0,...) at lf_setlock+0x181
 lf_advlock(ea9d8c90,c83c90b8,0,0,ea9d8c7c,...) at lf_advlock+0x203
 ufs_advlock(ea9d8c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea9d8c90) at VOP_ADVLOCK_APV+0xa9
 flock(c84e9600,ea9d8d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 823 tid 100147 td 0xc84e9480
 sched_switch(c84e9480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80c15c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c80c15c0) at sleepq_wait_sig+0x1d
 msleep(c80c15c0,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80c15c0,16,ffffffff,ffffffff,ea9d5cb0,...) at lf_setlock+0x181
 lf_advlock(ea9d5c90,c83c90b8,0,0,ea9d5c7c,...) at lf_advlock+0x203
 ufs_advlock(ea9d5c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea9d5c90) at VOP_ADVLOCK_APV+0xa9
 flock(c84e9480,ea9d5d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 822 tid 100148 td 0xc84e9300
 sched_switch(c84e9300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea8d00) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea8d00) at sleepq_wait_sig+0x1d
 msleep(c7ea8d00,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c7ea8d00,16,ffffffff,ffffffff,ea9d2cb0,...) at lf_setlock+0x181
 lf_advlock(ea9d2c90,c83c90b8,0,0,ea9d2c7c,...) at lf_advlock+0x203
 ufs_advlock(ea9d2c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea9d2c90) at VOP_ADVLOCK_APV+0xa9
 flock(c84e9300,ea9d2d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 821 tid 100149 td 0xc84e9180
 sched_switch(c84e9180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8112180) at sleepq_switch+0xc1
 sleepq_wait_sig(c8112180) at sleepq_wait_sig+0x1d
 msleep(c8112180,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c8112180,16,ffffffff,ffffffff,ea9cfcb0,...) at lf_setlock+0x181
 lf_advlock(ea9cfc90,c83c90b8,0,0,ea9cfc7c,...) at lf_advlock+0x203
 ufs_advlock(ea9cfc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea9cfc90) at VOP_ADVLOCK_APV+0xa9
 flock(c84e9180,ea9cfd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 820 tid 100042 td 0xc7e05480
 sched_switch(c7e05480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8113700) at sleepq_switch+0xc1
 sleepq_wait_sig(c8113700) at sleepq_wait_sig+0x1d
 msleep(c8113700,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c8113700,16,ffffffff,ffffffff,e8386cb0,...) at lf_setlock+0x181
 lf_advlock(e8386c90,c83c90b8,0,0,e8386c7c,...) at lf_advlock+0x203
 ufs_advlock(e8386c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,e8386c90) at VOP_ADVLOCK_APV+0xa9
 flock(c7e05480,e8386d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 819 tid 100057 td 0xc80e3d80
 sched_switch(c80e3d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c81123c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c81123c0) at sleepq_wait_sig+0x1d
 msleep(c81123c0,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c81123c0,16,ffffffff,ffffffff,ea860cb0,...) at lf_setlock+0x181
 lf_advlock(ea860c90,c83c90b8,0,0,ea860c7c,...) at lf_advlock+0x203
 ufs_advlock(ea860c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea860c90) at VOP_ADVLOCK_APV+0xa9
 flock(c80e3d80,ea860d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 818 tid 100093 td 0xc810fd80
 sched_switch(c810fd80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea8dc0) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea8dc0) at sleepq_wait_sig+0x1d
 msleep(c7ea8dc0,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c7ea8dc0,16,ffffffff,ffffffff,ea8a9cb0,...) at lf_setlock+0x181
 lf_advlock(ea8a9c90,c83c90b8,0,0,ea8a9c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8a9c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea8a9c90) at VOP_ADVLOCK_APV+0xa9
 flock(c810fd80,ea8a9d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 817 tid 100092 td 0xc8150000
 sched_switch(c8150000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80c1580) at sleepq_switch+0xc1
 sleepq_wait_sig(c80c1580) at sleepq_wait_sig+0x1d
 msleep(c80c1580,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80c1580,16,ffffffff,ffffffff,ea8b9cb0,...) at lf_setlock+0x181
 lf_advlock(ea8b9c90,c83c90b8,0,0,ea8b9c7c,...) at lf_advlock+0x203
 ufs_advlock(ea8b9c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea8b9c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8150000,ea8b9d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 816 tid 100106 td 0xc8331d80
 sched_switch(c8331d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea6b80) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea6b80) at sleepq_wait_sig+0x1d
 msleep(c7ea6b80,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c7ea6b80,16,ffffffff,ffffffff,ea98dcb0,...) at lf_setlock+0x181
 lf_advlock(ea98dc90,c83c90b8,0,0,ea98dc7c,...) at lf_advlock+0x203
 ufs_advlock(ea98dc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea98dc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8331d80,ea98dd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 815 tid 100054 td 0xc8087000
 sched_switch(c8087000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80c0a00) at sleepq_switch+0xc1
 sleepq_wait_sig(c80c0a00) at sleepq_wait_sig+0x1d
 msleep(c80c0a00,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80c0a00,16,ffffffff,ffffffff,e863bcb0,...) at lf_setlock+0x181
 lf_advlock(e863bc90,c83c90b8,0,0,e863bc7c,...) at lf_advlock+0x203
 ufs_advlock(e863bc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,e863bc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8087000,e863bd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 814 tid 100053 td 0xc8087180
 sched_switch(c8087180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80c0600) at sleepq_switch+0xc1
 sleepq_wait_sig(c80c0600) at sleepq_wait_sig+0x1d
 msleep(c80c0600,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80c0600,16,ffffffff,ffffffff,e863ecb0,...) at lf_setlock+0x181
 lf_advlock(e863ec90,c83c90b8,0,0,e863ec7c,...) at lf_advlock+0x203
 ufs_advlock(e863ec90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,e863ec90) at VOP_ADVLOCK_APV+0xa9
 flock(c8087180,e863ed04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 813 tid 100115 td 0xc8157c00
 sched_switch(c8157c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea8d40) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea8d40) at sleepq_wait_sig+0x1d
 msleep(c7ea8d40,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c7ea8d40,16,ffffffff,ffffffff,ea90dcb0,...) at lf_setlock+0x181
 lf_advlock(ea90dc90,c83c90b8,0,0,ea90dc7c,...) at lf_advlock+0x203
 ufs_advlock(ea90dc90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea90dc90) at VOP_ADVLOCK_APV+0xa9
 flock(c8157c00,ea90dd04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 812 tid 100066 td 0xc810f600
 sched_switch(c810f600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80bba00) at sleepq_switch+0xc1
 sleepq_wait_sig(c80bba00) at sleepq_wait_sig+0x1d
 msleep(c80bba00,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80bba00,16,ffffffff,ffffffff,ea89acb0,...) at lf_setlock+0x181
 lf_advlock(ea89ac90,c83c90b8,0,0,ea89ac7c,...) at lf_advlock+0x203
 ufs_advlock(ea89ac90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea89ac90) at VOP_ADVLOCK_APV+0xa9
 flock(c810f600,ea89ad04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 811 tid 100108 td 0xc8331a80
 sched_switch(c8331a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80bc7c0) at sleepq_switch+0xc1
 sleepq_wait_sig(c80bc7c0) at sleepq_wait_sig+0x1d
 msleep(c80bc7c0,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80bc7c0,16,ffffffff,ffffffff,ea987cb0,...) at lf_setlock+0x181
 lf_advlock(ea987c90,c83c90b8,0,0,ea987c7c,...) at lf_advlock+0x203
 ufs_advlock(ea987c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea987c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8331a80,ea987d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 810 tid 100109 td 0xc8331900
 sched_switch(c8331900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8112240) at sleepq_switch+0xc1
 sleepq_wait_sig(c8112240) at sleepq_wait_sig+0x1d
 msleep(c8112240,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c8112240,16,ffffffff,ffffffff,ea984cb0,...) at lf_setlock+0x181
 lf_advlock(ea984c90,c83c90b8,0,0,ea984c7c,...) at lf_advlock+0x203
 ufs_advlock(ea984c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea984c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8331900,ea984d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 809 tid 100110 td 0xc8331780
 sched_switch(c8331780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80c1e80) at sleepq_switch+0xc1
 sleepq_wait_sig(c80c1e80) at sleepq_wait_sig+0x1d
 msleep(c80c1e80,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c80c1e80,16,ffffffff,ffffffff,ea981cb0,...) at lf_setlock+0x181
 lf_advlock(ea981c90,c83c90b8,0,0,ea981c7c,...) at lf_advlock+0x203
 ufs_advlock(ea981c90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea981c90) at VOP_ADVLOCK_APV+0xa9
 flock(c8331780,ea981d04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 808 tid 100111 td 0xc8331600
 sched_switch(c8331600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8112300) at sleepq_switch+0xc1
 sleepq_wait_sig(c8112300) at sleepq_wait_sig+0x1d
 msleep(c8112300,0,168,c074f7c4,0) at msleep+0x26e
 lf_setlock(c8112300,16,ffffffff,ffffffff,ea97ecb0,...) at lf_setlock+0x181
 lf_advlock(ea97ec90,c83c90b8,0,0,ea97ec7c,...) at lf_advlock+0x203
 ufs_advlock(ea97ec90) at ufs_advlock+0x1c
 VOP_ADVLOCK_APV(c0764180,ea97ec90) at VOP_ADVLOCK_APV+0xa9
 flock(c8331600,ea97ed04) at flock+0x1a1
 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---
 
 Tracing command httpd pid 807 tid 100112 td 0xc8158180
 sched_switch(c8158180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,3b7,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
 poll(c8158180,ea916d04) at poll+0x3b9
 syscall(3b,3b,3b,2,28,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (209, FreeBSD ELF32, poll), eip = 0x282c3a87, esp = 0xbfbfebcc, ebp = 0xbfbfec18 ---
 
 Tracing command getty pid 806 tid 100089 td 0xc8150480
 sched_switch(c8150480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(dbd4b0bc,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(dbd4b0bc,0,c0715e2f,e78,c0727ac1,...) at sleepq_wait+0x46
 msleep(dbd4b0bc,c07cf5c0,44,c0727ce4,0) at msleep+0x27d
 bwait(dbd4b0bc,44,c0727ce4) at bwait+0x47
 vnode_pager_generic_getpages(c98dc414,ea8c2acc,5000,0,c9752294,...) at vnode_pager_generic_getpages+0x847
 ffs_getpages(ea8c2a88) at ffs_getpages+0x100
 VOP_GETPAGES_APV(c0764180,ea8c2a88) at VOP_GETPAGES_APV+0xa9
 vnode_pager_getpages(c9752294,ea8c2acc,5,0) at vnode_pager_getpages+0xa5
 exec_map_first_page(ea8c2be4) at exec_map_first_page+0x16d
 do_execve(c8150480,ea8c2cb4,0) at do_execve+0x1e4
 kern_execve(c8150480,ea8c2cb4,0) at kern_execve+0x74
 execve(c8150480,ea8c2d04) at execve+0x32
 syscall(3b,3b,3b,bfbfee18,bfbfedd0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (59, FreeBSD ELF32, execve), eip = 0x280df4df, esp = 0xbfbfedbc, ebp = 0xbfbfee08 ---
 
 Tracing command getty pid 805 tid 100084 td 0xc8150c00
 sched_switch(c8150c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e9c410) at sleepq_switch+0xc1
 sleepq_wait_sig(c7e9c410) at sleepq_wait_sig+0x1d
 msleep(c7e9c410,0,159,c0714584,0) at msleep+0x26e
 ttysleep(c7e9c400,c7e9c410,159,c0714584,0,...) at ttysleep+0x21
 ttread(c7e9c400,ea8d1cbc,0) at ttread+0x48f
 ttyread(c7e8f600,ea8d1cbc,0,ea8d1c14,c052f435,...) at ttyread+0x2f
 scread(c7e8f600,ea8d1cbc,0,c0781300,0,...) at scread+0x22
 giant_read(c7e8f600,ea8d1cbc,0) at giant_read+0x2d
 devfs_read_f(c80f37e0,ea8d1cbc,c7ce0780,0,c8150c00) at devfs_read_f+0x62
 dofileread(c8150c00,0,c80f37e0,ea8d1cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8150c00,0,ea8d1cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c8150c00,ea8d1d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 804 tid 100064 td 0xc810f900
 sched_switch(c810f900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea4010) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea4010) at sleepq_wait_sig+0x1d
 msleep(c7ea4010,0,159,c0714584,0) at msleep+0x26e
 ttysleep(c7ea4000,c7ea4010,159,c0714584,0,...) at ttysleep+0x21
 ttread(c7ea4000,ea8a0cbc,0) at ttread+0x48f
 ttyread(c7e8f700,ea8a0cbc,0,ea8a0c14,c052f435,...) at ttyread+0x2f
 scread(c7e8f700,ea8a0cbc,0,c0781300,0,...) at scread+0x22
 giant_read(c7e8f700,ea8a0cbc,0) at giant_read+0x2d
 devfs_read_f(c80f6048,ea8a0cbc,c7ce0780,0,c810f900) at devfs_read_f+0x62
 dofileread(c810f900,0,c80f6048,ea8a0cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c810f900,0,ea8a0cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c810f900,ea8a0d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 803 tid 100117 td 0xc8157900
 sched_switch(c8157900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea3c10) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea3c10) at sleepq_wait_sig+0x1d
 msleep(c7ea3c10,0,159,c0714584,0) at msleep+0x26e
 ttysleep(c7ea3c00,c7ea3c10,159,c0714584,0,...) at ttysleep+0x21
 ttread(c7ea3c00,ea907cbc,0) at ttread+0x48f
 ttyread(c7e90600,ea907cbc,0,ea907c14,c052f435,...) at ttyread+0x2f
 scread(c7e90600,ea907cbc,0,c0781300,0,...) at scread+0x22
 giant_read(c7e90600,ea907cbc,0) at giant_read+0x2d
 devfs_read_f(c80f3828,ea907cbc,c7ce0780,0,c8157900) at devfs_read_f+0x62
 dofileread(c8157900,0,c80f3828,ea907cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8157900,0,ea907cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c8157900,ea907d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 802 tid 100061 td 0xc80e3780
 sched_switch(c80e3780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea4c10) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea4c10) at sleepq_wait_sig+0x1d
 msleep(c7ea4c10,0,159,c0714584,0) at msleep+0x26e
 ttysleep(c7ea4c00,c7ea4c10,159,c0714584,0,...) at ttysleep+0x21
 ttread(c7ea4c00,ea854cbc,0) at ttread+0x48f
 ttyread(c7e90500,ea854cbc,0,ea854c14,c052f435,...) at ttyread+0x2f
 scread(c7e90500,ea854cbc,0,c0781300,0,...) at scread+0x22
 giant_read(c7e90500,ea854cbc,0) at giant_read+0x2d
 devfs_read_f(c82a14c8,ea854cbc,c7ce0780,0,c80e3780) at devfs_read_f+0x62
 dofileread(c80e3780,0,c82a14c8,ea854cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c80e3780,0,ea854cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c80e3780,ea854d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 801 tid 100078 td 0xc8153600
 sched_switch(c8153600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea4810) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea4810) at sleepq_wait_sig+0x1d
 msleep(c7ea4810,0,159,c0714584,0) at msleep+0x26e
 ttysleep(c7ea4800,c7ea4810,159,c0714584,0,...) at ttysleep+0x21
 ttread(c7ea4800,ea8e3cbc,0) at ttread+0x48f
 ttyread(c7e90400,ea8e3cbc,0,ea8e3c14,c052f435,...) at ttyread+0x2f
 scread(c7e90400,ea8e3cbc,0,c0781300,0,...) at scread+0x22
 giant_read(c7e90400,ea8e3cbc,0) at giant_read+0x2d
 devfs_read_f(c82a1678,ea8e3cbc,c7ce0780,0,c8153600) at devfs_read_f+0x62
 dofileread(c8153600,0,c82a1678,ea8e3cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8153600,0,ea8e3cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c8153600,ea8e3d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 800 tid 100088 td 0xc8150600
 sched_switch(c8150600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea2c10) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea2c10) at sleepq_wait_sig+0x1d
 msleep(c7ea2c10,0,159,c0714584,0) at msleep+0x26e
 ttysleep(c7ea2c00,c7ea2c10,159,c0714584,0,...) at ttysleep+0x21
 ttread(c7ea2c00,ea8c5cbc,0) at ttread+0x48f
 ttyread(c7e90200,ea8c5cbc,0,ea8c5c14,c052f435,...) at ttyread+0x2f
 scread(c7e90200,ea8c5cbc,0,c0781300,0,...) at scread+0x22
 giant_read(c7e90200,ea8c5cbc,0) at giant_read+0x2d
 devfs_read_f(c80f6240,ea8c5cbc,c7ce0780,0,c8150600) at devfs_read_f+0x62
 dofileread(c8150600,0,c80f6240,ea8c5cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c8150600,0,ea8c5cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c8150600,ea8c5d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 799 tid 100075 td 0xc80e6780
 sched_switch(c80e6780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea2810) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea2810) at sleepq_wait_sig+0x1d
 msleep(c7ea2810,0,159,c0714584,0) at msleep+0x26e
 ttysleep(c7ea2800,c7ea2810,159,c0714584,0,...) at ttysleep+0x21
 ttread(c7ea2800,ea872cbc,0) at ttread+0x48f
 ttyread(c7e90100,ea872cbc,0,ea872c14,c052f435,...) at ttyread+0x2f
 scread(c7e90100,ea872cbc,0,c0781300,0,...) at scread+0x22
 giant_read(c7e90100,ea872cbc,0) at giant_read+0x2d
 devfs_read_f(c80f6798,ea872cbc,c7ce0780,0,c80e6780) at devfs_read_f+0x62
 dofileread(c80e6780,0,c80f6798,ea872cbc,ffffffff,...) at dofileread+0x85
 kern_readv(c80e6780,0,ea872cbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c80e6780,ea872d04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command getty pid 798 tid 100040 td 0xc7e05780
 sched_switch(c7e05780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ea3810) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ea3810) at sleepq_wait_sig+0x1d
 msleep(c7ea3810,0,159,c0714584,0) at msleep+0x26e
 ttysleep(c7ea3800,c7ea3810,159,c0714584,0,...) at ttysleep+0x21
 ttread(c7ea3800,e838ccbc,0) at ttread+0x48f
 ttyread(c7e90000,e838ccbc,0,e838cc14,c052f435,...) at ttyread+0x2f
 scread(c7e90000,e838ccbc,0,c0781300,0,...) at scread+0x22
 giant_read(c7e90000,e838ccbc,0) at giant_read+0x2d
 devfs_read_f(c80f6120,e838ccbc,c7ce0780,0,c7e05780) at devfs_read_f+0x62
 dofileread(c7e05780,0,c80f6120,e838ccbc,ffffffff,...) at dofileread+0x85
 kern_readv(c7e05780,0,e838ccbc,bfbfee1f,1,...) at kern_readv+0x36
 read(c7e05780,e838cd04) at read+0x45
 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---
 
 Tracing command inetd pid 784 tid 100119 td 0xc8157600
 sched_switch(c8157600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
 kern_select(c8157600,7,bfbfe8a0,0,0,...) at kern_select+0x4bf
 select(c8157600,ea901d04) at select+0x44
 syscall(3b,3b,3b,bfbfe8a0,381,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2815a8d3, esp = 0xbfbfdb7c, ebp = 0xbfbfee68 ---
 
 Tracing command httpd pid 764 tid 100116 td 0xc8157a80
 sched_switch(c8157a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07cef64,c07cef64,3e9,ea90ab40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07cef64,c07cef40,3e9) at cv_timedwait_sig+0x194
 kern_select(c8157a80,0,0,0,0,...) at kern_select+0x4a9
 select(c8157a80,ea90ad04) at select+0x44
 syscall(3b,3b,3b,0,f4240,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2830c8d3, esp = 0xbfbfec3c, ebp = 0xbfbfec78 ---
 
 Tracing command mysqld pid 763 tid 100133 td 0xc8086c00
 sched_switch(c8086c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8154034) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e
 msleep(c8154034,c8156068,168,c070b128,7ba,...) at msleep+0x244
 kse_release(c8086c00,e8635d04) at kse_release+0x1df
 syscall(3b,3b,3b,8530000,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x8534f28, ebp = 0x8534f64 ---
 
 Tracing command mysqld pid 763 tid 100135 td 0xc8086900
 sched_switch(c8086900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8154034) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e
 msleep(c8154034,c8156068,168,c070b128,3e9,...) at msleep+0x244
 kse_release(c8086900,e862fd04) at kse_release+0x1df
 syscall(3b,3b,3b,8530b00,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7ef28, ebp = 0x13c7ef64 ---
 
 Tracing command mysqld pid 763 tid 100091 td 0xc8086780
 sched_switch(c8086780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8154034) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e
 msleep(c8154034,c8156068,168,c070b128,7ba,...) at msleep+0x244
 kse_release(c8086780,e862cd04) at kse_release+0x1df
 syscall(3b,3b,3b,8530a00,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7af28, ebp = 0x13c7af64 ---
 
 Tracing command mysqld pid 763 tid 100127 td 0xc8087780
 sched_switch(c8087780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8154034) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e
 msleep(c8154034,c8156068,168,c070b128,7ba,...) at msleep+0x244
 kse_release(c8087780,e864ad04) at kse_release+0x1df
 syscall(3b,3b,3b,8530900,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c76f28, ebp = 0x13c76f64 ---
 
 Tracing command mysqld pid 763 tid 100134 td 0xc8086000
 sched_switch(c8086000,c8086c00,1) at sched_switch+0x177
 mi_switch(1,c8086c00) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
 kern_select(c8086000,f,bfbfebf0,0,0,...) at kern_select+0x4bf
 select(c8086000,e861dd04) at select+0x44
 syscall(3b,3b,3b,8536000,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2861b8d3, esp = 0xbfbfeacc, ebp = 0xbfbfeb08 ---
 
 Tracing command mysqld pid 763 tid 100131 td 0xc80e3000
 sched_switch(c80e3000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(ea845c14) at sleepq_switch+0xc1
 sleepq_wait_sig(ea845c14) at sleepq_wait_sig+0x1d
 msleep(ea845c14,c8156068,168,c070b120,0,...) at msleep+0x26e
 kern_sigtimedwait(c80e3000,26005,0,0,0,ea845c80,0) at kern_sigtimedwait+0x30b
 sigtimedwait(c80e3000,ea845d04) at sigtimedwait+0x6b
 syscall(3b,3b,3b,13c83e00,bf0f5f37,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (345, FreeBSD ELF32, sigtimedwait), eip = 0x284735b7, esp = 0xbf0f5e2c, ebp = 0xbf0f5ee8 ---
 
 Tracing command mysqld pid 763 tid 100126 td 0xc80e3480
 sched_switch(c80e3480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8156138) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c8156138) at sleepq_timedwait_sig+0x1e
 msleep(c8156138,c8156068,168,c070b11d,7531,...) at msleep+0x244
 kse_release(c80e3480,ea84ed04) at kse_release+0x121
 syscall(3b,3b,3b,8530800,81,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0xbf9fef2c, ebp = 0xbf9fef88 ---
 
 Tracing command sh pid 733 tid 100096 td 0xc8158480
 sched_switch(c8158480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8156a78) at sleepq_switch+0xc1
 sleepq_wait_sig(c8156a78) at sleepq_wait_sig+0x1d
 msleep(c8156a78,c8156ae0,15c,c0713046,0) at msleep+0x26e
 kern_wait(c8158480,ffffffff,ea91cc84,2,0) at kern_wait+0x6bb
 wait4(c8158480,ea91cd04) at wait4+0x2a
 syscall(3b,3b,3b,bfbfe678,2d,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2813d51f, esp = 0xbfbfe62c, ebp = 0xbfbfe648 ---
 
 Tracing command cron pid 730 tid 100101 td 0xc8332600
 sched_switch(c8332600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c0781bec) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c0781bec) at sleepq_timedwait_sig+0x1e
 msleep(c0781bec,0,15c,c070eeb5,ea61,...) at msleep+0x244
 kern_nanosleep(c8332600,ea99cccc,ea99ccc4) at kern_nanosleep+0xab
 nanosleep(c8332600,ea99cd04) at nanosleep+0x4f
 syscall(3b,3b,3b,8050e6c,3c,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 ---
 
 Tracing command snmpd pid 722 tid 100080 td 0xc8153300
 sched_switch(c8153300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07cef64,c07cef64,1389,ea8ddb40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07cef64,c07cef40,1389) at cv_timedwait_sig+0x194
 kern_select(c8153300,9,bfbfed50,bfbfecd0,bfbfec50,...) at kern_select+0x4a9
 select(c8153300,ea8ddd04) at select+0x44
 syscall(3b,3b,3b,bfbfecd0,bfbfec38,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x284598d3, esp = 0xbfbfea4c, ebp = 0xbfbfee68 ---
 
 Tracing command cron pid 711 tid 100077 td 0xc8153780
 sched_switch(c8153780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c0781bec) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c0781bec) at sleepq_timedwait_sig+0x1e
 msleep(c0781bec,0,15c,c070eeb5,ea61,...) at msleep+0x244
 kern_nanosleep(c8153780,ea8e6ccc,ea8e6cc4) at kern_nanosleep+0xab
 nanosleep(c8153780,ea8e6d04) at nanosleep+0x4f
 syscall(3b,3b,3b,805008c,3c,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 ---
 
 Tracing command sendmail pid 705 tid 100060 td 0xc80e3900
 sched_switch(c80e3900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c80e2894) at sleepq_switch+0xc1
 sleepq_wait_sig(c80e2894) at sleepq_wait_sig+0x1d
 msleep(c80e2894,c80e28c8,168,c070e01c,0) at msleep+0x26e
 kern_sigsuspend(c80e3900,0,0,0,0,...) at kern_sigsuspend+0xa3
 sigsuspend(c80e3900,ea857d04) at sigsuspend+0x33
 syscall(3b,3b,3b,0,bfbfd000,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2827e3fb, esp = 0xbfbfcfec, ebp = 0xbfbfd018 ---
 
 Tracing command sendmail pid 699 tid 100051 td 0xc8087480
 sched_switch(c8087480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07cef64,c07cef64,1389,e8644b40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07cef64,c07cef40,1389) at cv_timedwait_sig+0x194
 kern_select(c8087480,6,bfbfc530,0,0,...) at kern_select+0x4a9
 select(c8087480,e8644d04) at select+0x44
 syscall(3b,3b,3b,2,5,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x282fc8d3, esp = 0xbfbfc44c, ebp = 0xbfbfd038 ---
 
 Tracing command sshd pid 694 tid 100079 td 0xc8153480
 sched_switch(c8153480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
 kern_select(c8153480,7,807b920,0,0,...) at kern_select+0x4bf
 select(c8153480,ea8e0d04) at select+0x44
 syscall(3b,3b,3b,1,a,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe16c, ebp = 0xbfbfee78 ---
 
 Tracing command ntpd pid 669 tid 100104 td 0xc8332180
 sched_switch(c8332180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
 kern_select(c8332180,a,bfbfed60,0,0,...) at kern_select+0x4bf
 select(c8332180,ea993d04) at select+0x44
 syscall(3b,3b,3b,bfbfed60,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2828a8d3, esp = 0xbfbfed1c, ebp = 0xbfbfedf8 ---
 
 Tracing command usbd pid 638 tid 100056 td 0xc80e6000
 sched_switch(c80e6000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07cef64,c07cef64,7531,ea863b40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07cef64,c07cef40,7531) at cv_timedwait_sig+0x194
 kern_select(c80e6000,6,bfbfeda0,bfbfed20,0,...) at kern_select+0x4a9
 select(c80e6000,ea863d04) at select+0x44
 syscall(3b,3b,3b,bfbfed20,1e,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2813c8d3, esp = 0xbfbfeccc, ebp = 0xbfbfee78 ---
 
 Tracing command accounting pid 591 tid 100065 td 0xc810f780
 sched_switch(c810f780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c077f760,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c077f760) at sleepq_timedwait+0x4a
 msleep(c077f760,0,7f,c0706bf1,3a98,c077f720,c07093bb,213) at msleep+0x259
 acct_thread(0,ea89dd38,0,c052cacc,0,...) at acct_thread+0xd5
 fork_exit(c052cacc,0,ea89dd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xea89dd6c, ebp = 0 ---
 
 Tracing command named pid 574 tid 100083 td 0xc8150d80
 sched_switch(c8150d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c07cef64,c07cef64,7531,ea8d4b40,246,...) at sleepq_timedwait_sig+0x1e
 cv_timedwait_sig(c07cef64,c07cef40,7531) at cv_timedwait_sig+0x194
 kern_select(c8150d80,1a,bfbfed60,bfbfece0,0,...) at kern_select+0x4a9
 select(c8150d80,ea8d4d04) at select+0x44
 syscall(3b,3b,3b,0,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2835e8d3, esp = 0xbfbfec9c, ebp = 0xbfbfedf8 ---
 
 Tracing command syslogd pid 554 tid 100087 td 0xc8150780
 sched_switch(c8150780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
 kern_select(c8150780,a,80541c0,0,0,...) at kern_select+0x4bf
 select(c8150780,ea8c8d04) at select+0x44
 syscall(3b,3b,3b,80541c4,80541c0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x2814c8d3, esp = 0xbfbfda0c, ebp = 0xbfbfee58 ---
 
 Tracing command ipmon pid 537 tid 100039 td 0xc7e05900
 sched_switch(c7e05900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c0781bec) at sleepq_switch+0xc1
 sleepq_timedwait_sig(c0781bec) at sleepq_timedwait_sig+0x1e
 msleep(c0781bec,0,15c,c070eeb5,3e9,...) at msleep+0x244
 kern_nanosleep(c7e05900,e838fccc,e838fcc4) at kern_nanosleep+0xab
 nanosleep(c7e05900,e838fd04) at nanosleep+0x4f
 syscall(3b,3b,3b,2,1,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2812f513, esp = 0xbfbfcd3c, ebp = 0xbfbfcd68 ---
 
 Tracing command devd pid 441 tid 100068 td 0xc810f300
 sched_switch(c810f300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cef64) at sleepq_switch+0xc1
 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
 cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
 kern_select(c810f300,5,bfbfe9d0,0,0,...) at kern_select+0x4bf
 select(c810f300,ea894d04) at select+0x44
 syscall(3b,3b,3b,0,8,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (93, FreeBSD ELF32, select), eip = 0x8075917, esp = 0xbfbfe96c, ebp = 0xbfbfee78 ---
 
 Tracing command adjkerntz pid 170 tid 100052 td 0xc8087300
 sched_switch(c8087300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c808567c) at sleepq_switch+0xc1
 sleepq_wait_sig(c808567c) at sleepq_wait_sig+0x1d
 msleep(c808567c,c80856b0,168,c070e01c,0) at msleep+0x26e
 kern_sigsuspend(c8087300,0,0,0,0,...) at kern_sigsuspend+0xa3
 sigsuspend(c8087300,e8641d04) at sigsuspend+0x33
 syscall(3b,3b,3b,0,ffffc7c0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x280bc3fb, esp = 0xbfbfed9c, ebp = 0xbfbfee78 ---
 
 Tracing command schedcpu pid 44 tid 100044 td 0xc7e05180
 sched_switch(c7e05180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(e8380d04,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(e8380d04) at sleepq_timedwait+0x4a
 msleep(e8380d04,0,0,c0706bf1,3e8) at msleep+0x259
 schedcpu_thread(0,e8380d38,0,c0565cbc,0,...) at schedcpu_thread+0x26
 fork_exit(c0565cbc,0,e8380d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8380d6c, ebp = 0 ---
 
 Tracing command softdepflush pid 43 tid 100045 td 0xc7e05000
 sched_switch(c7e05000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07dca74,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c07dca74) at sleepq_timedwait+0x4a
 msleep(c07dca74,c07dca40,44,c07229e3,3e8) at msleep+0x259
 softdep_flush(0,e837dd38,0,c0655db8,0,...) at softdep_flush+0x227
 fork_exit(c0655db8,0,e837dd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe837dd6c, ebp = 0 ---
 
 Tracing command vnlru pid 42 tid 100046 td 0xc7dbdd80
 sched_switch(c7dbdd80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c8084860,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c8084860) at sleepq_timedwait+0x4a
 msleep(c8084860,c07cf900,250,c0717bd5,3e8,c07cf980) at msleep+0x259
 vnlru_proc(0,e836dd38,0,c05aa450,0,...) at vnlru_proc+0xdf
 fork_exit(c05aa450,0,e836dd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe836dd6c, ebp = 0 ---
 
 Tracing command syncer pid 41 tid 100047 td 0xc7dbdc00
 sched_switch(c7dbdc00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c0766324,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c0766324,0,c07278c1,144,c0717b07,...) at sleepq_wait+0x46
 msleep(c0766324,c07dda00,44,c0727902,0) at msleep+0x27d
 getpbuf(c0766324,c99a1794,0,0,c99a1794,...) at getpbuf+0x44
 cluster_wbuild(c99a16cc,4000,0,0,4,...) at cluster_wbuild+0x16b
 vfs_bio_awrite(dbe7c9c4) at vfs_bio_awrite+0x1cd
 ffs_syncvnode(c99a16cc,3) at ffs_syncvnode+0x2c1
 ffs_sync(c822d000,3,c7dbdc00,c822d000,2,...) at ffs_sync+0x209
 sync_fsync(e836acbc) at sync_fsync+0x164
 VOP_FSYNC_APV(c0759ec0,e836acbc) at VOP_FSYNC_APV+0x9b
 sync_vnode(c822e7d8,c7dbdc00) at sync_vnode+0x106
 sched_sync(0,e836ad38,0,c05abf64,0,...) at sched_sync+0x1ed
 fork_exit(c05abf64,0,e836ad38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe836ad6c, ebp = 0 ---
 
 Tracing command bufdaemon pid 40 tid 100048 td 0xc7dbda80
 sched_switch(c7dbda80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07cf4e0,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c07cf4e0) at sleepq_timedwait+0x4a
 msleep(c07cf4e0,c07cf500,44,c07163bf,3e8) at msleep+0x259
 buf_daemon(0,e8367d38,0,c059d93c,0,...) at buf_daemon+0x108
 fork_exit(c059d93c,0,e8367d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8367d6c, ebp = 0 ---
 
 Tracing command pagezero pid 39 tid 100049 td 0xc7dbd900
 sched_switch(c7dbd900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07dda44,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c07dda44) at sleepq_timedwait+0x4a
 msleep(c07dda44,c07dd4c0,200,c0727a97,493e0,c07dd4c0,0,c0727a79,9d) at msleep+0x259
 vm_pagezero(0,e8364d38,0,c06870c4,0,...) at vm_pagezero+0x6c
 fork_exit(c06870c4,0,e8364d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8364d6c, ebp = 0 ---
 
 Tracing command vmdaemon pid 38 tid 100027 td 0xc7dbd180
 sched_switch(c7dbd180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07dd54c,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c07dd54c,0,c072771a,63e,0,...) at sleepq_wait+0x46
 msleep(c07dd54c,c07dd560,68,c07163bf,0,...) at msleep+0x27d
 vm_daemon(0,e8355d38,0,c06860d8,0,...) at vm_daemon+0x36
 fork_exit(c06860d8,0,e8355d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8355d6c, ebp = 0 ---
 
 Tracing command pagedaemon pid 37 tid 100028 td 0xc7dbd000
 sched_switch(c7dbd000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c07dd508,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c07dd508) at sleepq_timedwait+0x4a
 msleep(c07dd508,c07dd4c0,44,c07163bf,1388) at msleep+0x259
 vm_pageout(0,e8352d38,0,c0685d60,0,...) at vm_pageout+0x260
 fork_exit(c0685d60,0,e8352d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8352d6c, ebp = 0 ---
 
 Tracing command irq7: ppc0 pid 36 tid 100029 td 0xc7ce9d80
 fork_trampoline() at fork_trampoline
 
 Tracing command fdc0 pid 35 tid 100030 td 0xc7ce9c00
 sched_switch(c7ce9c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7e1e63c,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c7e1e63c) at sleepq_timedwait+0x4a
 msleep(c7e1e63c,c7e1e6f0,4c,c0706bf1,3e8) at msleep+0x259
 fdc_worker(c7e1e600,c7e1e6f0,0,c07293da,4a1) at fdc_worker+0x1fe
 fdc_thread(c7e1e600,e6929d38,c7e1e600,c0694bfc,0,...) at fdc_thread+0x54
 fork_exit(c0694bfc,c7e1e600,e6929d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6929d6c, ebp = 0 ---
 
 Tracing command swi0: sio pid 34 tid 100031 td 0xc7ce9a80
 sched_switch(c7ce9a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7e88c10,e6926d38,c7e88c10,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7e88c10,e6926d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6926d6c, ebp = 0 ---
 
 Tracing command irq1: atkbd0 pid 33 tid 100032 td 0xc7ce9900
 sched_switch(c7ce9900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7e89030,e6923d38,c7e89030,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7e89030,e6923d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6923d6c, ebp = 0 ---
 
 Tracing command irq22: fxp0 pid 32 tid 100033 td 0xc7ce9780
 sched_switch(c7ce9780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7e772d0,e6920d38,c7e772d0,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7e772d0,e6920d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6920d6c, ebp = 0 ---
 
 Tracing command irq20: mly0 pid 31 tid 100034 td 0xc7ce9600
 sched_switch(c7ce9600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7dfd920,e691dd38,c7dfd920,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7dfd920,e691dd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe691dd6c, ebp = 0 ---
 
 Tracing command irq15: ata1 pid 30 tid 100035 td 0xc7ce9480
 fork_trampoline() at fork_trampoline
 
 Tracing command irq14: ata0 pid 29 tid 100036 td 0xc7e05d80
 sched_switch(c7e05d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7e07300,e8398d38,c7e07300,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7e07300,e8398d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8398d6c, ebp = 0 ---
 
 Tracing command usb1 pid 28 tid 100037 td 0xc7e05c00
 sched_switch(c7e05c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7dae210,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c7dae210) at sleepq_timedwait+0x4a
 msleep(c7dae210,0,5c,c07057e6,ea60,c7debbc0) at msleep+0x259
 usb_event_thread(c7debbc0,e8395d38,c7debbc0,c050cf38,0,...) at usb_event_thread+0x70
 fork_exit(c050cf38,c7debbc0,e8395d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8395d6c, ebp = 0 ---
 
 Tracing command irq17: ehci0 pid 27 tid 100017 td 0xc7ce5900
 fork_trampoline() at fork_trampoline
 
 Tracing command usbtask pid 26 tid 100018 td 0xc7ce5780
 sched_switch(c7ce5780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c077ec04,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c077ec04,0,c054c4d4,c0781300,c07057cb,...) at sleepq_wait+0x46
 msleep(c077ec04,0,5c,c07057ed,0,...) at msleep+0x27d
 usb_task_thread(0,e6902d38,0,c050cfc8,0,...) at usb_task_thread+0x47
 fork_exit(c050cfc8,0,e6902d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6902d6c, ebp = 0 ---
 
 Tracing command usb0 pid 25 tid 100019 td 0xc7ce5600
 sched_switch(c7ce5600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7db4210,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c7db4210) at sleepq_timedwait+0x4a
 msleep(c7db4210,0,5c,c07057e6,ea60,c7dea580) at msleep+0x259
 usb_event_thread(c7dea580,e68ffd38,c7dea580,c050cf38,0,...) at usb_event_thread+0x70
 fork_exit(c050cf38,c7dea580,e68ffd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68ffd6c, ebp = 0 ---
 
 Tracing command irq16: ohci0 pid 24 tid 100020 td 0xc7ce5480
 fork_trampoline() at fork_trampoline
 
 Tracing command irq9: acpi0 pid 23 tid 100021 td 0xc7ce5300
 fork_trampoline() at fork_trampoline
 
 Tracing command swi5: + pid 22 tid 100022 td 0xc7ce5180
 fork_trampoline() at fork_trampoline
 
 Tracing command thread taskq pid 21 tid 100023 td 0xc7dbd780
 sched_switch(c7dbd780,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d43d80,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d43d80,0,c07113b9,49,c8053ab0,...) at sleepq_wait+0x46
 msleep(c7d43d80,c7d43d9c,0,c0706bf1,0) at msleep+0x27d
 taskqueue_thread_loop(c07872a8,e8361d38,c07872a8,c05753dc,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c05753dc,c07872a8,e8361d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8361d6c, ebp = 0 ---
 
 Tracing command swi6: Giant taskq pid 20 tid 100024 td 0xc7dbd600
 sched_switch(c7dbd600,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8b80,e835ed38,c7cc8b80,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7cc8b80,e835ed38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe835ed6c, ebp = 0 ---
 
 Tracing command swi6: task queue pid 19 tid 100025 td 0xc7dbd480
 sched_switch(c7dbd480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8b90,e835bd38,c7cc8b90,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7cc8b90,e835bd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe835bd6c, ebp = 0 ---
 
 Tracing command kqueue taskq pid 9 tid 100026 td 0xc7dbd300
 sched_switch(c7dbd300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d73100,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d73100,0,c07113b9,49,8,...) at sleepq_wait+0x46
 msleep(c7d73100,c7d7311c,0,c0706bf1,0) at msleep+0x27d
 taskqueue_thread_loop(c077f940,e8358d38,c077f940,c05753dc,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c05753dc,c077f940,e8358d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe8358d6c, ebp = 0 ---
 
 Tracing command acpi_task_2 pid 8 tid 100008 td 0xc7ce3300
 sched_switch(c7ce3300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d73180,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d73180,0,c07113b9,49,8,...) at sleepq_wait+0x46
 msleep(c7d73180,c7d7319c,0,c0706bf1,0) at msleep+0x27d
 taskqueue_thread_loop(c077bb08,e68dbd38,c077bb08,c05753dc,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c05753dc,c077bb08,e68dbd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68dbd6c, ebp = 0 ---
 
 Tracing command acpi_task_1 pid 7 tid 100009 td 0xc7ce3180
 sched_switch(c7ce3180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d73180,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d73180,0,c07113b9,49,8,...) at sleepq_wait+0x46
 msleep(c7d73180,c7d7319c,0,c0706bf1,0) at msleep+0x27d
 taskqueue_thread_loop(c077bb08,e68d8d38,c077bb08,c05753dc,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c05753dc,c077bb08,e68d8d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68d8d6c, ebp = 0 ---
 
 Tracing command acpi_task_0 pid 6 tid 100010 td 0xc7ce3000
 sched_switch(c7ce3000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7d73180,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c7d73180,0,c07113b9,49,c7ddf1e0,...) at sleepq_wait+0x46
 msleep(c7d73180,c7d7319c,0,c0706bf1,0) at msleep+0x27d
 taskqueue_thread_loop(c077bb08,e68d5d38,c077bb08,c05753dc,0,...) at taskqueue_thread_loop+0x78
 fork_exit(c05753dc,c077bb08,e68d5d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68d5d6c, ebp = 0 ---
 
 Tracing command swi2: cambio pid 18 tid 100011 td 0xc7ce9300
 sched_switch(c7ce9300,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8bc0,e6917d38,c7cc8bc0,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7cc8bc0,e6917d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6917d6c, ebp = 0 ---
 
 Tracing command xpt_thrd pid 5 tid 100012 td 0xc7ce9180
 sched_switch(c7ce9180,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c0776004,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
 sleepq_wait(c0776004,0,c054c4d4,c0781300,c06ecd75,...) at sleepq_wait+0x46
 msleep(c0776004,0,4c,c06ecd90,0) at msleep+0x27d
 xpt_scanner_thread(0,e6914d38,0,c0443a54,0,...) at xpt_scanner_thread+0x35
 fork_exit(c0443a54,0,e6914d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6914d6c, ebp = 0 ---
 
 Tracing command yarrow pid 17 tid 100013 td 0xc7ce9000
 sched_switch(c7ce9000,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c077cc00,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c077cc00) at sleepq_timedwait+0x4a
 msleep(c077cc00,0,0,c0706bf1,64) at msleep+0x259
 random_kthread(0,e6911d38,0,c04f5f30,0,...) at random_kthread+0x180
 fork_exit(c04f5f30,0,e6911d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6911d6c, ebp = 0 ---
 
 Tracing command g_down pid 4 tid 100014 td 0xc7ce5d80
 sched_switch(c7ce5d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c077f0a8,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c077f0a8) at sleepq_timedwait+0x4a
 msleep(c077f0a8,c077ef88,24c,c0706bf1,64) at msleep+0x259
 g_io_schedule_down(c7ce5d80) at g_io_schedule_down+0x56
 g_down_procbody(0,e690ed38,0,c051eeb8,0,...) at g_down_procbody+0x5a
 fork_exit(c051eeb8,0,e690ed38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe690ed6c, ebp = 0 ---
 
 Tracing command g_up pid 3 tid 100015 td 0xc7ce5c00
 sched_switch(c7ce5c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c077f0a4,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c077f0a4) at sleepq_timedwait+0x4a
 msleep(c077f0a4,c077efc8,24c,c0706bf1,64) at msleep+0x259
 g_io_schedule_up(c7ce5c00) at g_io_schedule_up+0x127
 g_up_procbody(0,e690bd38,0,c051ee58,0,...) at g_up_procbody+0x5a
 fork_exit(c051ee58,0,e690bd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe690bd6c, ebp = 0 ---
 
 Tracing command g_event pid 2 tid 100016 td 0xc7ce5a80
 sched_switch(c7ce5a80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c077f09c,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
 sleepq_timedwait(c077f09c) at sleepq_timedwait+0x4a
 msleep(c077f09c,0,4c,c0706bf1,64) at msleep+0x259
 g_event_procbody(0,e6908d38,0,c051ef18,0,...) at g_event_procbody+0x9e
 fork_exit(c051ef18,0,e6908d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe6908d6c, ebp = 0 ---
 
 Tracing command swi3: vm pid 16 tid 100000 td 0xc7ce5000
 fork_trampoline() at fork_trampoline
 
 Tracing command swi4: clock sio pid 15 tid 100001 td 0xc7ce3d80
 sched_switch(c7ce3d80,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8870,e68f0d38,c7cc8870,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7cc8870,e68f0d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68f0d6c, ebp = 0 ---
 
 Tracing command swi1: net pid 14 tid 100002 td 0xc7ce3c00
 sched_switch(c7ce3c00,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 ithread_loop(c7cc8880,e68edd38,c7cc8880,c0540f8c,0,...) at ithread_loop+0xda
 fork_exit(c0540f8c,c7cc8880,e68edd38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68edd6c, ebp = 0 ---
 
 Tracing command idle: cpu0 pid 13 tid 100003 td 0xc7ce3a80
 sched_switch(c07120f0,6a9,c07cee5c,e68eac40,c05798ee,...) at sched_switch+0x177
 (null)(c0712111,50000,c7ccb010,c07819d0,0,...) at 0
 __func__.0(c054cd38,0,2041,1,3db17,...) at 0xc0712111
 (null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa
 
 Tracing command idle: cpu1 pid 12 tid 100004 td 0xc7ce3900
 kdb_enter(c072a005) at kdb_enter+0x2b
 siointr1(c7e8c800,c07df9c0,0,c0729e0b,56e,...) at siointr1+0xce
 siointr(c7e8c800) at siointr+0x21
 intr_execute_handlers(c7cd74c8,e68e7c94,4,e68e7cd8,c06b1b23,...) at intr_execute_handlers+0xe1
 lapic_handle_intr(38) at lapic_handle_intr+0x2e
 Xapic_isr1() at Xapic_isr1+0x33
 --- interrupt, eip = 0xc06aaa4d, esp = 0xe68e7cd8, ebp = 0xe68e7cd8 ---
 acpi_cpu_c1(c07812c0,2,e68e7cf8,2,c7ce2860,...) at acpi_cpu_c1+0x5
 acpi_cpu_idle(e68e7d0c,c05402e9,c054028c,e68e7d24,c05400cc,...) at acpi_cpu_idle+0x15d
 cpu_idle(c054028c,e68e7d24,c05400cc,0,e68e7d38,...) at cpu_idle+0x28
 idle_proc(0,e68e7d38,0,c054028c,0,...) at idle_proc+0x5d
 fork_exit(c054028c,0,e68e7d38) at fork_exit+0xa0
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xe68e7d6c, ebp = 0 ---
 
 Tracing command idle: cpu2 pid 11 tid 100005 td 0xc7ce3780
 sched_switch(c07887c0,0,ffc06034) at sched_switch+0x177
 mi_switch(c054c729,c07cee58,e68e4c1c,c057a284,c07887c0,...) at mi_switch+0x270
 end(c0712111,50000,c7ccb010,c07819d0,0,...) at 0xe68e4bf0
 __func__.0(c054cd38,0,2041,1,3db17,...) at 0xc0712111
 (null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa
 
 Tracing command idle: cpu3 pid 10 tid 100006 td 0xc7ce3600
 sched_switch(28,12,c06b1eb1,c07f95b0,8,...) at sched_switch+0x177
 
 Tracing command init pid 1 tid 100007 td 0xc7ce3480
 sched_switch(c7ce3480,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c7ce8000) at sleepq_switch+0xc1
 sleepq_wait_sig(c7ce8000) at sleepq_wait_sig+0x1d
 msleep(c7ce8000,c7ce8068,15c,c0713046,0) at msleep+0x26e
 kern_wait(c7ce3480,ffffffff,e68dec84,0,0) at kern_wait+0x6bb
 wait4(c7ce3480,e68ded04) at wait4+0x2a
 syscall(3b,3b,3b,0,0,...) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8052f47, esp = 0xbfbfed7c, ebp = 0xbfbfed98 ---
 
 Tracing command swapper pid 0 tid 0 td 0xc077f3c0
 sched_switch(c077f3c0,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 scheduler(0,c1ec00,c1e000,0,c0440585,...) at scheduler+0x199
 mi_startup() at mi_startup+0x96
 begin() at begin+0x2c
 db> ps
   pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
   897   784   897     0  Ss      ufs      0xc820f46c inetd
   894   893   893     0  S       ufs      0xc82481b4 sh
   893   892   893     0  Ss      wait     0xc8964c90 sh
   892   711   711     0  S       piperd   0xc8107990 cron
   891   784   891     0  Ss      biord    0xdbdb3fdc inetd
   851   848   851     0  S+      getblk   0xdbfdafa4 mc
   848   847   848     0  S+      pause    0xc851dcc4 csh
   847   842   847  1001  S+      wait     0xc851da78 su
   842   841   842  1001  Ss+     pause    0xc835224c csh
   841   837   837  1001  S       select   0xc07cef64 sshd
   837   694   837     0  Ss      sbwait   0xc827720c sshd
   836   764   764 65534  S       lockf    0xc80bba80 httpd
   835   764   764 65534  S       lockf    0xc8113680 httpd
   834   764   764 65534  S       lockf    0xc80c0840 httpd
   833   764   764 65534  S       lockf    0xc8112880 httpd
   832   764   764 65534  S       lockf    0xc7ea6080 httpd
   831   764   764 65534  S       lockf    0xc80c08c0 httpd
   830   764   764 65534  S       lockf    0xc81124c0 httpd
   829   764   764 65534  S       lockf    0xc80c0800 httpd
   828   764   764 65534  S       lockf    0xc80bc840 httpd
   827   764   764 65534  S       lockf    0xc80bc600 httpd
   826   764   764 65534  S       lockf    0xc80c0980 httpd
   825   764   764 65534  S       lockf    0xc7ea8cc0 httpd
   824   764   764 65534  S       lockf    0xc80bbb00 httpd
   823   764   764 65534  S       lockf    0xc80c15c0 httpd
   822   764   764 65534  S       lockf    0xc7ea8d00 httpd
   821   764   764 65534  S       lockf    0xc8112180 httpd
   820   764   764 65534  S       lockf    0xc8113700 httpd
   819   764   764 65534  S       lockf    0xc81123c0 httpd
   818   764   764 65534  S       lockf    0xc7ea8dc0 httpd
   817   764   764 65534  S       lockf    0xc80c1580 httpd
   816   764   764 65534  S       lockf    0xc7ea6b80 httpd
   815   764   764 65534  S       lockf    0xc80c0a00 httpd
   814   764   764 65534  S       lockf    0xc80c0600 httpd
   813   764   764 65534  S       lockf    0xc7ea8d40 httpd
   812   764   764 65534  S       lockf    0xc80bba00 httpd
   811   764   764 65534  S       lockf    0xc80bc7c0 httpd
   810   764   764 65534  S       lockf    0xc8112240 httpd
   809   764   764 65534  S       lockf    0xc80c1e80 httpd
   808   764   764 65534  S       lockf    0xc8112300 httpd
   807   764   764 65534  S       select   0xc07cef64 httpd
   806     1   806     0  Ss+     vnread   0xdbd4b0bc getty
   805     1   805     0  Ss+     ttyin    0xc7e9c410 getty
   804     1   804     0  Ss+     ttyin    0xc7ea4010 getty
   803     1   803     0  Ss+     ttyin    0xc7ea3c10 getty
   802     1   802     0  Ss+     ttyin    0xc7ea4c10 getty
   801     1   801     0  Ss+     ttyin    0xc7ea4810 getty
   800     1   800     0  Ss+     ttyin    0xc7ea2c10 getty
   799     1   799     0  Ss+     ttyin    0xc7ea2810 getty
   798     1   798     0  Ss+     ttyin    0xc7ea3810 getty
   784     1   784     0  Ss      select   0xc07cef64 inetd
   764     1   764     0  Ss      select   0xc07cef64 httpd
   763   733    45   509  S+      (threaded)          mysqld
 100133                   S       kserel   0xc8154034 mysqld
 100135                   S       kserel   0xc8154034 mysqld
 100091                   S       kserel   0xc8154034 mysqld
 100127                   S       kserel   0xc8154034 mysqld
 100134                   S       select   0xc07cef64 mysqld
 100131                   S       sigwait  0xea845c14 mysqld
 100126                   S       ksesigwa 0xc8156138 mysqld
   733     1    45     0  S+      wait     0xc8156a78 sh
   730     1   730     0  Ss      nanslp   0xc0781bec cron
   722     1   721     0  S       select   0xc07cef64 snmpd
   711     1   711     0  Ss      nanslp   0xc0781bec cron
   705     1   705    25  Ss      pause    0xc80e2894 sendmail
   699     1   699     0  Ss      select   0xc07cef64 sendmail
   694     1   694     0  Ss      select   0xc07cef64 sshd
   669     1   669     0  Ss      select   0xc07cef64 ntpd
   638     1   638     0  Ss      select   0xc07cef64 usbd
   591     0     0     0  SL      -        0xc077f760 [accounting]
   574     1   574    53  Ss      select   0xc07cef64 named
   554     1   554     0  Ss      select   0xc07cef64 syslogd
   537     1   537     0  Ss      nanslp   0xc0781bec ipmon
   441     1   441     0  Ss      select   0xc07cef64 devd
   170     1   170     0  Ss      pause    0xc808567c adjkerntz
    44     0     0     0  SL      -        0xe8380d04 [schedcpu]
    43     0     0     0  SL      sdflush  0xc07dca74 [softdepflush]
    42     0     0     0  SL      vlruwt   0xc8084860 [vnlru]
    41     0     0     0  SL      wswbuf0  0xc0766324 [syncer]
    40     0     0     0  SL      psleep   0xc07cf4e0 [bufdaemon]
    39     0     0     0  SL      pgzero   0xc07dda44 [pagezero]
    38     0     0     0  SL      psleep   0xc07dd54c [vmdaemon]
    37     0     0     0  SL      psleep   0xc07dd508 [pagedaemon]
    36     0     0     0  WL                          [irq7: ppc0]
    35     0     0     0  SL      -        0xc7e1e63c [fdc0]
    34     0     0     0  WL                          [swi0: sio]
    33     0     0     0  WL                          [irq1: atkbd0]
    32     0     0     0  WL                          [irq22: fxp0]
    31     0     0     0  WL                          [irq20: mly0]
    30     0     0     0  WL                          [irq15: ata1]
    29     0     0     0  WL                          [irq14: ata0]
    28     0     0     0  SL      usbevt   0xc7dae210 [usb1]
    27     0     0     0  WL                          [irq17: ehci0]
    26     0     0     0  SL      usbtsk   0xc077ec04 [usbtask]
    25     0     0     0  SL      usbevt   0xc7db4210 [usb0]
    24     0     0     0  WL                          [irq16: ohci0]
    23     0     0     0  WL                          [irq9: acpi0]
    22     0     0     0  WL                          [swi5: +]
    21     0     0     0  SL      -        0xc7d43d80 [thread taskq]
    20     0     0     0  WL                          [swi6: Giant taskq]
    19     0     0     0  WL                          [swi6: task queue]
     9     0     0     0  SL      -        0xc7d73100 [kqueue taskq]
     8     0     0     0  SL      -        0xc7d73180 [acpi_task_2]
     7     0     0     0  SL      -        0xc7d73180 [acpi_task_1]
     6     0     0     0  SL      -        0xc7d73180 [acpi_task_0]
    18     0     0     0  WL                          [swi2: cambio]
     5     0     0     0  SL      ccb_scan 0xc0776004 [xpt_thrd]
    17     0     0     0  SL      -        0xc077cc00 [yarrow]
     4     0     0     0  SL      -        0xc077f0a8 [g_down]
     3     0     0     0  SL      -        0xc077f0a4 [g_up]
     2     0     0     0  SL      -        0xc077f09c [g_event]
    16     0     0     0  WL                          [swi3: vm]
    15     0     0     0  WL                          [swi4: clock sio]
    14     0     0     0  WL                          [swi1: net]
    13     0     0     0  RL      CPU 0               [idle: cpu0]
    12     0     0     0  RL      CPU 1               [idle: cpu1]
    11     0     0     0  RL      CPU 2               [idle: cpu2]
    10     0     0     0  RL      CPU 3               [idle: cpu3]
     1     0     1     0  SLs     wait     0xc7ce8000 [init]
     0     0     0     0  WLs                         [swapper]
 db>
 
 --
 Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
 Phone: +7 4732 539880
 Fax:   +7 4732 531415         http://www.vsi.ru
 CenterTelecom Voronezh ISP    http://isp.vsi.ru
 

From: Kostik Belousov <kostikbel@gmail.com>
To: Oleg Derevenetz <oleg@vsi.ru>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/104406: [ufs] Processes get stuck in "ufs" state under persistent CPU load
Date: Thu, 1 Nov 2007 19:00:34 +0200

 --vRjLMflJ/YJvC+Q3
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 On Thu, Nov 01, 2007 at 07:54:39PM +0300, Oleg Derevenetz wrote:
 > Here is a kernel dump:
 >=20
 > ftp://oleg.vsi.ru/private/vmcore.0.zip
 >=20
 > It was saved for another static hang state.
 
 I already analyzed one dump for you.
 
 --vRjLMflJ/YJvC+Q3
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.4 (FreeBSD)
 
 iD8DBQFHKgYxC3+MBN1Mb4gRAmLiAKDFbA5fRqBa/o8rEa6NrKg074OlxACg35SF
 4YrS9lx8uXfQTQK/DPcFB34=
 =zUz7
 -----END PGP SIGNATURE-----
 
 --vRjLMflJ/YJvC+Q3--

From: "Oleg Derevenetz" <oleg@vsi.ru>
To: <bug-followup@FreeBSD.org>, "Kostik Belousov" <kostikbel@gmail.com>
Cc:  
Subject: Re: kern/104406: [ufs] Processes get stuck in "ufs" state under persistent CPU load
Date: Thu, 1 Nov 2007 19:54:39 +0300

 Here is a kernel dump:
 
 ftp://oleg.vsi.ru/private/vmcore.0.zip
 
 It was saved for another static hang state.
 
 --
 Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
 Phone: +7 4732 539880
 Fax:   +7 4732 531415         http://www.vsi.ru
 CenterTelecom Voronezh ISP    http://isp.vsi.ru
 
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon May 18 03:02:12 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=104406 
>Unformatted:
