From moonshade@pnhz.kz  Mon Mar  6 13:30:09 2006
Return-Path: <moonshade@pnhz.kz>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 4DBC016A420
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  6 Mar 2006 13:30:09 +0000 (GMT)
	(envelope-from moonshade@pnhz.kz)
Received: from relay.pnhz.kz (relay.pnhz.kz [212.154.198.217])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 6740C43D46
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  6 Mar 2006 13:30:07 +0000 (GMT)
	(envelope-from moonshade@pnhz.kz)
Received: from abyss.pnhz.kz (abyss.pnhz.kz [192.168.1.29])
	by relay.pnhz.kz with ESMTP id k26DU54a086266
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 6 Mar 2006 19:30:05 +0600 (ALMT)
	(envelope-from moonshade@pnhz.kz)
Message-Id: <1141651795.28694@abyss.pnhz.kz>
Date: Mon, 6 Mar 2006 19:29:55 +0600
From: "Denis Eremenko <moonshade@pnhz.kz>" <moonshade@pnhz.kz>
To: "FreeBSD gnats submit" <FreeBSD-gnats-submit@freebsd.org>
Subject: amr broken with LSILogic MegaRAID Series 428 (HP NetRAID) on 6-STABLE: reading app hangs
X-Send-Pr-Version: gtk-send-pr 0.4.7 
X-GNATS-Notify:

>Number:         94139
>Category:       kern
>Synopsis:       [amr] [regression] amr broken with LSILogic MegaRAID Series 428 (HP NetRAID) on 6-STABLE: reading app hangs
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    scottl
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 06 13:40:02 GMT 2006
>Closed-Date:    
>Last-Modified:  Mon Jul 10 09:50:13 GMT 2006
>Originator:     Denis Eremenko <moonshade@pnhz.kz>
>Release:        FreeBSD 6.1-PRERELEASE i386
>Organization:
>Environment:


System: FreeBSD 6.1-PRERELEASE #0: Thu Feb 23 08:34:49 ALMT 2006
    root@abyss.pnhz.kz:/usr/obj/data/os/FreeBSD/src/6/src/sys/ABYSS-6



>Description:


Application hangs in D state (reading from disk) while reading from amrd
on 6-STABLE.  But amr(4) from 6.0-RELEASE works fine. I suppose it was
broken with "LSI MegaRAID improvements".

I can be wrong, but looks like it is most likely to hang about 8Gb...

Kernel built with:
options	SMP
options	apic
options	SCHED_4BSD
options	PREEMPTION
options	ADAPTIVE_GIANT



>How-To-Repeat:


dd if=/dev/amrd0 of=/dev/null bs=1m skip=8000



>Fix:


use amr(4) from 6.0-RELEASE


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->scottl 
Responsible-Changed-By: glebius 
Responsible-Changed-When: Mon Mar 13 11:20:18 UTC 2006 
Responsible-Changed-Why:  
Let Scott look at the PR, since he did the big change to this driver. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=94139 
State-Changed-From-To: open->feedback 
State-Changed-By: glebius 
State-Changed-When: Mon Mar 13 20:43:07 UTC 2006 
State-Changed-Why:  

A dump or a trace of the hung process from DDB would be appreciated. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=94139 

From: =?KOI8-R?Q?=E5=D2=C5=CD=C5=CE=CB=CF_=E4=C5=CE=C9=D3_=F7=CC=C1=C4?=
 =?KOI8-R?Q?=C9=CD=C9=D2=CF=D7=C9=DE?= <moonshade@pnhz.kz>
To: bug-followup@FreeBSD.org, moonshade@pnhz.kz
Cc:  
Subject: Re: kern/94139: [amr] [regression] amr broken with LSILogic MegaRAID
 Series 428 (HP NetRAID) on 6-STABLE: reading app hangs
Date: Thu, 23 Mar 2006 14:44:44 +0600

 I have managed to get more details on this problem.
 
 Previous note about 8Gb appeared to be wrong - application hung varies 
 (increasing load speeds up hung).
 
 Note about breakage with "LSI MegaRAID improvements" looks to be wrong 
 too. Change was big, but not breaking...
 I suppose breakage was done with last less massive change in amr(4) in 
 RELENG_6: "MFC: amr.c 1.75, amrvar.h 1.30". At least I have failed to 
 reproduce hang with amr.c and amrvar.h downgraded to versions 1.65.2.4 
 and 1.26.2.1 accordingly.
 
 I've run 'dd if=/dev/amrd0 of=/dev/null bs=1m' on one ttty 'find /' on 
 another.
 Some minutes later it hung...
 
 ctrl+t on "find" tty showed:
 load: 0.15  cmd: find 822 [getblk] 0.39u 3.00s 11% 884k
 
 ctrl+t on "dd" tty showed:
 load: 0.05  cmd: dd 800 [physrd] 0.00u 0.46s 0% 1688k
 
 DDB output (dropped to ddb with ctrl+alt+esc):
 db> bt
 Tracing pid 26 tid 100018 td 0xc2f7f780
 kdb_enter(c0613794) at kdb_enter+0x2b
 scgetc(c0673640,2,c066d40c,c304ee40,c0659520) at scgetc+0x4cc
 sckbdevent(c0659520,0,c0673640) at sckbdevent+0x1c8
 atkbd_intr(c0659520,0,d9a61cec,c04ab599,c0659520) at atkbd_intr+0x20
 atkbdintr(c0659520) at atkbdintr+0x16
 ithread_execute_handlers(c2fac830,c2f7c080) at 
 ithread_execute_handlers+0x125
 ithread_loop(c3060590,d9a61d38) at ithread_loop+0x55
 fork_exit(c04ab660,c3060590,d9a61d38) at fork_exit+0x71
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xd9a61d6c, ebp = 0 ---
 
 db> ps
   pid   proc     uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
 ...
   822 c31f6418    0   781   822 0004002 [SLPQ getblk 0xd3c60030][SLP] find
   800 c325a000    0   788   800 0004002 [SLPQ physrd 0xd3bbc118][SLP] dd
 ...
    21 c2ffaa3c    0     0     0 0000204 [IWAIT] irq9: amr0
 
 
 db> where 822
 Tracing pid 822 tid 100065 td 0xc31f7d80
 sched_switch(c31f7d80,0,1) at sched_switch+0x14b
 mi_switch(1,0,c31f7d80,db24e7e0,c04e07b8) at mi_switch+0x1ba
 sleepq_switch(d3c60030) at sleepq_switch+0x87
 sleepq_wait(d3c60030,0,c31f7d80,d3c60030,4) at sleepq_wait+0x5c
 msleep(d3c60030,c064e268,50,c06083ed,0) at msleep+0x261
 acquire(db24e860,120,60000,c31f7d80,0) at acquire+0x7b
 lockmgr(d3c60030,202122,c31887ec,c31f7d80) at lockmgr+0x3fe
 getblk(c3188770,b8800,0,4000,0) at getblk+0x1a5
 breadn(c3188770,b8800,0,4000,0) at breadn+0x31
 bread(c3188770,b8800,0,4000,0) at bread+0x20
 ffs_vget(c30f0400,cdc9,2,db24e9dc) at ffs_vget+0x330
 ufs_lookup(db24ea7c) at ufs_lookup+0xaa2
 VOP_CACHEDLOOKUP_APV(c063abc0,db24ea7c) at VOP_CACHEDLOOKUP_APV+0x38
 vfs_cache_lookup(db24eb18,c35ea440,c343be00,db24eb34,c0513e5d) at 
 vfs_cache_look
 up+0xb2
 VOP_LOOKUP_APV(c063abc0,db24eb18) at VOP_LOOKUP_APV+0x43
 lookup(db24eba0,db24eb78,1,c31f7d80,db24ebc8) at lookup+0x459
 namei(db24eba0,805d7c8,0,0,c312abb0) at namei+0x39e
 kern_lstat(c31f7d80,805d7c8,0,db24ec74) at kern_lstat+0x47
 lstat(c31f7d80,db24ed04,2,186,296) at lstat+0x1b
 syscall(805003b,bfbf003b,bfbf003b,bfbfeb60,805d780) at syscall+0x2bf
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (190, FreeBSD ELF32, lstat), eip = 0x2811d35f, esp = 
 0xbfbfeb3c, ebp
  = 0xbfbfebd8 ---
 
 db> where 800
 Tracing pid 800 tid 100077 td 0xc313fa80
 sched_switch(c313fa80,0,1) at sched_switch+0x14b
 mi_switch(1,0,c313fa80,db21dba4,c04e07b8) at mi_switch+0x1ba
 sleepq_switch(d3bbc118) at sleepq_switch+0x87
 sleepq_wait(d3bbc118,0,c313fa80,4c,d3bbc118) at sleepq_wait+0x5c
 msleep(d3bbc118,c0655e00,4c,c0603d3b,0) at msleep+0x261
 bwait(d3bbc118,4c,c0603d3b) at bwait+0x5f
 physio(c305d400,db21dcbc,0,100000,c305d400) at physio+0x207
 devfs_read_f(c2f4d630,db21dcbc,c32ce400,0,c313fa80) at devfs_read_f+0x62
 dofileread(c313fa80,3,c2f4d630,db21dcbc,ffffffff) at dofileread+0x85
 kern_readv(c313fa80,3,db21dcbc,80b1000,a0000) at kern_readv+0x36
 read(c313fa80,db21dd04,3,36,246) at read+0x45
 syscall(3b,3b,3b,2104,bfbfed2c) at syscall+0x2bf
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (3, FreeBSD ELF32, read), eip = 0x281260d3, esp = 
 0xbfbfec7c, ebp =
 0xbfbfecc8 ---
 
 db> where 21
 Tracing pid 21 tid 100026 td 0xc2ffb300
 sched_switch(c2ffb300,0,1) at sched_switch+0x14b
 mi_switch(1,0) at mi_switch+0x1ba
 ithread_loop(c2fee660,da67fd38) at ithread_loop+0xd9
 fork_exit(c04ab660,c2fee660,da67fd38) at fork_exit+0x71
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xda67fd6c, ebp = 0 ---
 
 
 Fix: downgrade amr.c to 1.65.2.4 and amrvar.h to 1.26.2.1.
 
 Hope it helps.
 
State-Changed-From-To: feedback->open 
State-Changed-By: glebius 
State-Changed-When: Sat May 6 09:42:25 UTC 2006 
State-Changed-Why:  
Feedback received. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=94139 

From: "John Archambeau" <jcarchambeau@gmail.com>
To: bug-followup@FreeBSD.org, moonshade@pnhz.kz
Cc:  
Subject: Re: kern/94139: [amr] [regression] amr broken with LSILogic MegaRAID Series 428 (HP NetRAID) on 6-STABLE: reading app hangs
Date: Wed, 17 May 2006 21:28:13 -0700

 I believe my PR may be related.
 
 It consistantly fails when attempting to do an installation of FreeBSD
 6.1 regardless of the installation medium to a RAID 5 on three channel
 LSILogic 428 controller (OEM HP).  I have only attempted ftp from a
 local server on my LAN and local CD ROM installation.  It consistantly
 fails after attempting to extract the /stand directory after the file
 systems are created on the array.  It may get to the first hunk of the
 /bin distribution then hang.
 
 I have not tried 6.0 yet to see if this is also the case.  The machine
 reliably installs FreeBSD 5.4 without issue to the RAID 5 and was
 running 4.6 without issue until a couple of weeks ago.  It was running
 5.4 until I backed everything off of it to do a 6.1 upgrade.
 
 I'm also going to test to see if I can get 6.1 to install on another
 machine with an LSILogic 428.  I have a few of them here.  I just
 successfully installed 6.1 on a uniprocessor machine that does not
 have an LSILogic 428.
 
 Is there anything you need from me to confirm or deny if this existing
 PR is related to my 6.1 installation issue?
 
 I can be reached via this e-mail address or jca at fundingworks dot biz.

From: Nikolay Pavlov <quetzal@zone3000.net>
To: bug-followup@FreeBSD.org, moonshade@pnhz.kz
Cc:  
Subject: Re: kern/94139: [amr] [regression] amr broken with LSILogic MegaRAID Series 428 (HP NetRAID) on 6-STABLE: reading app hangs
Date: Mon, 10 Jul 2006 12:46:41 +0300

 Hi. I think my problems is also related for this PR.
 I have many processes with D status in ps under load on this server.
 Now i can provide backtrace (at the end of this report) 
 of megarc process that also hangs now and than.
 
 Some kernel options:
 
 options         INVARIANTS
 options         INVARIANT_SUPPORT
 options         WITNESS
 options         DEBUG_LOCKS
 options         DEBUG_VFS_LOCKS
 options         DIAGNOSTIC
 options         DDB
 options         KDB
 options         KDB_TRACE
 options         GDB
 options         DEBUG_MEMGUARD
 options         KTRACE
 options         ALT_BREAK_TO_DEBUGGER
 options         SCHED_4BSD
 options         PREEMPTION
 options         ADAPTIVE_GIANT
 options         SMP
 
 device          apic
 device          amr
 
 
 root@ms1 /root #ps awx | grep D
   PID  TT  STAT      TIME COMMAND
     2  ??  DL     0:01.54 [g_event]
     3  ??  DL     1:24.65 [g_up]
     4  ??  DL     0:58.63 [g_down]
     5  ??  DL     0:00.00 [thread taskq]
     6  ??  DL     0:00.00 [kqueue taskq]
     7  ??  DL     0:00.00 [acpi_task0]
     8  ??  DL     0:00.00 [acpi_task1]
     9  ??  DL     0:00.00 [acpi_task2]
    10  ??  DL     0:00.00 [ktrace]
    18  ??  DL     0:31.49 [yarrow]
    32  ??  DL     0:02.59 [fdc0]
    33  ??  DL     0:23.33 [pagedaemon]
    34  ??  DL     0:00.00 [vmdaemon]
    35  ??  DL     0:03.81 [pagezero]
    36  ??  DL     0:00.16 [bufdaemon]
    37  ??  DL     0:13.09 [syncer]
    38  ??  DL     0:00.35 [vnlru]
    39  ??  DL     0:00.43 [softdepflush]
    44  ??  DL     0:06.75 [schedcpu]
   471  ??  DL     0:00.01 [usb0]
   472  ??  DL     0:00.02 [usbtask]
   474  ??  DL     0:00.02 [usb1]
   476  ??  DL     0:00.02 [usb2]
  1419  ??  D      0:00.01 /usr/local/sbin/megarc -ldInfo -LAll -a0
  2303  d0  R+     0:00.00 grep D
 
 root@ms1:~# /usr/local/sbin/megarc -ldInfo -LAll -a0
 
 
         **********************************************************************
               MEGARC MegaRAID Configuration Utility(FreeBSD)-1.04(03-02-2005)
               By LSI Logic Corp.,USA
         **********************************************************************
           [Note: For SATA-2, 4 and 6 channel controllers, please specify
           Ch=0 Id=0..15 for specifying physical drive(Ch=channel, Id=Target)]
 
         Type ? as command line arg for help
 
 
         Finding Devices On Each MegaRAID Adapter...
         Scanning Ha 0, Chnl 1 Target 15
 
                 *******Information Of Logical Drive 0*******
 
           Logical Drive : 0( Adapter: 0 ):  Status: OPTIMAL
         ---------------------------------------------------
         SpanDepth :01     RaidLevel: 5  RdAhead : Adaptive  Cache: DirectIo
         StripSz   :128KB   Stripes  : 14  WrPolicy: WriteBack
 
         Logical Drive 0 : SpanLevel_0 Disks
         Chnl  Target  StartBlock   Blocks      Physical Target Status
         ----  ------  ----------   ------      ----------------------
         0      00    0x00000000   0x11203000   ONLINE
         0      01    0x00000000   0x11203000   ONLINE
         0      02    0x00000000   0x11203000   ONLINE
         0      03    0x00000000   0x11203000   ONLINE
         0      04    0x00000000   0x11203000   ONLINE
         0      05    0x00000000   0x11203000   ONLINE
         0      06    0x00000000   0x11203000   ONLINE
         1      00    0x00000000   0x11203000   ONLINE
         1      01    0x00000000   0x11203000   ONLINE
         1      02    0x00000000   0x11203000   ONLINE
         1      03    0x00000000   0x11203000   ONLINE
         1      04    0x00000000   0x11203000   ONLINE
         1      05    0x00000000   0x11203000   ONLINE
         1      06    0x00000000   0x11203000   ONLINE
 
 root@ms1 /root #pciconf -vl
 hostb0@pci0:0:0:        class=0x060000 card=0x348015d9 chip=0x254c8086 rev=0x01 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = 'E7501 Host Controller'
     class    = bridge
     subclass = HOST-PCI
 none0@pci0:0:1: class=0xff0000 card=0x348015d9 chip=0x25418086 rev=0x01 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = 'E7500 System Controller (MCH, Hub Interface A) Error Reporter'
 pcib1@pci0:2:0: class=0x060400 card=0x00000000 chip=0x25438086 rev=0x01 hdr=0x01
     vendor   = 'Intel Corporation'
     device   = 'E7500/E7501 HI_B Virtual PCI-to-PCI Bridge'
     class    = bridge
     subclass = PCI-PCI
 pcib5@pci0:3:0: class=0x060400 card=0x00000000 chip=0x25458086 rev=0x01 hdr=0x01
     vendor   = 'Intel Corporation'
     device   = 'E7500/E7501 HI_C Virtual PCI-to-PCI Bridge'
     class    = bridge
     subclass = PCI-PCI
 uhci0@pci0:29:0:        class=0x0c0300 card=0x348015d9 chip=0x24828086 rev=0x02 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = '82801CA/CAM (ICH3-S/ICH3-M) USB Controller'
     class    = serial bus
     subclass = USB
 uhci1@pci0:29:1:        class=0x0c0300 card=0x348015d9 chip=0x24848086 rev=0x02 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = '82801CA/CAM (ICH3-S/ICH3-M) USB Controller'
     class    = serial bus
     subclass = USB
 uhci2@pci0:29:2:        class=0x0c0300 card=0x348015d9 chip=0x24878086 rev=0x02 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = '82801CA/CAM (ICH3-S/ICH3-M) USB Controller'
     class    = serial bus
     subclass = USB
 pcib8@pci0:30:0:        class=0x060400 card=0x00000000 chip=0x244e8086 rev=0x42 hdr=0x01
     vendor   = 'Intel Corporation'
     device   = '82801BA/CA/DB/DBL/EB/ER/FB (ICH2/3/4/4/5/5/6), 6300ESB Hub Interface to PCI Bridge'
     class    = bridge
     subclass = PCI-PCI
 isab0@pci0:31:0:        class=0x060100 card=0x00000000 chip=0x24808086 rev=0x02 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = '82801CA/CAM (ICH3-S/ICH3-M) LPC Interface'
     class    = bridge
     subclass = PCI-ISA
 atapci0@pci0:31:1:      class=0x01018a card=0x348015d9 chip=0x248b8086 rev=0x02 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = '82801CA (ICH3) UltraATA/100 EIDE Controller'
     class    = mass storage
     subclass = ATA
 none1@pci0:31:3:        class=0x0c0500 card=0x348015d9 chip=0x24838086 rev=0x02 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = '82801CA/CAM (ICH3-S/ICH3-M) SMBus Controller'
     class    = serial bus
     subclass = SMBus
 none2@pci1:28:0:        class=0x080020 card=0x348015d9 chip=0x14618086 rev=0x04 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = 'P64H2 I/O Advanced Programmable Interrupt Controller'
     class    = base peripheral
     subclass = interrupt controller
 pcib2@pci1:29:0:        class=0x060400 card=0x00000050 chip=0x14608086 rev=0x04 hdr=0x01
     vendor   = 'Intel Corporation'
     device   = '82870P2 P64H2 PCI/PCI-X Hub Controller'
     class    = bridge
     subclass = PCI-PCI
 none3@pci1:30:0:        class=0x080020 card=0x348015d9 chip=0x14618086 rev=0x04 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = 'P64H2 I/O Advanced Programmable Interrupt Controller'
     class    = base peripheral
     subclass = interrupt controller
 pcib4@pci1:31:0:        class=0x060400 card=0x00000050 chip=0x14608086 rev=0x04 hdr=0x01
     vendor   = 'Intel Corporation'
     device   = '82870P2 P64H2 PCI/PCI-X Hub Controller'
     class    = bridge
     subclass = PCI-PCI
 pcib3@pci2:1:0: class=0x060400 card=0x00000080 chip=0x01a71014 rev=0x02 hdr=0x01
     vendor   = 'International Business Machines Corp.'
     device   = 'IBM 133 PCI-X Bridge R1.1'
     class    = bridge
     subclass = PCI-PCI
 amr0@pci3:0:0:  class=0x010400 card=0x05321000 chip=0x04071000 rev=0x02 hdr=0x00
     vendor   = 'LSI Logic (Was: Symbios Logic, NCR)'
     device   = 'MegaRAID'
     class    = mass storage
     subclass = RAID
 em0@pci4:2:0:   class=0x020000 card=0x10118086 chip=0x10108086 rev=0x01 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = '82546EB Dual Port Gigabit Ethernet Controller (Copper)'
     class    = network
     subclass = ethernet
 em1@pci4:2:1:   class=0x020000 card=0x10118086 chip=0x10108086 rev=0x01 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = '82546EB Dual Port Gigabit Ethernet Controller (Copper)'
     class    = network
     subclass = ethernet
 none4@pci5:28:0:        class=0x080020 card=0x348015d9 chip=0x14618086 rev=0x04 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = 'P64H2 I/O Advanced Programmable Interrupt Controller'
     class    = base peripheral
     subclass = interrupt controller
 pcib6@pci5:29:0:        class=0x060400 card=0x00000050 chip=0x14608086 rev=0x04 hdr=0x01
     vendor   = 'Intel Corporation'
     device   = '82870P2 P64H2 PCI/PCI-X Hub Controller'
     class    = bridge
     subclass = PCI-PCI
 none5@pci5:30:0:        class=0x080020 card=0x348015d9 chip=0x14618086 rev=0x04 hdr=0x00
     vendor   = 'Intel Corporation'
     device   = 'P64H2 I/O Advanced Programmable Interrupt Controller'
     class    = base peripheral
     subclass = interrupt controller
 pcib7@pci5:31:0:        class=0x060400 card=0x00000050 chip=0x14608086 rev=0x04 hdr=0x01
     vendor   = 'Intel Corporation'
     device   = '82870P2 P64H2 PCI/PCI-X Hub Controller'
     class    = bridge
     subclass = PCI-PCI
 none6@pci8:1:0: class=0x030000 card=0x00081002 chip=0x47521002 rev=0x27 hdr=0x00
     vendor   = 'ATI Technologies Inc'
     device   = 'Rage XL PCI'
     class    = display
     subclass = VGA
 
 root@ms1 /root # vmstat 1
  procs      memory      page                   disk   faults      cpu
  r b w     avm    fre  flt  re  pi  po  fr  sr am0   in   sy  cs us sy id
 17 3 0  431904  80900   38   1   0   0 2857 2812   0 9011 4402 16732  1 26 73
 19 3 0  431904  66300    8   1   1   0 3677   0 110 9046 4186 17350  0 24 76
 28 3 0  431904 108712    0   3   0   0 2666 13229  93 9145 4130 17128  0 25 75
 25 3 0  431904 100136    0   0   0   0 2168   0  69 8880 3935 16665  0 22 78
 10 3 0  431904  89204    0   2   0   0 2731   0  83 9300 4024 17393  0 23 77
 28 4 0  431904  77860    0   3   0   0 2850   0  86 9179 4108 17361  0 23 77
 17 3 0  431904  65340    0   3   0   0 3105   0 100 9008 3958 17172  0 25 74
 34 3 2  431904 107116    2   1   0   0 2823 13218  86 9041 4285 17123  1 25 74
 41 3 0  431904  96688    0   0   0   0 2590   0  83 8896 4047 16688  1 25 75
 21 3 0  431904  86708    0   0   0   0 2510   0  80 9430 4445 17576  0 25 74
 14 3 0  431904  72636    0   0   0   0 3544   0 113 9287 4397 17457  0 27 73
 ^C
 root@ms1 /root #vmstat -m
          Type InUse MemUse HighUse Requests  Size(s)
        zombie     0     0K       -     1941  128
       ithread   169    18K       -      169  16,64,128
 CAM dev queue     1     1K       -        1  64
        KTRACE   100    13K       -      100  128
        linker    69   144K       -      119  16,32,256,1024,4096
         lockf     2     1K       -       66  64
        devbuf   251  1216K       -     1096  16,32,64,128,256,512,1024,2048,4096
          temp  1953   530K       -     7599  16,32,64,128,256,1024,2048,4096
       CAM SIM     1     1K       -        1  64
        module   201    13K       -      201  64,128
      mtx_pool     1     8K       -        1  
          pgrp    26     2K       -      179  64
       session    23     3K       -      144  128
          proc     2    16K       -        2  
          cred   322    41K       -    11464  128
       ata_dma     2     1K       -        2  128
        plimit    16     4K       -     1503  256
       uidinfo     4     3K       -       64  32,2048
        sysctl     0     0K       -     1138  16,32,64
     sysctloid  2450    74K       -     2450  16,32,64
     sysctltmp     0     0K       -      285  16,32,64,128
          umtx   460    29K       -      460  64
          SWAP     2   549K       -        2  64
           bus   651    30K       -     3931  16,32,64,128,1024
        bus-sc    57    31K       -     1900  16,32,64,128,256,512,2048,4096
       entropy  1024    64K       -     1024  64
       devstat     6    13K       -        6  16,4096
  eventhandler    42     3K       -       42  32,128
          kobj   121   242K       -      139  2048
        DEVFS3    96    12K       -       97  128
        DEVFS1    86    22K       -       86  256
          rman   169    11K       -      547  64
          sbuf     0     0K       -      206  16,32,64,128,256,512,1024,2048,4096
  sleep queues   461    15K       -      461  32
         DEVFS    10     1K       -       11  16,128
     taskqueue     6     1K       -        6  128
    turnstiles   461    29K       -      461  64
        Unitno     7     1K       -        9  16,64
     pfs_nodes    20     3K       -       20  128
      ioctlops     1     1K       -     3226  16,32,64,256,512,1024
           iov     0     0K       -    12878  16,64,128
           msg     4    25K       -        4  1024,4096
           sem     4     7K       -        4  512,1024,4096
           shm     1    12K       -        1  
          ttys  1057   151K       -     2399  128,1024
          ptys     1     1K       -        1  128
          accf     3     1K       -        3  16,32
      mbuf_tag     0     0K       -        2  32
        soname     4     1K       -    34805  16,32,128
           pcb    12     5K       -      221  16,32,64,2048
          GEOM    70     8K       -      422  16,32,64,128,256,512,1024,4096
    BIO buffer     3     6K       -     1200  2048
      vfscache     1   512K       -        1  
 cluster_save buffer     0     0K       -      122  32,64
      VFS hash     1   256K       -        1  
        vnodes     1     1K       -        1  128
         mount   118    11K       -      208  16,32,64,128,1024,2048
   vnodemarker     0     0K       -     3180  512
           BPF     3     1K       -        3  64
         ifnet     4     4K       -        4  256,1024
        ifaddr    22     5K       -       22  32,256,512,2048
   ether_multi     7     1K       -        8  16,32,64
         clone     3    12K       -        3  4096
        arpcom     2     1K       -        2  16
            lo     1     1K       -        1  16
        isadev    18     2K       -       18  64
      routetbl   143    34K       -      239  16,32,64,128,256
      in_multi     2     1K       -        2  32
     hostcache     1    24K       -        1  
      syncache     1     8K       -        1  
       NFS req     0     0K       -      393  128
    NFS daemon     1     8K       -        1  
      p1003.1b     1     1K       -        1  16
       pagedep     3    65K       -      129  64
      inodedep    14   258K       -      593  128
        newblk     1     1K       -     1919  64,256
     bmsafemap     2     1K       -      329  64
   allocdirect     3     1K       -     1131  128
      indirdep     0     0K       -      206  32
    allocindir     0     0K       -      787  64
      freefrag     0     0K       -      464  32
      freeblks     1     1K       -      259  256
      freefile     1     1K       -      256  32
        diradd     9     1K       -      595  64
         mkdir     0     0K       -        2  32
        dirrem     9     1K       -      588  32
      savedino     0     0K       -      253  256
   UFS dirhash   120    28K       -      261  16,32,64,512,2048,4096
     UFS mount    15    47K       -       15  512,2048,4096
       UMAHash     2   129K       -       12  256,512,1024,2048,4096
       CAM XPT     7     1K       -       11  16,64,512
     VM pgdata     2    65K       -        2  64
    CAM periph     1     1K       -        1  128
          cdev    24     3K       -       24  128
      atkbddev     2     1K       -        2  32
     CAM queue     3     1K       -        3  16
     file desc   688   173K       -     3018  16,256,512
      I/O APIC     5     5K       -        5  1024
       memdesc     1     4K       -        1  4096
      nexusdev     3     1K       -        3  16
         sigio     1     1K       -        1  32
          kenv    98     7K       -       99  16,32,64,4096
        kqueue     0     0K       -      162  256,1024
     proc-args    30     2K       -     1166  16,32,64,128,256
      PCI Link    16     2K       -       16  32,128
         pf_if    16     5K       -       16  16,256,2048
        acpica   996    55K       -    21842  16,32,64,128,256,512,1024,2048
      acpitask     0     0K       -        1  32
       acpidev    54     2K       -       54  32
       acpisem    18     2K       -       18  64
   IpFw/IpAcct     5     1K       -        5  64
        USBdev     3     1K       -       12  128,512
           USB    46     5K       -       46  16,32,64,128,256
 root@ms1 /root #vmstat -z
 ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
 
 UMA Kegs:        140,        0,      83,     13,       83
 UMA Zones:       480,        0,      83,      5,       83
 UMA Slabs:        64,        0,     458,    132,     4958
 UMA RCntSlabs:   104,        0,   15650,   2443,   154270
 UMA Hash:        128,        0,       4,     26,        6
 16 Bucket:        76,        0,      38,     62,       74
 32 Bucket:       140,        0,      30,     26,       63
 64 Bucket:       268,        0,      30,     26,       92
 128 Bucket:      524,        0,      79,     19,      661
 VM OBJECT:       132,        0,    6384,   1475,    59240
 MAP:             192,        0,       8,     32,        8
 KMAP ENTRY:       68,    65520,    1126,   2626,  3676246
 MAP ENTRY:        68,        0,   36916,    156,   142177
 PV ENTRY:         24,  2155135,  258680,  30160,  1355498
 DP fakepg:        72,        0,       0,      0,        0
 mt_zone:        1024,        0,     175,    125,      175
 16:               16,        0,    4096,    573,    62006
 32:               32,        0,    1628,    519,     6183
 64:               64,        0,    3850,    457,    24686
 128:             128,        0,    2025,    165,    22448
 256:             256,        0,     994,    176,     6742
 512:             512,        0,      66,     22,     3581
 1024:           1024,        0,      50,    114,     2378
 2048:           2048,        0,     141,     11,     1457
 4096:           4096,        0,      53,     38,     1999
 Files:            72,        0,     666,    129,    74772
 PROC:            524,        0,     386,     62,     2328
 THREAD:          372,        0,     448,     12,      448
 KSEGRP:           88,        0,     448,     72,      448
 UPCALL:           44,        0,       0,      0,        0
 VMSPACE:         300,        0,     336,     80,     2273
 mbuf_packet:     256,        0,   30975,     58, 93166998
 mbuf:            256,        0,     331,    300, 170303112
 mbuf_cluster:   2048,    65536,   31034,    266, 89834951
 mbuf_jumbo_pagesize: 4096,        0,       0,      0,        0
 mbuf_jumbo_9k:  9216,        0,       0,      0,        0
 mbuf_jumbo_16k: 16384,        0,       0,      0,        0
 ACL UMA zone:    388,        0,       0,      0,        0
 g_bio:           132,        0,       0,    145,  3787779
 ata_request:     204,        0,       0,      0,        0
 ata_composite:   196,        0,       0,      0,        0
 VNODE:           348,        0,   26742,    604,   205197
 VNODEPOLL:        76,        0,       0,      0,        0
 S VFS Cache:      68,        0,   26677,   1155,   198797
 L VFS Cache:     291,        0,    1782,     12,    12212
 NAMEI:          1024,        0,       0,     20,   522238
 NFSMOUNT:        480,        0,       1,     15,        1
 NFSNODE:         536,        0,       9,      5,       36
 DIRHASH:        1024,        0,     479,     21,      586
 PIPE:            408,        0,       4,     32,      920
 KNOTE:            68,        0,       0,    224,      164
 socket:          356,    65538,     408,    109,    17799
 unpcb:           140,    65548,       8,    104,      421
 ipq:              32,     2147,       0,      0,        0
 udpcb:           180,    65538,       4,     84,      348
 inpcb:           180,    65538,     400,    216,    17024
 tcpcb:           464,    65536,     396,     92,    17024
 tcptw:            48,    13182,       4,    464,     5439
 syncache:        100,    15366,       3,    192,    18661
 hostcache:        76,    15400,     261,     89,      422
 tcpreass:         20,     4225,       0,    169,       26
 sackhole:         20,        0,      26,    481,   558757
 ripcb:           180,    65538,       0,     44,        5
 rtentry:         132,        0,       6,     52,        7
 pfsrctrpl:       100,        0,       0,      0,        0
 pfrulepl:        604,        0,       0,      0,        0
 pfstatepl:       260,    10005,       0,      0,        0
 pfaltqpl:        128,        0,       0,      0,        0
 pfpooladdrpl:     68,        0,       0,      0,        0
 pfrktable:      1240,        0,       0,      0,        0
 pfrkentry:       156,        0,       0,      0,        0
 pfrkentry2:      156,        0,       0,      0,        0
 pffrent:          16,     5075,       0,      0,        0
 pffrag:           48,        0,       0,      0,        0
 pffrcache:        48,    10062,       0,      0,        0
 pffrcent:         12,    50141,       0,      0,        0
 pfstatescrub:     28,        0,       0,      0,        0
 pfiaddrpl:        92,        0,       0,      0,        0
 pfospfen:        108,        0,       0,      0,        0
 pfosfp:           28,        0,       0,      0,        0
 IPFW dynamic rule zone:  108,        0,       0,      0,        0
 SWAPMETA:        276,   121576,      25,      3,       49
 FFS inode:       132,        0,   26694,    711,   205097
 FFS1 dinode:     128,        0,       0,      0,        0
 FFS2 dinode:     256,        0,   26694,    591,   205097
 
 root@ms1 /root #vmstat -s
 134351978 cpu context switches
  72351904 device interrupts
    199026 software interrupts
    323329 traps
  35336126 system calls
        51 kernel threads created
      2106  fork() calls
       172 vfork() calls
         0 rfork() calls
       253 swap pager pageins
       265 swap pager pages paged in
        66 swap pager pageouts
        94 swap pager pages paged out
      1860 vnode pager pageins
      5780 vnode pager pages paged in
         0 vnode pager pageouts
         0 vnode pager pages paged out
      1306 page daemon wakeups
  22572806 pages examined by the page daemon
      9762 pages reactivated
     83627 copy-on-write faults
       215 copy-on-write optimized faults
    141196 zero fill pages zeroed
     83358 zero fill pages prezeroed
       878 intransit blocking page faults
    307140 total VM faults taken
         0 pages affected by kernel thread creation
    259906 pages affected by  fork()
     27090 pages affected by vfork()
         0 pages affected by rfork()
  22936815 pages freed
       112 pages freed by daemon
    152492 pages freed by exiting processes
     47199 pages active
    383694 pages inactive
     17183 pages in VM cache
     63764 pages wired down
       750 pages free
      4096 bytes per page
   1247234 total name lookups
           cache hits (81% pos + 1% neg) system 1% per-directory
           deletions 0%, falsehits 0%, toolong 0%
 
 db> ps
   pid   proc     uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
  2327 cc51820c   80   483   483 0000100 [SLPQ sbwait 0xd0029120][SLP] httpd
  2326 cc524830   80   483   483 0000100 [SLPQ sbwait 0xcd2b4c40][SLP] httpd
  2325 ce3d8000   80   483   483 0000100 [SLPQ sbwait 0xd0099284][SLP] httpd
  2324 cc52420c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
  2322 cd50720c   80   483   483 0000100 [SLPQ sbwait 0xca237978][SLP] httpd
  2321 cbff0a3c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
  2320 c9a0b20c   80   483   483 0000100 [SLPQ sbwait 0xcb676284][SLP] httpd
  2306 cbd29624   80   483   483 0000100 [SLPQ sbwait 0xcc9d2814][SLP] httpd
  2268 cbfe9418   80   483   483 0000100 [SLPQ sbwait 0xca66d120][SLP] httpd
  2267 ce3d8830   80   483   483 0000100 [SLPQ sbwait 0xcbe1754c][SLP] httpd
  2266 c96f5830   80   483   483 0000100 [SLPQ sbwait 0xcb67754c][SLP] httpd
  2265 cb36d830   80   483   483 0000100 [SLPQ sbwait 0xc9c97978][SLP] httpd
  2264 cc635000   80   483   483 0000100 [SLPQ sbwait 0xc9ed23e8][SLP] httpd
  2263 c9a0bc48   80   483   483 0000100 [SLPQ sbwait 0xcb677f08][SLP] httpd
  2262 cfaa4624   80   483   483 0000100 [SLPQ sbwait 0xcb442284][SLP] httpd
  2261 cbd29000   80   483   483 0000100 [SLPQ sbwait 0xc9334978][SLP] httpd
  2258 cc515a3c   80   483   483 0000100 [SLPQ sbwait 0xca236adc][SLP] httpd
  2257 cc635624   80   483   483 0000100 [SLPQ sbwait 0xcb442c40][SLP] httpd
  2256 cbfe9c48   80   483   483 0000100 [SLPQ sbwait 0xc9990f08][SLP] httpd
 --More--         2255 ce3d820c   80   483   483 0000100 [SLPQ sbwait 0xcb8536b0][SLP] httpd
  2254 cc515624   80   483   483 0000100 [SLPQ sbwait 0xca66d284][SLP] httpd
  2248 cb35b830   80   483   483 0000100 [SLPQ sbwait 0xc929dadc][SLP] httpd
  2245 cbfe9000   80   483   483 0000100 [SLPQ sbwait 0xca2363e8][SLP] httpd
  2244 cd507000   80   483   483 0000100 [SLPQ sbwait 0xcccfbc40][SLP] httpd
  2243 cb36a830   80   483   483 0000100 [SLPQ sbwait 0xcabb16b0][SLP] httpd
  2239 cbfef830   80   483   483 0000100 [SLPQ sbwait 0xc929cf08][SLP] httpd
  2237 cc524418   80   483   483 0000100 [SLPQ sbwait 0xca236f08][SLP] httpd
  2233 c9a8aa3c    0  2231  2233 0004002 [SLPQ ttyin 0xc8fd1810][SLP] tcsh
  2231 cfaa1418 1001  2228  2231 0004002 [SLPQ pause 0xcfaa144c][SLP] tcsh
  2230 cbff0c48   80   483   483 0000100 [SLPQ biord 0xdafe1514][SLP] httpd
  2229 cd51aa3c   80   483   483 0000100 [SLPQ sbwait 0xcb012c40][SLP] httpd
  2228 cc51520c 1001  2222  2222 0000100 [SLPQ select 0xc072ade4][SLP] sshd
  2227 cd50cc48   80   483   483 0000100 [SLPQ sbwait 0xcc16b3e8][SLP] httpd
  2226 cbd23c48   80   483   483 0000100 [SLPQ sbwait 0xc9ed2f08][SLP] httpd
  2225 cc518624   80   483   483 0000100 [SLPQ sbwait 0xcabb0978][SLP] httpd
  2224 cc635830   80   483   483 0000100 [SLPQ sbwait 0xce5a2120][SLP] httpd
  2222 cc23e830    0   502  2222 0004100 [SLPQ sbwait 0xcbe170a8][SLP] sshd
  2221 cbff0624   80   483   483 0000100 [SLPQ sbwait 0xcb442f08][SLP] httpd
  2220 c96f5a3c   80   483   483 0000100 [SLPQ sbwait 0xcd653284][SLP] httpd
 --More--         2219 cd51a20c   80   483   483 0000100 [SLPQ sbwait 0xc9d25adc][SLP] httpd
  2218 cd51a418   80   483   483 0000100 [SLPQ sbwait 0xcf3a854c][SLP] httpd
  2217 cb075830   80   483   483 0000100 [SLPQ sbwait 0xcc50d284][SLP] httpd
  2216 ce3d8a3c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
  2215 cd507830   80   483   483 0000100 [SLPQ sbwait 0xcbb5854c][SLP] httpd
  2140 cbd29418   80   483   483 0000100 [SLPQ sbwait 0xcaad9120][SLP] httpd
  2139 cbd23624   80   483   483 0000100 [SLPQ sbwait 0xcbeb0814][SLP] httpd
  2138 cec1dc48   80   483   483 0000100 [SLPQ biord 0xdb04eff4][SLP] httpd
  2137 ca29c000   80   483   483 0000100 [SLPQ sbwait 0xcbb58f08][SLP] httpd
  2136 cd50c418   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
  2135 cc635c48   80   483   483 0000100 [SLPQ sbwait 0xc934aadc][SLP] httpd
  2043 cfaa1a3c   80   483   483 0000100 [SLPQ sbwait 0xcd2b46b0][SLP] httpd
  2027 cc524000   80   483   483 0000100 [SLPQ sbwait 0xc962d978][SLP] httpd
  2026 cbff0000   80   483   483 0000100 [SLPQ sbwait 0xc962dc40][SLP] httpd
  1980 cc23e20c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
  1978 c9a8a418   80   483   483 0000100 [SLPQ sbwait 0xcd2b5da4][SLP] httpd
  1938 cfaa1830   80   483   483 0000100 [SLPQ sbwait 0xca2366b0][SLP] httpd
  1937 cc23ea3c   80   483   483 0000100 [SLPQ sbwait 0xcbe17adc][SLP] httpd
  1905 c9a8a20c   80   483   483 0000100 [SLPQ sbwait 0xcd2b56b0][SLP] httpd
  1902 cc515000   80   483   483 0000100 [SLPQ sbwait 0xc9c97120][SLP] httpd
 --More--         1896 cbd23830   80   483   483 0000100 [SLPQ sbwait 0xcbeb0f08][SLP] httpd
  1876 c9e07c48   80   483   483 0000100 [SLPQ sbwait 0xce5a2c40][SLP] httpd
  1748 cfaa420c   80   483   483 0000100 [SLPQ sbwait 0xcabb06b0][SLP] httpd
  1747 cfaa1c48   80   483   483 0000100 [SLPQ sbwait 0xcbb58978][SLP] httpd
  1746 cbfef000   80   483   483 0000100 [SLPQ sbwait 0xcd653978][SLP] httpd
  1613 cb36d000   80   483   483 0000100 [SLPQ sbwait 0xce80af08][SLP] httpd
  1556 c96f520c   80   483   483 0000100 [SLPQ sbwait 0xcbb57da4][SLP] httpd
  1544 cd50ca3c   80   483   483 0000100 [SLPQ sbwait 0xcb677120][SLP] httpd
  1543 cc515830   80   483   483 0000100 [SLPQ sbwait 0xca66d3e8][SLP] httpd
  1498 cc261000   80   483   483 0000100 [SLPQ sbwait 0xcb442da4][SLP] httpd
  1493 cbfef20c   80   483   483 0000100 [SLPQ sbwait 0xcb01254c][SLP] httpd
  1491 c9a0b830   80   483   483 0000100 [SLPQ sbwait 0xce80a120][SLP] httpd
  1490 cc518c48   80   483   483 0000100 [SLPQ sbwait 0xca3a5120][SLP] httpd
  1486 cd507c48   80   483   483 0000100 [SLPQ sbwait 0xcfe4a3e8][SLP] httpd
  1485 cc261c48   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
  1483 cc518830   80   483   483 0000100 [SLPQ sbwait 0xcbe173e8][SLP] httpd
  1482 ca29c624   80   483   483 0000100 [SLPQ sbwait 0xc9c973e8][SLP] httpd
  1481 c9a0b000   80   483   483 0000100 [SLPQ sbwait 0xcd2b43e8][SLP] httpd
  1419 c9a0b624    0  1370  1368 0004000 [SLPQ amrwcmd 0xc9188290][SLP] megarc
  1370 c96f5000    0  1368  1368 0004000 [SLPQ piperd 0xcc0fd000][SLP] perl5.8.8
 --More--         1368 cbd23418    0  1357  1368 0004000 [SLPQ wait 0xcbd23418][SLP] sh
  1357 cc23e624    0   514   514 0000000 [SLPQ piperd 0xc918ab28][SLP] cron
  1279 cd50c624   80   483   483 0000100 [SLPQ sbwait 0xcf3a8814][SLP] httpd
  1274 c96f5418   80   483   483 0000100 [SLPQ sbwait 0xcd6533e8][SLP] httpd
  1273 c96f5624   80   483   483 0000100 [SLPQ sbwait 0xcaad9adc][SLP] httpd
  1270 cbfefa3c   80   483   483 0000100 [SLPQ sbwait 0xcc16aadc][SLP] httpd
  1264 ce3d8624   80   483   483 0000100 [SLPQ sbwait 0xcb012120][SLP] httpd
  1231 cd50c000   80   483   483 0000100 [SLPQ sbwait 0xcbb57120][SLP] httpd
  1226 c9a0b418   80   483   483 0000100 [SLPQ sbwait 0xc929d3e8][SLP] httpd
  1225 c9a0ba3c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
  1125 cd51a000   80   483   483 0000100 [SLPQ sbwait 0xce5a2da4][SLP] httpd
  1099 cb36dc48   80   483   483 0000100 [SLPQ sbwait 0xc962d3e8][SLP] httpd
  1015 ca29c830   80   483   483 0000100 [SLPQ sbwait 0xcbb5754c][SLP] httpd
  1008 cc518418   80   483   483 0000100 [SLPQ sbwait 0xc991df08][SLP] httpd
  1006 cc23ec48   80   483   483 0000100 [SLPQ sbwait 0xca93af08][SLP] httpd
  1004 cbff0830   80   483   483 0000100 [SLPQ sbwait 0xcf3a8284][SLP] httpd
  1003 c9e07000   80   483   483 0000100 [SLPQ sbwait 0xcb44254c][SLP] httpd
  1002 cc261418   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   911 cc518000   80   483   483 0000100 [SLPQ sbwait 0xcabb1814][SLP] httpd
   877 ca29c20c   80   483   483 0000100 [SLPQ sbwait 0xc9c96da4][SLP] httpd
 --More--          870 cbd29c48   80   483   483 0000100 [SLPQ sbwait 0xc933154c][SLP] httpd
   857 cbfef418   80   483   483 0000100 [SLPQ sbwait 0xcb012978][SLP] httpd
   851 cbfef624   80   483   483 0000100 [SLPQ sbwait 0xcb85354c][SLP] httpd
   844 cbd2920c   80   483   483 0000100 [SLPQ sbwait 0xcb853814][SLP] httpd
   843 cb36d624   80   483   483 0000100 [SLPQ sbwait 0xc9331c40][SLP] httpd
   842 cb35b624   80   483   483 0000100 [SLPQ sbwait 0xcccfa54c][SLP] httpd
   839 cbd29830   80   483   483 0000100 [SLPQ sbwait 0xcfe4a120][SLP] httpd
   838 cbd29a3c   80   483   483 0000100 [SLPQ sbwait 0xcccfa978][SLP] httpd
   834 cb36aa3c   80   483   483 0000100 [SLPQ sbwait 0xc9c96f08][SLP] httpd
   833 cb35b418   80   483   483 0000100 [SLPQ sbwait 0xc9d2554c][SLP] httpd
   831 cb35d418   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   829 cb075418   80   483   483 0000100 [SLPQ sbwait 0xcccfa3e8][SLP] httpd
   828 cb35d624   80   483   483 0000100 [SLPQ sbwait 0xcbb57f08][SLP] httpd
   827 cb075000   80   483   483 0000100 [SLPQ sbwait 0xc929c6b0][SLP] httpd
   826 cb35d20c   80   483   483 0000100 [SLPQ sbwait 0xca93a6b0][SLP] httpd
   825 cb075c48   80   483   483 0000100 [SLPQ sbwait 0xca3a5c40][SLP] httpd
   824 cb36a624   80   483   483 0000100 [SLPQ sbwait 0xcd653c40][SLP] httpd
   823 cb36a20c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   822 cb075624   80   483   483 0000100 [SLPQ sbwait 0xcc50dc40][SLP] httpd
   821 cb075a3c   80   483   483 0000100 [SLPQ sbwait 0xcc16b284][SLP] httpd
 --More--          820 cadd1a3c   80   483   483 0000100 [SLPQ sbwait 0xcfe4a814][SLP] httpd
   819 cb35c830   80   483   483 0000100 [SLPQ sbwait 0xcd2b4f08][SLP] httpd
   817 cb07520c   80   483   483 0000100 [SLPQ sbwait 0xcc16badc][SLP] httpd
   816 cb36ac48   80   483   483 0000100 [SLPQ sbwait 0xcf3a8da4][SLP] httpd
   815 cb35d000   80   483   483 0000100 [SLPQ sbwait 0xc9c96814][SLP] httpd
   814 cb35ba3c   80   483   483 0000100 [SLPQ sbwait 0xcfe4af08][SLP] httpd
   813 cb35b000   80   483   483 0000100 [SLPQ sbwait 0xcb676c40][SLP] httpd
   812 cb35b20c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   797 cadd1c48   80   483   483 0000100 [SLPQ sbwait 0xc962d6b0][SLP] httpd
   796 cb35cc48   80   483   483 0000100 [SLPQ sbwait 0xcccfa284][SLP] httpd
   795 cb36d20c   80   483   483 0000100 [SLPQ sbwait 0xcb0126b0][SLP] httpd
   794 cb35ca3c   80   483   483 0000100 [SLPQ sbwait 0xcccfada4][SLP] httpd
   793 cb36a000   80   483   483 0000100 [SLPQ sbwait 0xc9d0ef08][SLP] httpd
   792 cb36a418   80   483   483 0000100 [SLPQ sbwait 0xcccfbda4][SLP] httpd
   791 cb35bc48   80   483   483 0000100 [SLPQ sbwait 0xca93a284][SLP] httpd
   790 cb35c000   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   789 cb35da3c   80   483   483 0000100 [SLPQ sbwait 0xca3a53e8][SLP] httpd
   788 cb35dc48   80   483   483 0000100 [SLPQ sbwait 0xcb67654c][SLP] httpd
   787 cb36d418   80   483   483 0000100 [SLPQ sbwait 0xca66d814][SLP] httpd
   786 cb35c20c   80   483   483 0000100 [SLPQ sbwait 0xc9ed26b0][SLP] httpd
 --More--          785 cb35d830   80   483   483 0000100 [SLPQ sbwait 0xcb853c40][SLP] httpd
   784 cb35c418   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   783 cb35c624   80   483   483 0000100 [SLPQ sbwait 0xcd2b5370][SLP] httpd
   754 cb07a000   80   483   483 0000100 [SLPQ sbwait 0xc9990adc][SLP] httpd
   753 cb07a20c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   752 cb07a418   80   483   483 0000100 [SLPQ sbwait 0xc9c9654c][SLP] httpd
   751 cb07a624   80   483   483 0000100 [SLPQ sbwait 0xcb6776b0][SLP] httpd
   750 cb07a830   80   483   483 0000100 [SLPQ sbwait 0xc962dadc][SLP] httpd
   749 cb07aa3c   80   483   483 0000100 [SLPQ sbwait 0xc9c96c40][SLP] httpd
   748 cb07ac48   80   483   483 0000100 [SLPQ sbwait 0xc9990c40][SLP] httpd
   747 ca84c624   80   483   483 0000100 [SLPQ sbwait 0xcd2b4adc][SLP] httpd
   746 ca84c830   80   483   483 0000100 [SLPQ sbwait 0xc9c97adc][SLP] httpd
   745 ca84ca3c   80   483   483 0000100 [SLPQ sbwait 0xca66e3e8][SLP] httpd
   744 ca84cc48   80   483   483 0000100 [SLPQ sbwait 0xc9c97c40][SLP] httpd
   743 ca7f2624   80   483   483 0000100 [SLPQ sbwait 0xcbb58da4][SLP] httpd
   742 cadcf000   80   483   483 0000100 [SLPQ sbwait 0xcd65354c][SLP] httpd
   741 cadcf20c   80   483   483 0000100 [SLPQ sbwait 0xcbeb06b0][SLP] httpd
   740 cadcf418   80   483   483 0000100 [SLPQ sbwait 0xcd2b5120][SLP] httpd
   739 ca7f220c   80   483   483 0000100 [SLPQ sbwait 0xcabb13e8][SLP] httpd
   738 cadcf624   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
 --More--          737 cadcf830   80   483   483 0000100 [SLPQ sbwait 0xca93a120][SLP] httpd
   736 cadcfa3c   80   483   483 0000100 [SLPQ sbwait 0xc9ed254c][SLP] httpd
   735 cadcfc48   80   483   483 0000100 [SLPQ sbwait 0xcc9d23e8][SLP] httpd
   734 cadd1000   80   483   483 0000100 [SLPQ sbwait 0xc9ed2814][SLP] httpd
   733 ca848c48   80   483   483 0000100 [SLPQ sbwait 0xcac3a284][SLP] httpd
   732 cadd120c   80   483   483 0000100 [SLPQ sbwait 0xcfe4aadc][SLP] httpd
   731 c93c6624   80   483   483 0000100 [SLPQ sbwait 0xca66e6b0][SLP] httpd
   730 cadd1418   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   729 cadd1624   80   483   483 0000100 [SLPQ sbwait 0xcccfb814][SLP] httpd
   728 c93c6418   80   483   483 0000100 [SLPQ sbwait 0xc9331da4][SLP] httpd
   727 ca7f320c   80   483   483 0000100 [SLPQ sbwait 0xcf3a8978][SLP] httpd
   726 ca7f3418   80   483   483 0000100 [SLPQ sbwait 0xc9d0e3e8][SLP] httpd
   725 ca7f3624   80   483   483 0000100 [SLPQ sbwait 0xcc16ada4][SLP] httpd
   724 ca7f3830   80   483   483 0000100 [SLPQ sbwait 0xc99903e8][SLP] httpd
   723 cadd1830   80   483   483 0000100 [SLPQ sbwait 0xcd653da4][SLP] httpd
   722 ca7f3a3c   80   483   483 0000100 [SLPQ sbwait 0xca236120][SLP] httpd
   721 ca7f3c48   80   483   483 0000100 [SLPQ sbwait 0xcbe176b0][SLP] httpd
   720 ca848000   80   483   483 0000100 [SLPQ sbwait 0xcbb57814][SLP] httpd
   719 c93c7830   80   483   483 0000100 [SLPQ sbwait 0xca237da4][SLP] httpd
   718 ca84820c   80   483   483 0000100 [SLPQ sbwait 0xce5a2f08][SLP] httpd
 --More--          717 ca848418   80   483   483 0000100 [SLPQ sbwait 0xc9c97814][SLP] httpd
   716 ca84c000   80   483   483 0000100 [SLPQ sbwait 0xce80ada4][SLP] httpd
   715 ca7f2418   80   483   483 0000100 [SLPQ sbwait 0xcabb0814][SLP] httpd
   714 c9468000   80   483   483 0000100 [SLPQ sbwait 0xc991dda4][SLP] httpd
   713 ca256c48   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   712 ca7f0000   80   483   483 0000100 [SLPQ sbwait 0xcabb0120][SLP] httpd
   711 ca7f020c   80   483   483 0000100 [SLPQ sbwait 0xcc9d2c40][SLP] httpd
   710 ca7f0418   80   483   483 0000100 [SLPQ sbwait 0xc9d25814][SLP] httpd
   709 ca848624   80   483   483 0000100 [SLPQ sbwait 0xce80a6b0][SLP] httpd
   708 ca7f0624   80   483   483 0000100 [SLPQ sbwait 0xca237adc][SLP] httpd
   707 ca7f0830   80   483   483 0000100 [SLPQ sbwait 0xc9c963e8][SLP] httpd
   706 ca7f0a3c   80   483   483 0000100 [SLPQ sbwait 0xcbb57284][SLP] httpd
   705 ca848830   80   483   483 0000100 [SLPQ sbwait 0xcb442814][SLP] httpd
   704 ca7f0c48   80   483   483 0000100 [SLPQ sbwait 0xce80a814][SLP] httpd
   703 ca7f2000   80   483   483 0000100 [SLPQ sbwait 0xc9c97284][SLP] httpd
   702 ca848a3c   80   483   483 0000100 [SLPQ sbwait 0xcfe4a6b0][SLP] httpd
   696 ca84c20c   80   483   483 0000100 [SLPQ sbwait 0xc9ed2978][SLP] httpd
   695 ca84c418   80   483   483 0000100 [SLPQ sbwait 0xca237284][SLP] httpd
   694 ca7f2830   80   483   483 0000100 [SLPQ sbwait 0xcb8533e8][SLP] httpd
   693 ca7f2a3c   80   483   483 0000100 [SLPQ sbwait 0xd0029c40][SLP] httpd
 --More--          692 ca7f2c48   80   483   483 0000100 [SLPQ sbwait 0xd0029f08][SLP] httpd
   691 ca7f3000   80   483   483 0000100 [SLPQ sbwait 0xc9ed2284][SLP] httpd
   690 c9d3c418   80   483   483 0000100 [SLPQ sbwait 0xca3a5978][SLP] httpd
   689 ca1d6830   80   483   483 0000100 [SLPQ sbwait 0xcd653adc][SLP] httpd
   688 c9d3c624   80   483   483 0000100 [SLPQ sbwait 0xcc50dda4][SLP] httpd
   687 c9d3c830   80   483   483 0000100 [SLPQ sbwait 0xc991d284][SLP] httpd
   686 ca1d6a3c   80   483   483 0000100 [SLPQ sbwait 0xc9ed2da4][SLP] httpd
   685 ca1d6c48   80   483   483 0000100 [SLPQ sbwait 0xc9d0e814][SLP] httpd
   684 ca24f000   80   483   483 0000100 [SLPQ sbwait 0xcccfa6b0][SLP] httpd
   683 ca24f20c   80   483   483 0000100 [SLPQ sbwait 0xcbeb0adc][SLP] httpd
   682 ca24f418   80   483   483 0000100 [SLPQ sbwait 0xcabb03e8][SLP] httpd
   681 c9d3ca3c   80   483   483 0000100 [SLPQ sbwait 0xcc16bc40][SLP] httpd
   680 c9d3cc48   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   679 ca1d5000   80   483   483 0000100 [CPU 0] httpd
   678 ca1d520c   80   483   483 0000100 [SLPQ sbwait 0xca236da4][SLP] httpd
   677 ca24f624   80   483   483 0000100 [SLPQ sbwait 0xcb853f08][SLP] httpd
   676 ca1d5418   80   483   483 0000100 [SLPQ sbwait 0xcbb58120][SLP] httpd
   675 ca1d5624   80   483   483 0000100 [SLPQ sbwait 0xcb677814][SLP] httpd
   674 ca24f830   80   483   483 0000100 [SLPQ sbwait 0xcabb1c40][SLP] httpd
   673 ca24fa3c   80   483   483 0000100 [SLPQ sbwait 0xcbb57978][SLP] httpd
 --More--          672 ca24fc48   80   483   483 0000100 [SLPQ sbwait 0xca93a54c][SLP] httpd
   671 ca256000   80   483   483 0000100 [SLPQ sbwait 0xcfe4ada4][SLP] httpd
   670 ca25620c   80   483   483 0000100 [SLPQ sbwait 0xcac3a6b0][SLP] httpd
   669 ca256418   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   668 ca256624   80   483   483 0000100 [SLPQ sbwait 0xcb012814][SLP] httpd
   667 ca256830   80   483   483 0000100 [SLPQ sbwait 0xc9d0eadc][SLP] httpd
   666 ca256a3c   80   483   483 0000100 [SLPQ sbwait 0xca237120][SLP] httpd
   665 c9d39000   80   483   483 0000100 [SLPQ sbwait 0xc9c97da4][SLP] httpd
   664 ca1d5830   80   483   483 0000100 [SLPQ sbwait 0xc9d25da4][SLP] httpd
   663 ca1d5a3c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   662 ca1d5c48   80   483   483 0000100 [SLPQ sbwait 0xca66ef08][SLP] httpd
   661 ca1d6000   80   483   483 0000100 [SLPQ sbwait 0xc9334adc][SLP] httpd
   660 c9d3920c   80   483   483 0000100 [SLPQ sbwait 0xcfe4a54c][SLP] httpd
   659 ca1d620c   80   483   483 0000100 [SLPQ sbwait 0xca236978][SLP] httpd
   658 ca1d6418   80   483   483 0000100 [SLPQ sbwait 0xcabb054c][SLP] httpd
   657 ca1d6624   80   483   483 0000100 [SLPQ sbwait 0xcb677284][SLP] httpd
   656 c9d39418   80   483   483 0000100 [SLPQ sbwait 0xc9331284][SLP] httpd
   655 c9d39624   80   483   483 0000100 [SLPQ sbwait 0xca3a554c][SLP] httpd
   654 c9d39830   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   653 c9d39a3c   80   483   483 0000100 [SLPQ sbwait 0xcd2b4284][SLP] httpd
 --More--          652 c9d39c48   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   651 c9d3b000   80   483   483 0000100 [SLPQ sbwait 0xcc9d254c][SLP] httpd
   650 c9d3b20c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   649 c9d3b418   80   483   483 0000100 [SLPQ sbwait 0xcbb58c40][SLP] httpd
   648 c9d3b624   80   483   483 0000100 [SLPQ sbwait 0xca93aadc][SLP] httpd
   647 c9852a3c   80   483   483 0000100 [SLPQ sbwait 0xc9d25120][SLP] httpd
   646 c9852c48   80   483   483 0000100 [SLPQ sbwait 0xc991d120][SLP] httpd
   645 c996b000   80   483   483 0000100 [SLPQ sbwait 0xcac3aadc][SLP] httpd
   644 c996b20c   80   483   483 0000100 [SLPQ sbwait 0xd00296b0][SLP] httpd
   643 c9d3b830   80   483   483 0000100 [SLPQ sbwait 0xca93a814][SLP] httpd
   642 c9d3ba3c   80   483   483 0000100 [SLPQ sbwait 0xca93a3e8][SLP] httpd
   641 c9d3bc48   80   483   483 0000100 [SLPQ sbwait 0xc9990814][SLP] httpd
   640 c996b418   80   483   483 0000100 [SLPQ sbwait 0xcc9d2da4][SLP] httpd
   639 c9d3c000   80   483   483 0000100 [SLPQ sbwait 0xcabb0284][SLP] httpd
   638 c996b624   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   637 c996b830   80   483   483 0000100 [SLPQ sbwait 0xd0029da4][SLP] httpd
   636 c9d3c20c   80   483   483 0000100 [SLPQ sbwait 0xcb012da4][SLP] httpd
   635 c9550624   80   483   483 0000100 [SLPQ sbwait 0xcac3a54c][SLP] httpd
   634 c9550830   80   483   483 0000100 [SLPQ sbwait 0xc9990120][SLP] httpd
   633 c996ba3c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
 --More--          632 c996bc48   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   631 c996c000   80   483   483 0000100 [SLPQ sbwait 0xcabb1120][SLP] httpd
   630 c996c20c   80   483   483 0000100 [SLPQ sbwait 0xc962d120][SLP] httpd
   629 c996c418    0   576   629 0004002 [SLPQ ttyin 0xc8fdf010][SLP] csh
   628 c9550a3c   80   483   483 0000100 [SLPQ sbwait 0xcd2b454c][SLP] httpd
   627 c9550c48   80   483   483 0000100 [SLPQ sbwait 0xc991d6b0][SLP] httpd
   626 c9851000   80   483   483 0000100 [SLPQ sbwait 0xcfe4a978][SLP] httpd
   625 c985120c   80   483   483 0000100 [SLPQ select 0xc072ade4][SLP] httpd
   624 c9851418   80   483   483 0000100 [SLPQ sbwait 0xcccfa814][SLP] httpd
   623 c996c624   80   483   483 0000100 [SLPQ sbwait 0xc9331978][SLP] httpd
   622 c996c830   80   483   483 0000100 [SLPQ sbwait 0xc9334120][SLP] httpd
   621 c9851624   80   483   483 0000100 [SLPQ sbwait 0xcabb0f08][SLP] httpd
   620 c996ca3c   80   483   483 0000100 [SLPQ sbwait 0xcbe17da4][SLP] httpd
   619 c996cc48   80   483   483 0000100 [SLPQ sbwait 0xce5a23e8][SLP] httpd
   618 c946820c   80   483   483 0000100 [SLPQ sbwait 0xcd2b4da4][SLP] httpd
   617 c9468418   80   483   483 0000100 [SLPQ sbwait 0xc9c96adc][SLP] httpd
   616 c9851830   80   483   483 0000100 [SLPQ sbwait 0xcccfac40][SLP] httpd
   615 c9851a3c   80   483   483 0000100 [SLPQ sbwait 0xca66e814][SLP] httpd
   614 c9468624   80   483   483 0000100 [SLPQ sbwait 0xc93316b0][SLP] httpd
   613 c9468830   80   483   483 0000100 [SLPQ sbwait 0xca66e978][SLP] httpd
 --More--          612 c9468a3c   80   483   483 0000100 [SLPQ sbwait 0xcc50dadc][SLP] httpd
   611 c9851c48   80   483   483 0000100 [SLPQ sbwait 0xd00293e8][SLP] httpd
   610 c9852000   80   483   483 0000100 [SLPQ sbwait 0xcb676814][SLP] httpd
   609 c985220c   80   483   483 0000100 [SLPQ sbwait 0xcd2b4120][SLP] httpd
   608 c9852418   80   483   483 0000100 [SLPQ sbwait 0xcb853adc][SLP] httpd
   607 c9852624   80   483   483 0000100 [SLPQ sbwait 0xcc16a120][SLP] httpd
   606 c9852830   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   605 c9468c48   80   483   483 0000100 [SLPQ sbwait 0xca236c40][SLP] httpd
   604 c954d000   80   483   483 0000100 [SLPQ sbwait 0xcd2b5adc][SLP] httpd
   603 c954d20c   80   483   483 0000100 [SLPQ sbwait 0xcbb57c40][SLP] httpd
   602 c954d418   80   483   483 0000100 [SLPQ sbwait 0xc9c966b0][SLP] httpd
   601 c93c6a3c   80   483   483 0000100 [SLPQ sbwait 0xca237814][SLP] httpd
   600 c954d624   80   483   483 0000100 [SLPQ sbwait 0xc9d0e6b0][SLP] httpd
   599 c954d830   80   483   483 0000100 [SLPQ sbwait 0xc9990284][SLP] httpd
   598 c954da3c   80   483   483 0000100 [SLPQ sbwait 0xca23754c][SLP] httpd
   597 c9278418   80   483   483 0000100 [SLPQ sbwait 0xcf3a86b0][SLP] httpd
   596 c913d20c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   595 c93c7624   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   594 c913ea3c   80   483   483 0000100 [SLPQ sbwait 0xd002954c][SLP] httpd
   593 c927820c   80   483   483 0000100 [SLPQ sbwait 0xca66ec40][SLP] httpd
 --More--          592 c93c7000   80   483   483 0000100 [SLPQ sbwait 0xce80aadc][SLP] httpd
   591 c93c6000   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   590 c954dc48   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   589 c9550000   80   483   483 0000100 [SLPQ sbwait 0xc9331f08][SLP] httpd
   588 c955020c   80   483   483 0000100 [SLPQ accept 0xc929c466][SLP] httpd
   587 c9550418   80   483   483 0000100 [SLPQ sbwait 0xc9331120][SLP] httpd
   586 c913e830   80   483   483 0000100 [SLPQ sbwait 0xc9334c40][SLP] httpd
   585 c9278830   80   483   483 0000100 [SLPQ sbwait 0xc9d0e284][SLP] httpd
   584 c9278624   80   483   483 0000100 [SLPQ sbwait 0xcccfa120][SLP] httpd
   583 c8fd9a3c   80   483   483 0000100 [SLPQ sbwait 0xc929d6b0][SLP] httpd
   582 c93c7a3c   80   483   483 0000100 [SLPQ sbwait 0xca66e120][SLP] httpd
   581 c9464000   80   483   483 0000100 [SLPQ sbwait 0xc929cadc][SLP] httpd
   580 c946420c   80   483   483 0000100 [SLPQ sbwait 0xcbe17814][SLP] httpd
   579 c9464418   80   483   483 0000100 [SLPQ sbwait 0xc962d54c][SLP] httpd
   578 c9464624   80   483   483 0000100 [SLPQ sbwait 0xcc16b54c][SLP] httpd
   577 c9464830   80   483   483 0000100 [SLPQ sbwait 0xcaad96b0][SLP] httpd
   576 c9464a3c    0     1   576 0004102 [SLPQ wait 0xc9464a3c][SLP] login
   575 c9464c48    0     1   575 0004002 [SLPQ ttyin 0xc8ff3c10][SLP] getty
   574 c9465000    0     1   574 0004002 [SLPQ ttyin 0xc8ff4010][SLP] getty
   573 c946520c    0     1   573 0004002 [SLPQ ttyin 0xc8ff4410][SLP] getty
 --More--          572 c9465418    0     1   572 0004002 [SLPQ ttyin 0xc8fedc10][SLP] getty
   571 c9465624    0     1   571 0004002 [SLPQ ttyin 0xc8fee010][SLP] getty
   570 c9465830    0     1   570 0004002 [SLPQ ttyin 0xc8feb010][SLP] getty
   569 c9465a3c    0     1   569 0004002 [SLPQ ttyin 0xc8feb810][SLP] getty
   568 c9465c48    0     1   568 0004002 [SLPQ ttyin 0xc8fec810][SLP] getty
   524 c93c6830    0     1   524 0000000 [SLPQ nanslp 0xc06ddacc][SLP] uptimed
   514 c913e624    0     1   514 0000000 [SLPQ nanslp 0xc06ddacc][SLP] cron
   502 c93c620c    0     1   502 0000100 [SLPQ select 0xc072ade4][SLP] sshd
   489 c93c7c48    0     1   489 0000000 [SLPQ select 0xc072ade4][SLP] rsync
   483 c93c6c48    0     1   483 0000000 [SLPQ select 0xc072ade4][SLP] httpd
   477 c913d000    0     1   477 0000000 [SLPQ select 0xc072ade4][SLP] usbd
   476 c913ec48    0     0     0 0000204 [SLPQ usbevt 0xc941c210][SLP] usb2
   475 c93c720c    0     0     0 0000204 [IWAIT] irq18: uhci2
   474 c9278a3c    0     0     0 0000204 [SLPQ usbevt 0xc941d210][SLP] usb1
   473 c93c7418    0     0     0 0000204 [IWAIT] irq19: uhci1
   472 c913e418    0     0     0 0000204 [SLPQ usbtsk 0xc93e7d84][SLP] usbtask
   471 c9278c48    0     0     0 0000204 [SLPQ usbevt 0xc9290210][SLP] usb0
   470 c8fd9624    0     0     0 0000204 [IWAIT] irq16: uhci0
   450 c9278000    0     1   450 0000000 [SLPQ select 0xc072ade4][SLP] ntpd
   369 c913e20c    0     1   369 0000000 [SLPQ select 0xc072ade4][SLP] syslogd
 --More--          326 c8fd9830    0     1   326 0000000 [SLPQ select 0xc072ade4][SLP] devd
   164 c8fd9c48    0     1   164 0000000 [SLPQ pause 0xc8fd9c7c][SLP] adjkerntz
    44 c913d418    0     0     0 0000204 [SLPQ - 0xe7eaed04][SLP] schedcpu
    43 c913d624    0     0     0 0000204 [SLPQ - 0xc072e90c][SLP] nfsiod 3
    42 c913d830    0     0     0 0000204 [SLPQ - 0xc072e908][SLP] nfsiod 2
    41 c913da3c    0     0     0 0000204 [SLPQ - 0xc072e904][SLP] nfsiod 1
    40 c913dc48    0     0     0 0000204 [SLPQ - 0xc072e900][SLP] nfsiod 0
    39 c913e000    0     0     0 0000204 [SLPQ sdflush 0xc0733eb4][SLP] softdepflush
    38 c8fa7c48    0     0     0 0000204 [SLPQ vlruwt 0xc8fa7c48][SLP] vnlru
    37 c8fd8000    0     0     0 0000204 [SLPQ syncer 0xc06dd83c][SLP] syncer
    36 c8fd820c    0     0     0 0000204 [SLPQ psleep 0xc072b34c][SLP] bufdaemon
    35 c8fd8418    0     0     0 000020c [SLPQ pgzero 0xc0735164][SLP] pagezero
    34 c8fd8624    0     0     0 0000204 [SLPQ psleep 0xc0734cb4][SLP] vmdaemon
    33 c8fd8830    0     0     0 0000204 [SLPQ psleep 0xc0734c70][SLP] pagedaemon
    32 c8fd8a3c    0     0     0 0000204 [SLPQ - 0xc8fa383c][SLP] fdc0
    31 c8fd8c48    0     0     0 0000204 [IWAIT] swi0: sio
    30 c8fd9000    0     0     0 0000204 [IWAIT] irq12: psm0
    29 c8fd920c    0     0     0 0000204 [IWAIT] irq1: atkbd0
    28 c8fd9418    0     0     0 0000204 [IWAIT] irq15: ata1
    27 c8f46624    0     0     0 0000204 [IWAIT] irq14: ata0
 --More--           26 c8f46830    0     0     0 0000204 [IWAIT] irq29: em1
    25 c8f46a3c    0     0     0 0000204 [IWAIT] irq28: em0
    24 c8f46c48    0     0     0 0000204 [IWAIT] irq48: amr0
    23 c8fa7000    0     0     0 0000204 [IWAIT] irq9: acpi0
    22 c8fa720c    0     0     0 0000204 [IWAIT] swi6: +
    21 c8fa7418    0     0     0 0000204 [IWAIT] swi6: task queue
     9 c8fa7624    0     0     0 0000204 [SLPQ - 0xc8fa6080][SLP] acpi_task2
     8 c8fa7830    0     0     0 0000204 [SLPQ - 0xc8fa6080][SLP] acpi_task1
     7 c8fa7a3c    0     0     0 0000204 [SLPQ - 0xc8fa6080][SLP] acpi_task0
    20 c8ee520c    0     0     0 0000204 [IWAIT] swi2: cambio
     6 c8ee5418    0     0     0 0000204 [SLPQ - 0xc8f44700][SLP] kqueue taskq
    19 c8ee5624    0     0     0 0000204 [IWAIT] swi5: +
     5 c8ee5830    0     0     0 0000204 [SLPQ - 0xc8f44880][SLP] thread taskq
    18 c8ee5a3c    0     0     0 0000204 [SLPQ - 0xc06da1e0][SLP] yarrow
     4 c8ee5c48    0     0     0 0000204 [SLPQ - 0xc06dafa8][SLP] g_down
     3 c8f46000    0     0     0 0000204 [SLPQ - 0xc06dafa4][SLP] g_up
     2 c8f4620c    0     0     0 0000204 [SLPQ - 0xc06daf9c][SLP] g_event
    17 c8f46418    0     0     0 0000204 [IWAIT] swi1: net
    16 c8ee0000    0     0     0 0000204 [IWAIT] swi3: vm
    15 c8ee020c    0     0     0 000020c [IWAIT] swi4: clock sio
 --More--           14 c8ee0418    0     0     0 000020c [Can run] idle: cpu0
    13 c8ee0624    0     0     0 000020c [CPU 1] idle: cpu1
    12 c8ee0830    0     0     0 000020c [CPU 2] idle: cpu2
    11 c8ee0a3c    0     0     0 000020c [CPU 3] idle: cpu3
     1 c8ee0c48    0     0     1 0004200 [SLPQ wait 0xc8ee0c48][SLP] init
    10 c8ee5000    0     0     0 0000204 [SLPQ ktrace 0xc06db9f8][SLP] ktrace
     0 c06db0a0    0     0     0 0000200 [IWAIT] swapper
 db> sh allpcpu
 Current CPU: 0
 
 cpuid        = 0
 curthread    = 0xc996dd80: pid 679 "httpd"
 curpcb       = 0xea255d90
 fpcurthread  = none
 idlethread   = 0xc8ee1c00: pid 14 "idle: cpu0"
 APIC ID      = 0
 currentldt   = 0x50
 spin locks held:
 exclusive spin mutex sio r = 0 (0xc07370e0) locked @ /shared/FREEBSD/src6/sys/dev/sio/sio.c:1390
 
 cpuid        = 1
 curthread    = 0xc8ee1a80: pid 13 "idle: cpu1"
 curpcb       = 0xe5ba2d90
 fpcurthread  = none
 idlethread   = 0xc8ee1a80: pid 13 "idle: cpu1"
 APIC ID      = 1
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 2
 curthread    = 0xc8ee1900: pid 12 "idle: cpu2"
 curpcb       = 0xe5b9fd90
 fpcurthread  = none
 idlethread   = 0xc8ee1900: pid 12 "idle: cpu2"
 APIC ID      = 6
 currentldt   = 0x50
 spin locks held:
 
 cpuid        = 3
 curthread    = 0xc8ee1780: pid 11 "idle: cpu3"
 curpcb       = 0xe5b9cd90
 fpcurthread  = none
 idlethread   = 0xc8ee1780: pid 11 "idle: cpu3"
 APIC ID      = 7
 currentldt   = 0x50
 spin locks held:
 
 db> sh alllocks
 db> sh lockedvnods
 Locked vnodes
 
 0xced99984: tag ufs, type VREG
     usecount 1, writecount 0, refcount 7 mountedhere 0
     flags ()
     v_object 0xcfc4aad4 ref 0 pages 80
      lock type ufs: EXCL (count 1) by thread 0xcbff1900 (pid 2230)#0 0xc04f9955 at lockmgr+0x4ed
 #1 0xc0550dfd at vop_stdlock+0x21
 #2 0xc06546af at VOP_LOCK_APV+0x87
 #3 0xc05e4a78 at ffs_lock+0x10
 #4 0xc06546af at VOP_LOCK_APV+0x87
 #5 0xc05657dc at vn_lock+0xac
 #6 0xc0564ef7 at vn_read+0xd7
 #7 0xc0528b11 at dofileread+0x85
 #8 0xc05289aa at kern_readv+0x36
 #9 0xc05288d5 at read+0x45
 #10 0xc0642207 at syscall+0x25b
 #11 0xc062f9af at Xint0x80_syscall+0x1f
 
 	ino 4487869, on dev amrd0s1g
 
 0xcd882828: tag ufs, type VREG
     usecount 1, writecount 0, refcount 17 mountedhere 0
     flags ()
     v_object 0xccd27108 ref 0 pages 1874
      lock type ufs: EXCL (count 1) by thread 0xce3d9780 (pid 2138)#0 0xc04f9955 at lockmgr+0x4ed
 #1 0xc0550dfd at vop_stdlock+0x21
 #2 0xc06546af at VOP_LOCK_APV+0x87
 #3 0xc05e4a78 at ffs_lock+0x10
 #4 0xc06546af at VOP_LOCK_APV+0x87
 #5 0xc05657dc at vn_lock+0xac
 #6 0xc0564ef7 at vn_read+0xd7
 #7 0xc0528b11 at dofileread+0x85
 #8 0xc05289aa at kern_readv+0x36
 #9 0xc05288d5 at read+0x45
 #10 0xc0642207 at syscall+0x25b
 #11 0xc062f9af at Xint0x80_syscall+0x1f
 
 	ino 5677318, on dev amrd0s1g
 db> bt
 Tracing pid 679 tid 100161 td 0xc996dd80
 kdb_enter(c06963c8) at kdb_enter+0x2b
 siointr1(c8fe7800,c07370e0,0,c06961c8,56e) at siointr1+0xd5
 siointr(c8fe7800) at siointr+0x21
 intr_execute_handlers(c8f358b4,ea255ab0,4,ea255af8,c062fd13) at intr_execute_handlers+0xe1
 lapic_handle_intr(38) at lapic_handle_intr+0x2e
 Xapic_isr1() at Xapic_isr1+0x33
 --- interrupt, eip = 0xc0638aef, esp = 0xea255af4, ebp = 0xea255af8 ---
 spinlock_exit(c10533c8,ea255b1c,c0524c08,c06e39dc,0) at spinlock_exit+0x27
 _mtx_unlock_spin_flags(c06e39dc,0,c0680cd0,1ef,ea255b30) at _mtx_unlock_spin_flags+0x8d
 turnstile_release(c10533c8,c10533c8,8af,ea255b54,c04fcc63) at turnstile_release+0x28
 _mtx_unlock_sleep(c10533c8,0,c069173a,8af) at _mtx_unlock_sleep+0x4b
 _mtx_unlock_flags(c10533c8,0,c069173a,8af,c1050d20,cc366fa8) at _mtx_unlock_flags+0x9b
 uma_zalloc_internal(c1050d20,ea255bfc,2,c10533c8,0) at uma_zalloc_internal+0x7d
 uma_zalloc_arg(c1050d20,ea255bfc,2) at uma_zalloc_arg+0x31f
 sosend(cd2b5164,0,ea255cbc,0,0) at sosend+0x3c5
 soo_write(c917db88,ea255cbc,c9530380,0,c996dd80) at soo_write+0x46
 dofilewrite(c996dd80,3,c917db88,ea255cbc,ffffffff) at dofilewrite+0x77
 kern_writev(c996dd80,3,ea255cbc,bfbfdb48,a58) at kern_writev+0x3b
 write(c996dd80,ea255d04,3,701,296) at write+0x45
 syscall(3b,281a003b,bfbf003b,2000,bfbfc5a0) at syscall+0x25b
 --More--        Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (4, FreeBSD ELF32, write), eip = 0x2818c7bf, esp = 0xbfbfc45c, ebp = 0xbfbfc478 ---
 
 db> where 1419
 Tracing pid 1419 tid 100409 td 0xc97a0900
 sched_switch(c97a0900,0,1) at sched_switch+0x177
 mi_switch(1,0) at mi_switch+0x270
 sleepq_switch(c9188290,c06dd1a0,0,c06805df,20c) at sleepq_switch+0xc1
 sleepq_wait(c9188290,0,c9188290,0,c067949a) at sleepq_wait+0x46
 msleep(c9188290,0,4c,c066d69e,0) at msleep+0x275
 amr_wait_command(c9188290,0,0,ce3dbcec,ce3dbcc0) at amr_wait_command+0x41
 amr_ioctl(c8fbc500,c0304301,ce3dbcc0,1,c97a0900) at amr_ioctl+0x2c1
 giant_ioctl(c8fbc500,c0304301,ce3dbcc0,1,c97a0900) at giant_ioctl+0x33
 devfs_ioctl_f(cbd33990,c0304301,ce3dbcc0,cbe2d700,c97a0900) at devfs_ioctl_f+0xaf
 ioctl(c97a0900,ea669d04,3,1,296) at ioctl+0x370
 syscall(3b,3b,3b,bfbfee68,bfbfee7c) at syscall+0x25b
 Xint0x80_syscall() at Xint0x80_syscall+0x1f
 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x8079d2f, esp = 0xbfbfe95c, ebp = 0xbfbfe9c8 ---
>Unformatted:
