From nobody@FreeBSD.org  Fri Dec 10 04:30:43 2004
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id BD28816A4CE
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 10 Dec 2004 04:30:43 +0000 (GMT)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 853C543D54
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 10 Dec 2004 04:30:43 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id iBA4UhCc086664
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 10 Dec 2004 04:30:43 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id iBA4Uhmc086663;
	Fri, 10 Dec 2004 04:30:43 GMT
	(envelope-from nobody)
Message-Id: <200412100430.iBA4Uhmc086663@www.freebsd.org>
Date: Fri, 10 Dec 2004 04:30:43 GMT
From: Geoffrey Lassner <freebsd53@zyni.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: SMP crash FreeBSD-5.3-p2 (swi1: net) / ipllog
X-Send-Pr-Version: www-2.3

>Number:         74909
>Category:       kern
>Synopsis:       SMP crash FreeBSD-5.3-p2 (swi1: net) / ipllog
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Dec 10 04:40:22 GMT 2004
>Closed-Date:    Fri Apr 08 02:25:45 GMT 2005
>Last-Modified:  Fri Apr 08 02:25:45 GMT 2005
>Originator:     Geoffrey Lassner
>Release:        5.3-RELEASE-p2
>Organization:
None
>Environment:
FreeBSD oak.zyni.com 5.3-RELEASE-p2 FreeBSD 5.3-RELEASE-p2 #2: Wed Dec  8 22:06:28 MST 2004     root@oak.zyni.com:/usr/src/sys/i386/compile/OAK-TEST  i386

>Description:
Crashes with SMP enabled on 5.3 and 5.3-p2 (two systems identical hardware)

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 00
fault virtual address   = 0x4
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xc0464757
stack pointer           = 0x10:0xd4233b10
frame pointer           = 0x10:0xd4233b1c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 31 (swi1: net)
[thread 100027]
Stopped at      ipllog+0x43:    incl    0x4(%eax)
db> show thread
  100115 (0xc1f2e7d0)  fork_trampoline() at fork_trampoline
  100100 (0xc1f2d000)  sched_switch(c1f2d000,0,1) at sched_switch+0x143
  100066 (0xc1a4e960)  sched_switch(c1a4e960,0,2) at sched_switch+0x143
console 3%Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 00
fault virtual address   = 0x4
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xc0464757
stack pointer           = 0x10:0xd4233b10
frame pointer           = 0x10:0xd4233b1c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 31 (swi1: net)
[thread 100027]
Stopped at      ipllog+0x43:    incl    0x4(%eax)
db> show thread
  100115 (0xc1f2e7d0)  fork_trampoline() at fork_trampoline
  100100 (0xc1f2d000)  sched_switch(c1f2d000,0,1) at sched_switch+0x143
  100066 (0xc1a4e960)  sched_switch(c1a4e960,0,2) at sched_switch+0x143
  100051 (0xc1a4b190)  sched_switch(c1a4b190,0,2) at sched_switch+0x143
  100118 (0xc1f2ec80)  sched_switch(c1f2ec80,0,1) at sched_switch+0x143
  100102 (0xc1f2d320)  sched_switch(c1f2d320,0,1) at sched_switch+0x143
  100144 (0xc20d0640)  sched_switch(c20d0640,0,1) at sched_switch+0x143
  100104 (0xc1f2d640)  sched_switch(c1f2d640,0,1) at sched_switch+0x143
  100145 (0xc20d07d0)  sched_switch(c20d07d0,0,1) at sched_switch+0x143
  100121 (0xc1f53190)  sched_switch(c1f53190,0,2) at sched_switch+0x143
  100186 (0xc237e960)  sched_switch(c237e960,0,1) at sched_switch+0x143
  100138 (0xc20cfc80)  sched_switch(c20cfc80,0,1) at sched_switch+0x143
  100119 (0xc1f2ee10)  sched_switch(c1f2ee10,0,2) at sched_switch+0x143
  100135 (0xc20cf7d0)  sched_switch(c20cf7d0,0,1) at sched_switch+0x143
  100134 (0xc20cf640)  sched_switch(c20cf640,0,1) at sched_switch+0x143
  100058 (0xc1a4bc80)  sched_switch(c1a4bc80,0,1) at sched_switch+0x143
  100122 (0xc1f53320)  sched_switch(c1f53320,0,1) at sched_switch+0x143
geoff@cyclone % more console
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 00
fault virtual address   = 0x4
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xc0464757
stack pointer           = 0x10:0xd4233b10
frame pointer           = 0x10:0xd4233b1c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 31 (swi1: net)
[thread 100027]
Stopped at      ipllog+0x43:    incl    0x4(%eax)
db> show thread
  100115 (0xc1f2e7d0)  fork_trampoline() at fork_trampoline
  100100 (0xc1f2d000)  sched_switch(c1f2d000,0,1) at sched_switch+0x143
  100066 (0xc1a4e960)  sched_switch(c1a4e960,0,2) at sched_switch+0x143
  100051 (0xc1a4b190)  sched_switch(c1a4b190,0,2) at sched_switch+0x143
  100118 (0xc1f2ec80)  sched_switch(c1f2ec80,0,1) at sched_switch+0x143
  100102 (0xc1f2d320)  sched_switch(c1f2d320,0,1) at sched_switch+0x143
  100144 (0xc20d0640)  sched_switch(c20d0640,0,1) at sched_switch+0x143
  100104 (0xc1f2d640)  sched_switch(c1f2d640,0,1) at sched_switch+0x143
  100145 (0xc20d07d0)  sched_switch(c20d07d0,0,1) at sched_switch+0x143
  100121 (0xc1f53190)  sched_switch(c1f53190,0,2) at sched_switch+0x143
  100186 (0xc237e960)  sched_switch(c237e960,0,1) at sched_switch+0x143
  100138 (0xc20cfc80)  sched_switch(c20cfc80,0,1) at sched_switch+0x143
  100119 (0xc1f2ee10)  sched_switch(c1f2ee10,0,2) at sched_switch+0x143
  100135 (0xc20cf7d0)  sched_switch(c20cf7d0,0,1) at sched_switch+0x143
  100134 (0xc20cf640)  sched_switch(c20cf640,0,1) at sched_switch+0x143
  100058 (0xc1a4bc80)  sched_switch(c1a4bc80,0,1) at sched_switch+0x143
  100122 (0xc1f53320)  sched_switch(c1f53320,0,1) at sched_switch+0x143
  100081 (0xc1c6f190)  sched_switch(c1c6f190,0,2) at sched_switch+0x143
  100083 (0xc1c6f4b0)  sched_switch(c1c6f4b0,0,1) at sched_switch+0x143
  100069 (0xc1a4ee10)  sched_switch(c1a4ee10,0,1) at sched_switch+0x143
  100075 (0xc1c6e7d0)  sched_switch(c1c6e7d0,0,1) at sched_switch+0x143
  100113 (0xc1f2e4b0)  sched_switch(c1f2e4b0,0,1) at sched_switch+0x143
  100093 (0xc1c724b0)  sched_switch(c1c724b0,0,1) at sched_switch+0x143
  100084 (0xc1c6f640)  sched_switch(c1c6f640,0,1) at sched_switch+0x143
  100055 (0xc1a4b7d0)  sched_switch(c1a4b7d0,0,1) at sched_switch+0x143
  100062 (0xc1a4e320)  sched_switch(c1a4e320,0,1) at sched_switch+0x143
  100070 (0xc1c6e000)  sched_switch(c1c6e000,0,1) at sched_switch+0x143
  100111 (0xc1f2e190)  sched_switch(c1f2e190,0,2) at sched_switch+0x143
  100078 (0xc1c6ec80)  sched_switch(c1c6ec80,0,1) at sched_switch+0x143
  100110 (0xc1f2e000)  sched_switch(c1f2e000,0,1) at sched_switch+0x143
  100098 (0xc1c72c80)  sched_switch(c1c72c80,0,1) at sched_switch+0x143
  100099 (0xc1c72e10)  sched_switch(c1c72e10,0,1) at sched_switch+0x143
  100036 (0xc19a1960)  sched_switch(c19a1960,0,1) at sched_switch+0x143
  100085 (0xc1c6f7d0)  sched_switch(c1c6f7d0,0,1) at sched_switch+0x143
  100082 (0xc1c6f320)  sched_switch(c1c6f320,0,1) at sched_switch+0x143
  100112 (0xc1f2e320)  sched_switch(c1f2e320,0,1) at sched_switch+0x143
  100056 (0xc1a4b960)  sched_switch(c1a4b960,0,1) at sched_switch+0x143
  100109 (0xc1f2de10)  sched_switch(c1f2de10,0,1) at sched_switch+0x143
  100117 (0xc1f2eaf0)  sched_switch(c1f2eaf0,0,1) at sched_switch+0x143
  100092 (0xc1c72320)  sched_switch(c1c72320,0,1) at sched_switch+0x143
  100090 (0xc1c72000)  sched_switch(c1c72000,0,1) at sched_switch+0x143
  100080 (0xc1c6f000)  sched_switch(c1c6f000,0,1) at sched_switch+0x143
  100087 (0xc1c6faf0)  sched_switch(c1c6faf0,0,1) at sched_switch+0x143
  100065 (0xc1a4e7d0)  sched_switch(c1a4e7d0,0,1) at sched_switch+0x143
  100067 (0xc1a4eaf0)  sched_switch(c1a4eaf0,0,1) at sched_switch+0x143
  100101 (0xc1f2d190)  sched_switch(c1f2d190,0,1) at sched_switch+0x143
  100106 (0xc1f2d960)  sched_switch(c1f2d960,0,1) at sched_switch+0x143
  100126 (0xc1f53960)  sched_switch(c1f53960,0,1) at sched_switch+0x143
  100089 (0xc1c6fe10)  sched_switch(c1c6fe10,0,1) at sched_switch+0x143
  100096 (0xc1c72960)  sched_switch(c1c72960,0,1) at sched_switch+0x143
  100086 (0xc1c6f960)  sched_switch(c1c6f960,0,1) at sched_switch+0x143
  100039 (0xc19a1e10)  sched_switch(c19a1e10,0,1) at sched_switch+0x143
  100095 (0xc1c727d0)  sched_switch(c1c727d0,0,1) at sched_switch+0x143
  100057 (0xc1a4baf0)  sched_switch(c1a4baf0,0,1) at sched_switch+0x143
  100068 (0xc1a4ec80)  sched_switch(c1a4ec80,0,1) at sched_switch+0x143
  100091 (0xc1c72190)  sched_switch(c1c72190,0,1) at sched_switch+0x143
  100064 (0xc1a4e640)  sched_switch(c1a4e640,0,1) at sched_switch+0x143
  100063 (0xc1a4e4b0)  sched_switch(c1a4e4b0,0,1) at sched_switch+0x143
  100052 (0xc1a4b320)  sched_switch(c1a4b320,0,1) at sched_switch+0x143
  100094 (0xc1c72640)  sched_switch(c1c72640,0,1) at sched_switch+0x143
  100127 (0xc1f53af0)  sched_switch(c1f53af0,0,2) at sched_switch+0x143
  100128 (0xc1f53c80)  sched_switch(c1f53c80,0,1) at sched_switch+0x143
  100103 (0xc1f2d4b0)  sched_switch(c1f2d4b0,0,1) at sched_switch+0x143
  100097 (0xc1c72af0)  sched_switch(c1c72af0,0,1) at sched_switch+0x143
  100123 (0xc1f534b0)  sched_switch(c1f534b0,0,1) at sched_switch+0x143
  100108 (0xc1f2dc80)  sched_switch(c1f2dc80,0,1) at sched_switch+0x143
  100088 (0xc1c6fc80)  sched_switch(c1c6fc80,0,1) at sched_switch+0x143
  100059 (0xc1a4be10)  sched_switch(c1a4be10,0,1) at sched_switch+0x143
  100073 (0xc1c6e4b0)  sched_switch(c1c6e4b0,0,1) at sched_switch+0x143
  100072 (0xc1c6e320)  sched_switch(c1c6e320,0,1) at sched_switch+0x143
  100060 (0xc1a4e000)  sched_switch(c1a4e000,0,1) at sched_switch+0x143
  100038 (0xc19a1c80)  sched_switch(c19a1c80,0,1) at sched_switch+0x143
  100071 (0xc1c6e190)  sched_switch(c1c6e190,0,1) at sched_switch+0x143
  100053 (0xc1a4b4b0)  sched_switch(c1a4b4b0,0,1) at sched_switch+0x143
  100079 (0xc1c6ee10)  sched_switch(c1c6ee10,0,1) at sched_switch+0x143
  100074 (0xc1c6e640)  sched_switch(c1c6e640,0,1) at sched_switch+0x143
  100054 (0xc1a4b640)  sched_switch(c1a4b640,0,1) at sched_switch+0x143
  100076 (0xc1c6e960)  sched_switch(c1c6e960,0,1) at sched_switch+0x143
  100061 (0xc1a4e190)  sched_switch(c1a4e190,0,1) at sched_switch+0x143
  100077 (0xc1c6eaf0)  sched_switch(c1c6eaf0,0,1) at sched_switch+0x143
  100025 (0xc196e7d0)  sched_switch(c196e7d0,0,1) at sched_switch+0x143
  100024 (0xc196e640)  sched_switch(c196e640,0,1) at sched_switch+0x143
  100023 (0xc196e4b0)  sched_switch(c196e4b0,0,1) at sched_switch+0x143
  100049 (0xc1a48e10)  sched_switch(c1a48e10,0,1) at sched_switch+0x143
  100048 (0xc1a48c80)  sched_switch(c1a48c80,0,1) at sched_switch+0x143
  100047 (0xc1a48af0)  sched_switch(c1a48af0,0,1) at sched_switch+0x143
  100046 (0xc1a48960)  sched_switch(c1a48960,0,1) at sched_switch+0x143
  100045 (0xc1a487d0)  sched_switch(c1a487d0,0,1) at sched_switch+0x143
  100044 (0xc1a48640)  sched_switch(c1a48640,0,1) at sched_switch+0x143
  100043 (0xc1a484b0)  sched_switch(c1a484b0,0,1) at sched_switch+0x143
  100042 (0xc1a48320)  sched_switch(c1a48320,0,1) at sched_switch+0x143
  100041 (0xc1a48190)  sched_switch(c1a48190,0,1) at sched_switch+0x143
  100040 (0xc1a48000)  fork_trampoline() at fork_trampoline
  100012 (0xc196a320)  fork_trampoline() at fork_trampoline
  100011 (0xc196a190)  sched_switch(c196a190,0,1) at sched_switch+0x143
  100035 (0xc19a17d0)  sched_switch(c19a17d0,0,1) at sched_switch+0x143
  100034 (0xc19a1640)  fork_trampoline() at fork_trampoline
  100033 (0xc19a14b0)  fork_trampoline() at fork_trampoline
  100032 (0xc19a1320)  sched_switch(c19a1320,0,1) at sched_switch+0x143
  100031 (0xc19a1190)  sched_switch(c19a1190,0,1) at sched_switch+0x143
  100030 (0xc19a1000)  sched_switch(18f1420,c1b0d800,c1c34800,c1bcab00,d5413c7c) at sched_switch+
0x143
  100029 (0xc196ee10)  sched_switch(c196ee10,0,1) at sched_switch+0x143
  100028 (0xc196ec80)  sched_switch(c196ec80,0,1) at sched_switch+0x143
  100027 (0xc196eaf0)  ipllog(0,d4233bd8,d4233b50,d4233b48,d4233b40) at ipllog+0x43
  100026 (0xc196e960)  fork_trampoline() at fork_trampoline
  100000 (0xc1965000)  sched_switch(c1965000,0,1) at sched_switch+0x143
  100022 (0xc196e320)  fork_trampoline() at fork_trampoline
  100021 (0xc196e190)  fork_trampoline() at fork_trampoline
  100020 (0xc196e000)  fork_trampoline() at fork_trampoline
  100019 (0xc196ae10)  fork_trampoline() at fork_trampoline
  100018 (0xc196ac80)  fork_trampoline() at fork_trampoline
  100017 (0xc196aaf0)  sched_switch(c196aaf0,0,1) at sched_switch+0x143
  100016 (0xc196a960)  sched_switch(c196a960,0,1) at sched_switch+0x143
  100015 (0xc196a7d0)  sched_switch(c196a7d0,0,1) at sched_switch+0x143
  100014 (0xc196a640)  fork_trampoline() at fork_trampoline
  100013 (0xc196a4b0)  sched_switch(c196a4b0,0,1) at sched_switch+0x143
  100010 (0xc196a000)  fork_trampoline() at fork_trampoline
  100009 (0xc1965e10)  fork_trampoline() at fork_trampoline
  100008 (0xc1965c80)  fork_trampoline() at fork_trampoline
  100007 (0xc1965af0)  fork_trampoline() at fork_trampoline
  100006 (0xc1965960)  fork_trampoline() at fork_trampoline
  100005 (0xc19657d0)  fork_trampoline() at fork_trampoline
  100004 (0xc1965640)  sched_switch(c1965640,0,1) at sched_switch+0x143
  100003 (0xc19654b0)  sched_switch(c19654b0,0,1) at sched_switch+0x143
  100002 (0xc1965320)  sched_switch(c1965320,0,1) at sched_switch+0x143
  100001 (0xc1965190)  sched_switch(c1965190,0,1) at sched_switch+0x143
       0 (0xc08bce00)  sched_switch(c08bce00,0,1) at sched_switch+0x143
db> show all procs
  pid   proc     uarea   uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
28532 c20d5388 d6a8a000    0   753   753 0000002 [RUNQ] find
27240 c20d554c d6a8b000    0   735 27240 0004002 [SLPQ nanslp 0xc08c256c][SLP] sleep
27239 c20d4e20 d6a87000    0   735 27239 0004002 [RUNQ] tcpdump
23076 c20d4000 d6a7f000 1000 23075 23075 0004002 [RUNQ] ssh
23075 c20d4710 d6a83000 1000   718 23075 0004002 [SLPQ pipdwt 0xc2268480][SLP] scp
17334 c20d41c4 d6a80000    0 17331 17320 0000002 [SLPQ pause 0xc20d41fc][SLP] dump
17333 c20ce710 d6a6b000    0 17331 17320 0000002 [SLPQ pause 0xc20ce748][SLP] dump
17332 c20d454c d6a82000    0 17331 17320 0000002 [SLPQ pipdwt 0xc2268c00][SLP] dump
17331 c20ce54c d6a6a000    0 17320 17320 0000002 [SLPQ sbwait 0xc209ee84][SLP] dump
17321 c20d51c4 d6a89000    0   732 17320 0004002 [RUNQ] gzip
17320 c2378388 d6aff000    0   732 17320 0004002 [SLPQ wait 0xc2378388][SLP] dump
  753 c20d11c4 d6a71000    0   724   753 0004003 [SLPQ wait 0xc20d11c4][SLP] find
  752 c20d4c5c d6a86000 1000   700   752 0004002 [RUNQ] bzip2
  743 c20d1710 d6a74000    0   740   738 0000002 [SLPQ pause 0xc20d1748][SLP] dump
  742 c20d18d4 d6a75000    0   740   738 0000002 [SLPQ pause 0xc20d190c][SLP] dump
  741 c1c5c388 d6896000    0   740   738 0000002 [SLPQ pipdwt 0xc1d9f180][SLP] dump
  740 c2039000 d689e000    0   738   738 0000002 [SLPQ sbwait 0xc209eab8][SLP] dump
  739 c1f29710 d693f000    0   729   738 0004002 [RUNQ] gzip
  738 c1f29388 d693d000    0   729   738 0004002 [SLPQ wait 0xc1f29388][SLP] dump
  735 c1c71388 d68e5000    0   712   735 0004003 [SLPQ pause 0xc1c713c0][SLP] tcsh
  732 c1c6d8d4 d68df000    0   706   732 0004003 [SLPQ pause 0xc1c6d90c][SLP] tcsh
  729 c20391c4 d6918000    0   694   729 0004003 [SLPQ pause 0xc20391fc][SLP] tcsh
  724 c1f281c4 d6928000 1000   723   724 0004002 [SLPQ pause 0xc1f281fc][SLP] tcsh
  723 c1f291c4 d693c000 1000   690   690 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  718 c1c5c8d4 d6899000 1000   717   718 0004002 [SLPQ pause 0xc1c5c90c][SLP] tcsh
  717 c1a4ac5c d54a5000 1000   687   687 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  712 c1c711c4 d68e4000 1000   711   712 0004002 [SLPQ pause 0xc1c711fc][SLP] tcsh
  711 c1f29a98 d697d000 1000   684   684 0000100 [RUNQ] sshd
  706 c1c6d388 d68dc000 1000   705   706 0004002 [SLPQ pause 0xc1c6d3c0][SLP] tcsh
  705 c1f29c5c d697e000 1000   681   681 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  700 c203954c d698d000 1000   699   700 0004002 [SLPQ pause 0xc2039584][SLP] tcsh
  699 c2039710 d698f000 1000   678   678 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  694 c1a4a710 d5466000 1000   693   694 0004002 [SLPQ pause 0xc1a4a748][SLP] tcsh
  693 c1f29000 d693b000 1000   675   675 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  690 c1f2954c d693e000    0   461   690 0000100 [SLPQ sbwait 0xc209b5a8][SLP] sshd
  687 c1f298d4 d6940000    0   461   687 0000100 [SLPQ sbwait 0xc1f85e84][SLP] sshd
  684 c1c5c710 d6898000    0   461   684 0000100 [SLPQ sbwait 0xc1f85bfc][SLP] sshd
  681 c1f29e20 d697f000    0   461   681 0000100 [SLPQ sbwait 0xc1d95d40][SLP] sshd
  678 c2039388 d692f000    0   461   678 0000100 [SLPQ sbwait 0xc1f86320][SLP] sshd
  675 c1f28388 d6930000    0   461   675 0000100 [SLPQ sbwait 0xc1f86974][SLP] sshd
  674 c1f28710 d6936000    0   617   617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5
  673 c1c6d000 d689d000    0   617   617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5
geoff@cyclone % 
geoff@cyclone % 
geoff@cyclone % 
geoff@cyclone % 
geoff@cyclone % 
geoff@cyclone % 
geoff@cyclone % 
geoff@cyclone % 
geoff@cyclone % ls
console                 instruction_pointer     mptable
geoff@cyclone % cat console 
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 00
fault virtual address   = 0x4
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xc0464757
stack pointer           = 0x10:0xd4233b10
frame pointer           = 0x10:0xd4233b1c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 31 (swi1: net)
[thread 100027]
Stopped at      ipllog+0x43:    incl    0x4(%eax)
db> show thread
  100115 (0xc1f2e7d0)  fork_trampoline() at fork_trampoline
  100100 (0xc1f2d000)  sched_switch(c1f2d000,0,1) at sched_switch+0x143
  100066 (0xc1a4e960)  sched_switch(c1a4e960,0,2) at sched_switch+0x143
  100051 (0xc1a4b190)  sched_switch(c1a4b190,0,2) at sched_switch+0x143
  100118 (0xc1f2ec80)  sched_switch(c1f2ec80,0,1) at sched_switch+0x143
  100102 (0xc1f2d320)  sched_switch(c1f2d320,0,1) at sched_switch+0x143
  100144 (0xc20d0640)  sched_switch(c20d0640,0,1) at sched_switch+0x143
  100104 (0xc1f2d640)  sched_switch(c1f2d640,0,1) at sched_switch+0x143
  100145 (0xc20d07d0)  sched_switch(c20d07d0,0,1) at sched_switch+0x143
  100121 (0xc1f53190)  sched_switch(c1f53190,0,2) at sched_switch+0x143
  100186 (0xc237e960)  sched_switch(c237e960,0,1) at sched_switch+0x143
  100138 (0xc20cfc80)  sched_switch(c20cfc80,0,1) at sched_switch+0x143
  100119 (0xc1f2ee10)  sched_switch(c1f2ee10,0,2) at sched_switch+0x143
  100135 (0xc20cf7d0)  sched_switch(c20cf7d0,0,1) at sched_switch+0x143
  100134 (0xc20cf640)  sched_switch(c20cf640,0,1) at sched_switch+0x143
  100058 (0xc1a4bc80)  sched_switch(c1a4bc80,0,1) at sched_switch+0x143
  100122 (0xc1f53320)  sched_switch(c1f53320,0,1) at sched_switch+0x143
  100081 (0xc1c6f190)  sched_switch(c1c6f190,0,2) at sched_switch+0x143
  100083 (0xc1c6f4b0)  sched_switch(c1c6f4b0,0,1) at sched_switch+0x143
  100069 (0xc1a4ee10)  sched_switch(c1a4ee10,0,1) at sched_switch+0x143
  100075 (0xc1c6e7d0)  sched_switch(c1c6e7d0,0,1) at sched_switch+0x143
  100113 (0xc1f2e4b0)  sched_switch(c1f2e4b0,0,1) at sched_switch+0x143
  100093 (0xc1c724b0)  sched_switch(c1c724b0,0,1) at sched_switch+0x143
  100084 (0xc1c6f640)  sched_switch(c1c6f640,0,1) at sched_switch+0x143
  100055 (0xc1a4b7d0)  sched_switch(c1a4b7d0,0,1) at sched_switch+0x143
  100062 (0xc1a4e320)  sched_switch(c1a4e320,0,1) at sched_switch+0x143
  100070 (0xc1c6e000)  sched_switch(c1c6e000,0,1) at sched_switch+0x143
  100111 (0xc1f2e190)  sched_switch(c1f2e190,0,2) at sched_switch+0x143
  100078 (0xc1c6ec80)  sched_switch(c1c6ec80,0,1) at sched_switch+0x143
  100110 (0xc1f2e000)  sched_switch(c1f2e000,0,1) at sched_switch+0x143
  100098 (0xc1c72c80)  sched_switch(c1c72c80,0,1) at sched_switch+0x143
  100099 (0xc1c72e10)  sched_switch(c1c72e10,0,1) at sched_switch+0x143
  100036 (0xc19a1960)  sched_switch(c19a1960,0,1) at sched_switch+0x143
  100085 (0xc1c6f7d0)  sched_switch(c1c6f7d0,0,1) at sched_switch+0x143
  100082 (0xc1c6f320)  sched_switch(c1c6f320,0,1) at sched_switch+0x143
  100112 (0xc1f2e320)  sched_switch(c1f2e320,0,1) at sched_switch+0x143
  100056 (0xc1a4b960)  sched_switch(c1a4b960,0,1) at sched_switch+0x143
  100109 (0xc1f2de10)  sched_switch(c1f2de10,0,1) at sched_switch+0x143
  100117 (0xc1f2eaf0)  sched_switch(c1f2eaf0,0,1) at sched_switch+0x143
  100092 (0xc1c72320)  sched_switch(c1c72320,0,1) at sched_switch+0x143
  100090 (0xc1c72000)  sched_switch(c1c72000,0,1) at sched_switch+0x143
  100080 (0xc1c6f000)  sched_switch(c1c6f000,0,1) at sched_switch+0x143
  100087 (0xc1c6faf0)  sched_switch(c1c6faf0,0,1) at sched_switch+0x143
  100065 (0xc1a4e7d0)  sched_switch(c1a4e7d0,0,1) at sched_switch+0x143
  100067 (0xc1a4eaf0)  sched_switch(c1a4eaf0,0,1) at sched_switch+0x143
  100101 (0xc1f2d190)  sched_switch(c1f2d190,0,1) at sched_switch+0x143
  100106 (0xc1f2d960)  sched_switch(c1f2d960,0,1) at sched_switch+0x143
  100126 (0xc1f53960)  sched_switch(c1f53960,0,1) at sched_switch+0x143
  100089 (0xc1c6fe10)  sched_switch(c1c6fe10,0,1) at sched_switch+0x143
  100096 (0xc1c72960)  sched_switch(c1c72960,0,1) at sched_switch+0x143
  100086 (0xc1c6f960)  sched_switch(c1c6f960,0,1) at sched_switch+0x143
  100039 (0xc19a1e10)  sched_switch(c19a1e10,0,1) at sched_switch+0x143
  100095 (0xc1c727d0)  sched_switch(c1c727d0,0,1) at sched_switch+0x143
  100057 (0xc1a4baf0)  sched_switch(c1a4baf0,0,1) at sched_switch+0x143
  100068 (0xc1a4ec80)  sched_switch(c1a4ec80,0,1) at sched_switch+0x143
  100091 (0xc1c72190)  sched_switch(c1c72190,0,1) at sched_switch+0x143
  100064 (0xc1a4e640)  sched_switch(c1a4e640,0,1) at sched_switch+0x143
  100063 (0xc1a4e4b0)  sched_switch(c1a4e4b0,0,1) at sched_switch+0x143
  100052 (0xc1a4b320)  sched_switch(c1a4b320,0,1) at sched_switch+0x143
  100094 (0xc1c72640)  sched_switch(c1c72640,0,1) at sched_switch+0x143
  100127 (0xc1f53af0)  sched_switch(c1f53af0,0,2) at sched_switch+0x143
  100128 (0xc1f53c80)  sched_switch(c1f53c80,0,1) at sched_switch+0x143
  100103 (0xc1f2d4b0)  sched_switch(c1f2d4b0,0,1) at sched_switch+0x143
  100097 (0xc1c72af0)  sched_switch(c1c72af0,0,1) at sched_switch+0x143
  100123 (0xc1f534b0)  sched_switch(c1f534b0,0,1) at sched_switch+0x143
  100108 (0xc1f2dc80)  sched_switch(c1f2dc80,0,1) at sched_switch+0x143
  100088 (0xc1c6fc80)  sched_switch(c1c6fc80,0,1) at sched_switch+0x143
  100059 (0xc1a4be10)  sched_switch(c1a4be10,0,1) at sched_switch+0x143
  100073 (0xc1c6e4b0)  sched_switch(c1c6e4b0,0,1) at sched_switch+0x143
  100072 (0xc1c6e320)  sched_switch(c1c6e320,0,1) at sched_switch+0x143
  100060 (0xc1a4e000)  sched_switch(c1a4e000,0,1) at sched_switch+0x143
  100038 (0xc19a1c80)  sched_switch(c19a1c80,0,1) at sched_switch+0x143
  100071 (0xc1c6e190)  sched_switch(c1c6e190,0,1) at sched_switch+0x143
  100053 (0xc1a4b4b0)  sched_switch(c1a4b4b0,0,1) at sched_switch+0x143
  100079 (0xc1c6ee10)  sched_switch(c1c6ee10,0,1) at sched_switch+0x143
  100074 (0xc1c6e640)  sched_switch(c1c6e640,0,1) at sched_switch+0x143
  100054 (0xc1a4b640)  sched_switch(c1a4b640,0,1) at sched_switch+0x143
  100076 (0xc1c6e960)  sched_switch(c1c6e960,0,1) at sched_switch+0x143
  100061 (0xc1a4e190)  sched_switch(c1a4e190,0,1) at sched_switch+0x143
  100077 (0xc1c6eaf0)  sched_switch(c1c6eaf0,0,1) at sched_switch+0x143
  100025 (0xc196e7d0)  sched_switch(c196e7d0,0,1) at sched_switch+0x143
  100024 (0xc196e640)  sched_switch(c196e640,0,1) at sched_switch+0x143
  100023 (0xc196e4b0)  sched_switch(c196e4b0,0,1) at sched_switch+0x143
  100049 (0xc1a48e10)  sched_switch(c1a48e10,0,1) at sched_switch+0x143
  100048 (0xc1a48c80)  sched_switch(c1a48c80,0,1) at sched_switch+0x143
  100047 (0xc1a48af0)  sched_switch(c1a48af0,0,1) at sched_switch+0x143
  100046 (0xc1a48960)  sched_switch(c1a48960,0,1) at sched_switch+0x143
  100045 (0xc1a487d0)  sched_switch(c1a487d0,0,1) at sched_switch+0x143
  100044 (0xc1a48640)  sched_switch(c1a48640,0,1) at sched_switch+0x143
  100043 (0xc1a484b0)  sched_switch(c1a484b0,0,1) at sched_switch+0x143
  100042 (0xc1a48320)  sched_switch(c1a48320,0,1) at sched_switch+0x143
  100041 (0xc1a48190)  sched_switch(c1a48190,0,1) at sched_switch+0x143
  100040 (0xc1a48000)  fork_trampoline() at fork_trampoline
  100012 (0xc196a320)  fork_trampoline() at fork_trampoline
  100011 (0xc196a190)  sched_switch(c196a190,0,1) at sched_switch+0x143
  100035 (0xc19a17d0)  sched_switch(c19a17d0,0,1) at sched_switch+0x143
  100034 (0xc19a1640)  fork_trampoline() at fork_trampoline
  100033 (0xc19a14b0)  fork_trampoline() at fork_trampoline
  100032 (0xc19a1320)  sched_switch(c19a1320,0,1) at sched_switch+0x143
  100031 (0xc19a1190)  sched_switch(c19a1190,0,1) at sched_switch+0x143
  100030 (0xc19a1000)  sched_switch(18f1420,c1b0d800,c1c34800,c1bcab00,d5413c7c) at sched_switch+0x143
  100029 (0xc196ee10)  sched_switch(c196ee10,0,1) at sched_switch+0x143
  100028 (0xc196ec80)  sched_switch(c196ec80,0,1) at sched_switch+0x143
  100027 (0xc196eaf0)  ipllog(0,d4233bd8,d4233b50,d4233b48,d4233b40) at ipllog+0x43
  100026 (0xc196e960)  fork_trampoline() at fork_trampoline
  100000 (0xc1965000)  sched_switch(c1965000,0,1) at sched_switch+0x143
  100022 (0xc196e320)  fork_trampoline() at fork_trampoline
  100021 (0xc196e190)  fork_trampoline() at fork_trampoline
  100020 (0xc196e000)  fork_trampoline() at fork_trampoline
  100019 (0xc196ae10)  fork_trampoline() at fork_trampoline
  100018 (0xc196ac80)  fork_trampoline() at fork_trampoline
  100017 (0xc196aaf0)  sched_switch(c196aaf0,0,1) at sched_switch+0x143
  100016 (0xc196a960)  sched_switch(c196a960,0,1) at sched_switch+0x143
  100015 (0xc196a7d0)  sched_switch(c196a7d0,0,1) at sched_switch+0x143
  100014 (0xc196a640)  fork_trampoline() at fork_trampoline
  100013 (0xc196a4b0)  sched_switch(c196a4b0,0,1) at sched_switch+0x143
  100010 (0xc196a000)  fork_trampoline() at fork_trampoline
  100009 (0xc1965e10)  fork_trampoline() at fork_trampoline
  100008 (0xc1965c80)  fork_trampoline() at fork_trampoline
  100007 (0xc1965af0)  fork_trampoline() at fork_trampoline
  100006 (0xc1965960)  fork_trampoline() at fork_trampoline
  100005 (0xc19657d0)  fork_trampoline() at fork_trampoline
  100004 (0xc1965640)  sched_switch(c1965640,0,1) at sched_switch+0x143
  100003 (0xc19654b0)  sched_switch(c19654b0,0,1) at sched_switch+0x143
  100002 (0xc1965320)  sched_switch(c1965320,0,1) at sched_switch+0x143
  100001 (0xc1965190)  sched_switch(c1965190,0,1) at sched_switch+0x143
       0 (0xc08bce00)  sched_switch(c08bce00,0,1) at sched_switch+0x143
db> show all procs
  pid   proc     uarea   uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
28532 c20d5388 d6a8a000    0   753   753 0000002 [RUNQ] find
27240 c20d554c d6a8b000    0   735 27240 0004002 [SLPQ nanslp 0xc08c256c][SLP] sleep
27239 c20d4e20 d6a87000    0   735 27239 0004002 [RUNQ] tcpdump
23076 c20d4000 d6a7f000 1000 23075 23075 0004002 [RUNQ] ssh
23075 c20d4710 d6a83000 1000   718 23075 0004002 [SLPQ pipdwt 0xc2268480][SLP] scp
17334 c20d41c4 d6a80000    0 17331 17320 0000002 [SLPQ pause 0xc20d41fc][SLP] dump
17333 c20ce710 d6a6b000    0 17331 17320 0000002 [SLPQ pause 0xc20ce748][SLP] dump
17332 c20d454c d6a82000    0 17331 17320 0000002 [SLPQ pipdwt 0xc2268c00][SLP] dump
17331 c20ce54c d6a6a000    0 17320 17320 0000002 [SLPQ sbwait 0xc209ee84][SLP] dump
17321 c20d51c4 d6a89000    0   732 17320 0004002 [RUNQ] gzip
17320 c2378388 d6aff000    0   732 17320 0004002 [SLPQ wait 0xc2378388][SLP] dump
  753 c20d11c4 d6a71000    0   724   753 0004003 [SLPQ wait 0xc20d11c4][SLP] find
  752 c20d4c5c d6a86000 1000   700   752 0004002 [RUNQ] bzip2
  743 c20d1710 d6a74000    0   740   738 0000002 [SLPQ pause 0xc20d1748][SLP] dump
  742 c20d18d4 d6a75000    0   740   738 0000002 [SLPQ pause 0xc20d190c][SLP] dump
  741 c1c5c388 d6896000    0   740   738 0000002 [SLPQ pipdwt 0xc1d9f180][SLP] dump
  740 c2039000 d689e000    0   738   738 0000002 [SLPQ sbwait 0xc209eab8][SLP] dump
  739 c1f29710 d693f000    0   729   738 0004002 [RUNQ] gzip
  738 c1f29388 d693d000    0   729   738 0004002 [SLPQ wait 0xc1f29388][SLP] dump
  735 c1c71388 d68e5000    0   712   735 0004003 [SLPQ pause 0xc1c713c0][SLP] tcsh
  732 c1c6d8d4 d68df000    0   706   732 0004003 [SLPQ pause 0xc1c6d90c][SLP] tcsh
  729 c20391c4 d6918000    0   694   729 0004003 [SLPQ pause 0xc20391fc][SLP] tcsh
  724 c1f281c4 d6928000 1000   723   724 0004002 [SLPQ pause 0xc1f281fc][SLP] tcsh
  723 c1f291c4 d693c000 1000   690   690 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  718 c1c5c8d4 d6899000 1000   717   718 0004002 [SLPQ pause 0xc1c5c90c][SLP] tcsh
  717 c1a4ac5c d54a5000 1000   687   687 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  712 c1c711c4 d68e4000 1000   711   712 0004002 [SLPQ pause 0xc1c711fc][SLP] tcsh
  711 c1f29a98 d697d000 1000   684   684 0000100 [RUNQ] sshd
  706 c1c6d388 d68dc000 1000   705   706 0004002 [SLPQ pause 0xc1c6d3c0][SLP] tcsh
  705 c1f29c5c d697e000 1000   681   681 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  700 c203954c d698d000 1000   699   700 0004002 [SLPQ pause 0xc2039584][SLP] tcsh
  699 c2039710 d698f000 1000   678   678 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  694 c1a4a710 d5466000 1000   693   694 0004002 [SLPQ pause 0xc1a4a748][SLP] tcsh
  693 c1f29000 d693b000 1000   675   675 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  690 c1f2954c d693e000    0   461   690 0000100 [SLPQ sbwait 0xc209b5a8][SLP] sshd
  687 c1f298d4 d6940000    0   461   687 0000100 [SLPQ sbwait 0xc1f85e84][SLP] sshd
  684 c1c5c710 d6898000    0   461   684 0000100 [SLPQ sbwait 0xc1f85bfc][SLP] sshd
  681 c1f29e20 d697f000    0   461   681 0000100 [SLPQ sbwait 0xc1d95d40][SLP] sshd
  678 c2039388 d692f000    0   461   678 0000100 [SLPQ sbwait 0xc1f86320][SLP] sshd
  675 c1f28388 d6930000    0   461   675 0000100 [SLPQ sbwait 0xc1f86974][SLP] sshd
  674 c1f28710 d6936000    0   617   617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5
  673 c1c6d000 d689d000    0   617   617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5
  672 c1f28c5c d6939000    0   617   617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5
  671 c1c71a98 d68e9000    0   617   617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5
  670 c1c71710 d68e7000    0   617   617 0000000 [SLPQ accept 0xc1d9b03a][SLP] perl5.8.5
  669 c20398d4 d69c2000    0     1   669 0004002 [SLPQ ttyin 0xc1c1d410][SLP] getty
  668 c2039a98 d69c3000    0     1   668 0004002 [SLPQ ttyin 0xc1afa210][SLP] getty
  667 c2039c5c d69c4000    0     1   667 0004002 [SLPQ ttyin 0xc1afa410][SLP] getty
  666 c1f288d4 d6937000    0     1   666 0004002 [SLPQ ttyin 0xc1afa610][SLP] getty
  665 c1c71c5c d68ea000    0     1   665 0004002 [SLPQ ttyin 0xc1afa810][SLP] getty
  664 c1f28e20 d693a000    0     1   664 0004002 [SLPQ ttyin 0xc1afaa10][SLP] getty
  663 c1a4a1c4 d5463000    0     1   663 0004002 [SLPQ ttyin 0xc1a62210][SLP] getty
  662 c1c71e20 d6909000    0     1   662 0004002 [SLPQ ttyin 0xc195b610][SLP] getty
  661 c1c5c54c d6897000    0     1   661 0004002 [SLPQ ttyin 0xc195b410][SLP] getty
  656 c1c7154c d68e6000    0     1   656 0000000 [SLPQ select 0xc08c7cc4][SLP] inetd
  625 c1f2854c d6935000    0     1   624 0000000 [SLPQ select 0xc08c7cc4][SLP] snmpd
  617 c1a4a8d4 d5467000    0     1   617 0000000 [SLPQ pause 0xc1a4a90c][SLP] perl5.8.5
  615 c1a4aa98 d54a4000 20001   560   560 0000100 [SLPQ lockf 0xc1f252c0][SLP] perl
  614 c1c5ce20 d689c000 20001   560   560 0000100 [SLPQ accept 0xc1ceb17e][SLP] perl
  605 c1f28000 d6919000    0     1   605 0000100 [SLPQ select 0xc08c7cc4][SLP] perl
  604 c1c718d4 d68e8000   88   579    47 000c182 (threaded)  mysqld
   thread 0xc1f53af0 ksegrp 0xc1a49b60 [RUNQ]
   thread 0xc1f53c80 ksegrp 0xc196c460 [SLPQ kserel 0xc196c4a0][SLP]
   thread 0xc1f2d4b0 ksegrp 0xc1a49b60 [SLPQ kserel 0xc1a49ba0][SLP]
   thread 0xc1c72af0 ksegrp 0xc1a49b60 [SLPQ select 0xc08c7cc4][SLP]
   thread 0xc1f534b0 ksegrp 0xc1a49150 [SLPQ sigwait 0xd699ec2c][SLP]
   thread 0xc1f2dc80 ksegrp 0xc1a491c0 [SLPQ ksesigwait 0xc1c719d4][SLP]
  579 c1f28a98 d6938000    0     1    47 0004002 [SLPQ wait 0xc1f28a98][SLP] sh
  567 c1c5c1c4 d6895000  106     1   567 0000100 [SLPQ accept 0xc1cebce2][SLP] clamd
  560 c1c6dc5c d68e1000 20001     1   560 0000100 [SLPQ select 0xc08c7cc4][SLP] perl
  474 c1c6de20 d68e2000    0     1   474 0000000 [SLPQ nanslp 0xc08c256c][SLP] cron
  461 c1c5c000 d6893000    0     1   461 0000100 [SLPQ select 0xc08c7cc4][SLP] sshd
  434 c1a4a388 d5464000    0     1    47 0004002 [SLPQ nanslp 0xc08c256c][SLP] perl
  433 c1c71000 d68e3000 20000   431   431 0004100 [SLPQ select 0xc08c7cc4][SLP] qmgr
  432 c1c5cc5c d689b000 20000   431   431 0004100 [SLPQ select 0xc08c7cc4][SLP] pickup
  431 c1c6d1c4 d68db000    0     1   431 0004100 [SLPQ select 0xc08c7cc4][SLP] master
  384 c1c6da98 d68e0000 65534     1   384 0000100 [SLPQ bpf 0xc1ce6400][SLP] snort
  290 c1c5ca98 d689a000    0     1   290 0000000 [SLPQ ffsfsn 0xc1cfb134][SLP] syslogd
  270 c1c6d710 d68de000    0     1   270 0000000 [SLPQ select 0xc08c7cc4][SLP] devd
  187 c1a4ae20 d54a6000    0     1   187 0000000 [SLPQ pause 0xc1a4ae58][SLP] adjkerntz
  165 c1c6d54c d68dd000    0     1   165 0000000 [SLPQ nanslp 0xc08c256c][SLP] ipmon
   46 c19b0a98 d5438000    0     0     0 0000204 [SLPQ - 0xd422dd18][SLP] schedcpu
   45 c19b0c5c d5439000    0     0     0 0000204 [SLPQ hptrdy 0xc0768be4][SLP] hpt_wt
   44 c19b0e20 d543a000    0     0     0 0000204 [SLPQ vlruwt 0xc19b0e20][SLP] vnlru
   43 c1a47000 d543b000    0     0     0 0000204 [SLPQ syncer 0xc08c22ec][SLP] syncer
   42 c1a471c4 d545a000    0     0     0 0000204 [SLPQ psleep 0xc08c828c][SLP] bufdaemon
   41 c1a47388 d545b000    0     0     0 000020c [RUNQ] pagezero
   40 c1a4754c d545c000    0     0     0 0000204 [SLPQ psleep 0xc08d0024][SLP] vmdaemon
   39 c1a47710 d545d000    0     0     0 0000204 [SLPQ psleep 0xc08cffe0][SLP] pagedaemon
   38 c1a478d4 d545e000    0     0     0 0000204 [IWAIT] swi0: sio
    9 c1a47a98 d545f000    0     0     0 0000204 [SLPQ - 0xc19a8d3c][SLP] fdc0
    8 c1a47c5c d5460000    0     0     0 0000204 [SLPQ idle 0xc195a400][SLP] aic_recovery0
    7 c1a47e20 d5461000    0     0     0 0000204 [SLPQ idle 0xc195a400][SLP] aic_recovery0
   37 c1a4a000 d5462000    0     0     0 0000204 [IWAIT] swi6:+
   36 c19a054c d540e000    0     0     0 0000204 [IWAIT] swi6: task queue
    6 c19a0710 d540f000    0     0     0 0000204 [SLPQ - 0xc19ec580][SLP] kqueue taskq
   35 c19a08d4 d5410000    0     0     0 0000204 [IWAIT] swi3: cambio
   34 c19a0a98 d542f000    0     0     0 0000204 [IWAIT] swi2: camnet
   33 c19a0c5c d5430000    0     0     0 0000204 [IWAIT] swi6:+
    5 c19a0e20 d5431000    0     0     0 0000204 [SLPQ - 0xc19ec7c0][SLP] thread taskq
   32 c19b0000 d5432000    0     0     0 0000204 [SLPQ - 0xc08b8100][SLP] yarrow
    4 c19b01c4 d5433000    0     0     0 0000204 [CPU 0] g_down
    3 c19b0388 d5434000    0     0     0 0000204 [SLPQ - 0xc08bcb24][SLP] g_up
    2 c19b054c d5435000    0     0     0 0000204 [SLPQ - 0xc08bcb1c][SLP] g_event
   31 c19b0710 d5436000    0     0     0 0000204 [CPU 1] swi1: net
   30 c19b08d4 d5437000    0     0     0 0000204 [IWAIT] swi4: vm
   29 c196d1c4 d421a000    0     0     0 000020c [RUNQ] swi5: clock sio
   28 c196d388 d421b000    0     0     0 0000204 [IWAIT] irq0: clk
   27 c196d54c d423a000    0     0     0 0000204 [IWAIT] irq15:
   26 c196d710 d423b000    0     0     0 0000204 [IWAIT] irq14: atapci0 ata0
   25 c196d8d4 d423c000    0     0     0 0000204 [IWAIT] irq13:
   24 c196da98 d423d000    0     0     0 0000204 [IWAIT] irq12:
   23 c196dc5c d423e000    0     0     0 0000204 [IWAIT] irq11: fxp1
   22 c196de20 d423f000    0     0     0 0000204 [IWAIT] irq10: xl0 fxp0
   21 c19a0000 d540b000    0     0     0 0000204 [IWAIT] irq9: ahc0
   20 c19a01c4 d540c000    0     0     0 0000204 [IWAIT] irq8: rtc
   19 c19a0388 d540d000    0     0     0 0000204 [IWAIT] irq7: ep0
   18 c1964000 d41d4000    0     0     0 0000204 [IWAIT] irq6: fdc0
   17 c19641c4 d4211000    0     0     0 0000204 [IWAIT] irq5:
   16 c1964388 d4212000    0     0     0 0000204 [IWAIT] irq4: sio0
   15 c196454c d4213000    0     0     0 0000204 [IWAIT] irq3: sio1
   14 c1964710 d4214000    0     0     0 0000204 [IWAIT] irq2:
   13 c19648d4 d4215000    0     0     0 0000204 [IWAIT] irq1: atkbd0
   12 c1964a98 d4216000    0     0     0 000020c [Can run] idle: cpu0
   11 c1964c5c d4217000    0     0     0 000020c [Can run] idle: cpu1
    1 c1964e20 d4218000    0     0     1 0004200 [SLPQ wait 0xc1964e20][SLP] init
   10 c196d000 d4219000    0     0     0 0000204 [SLPQ ktrace 0xc08c0518][SLP] ktrace
    0 c08bcc20 c0c1f000    0     0     0 0000200 [SLPQ sched 0xc08bcc20][SLP] swapper
db> show pcpu 0
cpuid        = 0
curthread    = 0xc19a1000: pid 4 "g_down"
curpcb       = 0xd5413da0
fpcurthread  = none
idlethread   = 0xc1965640: pid 12 "idle: cpu0"
APIC ID      = 1
currentldt   = 0x28
db> show pcpu 1
cpuid        = 1
curthread    = 0xc196eaf0: pid 31 "swi1: net"
curpcb       = 0xd4233da0
fpcurthread  = none
idlethread   = 0xc19654b0: pid 11 "idle: cpu1"
APIC ID      = 0
currentldt   = 0x28
db> trace
ipllog(0,d4233bd8,d4233b50,d4233b48,d4233b40) at ipllog+0x43
ipflog(1a,c2692820,d4233bd8,c1a72500) at ipflog+0x18f
fr_check(c2692820,14,c1a44000,0,d4233c80) at fr_check+0x820
fr_check_wrapper(0,d4233c80,c1a44000,1,0) at fr_check_wrapper+0x2a
pfil_run_hooks(c08c99c0,d4233ccc,c1a44000,1,0) at pfil_run_hooks+0xe3
ip_input(c1a72500) at ip_input+0x211
netisr_processqueue(c08c8c58) at netisr_processqueue+0xa3
swi_net(0) at swi_net+0xf2
ithread_loop(c1962680,d4233d48) at ithread_loop+0x159
fork_exit(c060bb5c,c1962680,d4233d48) at fork_exit+0x75
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd4233d7c, ebp = 0 ---
db> call doadump
Dumping 512 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496
Dump complete
0xf
db> reset
cpu_reset called on cpu#1
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 1



oak# nm -n /boot/kernel/kernel | grep c04647
c0464714 T ipllog
oak# 



oak# mptable -dmesg -verbose -grope

===============================================================================

MPTable, version 2.0.15

 looking for EBDA pointer @ 0x040e, found, searching EBDA @ 0x0009f800
 searching CMOS 'top of mem' @ 0x0009f400 (637K)
 searching default 'top of mem' @ 0x0009fc00 (639K)
 searching BIOS @ 0x000f0000

 MP FPS found in BIOS @ physical addr: 0x000f99a0

-------------------------------------------------------------------------------

MP Floating Pointer Structure:

  location:                     BIOS
  physical address:             0x000f99a0
  signature:                    '_MP_'
  length:                       16 bytes
  version:                      1.1
  checksum:                     0x4b
  mode:                         Virtual Wire

-------------------------------------------------------------------------------

MP Config Table Header:

  physical address:             0x000f99b0
  signature:                    'PCMP'
  base table length:            236
  version:                      1.1
  checksum:                     0xe0
  OEM ID:                       'HP      '
  Product ID:                   'XX          '
  OEM table pointer:            0x00000000
  OEM table size:               0
  entry count:                  21
  local APIC address:           0xfee00000
  extended table length:        0
  extended table checksum:      0

-------------------------------------------------------------------------------

MP Config Base Table Entries:

--
Processors:     APIC ID Version State           Family  Model   Step    Flags
                 1       0x11    BSP, usable     6       1       0       0xfbff
                 0       0x11    AP, usable      6       1       0       0xfbff
--
Bus:            Bus ID  Type
                 1       ISA   
                 0       PCI   
--
I/O APICs:      APIC ID Version State           Address
                16       0x11    usable          0xfec00000
--
I/O Ints:       Type    Polarity    Trigger     Bus ID   IRQ    APIC ID PIN#
                INT      conforms    conforms        1     1        255    1
                INT      conforms    conforms        1     0        255    2
                INT      conforms    conforms        1     3        255    3
                INT      conforms    conforms        1     4        255    4
                INT      conforms    conforms        1     5        255    5
                INT      conforms    conforms        1     6        255    6
                INT      conforms    conforms        1     7        255    7
                INT      conforms    conforms        1     8        255    8
                INT      conforms    conforms        1     9        255    9
                INT      conforms    conforms        1    10        255   10
                INT      conforms    conforms        1    11        255   11
                INT      conforms    conforms        1    12        255   12
                INT      conforms    conforms        1    13        255   13
                INT      conforms    conforms        1    14        255   14
                INT      conforms    conforms        1    15        255   15
--
Local Ints:     Type    Polarity    Trigger     Bus ID   IRQ    APIC ID PIN#
                NMI      conforms    conforms        1     2        255    1

-------------------------------------------------------------------------------

dmesg output:

Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 5.3-RELEASE-p2 #2: Wed Dec  8 22:06:28 MST 2004
    root@oak.zyni.com:/usr/src/sys/i386/compile/OAK-TEST
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Pentium Pro (199.74-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x619  Stepping = 9
  Features=0xfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV>
real memory  = 536870912 (512 MB)
avail memory = 515813376 (491 MB)
MPTable: <HP       XX          >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  1
 cpu1 (AP): APIC ID:  0
ioapic0: Changing APIC ID to 16
ioapic0: Assuming intbase of 0
MPTable: Ignoring global interrupt entry for pin 1
MPTable: Ignoring global interrupt entry for pin 2
MPTable: Ignoring global interrupt entry for pin 3
MPTable: Ignoring global interrupt entry for pin 4
MPTable: Ignoring global interrupt entry for pin 5
MPTable: Ignoring global interrupt entry for pin 6
MPTable: Ignoring global interrupt entry for pin 7
MPTable: Ignoring global interrupt entry for pin 8
MPTable: Ignoring global interrupt entry for pin 9
MPTable: Ignoring global interrupt entry for pin 10
MPTable: Ignoring global interrupt entry for pin 11
MPTable: Ignoring global interrupt entry for pin 12
MPTable: Ignoring global interrupt entry for pin 13
MPTable: Ignoring global interrupt entry for pin 14
MPTable: Ignoring global interrupt entry for pin 15
ioapic0 <Version 1.1> irqs 0-15 on motherboard
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Intel 82454KX/GX (Orion) host to PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 0.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <CMD 646 WDMA2 controller> port 0xfc00-0xfc0f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 irq 14 at device 1.0 on pci0
atapci0: HW has secondary channel disabled
ata0: channel #0 on atapci0
ahc0: <Adaptec aic7880 Ultra SCSI adapter> port 0xf800-0xf8ff mem 0xffbfc000-0xffbfcfff irq 9 at device 2.0 on pci0
ahc0: Using left over BIOS settings
ahc0: [GIANT-LOCKED]
aic7880: Ultra Single Channel A, SCSI Id=7, 16/253 SCBs
xl0: <3Com 3c905B-TX Fast Etherlink XL> port 0xf400-0xf47f mem 0xffbec000-0xffbec07f irq 10 at device 11.0 on pci0
miibus0: <MII bus> on xl0
xlphy0: <3Com internal media interface> on miibus0
xlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
xl0: Ethernet address: 00:10:5a:0d:79:54
fxp0: <Intel 82557 Pro/100 Ethernet> port 0xf000-0xf01f mem 0xffa00000-0xffafffff,0xffbdc000-0xffbdcfff irq 10 at device 12.0 on pci0
miibus1: <MII bus> on fxp0
nsphy0: <DP83840 10/100 media interface> on miibus1
nsphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:a0:c9:57:e8:69
fxp1: <Intel 82557 Pro/100 Ethernet> port 0xec00-0xec1f mem 0xff900000-0xff9fffff,0xffbcc000-0xffbccfff irq 11 at device 13.0 on pci0
miibus2: <MII bus> on fxp1
nsphy1: <DP83840 10/100 media interface> on miibus2
nsphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: Ethernet address: 00:a0:c9:3e:19:1d
pci0: <display, VGA> at device 14.0 (no driver attached)
pci0: <memory, RAM> at device 20.0 (no driver attached)
cpu0 on motherboard
cpu1 on motherboard
orm0: <ISA Option ROMs> at iomem 0xc8000-0xcc7ff,0xc0000-0xc7fff on isa0
pmtimer0 on isa0
ep0: <3Com 3C509-TP EtherLink III> at port 0x290-0x29f irq 7 on isa0
ep0: Ethernet address: 00:60:8c:b9:5e:18
atkbdc0: <Keyboard controller (i8042)> at port 0x64,0x60 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5 irq 6 drq 2 on isa0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0303> can't assign resources (port)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0501> can't assign resources (port)
ppc1: parallel port not found.
unknown: <PNP0700> can't assign resources (port)
Timecounters tick every 10.000 msec
IP Filter: v3.4.35 initialized.  Default = pass all, Logging = enabled
Waiting 15 seconds for SCSI devices to settle
da0 at ahc0 bus 0 target 1 lun 0
da0: <SEAGATE ST318416N 0010> Fixed Direct Access SCSI-3 device 
da0: 20.000MB/s transfers (20.000MHz, offset 15), Tagged Queueing Enabled
da0: 17522MB (35885168 512 byte sectors: 64H 32S/T 17522C)
SMP: AP CPU #1 Launched!
Mounting root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
ep0: promiscuous mode enabled

===============================================================================



oak# diff GENERIC OAK-TEST 
25c25
< ident         GENERIC
---
> ident         OAK-TEST
29a30,35
> makeoptions   DEBUG=-g                #Build kernel with gdb(1) debug symbols
> options               IPFILTER                #ipfilter support
> options               IPFILTER_LOG            #ipfilter logging
> options               SMP
> options               DDB
> options               KDB
38,40c44,46
< options       NFSCLIENT               # Network Filesystem Client
< options       NFSSERVER               # Network Filesystem Server
< options       NFS_ROOT                # NFS usable as /, requires NFSCLIENT
---
> #options      NFSCLIENT               # Network Filesystem Client
> #options      NFSSERVER               # Network Filesystem Server
> #options      NFS_ROOT                # NFS usable as /, requires NFSCLIENT
oak#


I also have the core file from the crash if any additional information can be obtained from that please let me know.
>How-To-Repeat:

Running these commands simultaneously will get a crash out within
45 minutes w/o debugging kernel on my systems.  With the kdb/ddb stuff turned on it seems to be more on the order of hours rather than minutes.  Also when the tcpdump starts and stops there is around 10 seconds or so where network traffic stops moving.

find / -type f -exec strings {} \; > /dev/null
dump 0sf 300000 - /home | gzip -9 > dumphome.gz
bzip2 <large image>.iso
scp <large image>.iso <another system>:/var/tmp
dump 0sf 300000 - / | gzip -9 > dumproot.gz


while 1
tcpdump -i fxp0 &
sleep 360
kill -9 `ps -auxww | grep tcpdump | grep -v grep | awk '{print $2}'`
sleep 60
end


>Fix:

>Release-Note:
>Audit-Trail:

From: Giorgos Keramidas <keramida@freebsd.org>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/74909: SMP crash FreeBSD-5.3-p2 (swi1: net) / ipllog
Date: Wed, 19 Jan 2005 03:04:34 +0200

 Adding to the audit trail:
 
 PR:		kern/76394
 Submitted by:	Geoffrey Lassner <geoff@zyni.com>
 
 % Tested with FreeBSD 5.3 p5:
 % Fatal trap 12: page fault while in kernel mode
 % cpuid = 0; apic id = 01
 % fault virtual address   = 0x4
 % fault code              = supervisor write, page not present
 % instruction pointer     = 0x8:0xc04646f7
 % stack pointer           = 0x10:0xd4233b10
 % frame pointer           = 0x10:0xd4233b1c
 % code segment            = base 0x0, limit 0xfffff, type 0x1b
 %                         = DPL 0, pres 1, def32 1, gran 1
 % processor eflags        = interrupt enabled, resume, IOPL = 0
 % current process         = 31 (swi1: net)
 % [thread 100027]
 % Stopped at      ipllog+0x43:    incl    0x4(%eax)
 % db> show thread
 %   100157 (0xc20bcaf0)  sched_switch(c20bcaf0,0,2) at sched_switch+0x143
 %   100139 (0xc206fe10)  sched_switch(c206fe10,0,1) at sched_switch+0x143
 %   100166 (0xc20bd960)  sched_switch(c20bd960,0,1) at sched_switch+0x143
 %   100056 (0xc1a4b960)  sched_switch(c1a4b960,0,1) at sched_switch+0x143
 %   100153 (0xc20bc4b0)  sched_switch(c20bc4b0,0,1) at sched_switch+0x143
 %   100133 (0xc206f4b0)  sched_switch(d6a02cc4,8,d6a02d0c,c078cc13,38) at sched_switch+0x143
 %   100052 (0xc1a4b320)  sched_switch(c1a4b320,0,1) at sched_switch+0x143
 %   100160 (0xc20bd000)  sched_switch(c20bd000,0,1) at sched_switch+0x143
 %   100161 (0xc20bd190)  sched_switch(c20bd190,0,1) at sched_switch+0x143
 %   100134 (0xc206f640)  sched_switch(c206f640,0,1) at sched_switch+0x143
 %   100075 (0xc1c517d0)  sched_switch(c1c517d0,0,1) at sched_switch+0x143
 %   100158 (0xc20bcc80)  sched_switch(c20bcc80,0,2) at sched_switch+0x143
 %   100137 (0xc206faf0)  sched_switch(c206faf0,0,1) at sched_switch+0x143
 %   100131 (0xc206f190)  sched_switch(c206f190,0,1) at sched_switch+0x143
 %   100061 (0xc1a4e190)  sched_switch(c1a4e190,0,1) at sched_switch+0x143
 %   100096 (0xc1c6e960)  sched_switch(c1c6e960,0,1) at sched_switch+0x143
 %   100169 (0xc20bde10)  sched_switch(c20bde10,0,1) at sched_switch+0x143
 %   100159 (0xc20bce10)  sched_switch(c20bce10,0,1) at sched_switch+0x143
 %   100071 (0xc1c51190)  sched_switch(c1c51190,0,1) at sched_switch+0x143
 %   100050 (0xc1a4b000)  sched_switch(c1a4b000,0,2) at sched_switch+0x143
 %   100109 (0xc1d2de10)  sched_switch(c1d2de10,0,1) at sched_switch+0x143
 %   100062 (0xc1a4e320)  sched_switch(c1a4e320,0,1) at sched_switch+0x143
 %   100122 (0xc1f1e320)  sched_switch(c1f1e320,0,1) at sched_switch+0x143
 %   100130 (0xc206f000)  sched_switch(c206f000,0,1) at sched_switch+0x143
 %   100065 (0xc1a4e7d0)  sched_switch(c1a4e7d0,0,1) at sched_switch+0x143
 %   100145 (0xc20727d0)  sched_switch(c20727d0,0,1) at sched_switch+0x143
 %   100113 (0xc1d2e4b0)  sched_switch(c1d2e4b0,0,1) at sched_switch+0x143
 %   100128 (0xc1f1ec80)  sched_switch(c1f1ec80,0,1) at sched_switch+0x143
 %   100082 (0xc1c52320)  sched_switch(c1c52320,0,1) at sched_switch+0x143
 %   100127 (0xc1f1eaf0)  sched_switch(c1f1eaf0,0,1) at sched_switch+0x143
 %   100141 (0xc2072190)  sched_switch(c2072190,0,1) at sched_switch+0x143
 %   100085 (0xc1c527d0)  sched_switch(c1c527d0,0,1) at sched_switch+0x143
 %   100125 (0xc1f1e7d0)  sched_switch(c1f1e7d0,0,1) at sched_switch+0x143
 %   100114 (0xc1d2e640)  sched_switch(c1d2e640,0,1) at sched_switch+0x143
 %   100069 (0xc1a4ee10)  sched_switch(c1a4ee10,0,1) at sched_switch+0x143
 %   100147 (0xc2072af0)  sched_switch(c2072af0,0,2) at sched_switch+0x143
 %   100039 (0xc19a1e10)  sched_switch(c19a1e10,0,1) at sched_switch+0x143
 %   100064 (0xc1a4e640)  sched_switch(c1a4e640,0,1) at sched_switch+0x143
 %   100120 (0xc1f1e000)  sched_switch(c1f1e000,0,1) at sched_switch+0x143
 %   100106 (0xc1d2d960)  sched_switch(c1d2d960,0,1) at sched_switch+0x143
 %   100148 (0xc2072c80)  sched_switch(c2072c80,0,1) at sched_switch+0x143
 %   100107 (0xc1d2daf0)  sched_switch(c1d2daf0,0,1) at sched_switch+0x143
 %   100098 (0xc1c6ec80)  sched_switch(c1c6ec80,0,1) at sched_switch+0x143
 %   100094 (0xc1c6e640)  sched_switch(c1c6e640,0,1) at sched_switch+0x143
 %   100101 (0xc1d2d190)  sched_switch(c1d2d190,0,1) at sched_switch+0x143
 %   100100 (0xc1d2d000)  sched_switch(c1d2d000,0,1) at sched_switch+0x143
 %   100105 (0xc1d2d7d0)  sched_switch(c1d2d7d0,0,1) at sched_switch+0x143
 %   100076 (0xc1c51960)  sched_switch(c1c51960,0,1) at sched_switch+0x143
 %   100091 (0xc1c6e190)  sched_switch(c1c6e190,0,1) at sched_switch+0x143
 %   100079 (0xc1c51e10)  sched_switch(c1c51e10,0,1) at sched_switch+0x143
 %   100090 (0xc1c6e000)  sched_switch(c1c6e000,0,1) at sched_switch+0x143
 %   100111 (0xc1d2e190)  sched_switch(c1d2e190,0,1) at sched_switch+0x143
 %   100099 (0xc1c6ee10)  sched_switch(c1c6ee10,0,1) at sched_switch+0x143
 %   100104 (0xc1d2d640)  sched_switch(c1d2d640,0,1) at sched_switch+0x143
 %   100070 (0xc1c51000)  sched_switch(c1c51000,0,1) at sched_switch+0x143
 %   100102 (0xc1d2d320)  sched_switch(c1d2d320,0,1) at sched_switch+0x143
 %   100103 (0xc1d2d4b0)  sched_switch(c1d2d4b0,0,1) at sched_switch+0x143
 %   100077 (0xc1c51af0)  sched_switch(c1c51af0,0,1) at sched_switch+0x143
 %   100110 (0xc1d2e000)  sched_switch(c1d2e000,0,1) at sched_switch+0x143
 %   100080 (0xc1c52000)  sched_switch(c1c52000,0,1) at sched_switch+0x143
 %   100092 (0xc1c6e320)  sched_switch(c1c6e320,0,1) at sched_switch+0x143
 %   100073 (0xc1c514b0)  sched_switch(c1c514b0,0,1) at sched_switch+0x143
 %   100086 (0xc1c52960)  sched_switch(c1c52960,0,1) at sched_switch+0x143
 %   100123 (0xc1f1e4b0)  sched_switch(c1f1e4b0,0,1) at sched_switch+0x143
 %   100097 (0xc1c6eaf0)  sched_switch(c1c6eaf0,0,1) at sched_switch+0x143
 %   100118 (0xc1d2ec80)  sched_switch(c1d2ec80,0,1) at sched_switch+0x143
 %   100117 (0xc1d2eaf0)  sched_switch(c1d2eaf0,0,1) at sched_switch+0x143
 %   100124 (0xc1f1e640)  sched_switch(c1f1e640,0,1) at sched_switch+0x143
 %   100112 (0xc1d2e320)  sched_switch(c1d2e320,0,1) at sched_switch+0x143
 %   100067 (0xc1a4eaf0)  sched_switch(c1a4eaf0,0,1) at sched_switch+0x143
 %   100059 (0xc1a4be10)  sched_switch(c1a4be10,0,1) at sched_switch+0x143
 %   100093 (0xc1c6e4b0)  sched_switch(c1c6e4b0,0,1) at sched_switch+0x143
 %   100066 (0xc1a4e960)  sched_switch(c1a4e960,0,1) at sched_switch+0x143
 %   100095 (0xc1c6e7d0)  sched_switch(c1c6e7d0,0,1) at sched_switch+0x143
 %   100078 (0xc1c51c80)  sched_switch(c1c51c80,0,1) at sched_switch+0x143
 %   100058 (0xc1a4bc80)  sched_switch(c1a4bc80,0,1) at sched_switch+0x143
 %   100068 (0xc1a4ec80)  sched_switch(c1a4ec80,0,1) at sched_switch+0x143
 %   100055 (0xc1a4b7d0)  sched_switch(c1a4b7d0,0,1) at sched_switch+0x143
 %   100036 (0xc19a1960)  sched_switch(c19a1960,0,1) at sched_switch+0x143
 %   100060 (0xc1a4e000)  sched_switch(c1a4e000,0,1) at sched_switch+0x143
 %   100037 (0xc19a1af0)  sched_switch(c19a1af0,0,1) at sched_switch+0x143
 %   100054 (0xc1a4b640)  sched_switch(c1a4b640,0,1) at sched_switch+0x143
 %   100025 (0xc196e7d0)  sched_switch(c196e7d0,0,1) at sched_switch+0x143
 %   100024 (0xc196e640)  sched_switch(c196e640,0,1) at sched_switch+0x143
 %   100023 (0xc196e4b0)  sched_switch(c196e4b0,0,1) at sched_switch+0x143
 %   100049 (0xc1a48e10)  sched_switch(c1a48e10,0,1) at sched_switch+0x143
 %   100048 (0xc1a48c80)  sched_switch(c1a48c80,0,1) at sched_switch+0x143
 %   100047 (0xc1a48af0)  sched_switch(c1a48af0,0,1) at sched_switch+0x143
 %   100046 (0xc1a48960)  sched_switch(c1a48960,0,1) at sched_switch+0x143
 %   100045 (0xc1a487d0)  sched_switch(c1a487d0,0,1) at sched_switch+0x143
 %   100044 (0xc1a48640)  sched_switch(c1a48640,0,1) at sched_switch+0x143
 %   100043 (0xc1a484b0)  sched_switch(c1a484b0,0,1) at sched_switch+0x143
 %   100042 (0xc1a48320)  sched_switch(c1a48320,0,1) at sched_switch+0x143
 %   100041 (0xc1a48190)  sched_switch(c1a48190,0,1) at sched_switch+0x143
 %   100040 (0xc1a48000)  fork_trampoline() at fork_trampoline
 %   100012 (0xc196a320)  fork_trampoline() at fork_trampoline
 %   100011 (0xc196a190)  sched_switch(c196a190,0,1) at sched_switch+0x143
 %   100035 (0xc19a17d0)  sched_switch(c19a17d0,0,1) at sched_switch+0x143
 %   100034 (0xc19a1640)  fork_trampoline() at fork_trampoline
 %   100033 (0xc19a14b0)  fork_trampoline() at fork_trampoline
 %   100032 (0xc19a1320)  sched_switch(c19a1320,0,1) at sched_switch+0x143
 %   100031 (0xc19a1190)  sched_switch(c19a1190,0,1) at sched_switch+0x143
 %   100030 (0xc19a1000)  sched_switch(c19a1000,0,1) at sched_switch+0x143
 %   100029 (0xc196ee10)  sched_switch(c196ee10,0,1) at sched_switch+0x143
 %   100028 (0xc196ec80)  sched_switch(c196ec80,0,1) at sched_switch+0x143
 %   100027 (0xc196eaf0)  ipllog(0,d4233bd8,d4233b50,d4233b48,d4233b40) at ipllog+0x43
 %   100026 (0xc196e960)  fork_trampoline() at fork_trampoline
 %   100000 (0xc1965000)  sched_switch(c1965000,0,1) at sched_switch+0x143
 %   100022 (0xc196e320)  fork_trampoline() at fork_trampoline
 %   100021 (0xc196e190)  fork_trampoline() at fork_trampoline
 %   100020 (0xc196e000)  fork_trampoline() at fork_trampoline
 %   100019 (0xc196ae10)  fork_trampoline() at fork_trampoline
 %   100018 (0xc196ac80)  fork_trampoline() at fork_trampoline
 %   100017 (0xc196aaf0)  sched_switch(c196aaf0,0,1) at sched_switch+0x143
 %   100016 (0xc196a960)  sched_switch(c196a960,0,1) at sched_switch+0x143
 %   100015 (0xc196a7d0)  sched_switch(c196a7d0,0,1) at sched_switch+0x143
 %   100014 (0xc196a640)  fork_trampoline() at fork_trampoline
 %   100013 (0xc196a4b0)  sched_switch(c196a4b0,0,1) at sched_switch+0x143
 %   100010 (0xc196a000)  fork_trampoline() at fork_trampoline
 %   100009 (0xc1965e10)  fork_trampoline() at fork_trampoline
 %   100008 (0xc1965c80)  fork_trampoline() at fork_trampoline
 %   100007 (0xc1965af0)  fork_trampoline() at fork_trampoline
 %   100006 (0xc1965960)  fork_trampoline() at fork_trampoline
 %   100005 (0xc19657d0)  fork_trampoline() at fork_trampoline
 %   100004 (0xc1965640)  sched_switch(c1965640,0,1) at sched_switch+0x143
 %   100003 (0xc19654b0)  sched_switch(c19654b0,0,1) at sched_switch+0x143
 %   100002 (0xc1965320)  sched_switch(c1965320,0,1) at sched_switch+0x143
 %   100001 (0xc1965190)  sched_switch(c1965190,0,1) at sched_switch+0x143
 %        0 (0xc08bce40)  sched_switch(c08bce40,0,1) at sched_switch+0x143
 % db> show all procs
 %   pid   proc     uarea   uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
 % 15497 c218954c d6af5000    0  6467  6467 0004002 [RUNQ] strings
 % 15314 c2073c5c d6a3b000    0  6846 15314 0004002 [SLPQ nanslp 0xc08c25ac][SLP] sleep
 % 15313 c20bb54c d6adb000    0  6846 15313 0004002 [SLPQ bpf 0xc22c8400][SLP] tcpdump
 % 12472 c1c6c710 d68eb000 1000 12459 12459 0004000 [SLPQ sbwait 0xc2634098][SLP] scp
 % 12459 c2189c5c d6af9000 1000 12458 12459 0004000 [SLPQ pause 0xc2189c94][SLP] tcsh
 % 12458 c20ba710 d6a97000 1000 12393 12393 0000100 [CPU 1] sshd
 % 12393 c1c6ce20 d68ef000    0   461 12393 0000100 [SLPQ sbwait 0xc20ad6ec][SLP] sshd
 % 12171 c2189000 d6af2000    0 12168 12166 0000002 [SLPQ pause 0xc2189038][SLP] dump
 % 12170 c20bbe20 d6ae0000    0 12168 12166 0000002 [SLPQ pipdwt 0xc1d8aa80][SLP] dump
 % 12169 c20ba54c d6a96000    0 12168 12166 0000002 [SLPQ pause 0xc20ba584][SLP] dump
 % 12168 c1c508d4 d68d5000    0 12166 12166 0000002 [SLPQ sbwait 0xc222c1dc][SLP] dump
 % 12167 c2189388 d6af4000    0   749 12166 0004002 [RUNQ] gzip
 % 12166 c20ba000 d6a92000    0   749 12166 0004002 [SLPQ wait 0xc20ba000][SLP] dump
 %  6846 c20baa98 d6a99000    0  6828  6846 0004003 [SLPQ pause 0xc20baad0][SLP] tcsh
 %  6828 c1a4ae20 d54a6000 1000  6827  6828 0004002 [SLPQ pause 0xc1a4ae58][SLP] tcsh
 %  6827 c1c6d388 d68f2000 1000  6809  6809 0000100 [SLPQ select 0xc08c7d04][SLP] sshd
 %  6809 c20bb000 d6a9c000    0   461  6809 0000100 [SLPQ sbwait 0xc1f525a8][SLP] sshd
 %  6467 c21891c4 d6af3000    0  6464  6467 0004002 [SLPQ wait 0xc21891c4][SLP] find
 %  6464 c1d2c000 d6919000    0   778  6464 0004003 [SLPQ pause 0xc1d2c038][SLP] tcsh
 %  3843 c1c6d1c4 d68f1000    0  3812  3843 0004002 [RUNQ] bzip2
 %  3812 c1d2f1c4 d696e000    0   762  3812 0004003 [SLPQ pause 0xc1d2f1fc][SLP] tcsh
 %   778 c1a4ac5c d54a5000 1000   777   778 0004002 [SLPQ pause 0xc1a4ac94][SLP] tcsh
 %   777 c206d54c d69f2000 1000   774   774 0000100 [SLPQ select 0xc08c7d04][SLP] sshd
 %   774 c20bac5c d6a9a000    0   461   774 0000100 [SLPQ sbwait 0xc1f52ab8][SLP] sshd
 %   762 c1d2ca98 d692e000 1000   761   762 0004002 [SLPQ pause 0xc1d2cad0][SLP] tcsh
 %   761 c20731c4 d6a35000 1000   758   758 0000100 [SLPQ select 0xc08c7d04][SLP] sshd
 %   758 c206d8d4 d69f4000    0   461   758 0000100 [SLPQ sbwait 0xc20ad320][SLP] sshd
 %   749 c1f1d8d4 d6994000    0   744   749 0004003 [SLPQ pause 0xc1f1d90c][SLP] tcsh
 %   744 c206d1c4 d69f0000 1000   743   744 0004002 [SLPQ pause 0xc206d1fc][SLP] tcsh
 %   743 c1f1da98 d69b3000 1000   740   740 0000100 [SLPQ select 0xc08c7d04][SLP] sshd
 %   740 c20738d4 d6a39000    0   461   740 0000100 [SLPQ sbwait 0xc1f51464][SLP] sshd
 %   739 c206da98 d69f5000    0   736   734 0000002 [SLPQ pause 0xc206dad0][SLP] dump
 %   738 c1f1de20 d69b5000    0   736   734 0000002 [SLPQ pause 0xc1f1de58][SLP] dump
 %   737 c206d000 d6917000    0   736   734 0000002 [SLPQ pipdwt 0xc1d93000][SLP] dump
 %   736 c1d2c388 d692a000    0   734   734 0000002 [SLPQ sbwait 0xc1cfa974][SLP] dump
 %   735 c206de20 d6a33000    0   728   734 0004002 [RUNQ] gzip
 %   734 c1a4a1c4 d5463000    0   728   734 0004002 [SLPQ wait 0xc1a4a1c4][SLP] dump
 %   728 c1a4a8d4 d5467000    0   723   728 0004003 [SLPQ pause 0xc1a4a90c][SLP] tcsh
 %   723 c206d710 d69f3000 1000   722   723 0004002 [SLPQ pause 0xc206d748][SLP] tcsh
 %   722 c1d2f710 d6971000 1000   719   719 0000100 [SLPQ select 0xc08c7d04][SLP] sshd
 %   719 c206dc5c d69f6000    0   461   719 0000100 [SLPQ sbwait 0xc1d87ab8][SLP] sshd
 %   694 c1d2f54c d6970000    0   667   694 0004002 [SLPQ ttyin 0xc1afc810][SLP] csh
 %   674 c1f1d54c d6992000    0   615   615 0000000 [SLPQ accept 0xc1d87e26][SLP] perl5.8.5
 %   673 c1c6d710 d6912000    0   615   615 0000000 [SLPQ accept 0xc1d87e26][SLP] perl5.8.5
 %   672 c1f1d000 d6924000    0   615   615 0000000 [SLPQ accept 0xc1d87e26][SLP] perl5.8.5
 %   671 c1f1d1c4 d6990000    0   615   615 0000000 [SLPQ accept 0xc1d87e26][SLP] perl5.8.5
 %   670 c1d2f8d4 d6972000    0   615   615 0000000 [SLPQ accept 0xc1d87e26][SLP] perl5.8.5
 %   669 c1c50710 d68d4000 20001   560   560 0000100 [SLPQ accept 0xc1d8717e][SLP] perl
 %   668 c1c6dc5c d6915000 20001   560   560 0000100 [SLPQ lockf 0xc1c31c00][SLP] perl
 %   667 c1c501c4 d68d1000    0     1   667 0004102 [SLPQ wait 0xc1c501c4][SLP] login
 %   666 c1c6de20 d6916000    0     1   666 0004002 [SLPQ ttyin 0xc1afa210][SLP] getty
 %   665 c1d2ce20 d696c000    0     1   665 0004002 [SLPQ ttyin 0xc1afa410][SLP] getty
 %   664 c1f1d388 d6991000    0     1   664 0004002 [SLPQ ttyin 0xc1afa610][SLP] getty
 %   663 c1d2fa98 d6973000    0     1   663 0004002 [SLPQ ttyin 0xc1afa810][SLP] getty
 %   662 c1d2c1c4 d691a000    0     1   662 0004002 [SLPQ ttyin 0xc1afaa10][SLP] getty
 %   661 c1d2fe20 d6975000    0     1   661 0004002 [SLPQ ttyin 0xc1a62210][SLP] getty
 %   660 c1d2fc5c d6974000    0     1   660 0004002 [SLPQ ttyin 0xc195b610][SLP] getty
 %   659 c1c5054c d68d3000    0     1   659 0004002 [SLPQ ttyin 0xc195b410][SLP] getty
 %   654 c1d2f000 d696d000    0     1   654 0000000 [SLPQ select 0xc08c7d04][SLP] inetd
 %   623 c1c50000 d6893000    0     1   622 0000000 [SLPQ select 0xc08c7d04][SLP] snmpd
 %   615 c1c6da98 d6914000    0     1   615 0000000 [SLPQ pause 0xc1c6dad0][SLP] perl5.8.5
 %   605 c1c50c5c d68d7000    0     1   605 0000100 [SLPQ select 0xc08c7d04][SLP] perl
 %   604 c1f1d710 d6993000   88   579    47 000c182 (threaded)  mysqld
 %    thread 0xc1c52960 ksegrp 0xc1a493f0 [SLPQ kserel 0xc1a49430][SLP]
 %    thread 0xc1f1e4b0 ksegrp 0xc1a493f0 [SLPQ kserel 0xc1a49430][SLP]
 %    thread 0xc1c6eaf0 ksegrp 0xc1f3c3f0 [SLPQ kserel 0xc1f3c430][SLP]
 %    thread 0xc1d2ec80 ksegrp 0xc1a493f0 [SLPQ select 0xc08c7d04][SLP]
 %    thread 0xc1d2eaf0 ksegrp 0xc196c070 [SLPQ sigwait 0xd6965c2c][SLP]
 %    thread 0xc1f1e640 ksegrp 0xc1a491c0 [SLPQ ksesigwait 0xc1f1d810][SLP]
 %   579 c1d2cc5c d692f000    0     1    47 0004002 [SLPQ wait 0xc1d2cc5c][SLP] sh
 %   567 c1d2c710 d692c000  106     1   567 0000100 [SLPQ accept 0xc1d8703a][SLP] clamd
 %   560 c1c6c1c4 d68e8000 20001     1   560 0000100 [SLPQ select 0xc08c7d04][SLP] perl
 %   474 c1c6d8d4 d6913000    0     1   474 0000000 [SLPQ nanslp 0xc08c25ac][SLP] cron
 %   461 c1d2c8d4 d692d000    0     1   461 0000100 [SLPQ select 0xc08c7d04][SLP] sshd
 %   437 c1c6d54c d6911000 20000   431   431 0004100 [SLPQ select 0xc08c7d04][SLP] qmgr
 %   436 c1c50388 d68d2000 20000   431   431 0004100 [SLPQ select 0xc08c7d04][SLP] pickup
 %   432 c1c6c388 d68e9000    0     1    47 0004002 [SLPQ nanslp 0xc08c25ac][SLP] perl
 %   431 c1d2c54c d692b000    0     1   431 0004100 [SLPQ select 0xc08c7d04][SLP] master
 %   384 c1c6c8d4 d68ec000 65534     1   384 0000100 [SLPQ bpf 0xc1d1da00][SLP] snort
 %   290 c1a4a710 d5466000    0     1   290 0000000 [SLPQ biowr 0xcbf8dcd8][SLP] syslogd
 %   270 c1c6c000 d68e6000    0     1   270 0000000 [SLPQ select 0xc08c7d04][SLP] devd
 %   187 c1a4a54c d5465000    0     1   187 0000000 [SLPQ pause 0xc1a4a584][SLP] adjkerntz
 %   165 c1c6ca98 d68ed000    0     1   165 0000000 [SLPQ nanslp 0xc08c25ac][SLP] ipmon
 %    46 c19b0a98 d5438000    0     0     0 0000204 [SLPQ - 0xd422dd18][SLP] schedcpu
 %    45 c19b0c5c d5439000    0     0     0 0000204 [SLPQ hptrdy 0xc0768c30][SLP] hpt_wt
 %    44 c19b0e20 d543a000    0     0     0 0000204 [SLPQ vlruwt 0xc19b0e20][SLP] vnlru
 %    43 c1a47000 d543b000    0     0     0 0000204 [SLPQ syncer 0xc08c232c][SLP] syncer
 %    42 c1a471c4 d545a000    0     0     0 0000204 [SLPQ psleep 0xc08c82cc][SLP] bufdaemon
 %    41 c1a47388 d545b000    0     0     0 000020c [RUNQ] pagezero
 %    40 c1a4754c d545c000    0     0     0 0000204 [SLPQ psleep 0xc08d0064][SLP] vmdaemon
 %    39 c1a47710 d545d000    0     0     0 0000204 [SLPQ psleep 0xc08d0020][SLP] pagedaemon
 %    38 c1a478d4 d545e000    0     0     0 0000204 [IWAIT] swi0: sio
 %     9 c1a47a98 d545f000    0     0     0 0000204 [SLPQ - 0xc19a8d3c][SLP] fdc0
 %     8 c1a47c5c d5460000    0     0     0 0000204 [SLPQ idle 0xc195a400][SLP] aic_recovery0
 %     7 c1a47e20 d5461000    0     0     0 0000204 [SLPQ idle 0xc195a400][SLP] aic_recovery0
 %    37 c1a4a000 d5462000    0     0     0 0000204 [IWAIT] swi6:+
 %    36 c19a054c d540e000    0     0     0 0000204 [IWAIT] swi6: task queue
 %     6 c19a0710 d540f000    0     0     0 0000204 [SLPQ - 0xc19ec580][SLP] kqueue taskq
 %    35 c19a08d4 d5410000    0     0     0 0000204 [IWAIT] swi3: cambio
 %    34 c19a0a98 d542f000    0     0     0 0000204 [IWAIT] swi2: camnet
 %    33 c19a0c5c d5430000    0     0     0 0000204 [IWAIT] swi6:+
 %     5 c19a0e20 d5431000    0     0     0 0000204 [SLPQ - 0xc19ec7c0][SLP] thread taskq
 %    32 c19b0000 d5432000    0     0     0 0000204 [SLPQ - 0xc08b8140][SLP] yarrow
 %     4 c19b01c4 d5433000    0     0     0 0000204 [SLPQ - 0xc08bcb68][SLP] g_down
 %     3 c19b0388 d5434000    0     0     0 0000204 [SLPQ - 0xc08bcb64][SLP] g_up
 %     2 c19b054c d5435000    0     0     0 0000204 [SLPQ - 0xc08bcb5c][SLP] g_event
 %    31 c19b0710 d5436000    0     0     0 0000204 [CPU 0] swi1: net
 %    30 c19b08d4 d5437000    0     0     0 0000204 [IWAIT] swi4: vm
 %    29 c196d1c4 d421a000    0     0     0 000020c [IWAIT] swi5: clock sio
 %    28 c196d388 d421b000    0     0     0 0000204 [IWAIT] irq0: clk
 %    27 c196d54c d423a000    0     0     0 0000204 [IWAIT] irq15:
 %    26 c196d710 d423b000    0     0     0 0000204 [IWAIT] irq14: atapci0 ata0
 %    25 c196d8d4 d423c000    0     0     0 0000204 [IWAIT] irq13:
 %    24 c196da98 d423d000    0     0     0 0000204 [IWAIT] irq12:
 %    23 c196dc5c d423e000    0     0     0 0000204 [IWAIT] irq11: fxp1
 %    22 c196de20 d423f000    0     0     0 0000204 [IWAIT] irq10: xl0 fxp0
 %    21 c19a0000 d540b000    0     0     0 0000204 [IWAIT] irq9: ahc0
 %    20 c19a01c4 d540c000    0     0     0 0000204 [IWAIT] irq8: rtc
 %    19 c19a0388 d540d000    0     0     0 0000204 [IWAIT] irq7: ep0
 %    18 c1964000 d41d4000    0     0     0 0000204 [IWAIT] irq6: fdc0
 %    17 c19641c4 d4211000    0     0     0 0000204 [IWAIT] irq5:
 %    16 c1964388 d4212000    0     0     0 0000204 [IWAIT] irq4: sio0
 %    15 c196454c d4213000    0     0     0 0000204 [IWAIT] irq3: sio1
 %    14 c1964710 d4214000    0     0     0 0000204 [IWAIT] irq2:
 %    13 c19648d4 d4215000    0     0     0 0000204 [IWAIT] irq1: atkbd0
 %    12 c1964a98 d4216000    0     0     0 000020c [Can run] idle: cpu0
 %    11 c1964c5c d4217000    0     0     0 000020c [Can run] idle: cpu1
 %     1 c1964e20 d4218000    0     0     1 0004200 [SLPQ wait 0xc1964e20][SLP] init
 %    10 c196d000 d4219000    0     0     0 0000204 [SLPQ ktrace 0xc08c0558][SLP] ktrace
 %     0 c08bcc60 c0c1f000    0     0     0 0000200 [SLPQ sched 0xc08bcc60][SLP] swapper
 % db> show pcpu 0
 % cpuid        = 0
 % curthread    = 0xc196eaf0: pid 31 "swi1: net"
 % curpcb       = 0xd4233da0
 % fpcurthread  = none
 % idlethread   = 0xc1965640: pid 12 "idle: cpu0"
 % APIC ID      = 1
 % currentldt   = 0x28
 % db> show pcpu 1
 % cpuid        = 1
 % curthread    = 0xc206f4b0: pid 12458 "sshd"
 % curpcb       = 0xd6a02da0
 % fpcurthread  = none
 % idlethread   = 0xc19654b0: pid 11 "idle: cpu1"
 % APIC ID      = 0
 % currentldt   = 0x30
 % db> trace
 % ipllog(0,d4233bd8,d4233b50,d4233b48,d4233b40) at ipllog+0x43
 % ipflog(1a,c26fb820,d4233bd8,c2489000) at ipflog+0x18f
 % fr_check(c26fb820,14,c1a44000,0,d4233c80) at fr_check+0x820
 % fr_check_wrapper(0,d4233c80,c1a44000,1,0) at fr_check_wrapper+0x2a
 % pfil_run_hooks(c08c9a00,d4233ccc,c1a44000,1,0) at pfil_run_hooks+0xe3
 % ip_input(c2489000) at ip_input+0x211
 % netisr_processqueue(c08c8c98) at netisr_processqueue+0xa3
 % swi_net(0) at swi_net+0xf2
 % ithread_loop(c1962680,d4233d48) at ithread_loop+0x159
 % fork_exit(c060bafc,c1962680,d4233d48) at fork_exit+0x75
 % fork_trampoline() at fork_trampoline+0x8
 % --- trap 0x1, eip = 0, esp = 0xd4233d7c, ebp = 0 ---
 % db> show registers
 % cs                   8
 % ds          3228893200  vm_pageout_scan+1928
 % es                  16
 % fs                  24
 % ss                  16
 % eax                  0
 % ecx                  0
 % edx                 76
 % ebx         3559078872
 % esp         3559078672
 % ebp         3559078684
 % esi         3230277712  ipll
 % edi                 40
 % eip         3225831159  ipllog+67
 % efl              66134
 % dr0                  0
 % dr1                  0
 % dr2                  0
 % dr3                  0
 % dr4         4294905840
 % dr5               1024
 % dr6         4294905840
 % dr7               1024
 % ipllog+67:      incl    4(%eax)
 % db> show intrcnt
 % irq4: sio0              2993
 % irq6: fdc0              10
 % irq7: ep0               2
 % irq8: rtc               489396
 % irq9: ahc0              240921
 % irq10: xl0 fxp0         581217
 % irq11: fxp1             245
 % irq13: npx0             1
 % irq0: clk               382325
 % db> call doadump
 % Dumping 512 MB
 %  16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496
 % Dump complete
 % db> reset
 % cpu_reset called on cpu#0

From: Geoffrey Lassner <geoff@zyni.com>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/74909: SMP crash FreeBSD-5.3-p2 (swi1: net) / ipllog
Date: Wed, 02 Feb 2005 21:56:48 -0700

 Please close this bug.  Using "option NET_WITH_GIANT" makes it
 stable.  Removing "option IPFILTER_LOG" and disabling ipmon 
 also works.  I did not realize that ipfilter was not MPSAFE.  
 Is there some documentation that noted this when 5.3 was 
 released?  Also is there a status web page on making ipfilter
 MPSAFE?  Thanks.

From: freebsd 53 <freebsd53@zyni.com>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/74909: SMP crash FreeBSD-5.3-p2 (swi1: net) / ipllog
Date: Thu, 07 Apr 2005 18:10:17 -0600

 This issue appears to have been fixed in FreeBSD 5.4 RC1, thank you
 Robert Watson.  Please feel free to close this problem out.
State-Changed-From-To: open->closed 
State-Changed-By: linimon 
State-Changed-When: Fri Apr 8 02:25:28 GMT 2005 
State-Changed-Why:  
Submitter notes that a recent commit has fixed the problem. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=74909 
>Unformatted:
