From nobody@FreeBSD.org  Sat Jan 13 09:39:42 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id E3E4916A403
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 13 Jan 2007 09:39:41 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id D3D3613C465
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 13 Jan 2007 09:39:41 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l0D9ddYJ041070
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 13 Jan 2007 09:39:39 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id l0D9ddeu041069;
	Sat, 13 Jan 2007 09:39:39 GMT
	(envelope-from nobody)
Message-Id: <200701130939.l0D9ddeu041069@www.freebsd.org>
Date: Sat, 13 Jan 2007 09:39:39 GMT
From: Mikle Davidkin<skylord@vt.net.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: kernel panic in 6.2-RC2 on heavy network load
X-Send-Pr-Version: www-3.0

>Number:         107864
>Category:       kern
>Synopsis:       [panic] kernel panic in 6.2-RC2 on heavy network load
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Jan 13 09:40:13 GMT 2007
>Closed-Date:    Fri Nov 16 09:28:04 UTC 2007
>Last-Modified:  Fri Nov 16 09:28:04 UTC 2007
>Originator:     Mikle Davidkin
>Release:        6.2-RC2
>Organization:
Vector Telecom
>Environment:
FreeBSD router05.vt.net.ru 6.2-RC2 FreeBSD 6.2-RC2 #0: Mon Jan  8 13:17:00 MSK 2007     root@router05.vt.net.ru:/usr/src/sys/i386/compile/skykernel  i386

>Description:
Kernel panics almost every evening when the most heavy network load is
on the net. Machine is just a software router for 3 subnets with routed,
ipfw, pf (for binat) and no more... 

Similar symptoms I have on other software router - will report in other
PR and add link to it here.

Kernel is GENERIC with some devices disabled (SCSI/RAID) and few options added:
options DEVICE_POLLING
options HZ=1000
options IPFIREWALL
options DUMMYNET
options IPFIREWALL_FORWARD

dmesg
==============
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RC2 #0: Mon Jan  8 13:17:00 MSK 2007
    root@router05.vt.net.ru:/usr/src/sys/i386/compile/skykernel
ACPI APIC Table: <KT600  AWRDACPI>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) XP 2000+ (1661.94-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x681  Stepping = 1
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
  AMD Features=0xc0400800<SYSCALL,MMX+,3DNow+,3DNow>
real memory  = 268369920 (255 MB)
avail memory = 252968960 (241 MB)
ioapic0 <Version 0.3> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <KT600 AWRDACPI> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8377 (Apollo KT400/KT400A/KT600) host to PCI bridge> mem 0xe5000000-0xe53fffff at device 0.0 on pci0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
stge0: <Sundance ST-1023 Gigabit Ethernet> port 0xc000-0xc0ff mem 0xe5703000-0xe57030ff irq 17 at device 9.0 on pci0
miibus0: <MII bus> on stge0
ip1000phy0: <IC Plus 10/100/1000 media interface> on miibus0
ip1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
stge0: Ethernet address: 00:15:f2:ef:de:a9
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xc400-0xc43f mem 0xe5701000-0xe5701fff,0xe5400000-0xe54fffff irq 18 at device 10.0 on pci0
miibus1: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus1
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:d0:43:7a:fd:77
fxp1: <Intel 82559 Pro/100 Ethernet> port 0xc800-0xc83f mem 0xe5700000-0xe5700fff,0xe5500000-0xe55fffff irq 19 at device 11.0 on pci0
miibus2: <MII bus> on fxp1
inphy1: <i82555 10/100 media interface> on miibus2
inphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: Ethernet address: 00:02:b3:bc:f2:60
fxp2: <Intel 82559 Pro/100 Ethernet> port 0xcc00-0xcc3f mem 0xe5702000-0xe5702fff,0xe5600000-0xe56fffff irq 16 at device 12.0 on pci0
miibus3: <MII bus> on fxp2
inphy2: <i82555 10/100 media interface> on miibus3
inphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp2: Ethernet address: 00:d0:43:7a:fa:99
atapci0: <VIA 8237 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 15.0 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
uhci0: <VIA 83C572 USB controller> port 0xd400-0xd41f at device 16.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xd800-0xd81f at device 16.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xdc00-0xdc1f at device 16.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0xe000-0xe01f at device 16.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xe5704000-0xe57040ff at device 16.4 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xe400-0xe4ff mem 0xe5705000-0xe57050ff irq 23 at device 18.0 on pci0
miibus4: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> on miibus4
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:11:5b:93:c2:3d
acpi_tz0: <Thermal Zone> on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
ppc0: <Standard parallel printer port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcffff,0xd0000-0xd0fff,0xd1000-0xd27ff,0xd3000-0xd3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1661943183 Hz quality 800
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to deny, logging disabled
ad0: 38166MB <WDC WD400JB-32ENA0 05.03E05> at ata0-master UDMA100
=================


backtrace
============================
1) Jan 8 20:26
# kgdb /boot/kernel/kernel.debug /var/crash/vmcore.6
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x78
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc0578abd
stack pointer           = 0x28:0xd40b8ab4
frame pointer           = 0x28:0xd40b8ab8
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = resume, IOPL = 0
current process         = 11 (swi1: net)
trap number             = 12
panic: page fault
Uptime: 6h42m36s
Physical memory: 247 MB
Dumping 34 MB: 19 3

#0  doadump () at pcpu.h:165
165     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc05584ee in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2  0xc0558784 in panic (fmt=0xc0738cc7 "%s") at ../../../kern/kern_shutdown.c:565
#3  0xc070fa60 in trap_fatal (frame=0xd40b8a74, eva=120) at ../../../i386/i386/trap.c:837
#4  0xc070f242 in trap (frame=
      {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -737440936, tf_esi = -1038653568, tf_ebp = -737441096, tf_isp = -737441120, tf_ebx = -1038686208, tf_edx = -1038686208, tf_ecx = 4, tf_eax = -1038653536, tf_trapno = 12, tf_err = 0, tf_eip = -1068004675, tf_cs = 32, tf_eflags = 589831, tf_esp = -1038653568, tf_ss = -737441060}) at ../../../i386/i386/trap.c:270
#5  0xc06fe3fa in calltrap () at ../../../i386/i386/exception.s:139
#6  0xc0578abd in turnstile_setowner (ts=0xc216e800, owner=0x4) at ../../../kern/subr_turnstile.c:432
#7  0xc0578db4 in turnstile_wait (lock=0xc28de168, owner=0x4) at ../../../kern/subr_turnstile.c:591
#8  0xc054ec70 in _mtx_lock_sleep (m=0xc28de168, tid=3256313728, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:579
#9  0xc05d5434 in rt_check (lrt=0xc21767a0, lrt0=0xd40b8b58, dst=0xc288e210) at ../../../net/route.c:1280
#10 0xc05d8432 in arpresolve (ifp=0xc2245400, rt0=0xc25fd420, m=0xc24baa00, dst=0xc288e210, desten=0xd40b8b78 "")
    at ../../../netinet/if_ether.c:404
#11 0xc05c7633 in ether_output (ifp=0xc2245400, m=0xc24baa00, dst=0xc288e210, rt0=0xc216e800) at ../../../net/if_ethersubr.c:174
#12 0xc05ed765 in ip_output (m=0xc24baa00, opt=0xc2245400, ro=0xd40b8bd4, flags=1, imo=0x0, inp=0x0)
    at ../../../netinet/ip_output.c:777
#13 0xc05ecb8c in ip_forward (m=0xc24baa00, srcrt=0) at ../../../netinet/ip_input.c:1917
#14 0xc05eb6bb in ip_input (m=0xc24baa00) at ../../../netinet/ip_input.c:688
#15 0xc05cff6b in netisr_processqueue (ni=0xc07fc7b8) at ../../../net/netisr.c:236
#16 0xc05d0166 in swi_net (dummy=0x0) at ../../../net/netisr.c:349
#17 0xc05428fd in ithread_execute_handlers (p=0xc2175a78, ie=0xc21a9300) at ../../../kern/kern_intr.c:682
#18 0xc0542a08 in ithread_loop (arg=0xc2143720) at ../../../kern/kern_intr.c:765
#19 0xc0541884 in fork_exit (callout=0xc05429b4 <ithread_loop>, arg=0xc2143720, frame=0xd40b8d38) at ../../../kern/kern_fork.c:821
#20 0xc06fe45c in fork_trampoline () at ../../../i386/i386/exception.s:208
(kgdb) list *0xc0578abd
0xc0578abd is in turnstile_setowner (../../../kern/subr_turnstile.c:433).
428
429             mtx_assert(&td_contested_lock, MA_OWNED);
430             MPASS(owner->td_proc->p_magic == P_MAGIC);
431             MPASS(ts->ts_owner == NULL);
432             ts->ts_owner = owner;
433             LIST_INSERT_HEAD(&owner->td_contested, ts, ts_link);
434     }
435
436     /*
437      * Malloc a turnstile for a new thread, initialize it and return it.

2) Jan 9 15:26
# kgdb /boot/kernel/kernel.debug /var/crash/vmcore.7
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:


#0  doadump () at pcpu.h:165
165     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc05584ee in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2  0xc0558784 in panic (fmt=0xc0738cc7 "%s") at ../../../kern/kern_shutdown.c:565
#3  0xc070fa60 in trap_fatal (frame=0xd40b8a74, eva=120) at ../../../i386/i386/trap.c:837
#4  0xc070f242 in trap (frame=
      {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -737440936, tf_esi = -1038653568, tf_ebp = -737441096, tf_isp = -737441120, tf_ebx = -1038686208, tf_edx = -1038686208, tf_ecx = 4, tf_eax = -1038653536, tf_trapno = 12, tf_err = 0, tf_eip = -1068004675, tf_cs = 32, tf_eflags = 589831, tf_esp = -1038653568, tf_ss = -737441060}) at ../../../i386/i386/trap.c:270
#5  0xc06fe3fa in calltrap () at ../../../i386/i386/exception.s:139
#6  0xc0578abd in turnstile_setowner (ts=0xc216e800, owner=0x4) at ../../../kern/subr_turnstile.c:432
#7  0xc0578db4 in turnstile_wait (lock=0xc2806378, owner=0x4) at ../../../kern/subr_turnstile.c:591
#8  0xc054ec70 in _mtx_lock_sleep (m=0xc2806378, tid=3256313728, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:579
#9  0xc05d5434 in rt_check (lrt=0xc21767a0, lrt0=0xd40b8b58, dst=0xc266ad50) at ../../../net/route.c:1280
#10 0xc05d8432 in arpresolve (ifp=0xc2245400, rt0=0xc2806000, m=0xc25b4100, dst=0xc266ad50, desten=0xd40b8b78 "")
    at ../../../netinet/if_ether.c:404
#11 0xc05c7633 in ether_output (ifp=0xc2245400, m=0xc25b4100, dst=0xc266ad50, rt0=0xc216e800) at ../../../net/if_ethersubr.c:174
#12 0xc05ed765 in ip_output (m=0xc25b4100, opt=0xc2245400, ro=0xd40b8bd4, flags=1, imo=0x0, inp=0x0)
    at ../../../netinet/ip_output.c:777
#13 0xc05ecb8c in ip_forward (m=0xc25b4100, srcrt=0) at ../../../netinet/ip_input.c:1917
#14 0xc05eb6bb in ip_input (m=0xc25b4100) at ../../../netinet/ip_input.c:688
#15 0xc05cff6b in netisr_processqueue (ni=0xc07fc7b8) at ../../../net/netisr.c:236
#16 0xc05d0166 in swi_net (dummy=0x0) at ../../../net/netisr.c:349
#17 0xc05428fd in ithread_execute_handlers (p=0xc2175a78, ie=0xc21a9300) at ../../../kern/kern_intr.c:682
#18 0xc0542a08 in ithread_loop (arg=0xc2143720) at ../../../kern/kern_intr.c:765
#19 0xc0541884 in fork_exit (callout=0xc05429b4 <ithread_loop>, arg=0xc2143720, frame=0xd40b8d38) at ../../../kern/kern_fork.c:821
#20 0xc06fe45c in fork_trampoline () at ../../../i386/i386/exception.s:208
(kgdb) quit

>How-To-Repeat:
panic occurs in 1-3 days uptume
>Fix:

>Release-Note:
>Audit-Trail:

From: Mikle Davidkin <skylord@vt.net.ru>
To: bug-followup@FreeBSD.org, skylord@vt.net.ru
Cc:  
Subject: Re: kern/107864: kernel panic in 6.2-RC2 on heavy network load
Date: Sat, 13 Jan 2007 13:02:07 +0300

 May be it's related to PR 107865
 
 [linimon note: backtrace from that is included here]

 Kernel is GENERIC with some devices disabled (SCSI/RAID) and few options added:
 options DEVICE_POLLING
 options HZ=1000
 options IPFIREWALL
 options DUMMYNET
 options IPFIREWALL_FORWARD

 dmesg
 ==============
 Copyright (c) 1992-2007 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 6.2-RC2 #0: Mon Jan  8 13:29:30 MSK 2007
     root@router03.vt.net.ru:/usr/src/sys/i386/compile/skykernel
 ACPI APIC Table: <Nvidia AWRDACPI>
 Timecounter "i8254" frequency 1193182 Hz quality 0
 CPU: AMD Sempron(tm) Processor 2600+ (1607.34-MHz 686-class CPU)
   Origin = "AuthenticAMD"  Id = 0x20fc2  Stepping = 2
   Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
   Features2=0x1<SSE3>
   AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
   AMD Features2=0x1<LAHF>
 real memory  = 268369920 (255 MB)
 avail memory = 252952576 (241 MB)
 ioapic0 <Version 1.1> irqs 0-23 on motherboard
 kbd1 at kbdmux0
 acpi0: <Nvidia AWRDACPI> on motherboard
 acpi0: Power Button (fixed)
 Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
 cpu0: <ACPI CPU> on acpi0
 acpi_button0: <Power Button> on acpi0
 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0xcf0-0xcf3 on acpi0
 pci0: <ACPI PCI bus> on pcib0
 agp0: <NVIDIA nForce3-250 AGP Controller> mem 0xee000000-0xefffffff at device 0.0 on pci0
 isab0: <PCI-ISA bridge> at device 1.0 on pci0
 isa0: <ISA bus> on isab0
 pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
 ohci0: <OHCI (generic) USB controller> mem 0xf6002000-0xf6002fff irq 20 at device 2.0 on pci0
 ohci0: [GIANT-LOCKED]
 usb0: OHCI version 1.0, legacy support
 usb0: SMM does not respond, resetting
 usb0: <OHCI (generic) USB controller> on ohci0
 usb0: USB revision 1.0
 uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
 uhub0: 4 ports with 4 removable, self powered
 ohci1: <OHCI (generic) USB controller> mem 0xf6003000-0xf6003fff irq 21 at device 2.1 on pci0
 ohci1: [GIANT-LOCKED]
 usb1: OHCI version 1.0, legacy support
 usb1: SMM does not respond, resetting
 usb1: <OHCI (generic) USB controller> on ohci1
 usb1: USB revision 1.0
 uhub1: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
 uhub1: 4 ports with 4 removable, self powered
 ehci0: <NVIDIA nForce3 250 USB 2.0 controller> mem 0xf6004000-0xf60040ff irq 22 at device 2.2 on pci0
 ehci0: [GIANT-LOCKED]
 usb2: EHCI version 1.0
 usb2: companion controllers, 4 ports each: usb0 usb1
 usb2: <NVIDIA nForce3 250 USB 2.0 controller> on ehci0
 usb2: USB revision 2.0
 uhub2: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
 uhub2: 8 ports with 8 removable, self powered
 nve0: <NVIDIA nForce MCP7 Networking Adapter> port 0xb800-0xb807 mem 0xf6000000-0xf6000fff irq 20 at device 5.0 on pci0
 nve0: Ethernet address 00:04:61:a5:08:01
 miibus0: <MII bus> on nve0
 rlphy0: <RTL8201L 10/100 media interface> on miibus0
 rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 nve0: Ethernet address: 00:04:61:a5:08:01
 atapci0: <nVidia nForce3 Pro UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 8.0 on pci0
 ata0: <ATA channel 0> on atapci0
 ata1: <ATA channel 1> on atapci0
 pcib1: <ACPI PCI-PCI bridge> at device 11.0 on pci0
 pci1: <ACPI PCI bus> on pcib1
 pci1: <display, VGA> at device 0.0 (no driver attached)
 pcib2: <ACPI PCI-PCI bridge> at device 14.0 on pci0
 pci2: <ACPI PCI bus> on pcib2
 em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x9000-0x903f mem 0xf5100000-0xf511ffff,0xf5120000-0xf513ffff irq 19 at device 7.0 on pci2
 em0: Ethernet address: 00:07:e9:0f:65:32
 xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0x9400-0x947f mem 0xf5143000-0xf514307f irq 16 at device 8.0 on pci2
 miibus1: <MII bus> on xl0
 ukphy0: <Generic IEEE 802.3u media interface> on miibus1
 ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 xl0: Ethernet address: 00:0a:5e:4f:f3:d6
 xl1: <3Com 3c905C-TX Fast Etherlink XL> port 0x9800-0x987f mem 0xf5140000-0xf514007f irq 17 at device 9.0 on pci2
 miibus2: <MII bus> on xl1
 ukphy1: <Generic IEEE 802.3u media interface> on miibus2
 ukphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 xl1: Ethernet address: 00:0a:5e:4f:f3:d9
 xl2: <3Com 3c905C-TX Fast Etherlink XL> port 0x9c00-0x9c7f mem 0xf5141000-0xf514107f irq 18 at device 10.0 on pci2
 miibus3: <MII bus> on xl2
 ukphy2: <Generic IEEE 802.3u media interface> on miibus3
 ukphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 xl2: Ethernet address: 00:0a:5e:4f:f3:e2
 fxp0: <Intel 82559 Pro/100 Ethernet> port 0xa000-0xa03f mem 0xf5142000-0xf5142fff,0xf5000000-0xf50fffff irq 19 at device 11.0 on pci2
 miibus4: <MII bus> on fxp0
 inphy0: <i82555 10/100 media interface> on miibus4
 inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 fxp0: Ethernet address: 00:d0:b7:07:c1:cc
 acpi_tz0: <Thermal Zone> on acpi0
 fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
 fdc0: [FAST]
 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
 sio0: type 16550A
 sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
 sio1: type 16550A
 ppc0: <Standard parallel printer port> port 0x378-0x37f,0x778-0x77b irq 7 on acpi0
 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
 ppbus0: <Parallel port bus> on ppc0
 plip0: <PLIP network interface> on ppbus0
 lpt0: <Printer> on ppbus0
 lpt0: Interrupt-driven port
 ppi0: <Parallel I/O> on ppbus0
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 pmtimer0 on isa0
 orm0: <ISA Option ROMs> at iomem 0xc0000-0xcffff,0xd0000-0xd0fff,0xd1000-0xd17ff,0xd2000-0xd27ff,0xd3000-0xd37ff,0xd4000-0xd4fff on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=0x300>
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
 Timecounter "TSC" frequency 1607340759 Hz quality 800
 Timecounters tick every 1.000 msec
 ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to deny, logging disabled
 ad0: 38204MB <SAMSUNG SP0411N TW100-13> at ata0-master UDMA100
 =================


 backtrace
 ============================
 kgdb: kvm_nlist(_stopped_cpus):
 kgdb: kvm_nlist(_stoppcbs):
 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd".

 Unread portion of the kernel message buffer:
 kernel trap 12 with interrupts disabled


 Fatal trap 12: page fault while in kernel mode
 fault virtual address   = 0x78
 fault code              = supervisor read, page not present
 instruction pointer     = 0x20:0xc05719ad
 stack pointer           = 0x28:0xd6e18a7c
 frame pointer           = 0x28:0xd6e18a80
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = resume, IOPL = 0
 current process         = 306 (routed)
 trap number             = 12
 panic: page fault
 Uptime: 2d13h32m45s
 Physical memory: 247 MB
 Dumping 65 MB: 50 34 18 2

 #0  doadump () at pcpu.h:165
 165     pcpu.h: No such file or directory.
         in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:165
 #1  0xc05513de in boot (howto=260) at ../../../kern/kern_shutdown.c:409
 #2  0xc0551674 in panic (fmt=0xc07318a5 "%s") at ../../../kern/kern_shutdown.c:565
 #3  0xc0708790 in trap_fatal (frame=0xd6e18a3c, eva=120) at ../../../i386/i386/trap.c:837
 #4  0xc0707f72 in trap (frame=
       {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 16, tf_esi = -1035997184, tf_ebp = -689862016, tf_isp = -689862040, tf_ebx = -1032266368, tf_edx = -1032266368, tf_ecx = 4, tf_eax = -1035997152, tf_trapno = 12, tf_err = 0, tf_eip = -1068033619, tf_cs = 32, tf_eflags = 65539, tf_esp = -1035997184, tf_ss = -689861980})
     at ../../../i386/i386/trap.c:270
 #5  0xc06f712a in calltrap () at ../../../i386/i386/exception.s:139
 #6  0xc05719ad in turnstile_setowner (ts=0xc278dd80, owner=0x4) at ../../../kern/subr_turnstile.c:432
 #7  0xc0571ca4 in turnstile_wait (lock=0xc2809168, owner=0x4) at ../../../kern/subr_turnstile.c:591
 #8  0xc0547b60 in _mtx_lock_sleep (m=0xc2809168, tid=3258970112, opts=0, file=0x0, line=0)
     at ../../../kern/kern_mutex.c:579
 #9  0xc05cdcdc in rt_setgate (rt=0xc27e0bdc, dst=0xc27b40e0, gate=0xc363b66c) at ../../../net/route.c:1041
 #10 0xc05ceecd in route_output (m=0xc36fed00, so=0xc247b000) at ../../../net/rtsock.c:487
 #11 0xc05cc634 in raw_usend (so=0x4, flags=0, m=0xc278dd80, nam=0x0, control=0xc23ff020, td=0xc23ff000)
     at ../../../net/raw_usrreq.c:263
 #12 0xc05ce7cb in rts_send (so=0xc247b000, flags=0, m=0xc36fed00, nam=0x0, control=0x0, td=0xc23ff000)
     at ../../../net/rtsock.c:269
 #13 0xc058a4d3 in sosend (so=0xc247b000, addr=0x0, uio=0xd6e18cbc, top=0xc36fed00, control=0x0, flags=0,
     td=0xc23ff000) at ../../../kern/uipc_socket.c:836
 #14 0xc0578d5e in soo_write (fp=0xc23ff020, uio=0xd6e18cbc, active_cred=0xc2173d80, flags=0,
     td=0xc23ff000) at ../../../kern/sys_socket.c:118
 #15 0xc0573467 in dofilewrite (td=0xc23ff000, fd=3, fp=0xc2424288, auio=0xd6e18cbc, offset=Unhandled dwarf expression opcode 0x93
 ) at file.h:252
 #16 0xc057330b in kern_writev (td=0xc23ff000, fd=3, auio=0xd6e18cbc) at ../../../kern/sys_generic.c:402
 #17 0xc0573231 in write (td=0xc23ff000, uap=0x4) at ../../../kern/sys_generic.c:326
 #18 0xc0708aa7 in syscall (frame=
       {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = -1024, tf_esi = 3, tf_ebp = -1077941176, tf_isp = -689861276, tf_ebx = 1, tf_edx = -1077941220, tf_ecx = 2, tf_eax = 4, tf_trapno = 22, tf_err = 2, tf_eip = 672494419, tf_cs = 51, tf_eflags = 642, tf_esp = -1077941380, tf_ss = 59}) at ../../../i386/i386/trap.c:983
 #19 0xc06f717f in Xint0x80_syscall () at ../../../i386/i386/exception.s:200
 #20 0x00000033 in ?? ()
 Previous frame inner to this frame (corrupt stack?)
 (kgdb) list *0xc05719ad
 0xc05719ad is in turnstile_setowner (../../../kern/subr_turnstile.c:433).
 428      * Malloc a turnstile for a new thread, initialize it and return it.
 429      */
 430     struct turnstile *
 431     turnstile_alloc(void)
 432     {
 433             struct turnstile *ts;
 434
 435             ts = malloc(sizeof(struct turnstile), M_TURNSTILE, M_WAITOK | M_ZERO);
 436             TAILQ_INIT(&ts->ts_blocked);
 437             TAILQ_INIT(&ts->ts_pending);

State-Changed-From-To: open->feedback 
State-Changed-By: kmacy 
State-Changed-When: Fri Nov 16 02:24:43 UTC 2007 
State-Changed-Why:  

Need follow-up confirmation. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=107864 

From: "Mikle Davidkin" <skylord@vt.net.ru>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/107864: [panic] kernel panic in 6.2-RC2 on heavy network 
     load
Date: Fri, 16 Nov 2007 11:38:28 +0300 (MSK)

 It was some "routed" bug. I change it to quagga and all is working good
 with it...
 
State-Changed-From-To: feedback->closed 
State-Changed-By: remko 
State-Changed-When: Fri Nov 16 09:28:03 UTC 2007 
State-Changed-Why:  
David mentions that his had been resolved (Quagga was at fault) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=107864 
>Unformatted:
A number of fixes have gone in to related areas. Can you confirm that this still happens on RELENG_6?
