From gadm@mowgli.rbc.ru  Thu Apr 27 13:37:58 2006
Return-Path: <gadm@mowgli.rbc.ru>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 2C67216A460
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 27 Apr 2006 13:37:58 +0000 (UTC)
	(envelope-from gadm@mowgli.rbc.ru)
Received: from relay.rbc.ru (relay.rbc.ru [80.68.241.103])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 6A65D43D4C
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 27 Apr 2006 13:37:57 +0000 (GMT)
	(envelope-from gadm@mowgli.rbc.ru)
Received: from mowgli.rbc.ru (ws-80-68-243-91.rbc.ru [80.68.243.91])
	by relay.rbc.ru (Postfix) with ESMTP id 8074626D19C
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 27 Apr 2006 17:37:55 +0400 (MSD)
Received: by mowgli.rbc.ru (Postfix, from userid 290)
	id 98C96109B6; Thu, 27 Apr 2006 17:21:26 +0400 (MSD)
Message-Id: <20060427132126.98C96109B6@mowgli.rbc.ru>
Date: Thu, 27 Apr 2006 17:21:26 +0400 (MSD)
From: Andrew Kolchoogin <gadm@rbc.ru>
Reply-To: Andrew Kolchoogin <andrew@rinet.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: FreeBSD 6.1-RC Kernel Panic
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         96413
>Category:       kern
>Synopsis:       NULL inpcb pointer in tcp_timewait()
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    silby
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Apr 27 13:40:09 GMT 2006
>Closed-Date:    Tue Jun 12 07:37:24 GMT 2007
>Last-Modified:  Tue Jun 12 07:37:24 GMT 2007
>Originator:     Andrew Kolchoogin
>Release:        FreeBSD 6.1-RC i386
>Organization:
RosBusinessConsulting LLC
>Environment:
System: FreeBSD rtcomm.avalon-island.ru 6.1-RC FreeBSD 6.1-RC #0: Thu Apr 27 15:30:19 MSD 2006 andrew@rtcomm.avalon-island.ru:/usr/obj/usr/src/sys/GENERIC  i386

Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 6.1-RC #0: Thu Apr 27 15:30:19 MSD 2006
    andrew@rtcomm.avalon-island.ru:/usr/obj/usr/src/sys/GENERIC
ACPI APIC Table: <A M I  OEMAPIC >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.00GHz (2992.51-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf41  Stepping = 1
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x641d<SSE3,RSVD2,MON,DS_CPL,CNTX-ID,CX16,<b14>>
  AMD Features=0x20100000<NX,LM>
  Logical CPUs per core: 2
real memory  = 1073532928 (1023 MB)
avail memory = 1041596416 (993 MB)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <A M I OEMRSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pci2: <network, ethernet> at device 0.0 (no driver attached)
pcib3: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci3: <ACPI PCI bus> on pcib3
uhci0: <UHCI (generic) USB controller> port 0xc880-0xc89f irq 16 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0xcc00-0xcc1f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
pci0: <base peripheral> at device 29.4 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 29.5 (no driver attached)
ehci0: <Intel 6300ESB USB 2.0 controller> mem 0xdedffc00-0xdedfffff irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb2: EHCI version 1.0
usb2: companion controllers, 2 ports each: usb0 usb1
usb2: <Intel 6300ESB USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 4 ports with 4 removable, self powered
pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pci4: <display, VGA> at device 2.0 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection Version - 3.2.18> port 0xec00-0xec3f mem 0xdefa0000-0xdefbffff irq 17 at device 3.0 on pci4
em0: Ethernet address: 00:0e:0c:4c:16:89
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 6300ESB SATA150 controller> port 0xc800-0xc807,0xc480-0xc483,0xc400-0xc407,0xc080-0xc083,0xc000-0xc00f irq 18 at device 31.2 on pci0
ata2: <ATA channel 0> on atapci0
ata3: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata1 at port 0x170-0x177,0x376 irq 15 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2992513988 Hz quality 800
Timecounters tick every 1.000 msec
ad4: 381554MB <Seagate ST3400832AS 3.03> at ata2-master SATA150
Trying to mount root from ufs:/dev/ad4s1a
WARNING: / was not properly dismounted
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, default to deny, logging disabled
em0: link state changed to UP

>Description:
	GENERIC kernel from RELENG_6_1 sporadically traps in network stack:

[root@rtcomm /var/crash]# kgdb /usr/obj/usr/src/sys/GENERIC/kernel.debug vmcore.5
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0xac
fault code              = supervisor write, page not present
instruction pointer     = 0x20:0xc075fff1
stack pointer           = 0x28:0xe3599b08
frame pointer           = 0x28:0xe3599b30
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi1: net)
trap number             = 12
panic: page fault
Uptime: 8m0s
Dumping 1023 MB (2 chunks)
  chunk 0: 1MB (159 pages) ... ok
  chunk 1: 1023MB (261837 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15

#0  doadump () at pcpu.h:165
165     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc06a147d in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402
#2  0xc06a1787 in panic (fmt=0xc090de4e "%s") at /usr/src/sys/kern/kern_shutdown.c:558
#3  0xc08c0a9d in trap_fatal (frame=0xe3599ac8, eva=0) at /usr/src/sys/i386/i386/trap.c:836
#4  0xc08c07bd in trap_pfault (frame=0xe3599ac8, usermode=0, eva=172) at /usr/src/sys/i386/i386/trap.c:744
#5  0xc08c0387 in trap (frame=
      {tf_fs = 8, tf_es = 40, tf_ds = -981401560, tf_edi = -991340508, tf_esi = -985873232, tf_ebp = -480666832, tf_isp = -480666892, tf_ebx = 4, tf_edx = 0, tf_ecx = -995464960, tf_eax = -995464960, tf_trapno = 12, tf_err = 2, tf_eip = -1066008591, tf_cs = 32, tf_eflags = 66194, tf_esp = -985873232, tf_ss = 16}) at /usr/src/sys/i386/i386/trap.c:434
#6  0xc08adaea in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc075fff1 in tcp_timewait (tw=0xc53cc4b0, to=0xe3599bf8, th=0xc4e95824, m=0xc4aa6900, tlen=0) at atomic.h:149
#8  0xc075ccf8 in tcp_input (m=0xc4e7f600, off0=20) at /usr/src/sys/netinet/tcp_input.c:762
#9  0xc0754af7 in ip_input (m=0xc4e7f600) at /usr/src/sys/netinet/ip_input.c:786
#10 0xc072bc87 in netisr_processqueue (ni=0xc0a02a58) at /usr/src/sys/net/netisr.c:236
#11 0xc072bed4 in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349
#12 0xc0688a48 in ithread_execute_handlers (p=0xc4aa5830, ie=0xc4aa4580) at /usr/src/sys/kern/kern_intr.c:684
#13 0xc0688b92 in ithread_loop (arg=0xc4a896a0) at /usr/src/sys/kern/kern_intr.c:767
#14 0xc068766c in fork_exit (callout=0xc0688b2f <ithread_loop>, arg=0xc4aa6900, frame=0xc4aa6900) at /usr/src/sys/kern/kern_fork.c:805
#15 0xc08adb4c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208

>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:

From: Andrew Kolchoogin <andrew@rinet.ru>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/96413: FreeBSD 6.1-RC Kernel Panic
Date: Wed, 10 May 2006 19:22:50 +0400

 6.1-RELEASE also affected with this bug:
 
 ===
 (kgdb) bt
 #0  doadump () at pcpu.h:165
 #1  0xc06a24f5 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402
 #2  0xc06a27ff in panic (fmt=0xc092e22e "%s") at /usr/src/sys/kern/kern_shutdown.c:558
 #3  0xc08e0b8d in trap_fatal (frame=0xe3599ac8, eva=0) at /usr/src/sys/i386/i386/trap.c:836
 #4  0xc08e08ad in trap_pfault (frame=0xe3599ac8, usermode=0, eva=172) at /usr/src/sys/i386/i386/trap.c:744
 #5  0xc08e0477 in trap (frame=
       {tf_fs = 8, tf_es = 40, tf_ds = -985595864, tf_edi = -990918620, tf_esi = -977145856, tf_ebp = -480666832, tf_isp = -480666892, tf_ebx = 4, tf_edx = 0, tf_ecx = -995464960, tf_eax = -995464960, tf_trapno = 12, tf_err = 2, tf_eip = -1066003983, tf_cs
   = 32, tf_eflags = 66194, tf_esp = -977145856, tf_ss = 16}) at /usr/src/sys/i386/i386/trap.c:434
 #6  0xc08cdbda in calltrap () at /usr/src/sys/i386/i386/exception.s:139
 #7  0xc07611f1 in tcp_timewait (tw=0xc5c1f000, to=0xe3599bf8, th=0xc4efc824, m=0xc4aa6900, tlen=0) at atomic.h:149
 #8  0xc075def8 in tcp_input (m=0xc4ed1900, off0=20) at /usr/src/sys/netinet/tcp_input.c:762
 #9  0xc0755cf7 in ip_input (m=0xc4ed1900) at /usr/src/sys/netinet/ip_input.c:786
 #10 0xc072ce87 in netisr_processqueue (ni=0xc0a23758) at /usr/src/sys/net/netisr.c:236
 #11 0xc072d0d4 in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349
 #12 0xc0689ac0 in ithread_execute_handlers (p=0xc4aa5830, ie=0xc4aa4580) at /usr/src/sys/kern/kern_intr.c:684
 #13 0xc0689c0a in ithread_loop (arg=0xc4a896a0) at /usr/src/sys/kern/kern_intr.c:767
 #14 0xc06886e4 in fork_exit (callout=0xc0689ba7 <ithread_loop>, arg=0xc4aa6900, frame=0xc4aa6900) at /usr/src/sys/kern/kern_fork.c:805
 #15 0xc08cdc3c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208
 (kgdb) x/a 0xc5c1f000
 0xc5c1f000:     0x0
 (kgdb)
 ===
 
     The first element of struct tcptw is a pointer to struct inpcb. As we
 could see, it is NULL in our case. Null pointer dereference => kernel panic.
 -- 
     Yours
         Andrew Kolchoogin.                              [DREW-RIPE, AKOL-RIPN]
 
 GOD#killall -KILL lifed && dd if=/dev/zero of=/dev/world; cd /src/world && make deinstall && make distclean && cat /patches/world0.01-0.59.patch | patch -p0 && make world && make installworld && /etc/rc.d/lifed start (C) someoneelse
State-Changed-From-To: open->feedback 
State-Changed-By: bms 
State-Changed-When: Mon Sep 25 12:10:39 UTC 2006 
State-Changed-Why:  
This may have been fixed by rev 1.281.2.9 on RELENG_6. 
Can you update sources and verify if the problem has been fixed? 
It would be helpful if you could give more detailed information about what 
you're doing with the system when this happens. 
Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=96413 
Responsible-Changed-From-To: freebsd-bugs->silby 
Responsible-Changed-By: bms 
Responsible-Changed-When: Mon Sep 25 12:42:11 UTC 2006 
Responsible-Changed-Why:  
silby, your starter for 10! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=96413 
State-Changed-From-To: feedback->closed 
State-Changed-By: linimon 
State-Changed-When: Tue Jun 12 07:37:01 UTC 2007 
State-Changed-Why:  
Feedback timeout (> 3 months). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=96413 
>Unformatted:
