From nsayer@quack.kfu.com  Mon Mar 31 06:48:03 2008
Return-Path: <nsayer@quack.kfu.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 10C25106566C
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Mar 2008 06:48:03 +0000 (UTC)
	(envelope-from nsayer@quack.kfu.com)
Received: from quack.kfu.com (6to4.kfu.com [IPv6:2002:478d:4001::1])
	by mx1.freebsd.org (Postfix) with ESMTP id C53DD8FC17
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Mar 2008 06:48:02 +0000 (UTC)
	(envelope-from nsayer@quack.kfu.com)
Received: from quack.kfu.com (localhost [127.0.0.1])
	by quack.kfu.com (8.14.2/8.14.2) with ESMTP id m2V6lxrH012833
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 30 Mar 2008 23:48:01 -0700 (PDT)
	(envelope-from nsayer@quack.kfu.com)
Received: (from nsayer@localhost)
	by quack.kfu.com (8.14.2/8.14.2/Submit) id m2V6lxIp012832;
	Sun, 30 Mar 2008 23:47:59 -0700 (PDT)
	(envelope-from nsayer)
Message-Id: <200803310647.m2V6lxIp012832@quack.kfu.com>
Date: Sun, 30 Mar 2008 23:47:59 -0700 (PDT)
From: Nick Sayer <nsayer@kfu.com>
Reply-To: Nick Sayer <nsayer@kfu.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Panic in ip_output related to IPv6 routes
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         122283
>Category:       kern
>Synopsis:       [ip6] [panic] Panic in ip_output related to IPv6 routes
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 31 06:50:01 UTC 2008
>Closed-Date:    Mon Oct 27 23:14:47 UTC 2008
>Last-Modified:  Mon Oct 27 23:14:47 UTC 2008
>Originator:     Nick Sayer
>Release:        FreeBSD 7.0-RELEASE i386
>Organization:
self
>Environment:
System: FreeBSD quack.kfu.com 7.0-RELEASE FreeBSD 7.0-RELEASE #2: Sun Feb 24 11:38:42 PST 2008 root@quack.kfu.com:/usr/obj/usr/src/sys/QUACK i386

The machine is a Core 2 Duo based desktop machine (configured for SMP).
It has a single interface connected to a DSL modem. It has the stf
nterface configured as its means of IPv6 connectivity. There is a
non-trivial amount of IPv6 traffic hitting this machine.

	
>Description:
About every 7-10 days, the machine panics. Almost always the
panic is in line 169 of ip_output.c, although a recent example
had the panic in line 235. In every case, *ro was an IPv6 route,
and ro->ro_rt->rt_flags was a crap address.

I haven't really tried to tackle this one by myself yet. I'm
instead hoping that someone else has already done the heavy
lifting.

>How-To-Repeat:
	
Just wait a week.
>Fix:
Unknown.

	


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: vwe 
State-Changed-When: Tue Apr 8 22:04:17 UTC 2008 
State-Changed-Why:  

submitter: Please don't be afraid to show us the kernel dump 
(panic message + backtrace) 
also an `ifconfig', `netstat -rn' and your kernel modifications may help 

http://www.freebsd.org/cgi/query-pr.cgi?pr=122283 

From: Nick Sayer <nsayer@kfu.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/122283: [ipv6] [panic] Panic in ip_output related to IPv6 routes
Date: Tue, 8 Apr 2008 15:30:49 -0700

 Latest example:
 
 Fatal trap 12: page fault while in kernel mode
 cpuid = 0; apic id = 00
 fault virtual address	= 0x34
 fault code		= supervisor read, page not present
 instruction pointer	= 0x20:0xc06f25b4
 stack pointer	        = 0x28:0xe688f798
 frame pointer	        = 0x28:0xe688f804
 code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, def32 1, gran 1
 processor eflags	= interrupt enabled, resume, IOPL = 0
 current process		= 1174 (imapd)
 trap number		= 12
 panic: page fault
 cpuid = 0
 Uptime: 6h12m31s
 Physical memory: 1015 MB
 Dumping 154 MB: 139 123 107 91 75 59 43 27 11
 
 
 (kgdb) bt
 #0  doadump () at pcpu.h:195
 #1  0xc062e2a7 in boot (howto=260) at /usr/src/sys/kern/ 
 kern_shutdown.c:409
 #2  0xc062e569 in panic (fmt=Variable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:563
 #3  0xc084ce2c in trap_fatal (frame=0xe688f758, eva=52)
      at /usr/src/sys/i386/i386/trap.c:899
 #4  0xc084d0b0 in trap_pfault (frame=0xe688f758, usermode=0, eva=52)
      at /usr/src/sys/i386/i386/trap.c:812
 #5  0xc084da5c in trap (frame=0xe688f758) at /usr/src/sys/i386/i386/ 
 trap.c:490
 #6  0xc0833d3b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
 #7  0xc06f25b4 in ip_output (m=0xc402f400, opt=0x0, ro=0xc3ee3344,  
 flags=0,
      imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:235
 #8  0xc06ca751 in stf_output (ifp=0xc3ee8400, m=0xc402f400,  
 dst=0xe688fa0c,
      rt=0xc3fd6d20) at /usr/src/sys/net/if_stf.c:533
 #9  0xc077772d in nd6_output (ifp=0xc3ee8400, origifp=0xc3ee8400,
      m0=0xc3e8b200, dst=0xe688fa0c, rt0=0xc3fd6d20)
      at /usr/src/sys/netinet6/nd6.c:2123
 #10 0xc07749f2 in ip6_output (m0=0xc3e8b200, opt=0x0, ro=0xe688fa08,  
 flags=0,
      im6o=0x0, ifpp=0x0, inp=0xc461f654)
      at /usr/src/sys/netinet6/ip6_output.c:927
 #11 0xc0750c21 in tcp_output (tp=0xc4a253a0)
      at /usr/src/sys/netinet/tcp_output.c:1114
 #12 0xc075af7a in tcp_usr_send (so=0xc48924a4, flags=Variable "flags"  
 is not available.
 )
      at /usr/src/sys/netinet/tcp_usrreq.c:843
 ---Type <return> to continue, or q <return> to quit---
 #13 0xc0681785 in sosend_generic (so=0xc48924a4, addr=0x0,  
 uio=0xe688fc60,
      top=0xc402f300, control=0x0, flags=0, td=0xc4a0c210)
      at /usr/src/sys/kern/uipc_socket.c:1240
 #14 0xc067d74f in sosend (so=0xc48924a4, addr=0x0, uio=0xe688fc60,  
 top=0x0,
      control=0x0, flags=0, td=0xc4a0c210)
      at /usr/src/sys/kern/uipc_socket.c:1286
 #15 0xc0667d4b in soo_write (fp=0xc43ba288, uio=0xe688fc60,
      active_cred=0xc4640400, flags=0, td=0xc4a0c210)
      at /usr/src/sys/kern/sys_socket.c:103
 #16 0xc06613f7 in dofilewrite (td=0xc4a0c210, fd=1, fp=0xc43ba288,
      auio=0xe688fc60, offset=-1, flags=0) at file.h:254
 #17 0xc06616d8 in kern_writev (td=0xc4a0c210, fd=1, auio=0xe688fc60)
      at /usr/src/sys/kern/sys_generic.c:401
 #18 0xc066174f in write (td=0xc4a0c210, uap=0xe688fcfc)
      at /usr/src/sys/kern/sys_generic.c:317
 #19 0xc084d405 in syscall (frame=0xe688fd38)
      at /usr/src/sys/i386/i386/trap.c:1035
 #20 0xc0833da0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/ 
 exception.s:196
 #21 0x00000033 in ?? ()
 Previous frame inner to this frame (corrupt stack?)
 
 ifconfig -au
 re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu  
 1500
 	options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
 	ether 00:17:31:e9:bc:66
 	inet6 fe80::217:31ff:fee9:bc66%re0 prefixlen 64 scopeid 0x1
 	inet 71.141.64.1 netmask 0xfffffff0 broadcast 71.141.64.15
 	inet6 2002:478d:4001:0:217:31ff:fee9:bc66 prefixlen 64
 	inet6 2002:478d:4001:: prefixlen 64 anycast
 	media: Ethernet autoselect (1000baseTX <full-duplex>)
 	status: active
 lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 	inet6 ::1 prefixlen 128
 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
 	inet 127.0.0.1 netmask 0xff000000
 stf0: flags=1<UP> metric 0 mtu 1280
 	inet6 2002:478d:4001::1 prefixlen 16
 
 
 Routing tables
 
 Internet:
 Destination        Gateway            Flags    Refs      Use  Netif  
 Expire
 default            71.141.64.14       UGS         0  4858740    re0
 71.141.64.0/28     link#1             UC          0        0    re0
 71.141.64.1        00:17:31:e9:bc:66  UHLW        1      558    lo0
 71.141.64.2        00:1b:63:f4:52:c8  UHLW        2   181265    re0     
 920
 71.141.64.14       00:02:3b:02:a7:51  UHLW        2        0    re0    
 1198
 127.0.0.1          127.0.0.1          UH          0  1736581    lo0
 
 Internet6:
 Destination                       Gateway                        
 Flags      Netif Expire
 ::/96                             ::1                            
 UGRS        lo0 =>
 default                           2002:c058:6301::               
 UGS        stf0
 ::1                               ::1                            
 UHL         lo0
 ::ffff:0.0.0.0/96                 ::1                            
 UGRS        lo0
 2002::/24                         ::1                            
 UGRS        lo0 =>
 2002::/16                         2002:478d:4001::1              
 U          stf0
 2002:478d:4001::                  00:17:31:e9:bc:66              
 UHL         lo0 =>
 2002:478d:4001::/64               link#1                         
 UC          re0
 2002:478d:4001::1                 link#3                         
 UHL         lo0
 2002:478d:4001:0:217:31ff:fee9:bc66 00:17:31:e9:bc:66              
 UHL         lo0
 2002:7f00::/24                    ::1                            
 UGRS        lo0
 2002:e000::/20                    ::1                            
 UGRS        lo0
 2002:ff00::/24                    ::1                            
 UGRS        lo0
 fe80::/10                         ::1                            
 UGRS        lo0
 fe80::%re0/64                     link#1                         
 UC          re0
 fe80::217:31ff:fee9:bc66%re0      00:17:31:e9:bc:66              
 UHL         lo0
 fe80::%lo0/64                     fe80::1%lo0                    
 U           lo0
 fe80::1%lo0                       link#2                         
 UHL         lo0
 ff01:1::/32                       link#1                         
 UC          re0
 ff01:2::/32                       ::1                            
 UC          lo0
 ff02::/16                         ::1                            
 UGRS        lo0
 ff02::%re0/32                     link#1                         
 UC          re0
 ff02::%lo0/32                     ::1                            
 UC          lo0
 
 

From: Pekka Savola <pekkas@netcore.fi>
To: bug-followup@freebsd.org
Cc:  
Subject: kern/122283: [ip6] [panic] Panic in ip_output related to IPv6 routes
 (fwd)
Date: Tue, 13 May 2008 14:04:31 +0300 (EEST)

 FYI,
 
 I got hit by this after upgrading a dual-CPU system (6to4 relay) from 6.3 to 
 7.0.  At the same time I enabled SMP.  I'm going to try to disable SMP but I 
 don't know if it helps.
 
 The backtrace seems somewhat similar, unfortunately I have less information 
 than you do:
 
 Tracing pid 12 tid 100003 td 0xc4d08880
 ip_output(c5434700,0,c5001804,0,0,...) at ip_output+0x15c
 stf_output(c4fd3c00,c5430a00,c08166c4,c51ced20,0,...) at stf_output+0x431
 nd6_output(c4fd3c00,c4fd3c00,c5430a00,c08166c4,c51ced20,...) at 
 nd6_output+0x70d
 ip6_forward(c5430a00,0,10,1,0,...) at ip6_forward+0x88d
 ip6_input(c5430a00,c058019e,1,93472,c4d08880,...) at ip6_input+0xd7e
 netisr_processqueue(1,e0633985,202,1000000,c4d08880,...) at 
 netisr_processqueue+0xcd
 swi_net(0,0,c0788ef0,46b,0,...) at swi_net+0xbe
 ithread_loop(c4cc58f0,e529dd38,0,0,0,...) at ithread_loop+0x1ab
 fork_exit(c056d8e0,c4cc58f0,e529dd38) at fork_exit+0x99
 fork_trampoline() at fork_trampoline+0x8
 
 -- 
 Pekka Savola                 "You each name yourselves king, yet the
 Netcore Oy                    kingdom bleeds."
 Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

From: Nick Sayer <nsayer@kfu.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/122283: [ip6] [panic] Panic in ip_output related to IPv6 routes
Date: Fri, 6 Jun 2008 10:10:56 -0700

 That stack trace looks like one of the alternate stack traces I have  
 observed. With the debugging symbols, it turns out to be in line 518  
 of if_stf.c, which says
 
 RTFREE(sc->sc_ro.ro_rt);
 
 which, once again, points back to something being pooched in the route  
 table.
 
 The full stack trace is
 
 #0  doadump () at pcpu.h:195
 #1  0xc062e277 in boot (howto=260) at /usr/src/sys/kern/ 
 kern_shutdown.c:409
 #2  0xc062e539 in panic (fmt=Variable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:563
 #3  0xc084cdfc in trap_fatal (frame=0xe683e7cc, eva=76)
      at /usr/src/sys/i386/i386/trap.c:899
 #4  0xc084d080 in trap_pfault (frame=0xe683e7cc, usermode=0, eva=76)
      at /usr/src/sys/i386/i386/trap.c:812
 #5  0xc084da2c in trap (frame=0xe683e7cc) at /usr/src/sys/i386/i386/ 
 trap.c:490
 #6  0xc0833d0b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
 #7  0xc06ca63b in stf_output (ifp=0xc3ef8800, m=0xc75bcc00,  
 dst=0xe683ea0c,
      rt=0xc3fbaca8) at /usr/src/sys/net/if_stf.c:518
 #8  0xc07776fd in nd6_output (ifp=0xc3ef8800, origifp=0xc3ef8800,
      m0=0xc6386700, dst=0xe683ea0c, rt0=0xc3fbaca8)
      at /usr/src/sys/netinet6/nd6.c:2123
 #9  0xc07749c2 in ip6_output (m0=0xc6386700, opt=0x0, ro=0xe683ea08,  
 flags=0,
      im6o=0x0, ifpp=0x0, inp=0xc49ddb40)
      at /usr/src/sys/netinet6/ip6_output.c:927
 #10 0xc0750bf1 in tcp_output (tp=0xc5fb2570)
      at /usr/src/sys/netinet/tcp_output.c:1114
 #11 0xc075af4a in tcp_usr_send (so=0xc4ae5948, flags=Variable "flags"  
 is not available.
 )
      at /usr/src/sys/netinet/tcp_usrreq.c:843
 #12 0xc0681755 in sosend_generic (so=0xc4ae5948, addr=0x0,  
 uio=0xe683ec60,
      top=0xc3fc4300, control=0x0, flags=0, td=0xc4679630)
      at /usr/src/sys/kern/uipc_socket.c:1240
 #13 0xc067d71f in sosend (so=0xc4ae5948, addr=0x0, uio=0xe683ec60,  
 top=0x0,
      control=0x0, flags=0, td=0xc4679630)
      at /usr/src/sys/kern/uipc_socket.c:1286
 #14 0xc0667d1b in soo_write (fp=0xc4add708, uio=0xe683ec60,
      active_cred=0xc4601d00, flags=0, td=0xc4679630)
      at /usr/src/sys/kern/sys_socket.c:103
 #15 0xc06613c7 in dofilewrite (td=0xc4679630, fd=1, fp=0xc4add708,
      auio=0xe683ec60, offset=-1, flags=0) at file.h:254
 #16 0xc06616a8 in kern_writev (td=0xc4679630, fd=1, auio=0xe683ec60)
      at /usr/src/sys/kern/sys_generic.c:401
 #17 0xc066171f in write (td=0xc4679630, uap=0xe683ecfc)
      at /usr/src/sys/kern/sys_generic.c:317
 #18 0xc084d3d5 in syscall (frame=0xe683ed38)
      at /usr/src/sys/i386/i386/trap.c:1035
 #19 0xc0833d70 in Xint0x80_syscall () at /usr/src/sys/i386/i386/ 
 exception.s:196
 #20 0x00000033 in ?? ()
 Previous frame inner to this frame (corrupt stack?)
 
 print *sc says
 
 $2 = {sc_ifp = 0xc3ef8800, __sc_ro46 = {__sc_ro4 = {ro_rt = 0xc3fba9d8,
        ro_dst = {sa_len = 16 '\020', sa_family = 2 '\002',
          sa_data = "\000\000G\215@\002\000\000\000\000\000\000\000"}},
      __sc_ro6 = {ro_rt = 0xc3fba9d8, ro_dst = {sin6_len = 16 '\020',
          sin6_family = 2 '\002', sin6_port = 0, sin6_flowinfo =  
 37784903,
          sin6_addr = {__u6_addr = {__u6_addr8 = '\0' <repeats 15 times>,
              __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,  
 0, 0,
                0}}}, sin6_scope_id = 0}}}, encap_cookie = 0xc3ef7c00}
 
 but print sc->sc_ro.ro_rt says that there is no member sc_ro in sc.
 
 I'll hang on to this core dump in the hopes that someone will want to  
 examine it.
 

From: Nick Sayer <nsayer@kfu.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/122283: [ip6] [panic] Panic in ip_output related to IPv6 routes
Date: Sat, 28 Jun 2008 15:53:15 -0700

 It appears that this is definitely SMP related. Setting  
 kern.smp.disabled=1 appears to keep this from happening.
 

From: Nick Sayer <nsayer@kfu.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/122283: [ip6] [panic] Panic in ip_output related to IPv6 routes
Date: Mon, 7 Jul 2008 11:39:48 -0700

 Well, I have kern.smp.disabled=1, but I had the machine panic over the  
 weekend despite that. The stack trace is, more or less, the same as  
 all the rest of them related to this problem.
 
 Perhaps it's not SMP related, then?
 

From: Pekka Savola <pekkas@netcore.fi>
To: bug-followup@freebsd.org
Cc:  
Subject: kern/122283: [ip6] [panic] Panic in ip_output related to IPv6
 routes
Date: Thu, 21 Aug 2008 11:02:42 +0300 (EEST)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --1589707168-720828604-1219305515=:23194
 Content-Type: TEXT/PLAIN; CHARSET=ISO-8859-15; format=flowed
 Content-Transfer-Encoding: 8BIT
 Content-ID: <alpine.LRH.1.10.0808211059421.23194@netcore.fi>
 
 FYI,
 
 I've just updated to a newer version of 7.0-STABLE (about Mon Aug 18 
 22:56:38 EEST 2008), and when I tried re-enabling SMP, I think I hit 
 the same, or very similar thing (the line is slightly different) 
 again:
 
 (kgdb) up 7
 #7  0xc065450f in ip_output (m=0xc551a200, opt=0x0, ro=0xc5037344, flags=0, imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:259
 259                     mtu = ro->ro_rt->rt_rmx.rmx_mtu;
 
 (kgdb) print *m
 $6 = {m_hdr = {mh_next = 0xc5514300, mh_nextpkt = 0x0, mh_data = 
 0xc551a2ec "E", mh_len = 20, mh_flags = 2, mh_type = 1, pad = "\000"},
    M_dat = {MH = {MH_pkthdr = {rcvif = 0xc4e4f800, header = 0x0, len = 
 80, csum_flags = 0, csum_data = 0, tso_segsz = 0, ether_vtag = 0,
          tags = {slh_first = 0x0}}, MH_dat = {MH_ext = {ext_buf = 
 0x1c000000 <Address 0x1c000000 out of bounds>, ext_free = 0x60,
            ext_args = 0x7f062000, ext_size = 288, ref_cnt = 0x509e3741, 
 ext_type = -1808119544},
   [[ removed MH_databuf and M_databuf here ]]
 
 Is the '<address 0x1c000000 out of bounds>' relevant here?  If not, 
 I'm not seeing anything very relevant here, except perhaps locking 
 problems.
 
 (kgdb) print *ro
 $1 = {ro_rt = 0xc51ed000, ro_dst = {sa_len = 16 '\020', sa_family = 2 '\002', sa_data = "\000\000O]\000\000\000\000\000\000\000"}}
 
 (kgdb) print *ro->ro_rt
 $3 = {rt_nodes = {{rn_mklist = 0xc4e5abf0, rn_parent = 0xc4fc1434, 
 rn_bit = -1, rn_bmask = 0 '\0', rn_flags = 4 '\004', rn_u = {
          rn_leaf = {rn_Key = 0xc4f9f960 "\020\002", rn_Mask = 
 0xc4e57800 "", rn_Dupedkey = 0x0}, rn_node = {rn_Off = -990250656,
            rn_L = 0xc4e57800, rn_R = 0x0}}}, {rn_mklist = 0x0, 
 rn_parent = 0x0, rn_bit = 0, rn_bmask = 0 '\0', rn_flags = 0 '\0', 
 rn_u = {
          rn_leaf = {rn_Key = 0x0, rn_Mask = 0x0, rn_Dupedkey = 0x0}, 
 rn_node = {rn_Off = 0, rn_L = 0x0, rn_R = 0x0}}}},
    rt_gateway = 0xc4f9f970, rt_flags = 2051, rt_ifp = 0xc4dd3400, 
 rt_ifa = 0xc506ce00, rt_rmx = {rmx_mtu = 1500, rmx_expire = 0,
      rmx_pksent = 346345}, rt_refcnt = 1, rt_genmask = 0x0, rt_llinfo = 
 0x0, rt_gwroute = 0xc51dfe88, rt_parent = 0x0, rt_fibnum = 0,
    rt_mtx = {lock_object = {lo_name = 0xc0788254 "rtentry", lo_type = 
 0xc0788254 "rtentry", lo_flags = 21168128, lo_witness_data = {
          lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 
 4, mtx_recurse = 0}}
 
 Therein is "rt_rmx = {rmx_mtu = 1500, rmx_expire = 0, rmx_pksent = 
 346345}".
 
 Also:
 
 When I disabled SMP and recompiled, I haven't hit this again.  On the 
 other hand, I've hit various other memory corruption problems on a 
 less frequent basis.
 
 ==================
 
 Fatal trap 12: page fault while in kernel mode
 cpuid = 0; apic id = 00
 fault virtual address   = 0x40
 fault code              = supervisor read, page not present
 instruction pointer     = 0x20:0xc065450f
 stack pointer           = 0x28:0xe530c9c0
 frame pointer           = 0x28:0xe530ca30
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                           = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 14 (swi1: net)
 trap number             = 12
 panic: page fault
 cpuid = 0
 Uptime: 4m24s
 Physical memory: 2039 MB
 Dumping 67 MB: 52 36 20 4
 
 #0  doadump () at pcpu.h:195
 195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
 (kgdb) bt
 #0  doadump () at pcpu.h:195
 #1  0xc058bc37 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
 #2  0xc058bef9 in panic (fmt=Variable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:572
 #3  0xc073a48c in trap_fatal (frame=0xe530c980, eva=64) at /usr/src/sys/i386/i386/trap.c:899
 #4  0xc073a710 in trap_pfault (frame=0xe530c980, usermode=0, eva=64) at /usr/src/sys/i386/i386/trap.c:812
 #5  0xc073b08c in trap (frame=0xe530c980) at /usr/src/sys/i386/i386/trap.c:490
 #6  0xc0720b1b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
 #7  0xc065450f in ip_output (m=0xc551a200, opt=0x0, ro=0xc5037344, flags=0, imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:259
 #8  0xc0628e26 in stf_output (ifp=0xc5070c00, m=0xc551a200, dst=0xc08078e4, rt=0xc51de364) at /usr/src/sys/net/if_stf.c:537
 #9  0xc068708d in nd6_output (ifp=0xc5070c00, origifp=0xc5070c00, m0=0xc5514300, dst=0xc08078e4, rt0=0xc51de364)
       at /usr/src/sys/netinet6/nd6.c:2123
 #10 0xc067c0bd in ip6_forward (m=0xc5514300, srcrt=0) at /usr/src/sys/netinet6/ip6_forward.c:605
 #11 0xc067e0ee in ip6_input (m=0xc5514300) at /usr/src/sys/netinet6/ip6_input.c:717
 #12 0xc062b87d in netisr_processqueue (ni=0xc0800d64) at /usr/src/sys/net/netisr.c:143
 #13 0xc062bb0e in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:250
 #14 0xc056c31b in ithread_loop (arg=0xc4cc58d0) at /usr/src/sys/kern/kern_intr.c:1088
 #15 0xc0568eb9 in fork_exit (callout=0xc056c160 <ithread_loop>, arg=0xc4cc58d0, frame=0xe530cd38) at /usr/src/sys/kern/kern_fork.c:781
 #16 0xc0720b90 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205
 --1589707168-720828604-1219305515=:23194--

From: Pekka Savola <pekkas@netcore.fi>
To: bug-followup@freebsd.org
Cc:  
Subject: kern/122283: [ip6] [panic] Panic in ip_output related to IPv6
 routes
Date: Thu, 21 Aug 2008 11:11:28 +0300 (EEST)

 FYI,
 
 Here's another, slightly different, crash also with SMP, which occurs 
 in the same place as Nick's first crash:
 
 (kgdb) up 7
 #7  0xc065427c in ip_output (m=0xc51ef800, opt=0x0, ro=0xc50fec84, 
 flags=0, imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:171
 171                     RTFREE(ro->ro_rt);
 
 (kgdb) list
 166              * cache with IPv6.
 167              */
 168             if (ro->ro_rt && ((ro->ro_rt->rt_flags & RTF_UP) == 0 ||
 169                               dst->sin_family != AF_INET ||
 170                               dst->sin_addr.s_addr != ip->ip_dst.s_addr)) {
 171                     RTFREE(ro->ro_rt);
 172                     ro->ro_rt = (struct rtentry *)NULL;
 173             }
 174     #ifdef IPFIREWALL_FORWARD
 175             if (ro->ro_rt == NULL && fwd_tag == NULL) {
 
 
 (kgdb) print *ro
 $1 = {ro_rt = 0x0, ro_dst = {sa_len = 16 '\020', sa_family = 2 '\002', 
 sa_data = "\000\000SYB\224\000\000\000\000\000\000\000"}}
 
 so ro->ro_rt is zero, and RTFREE is doing locking here which gives a 
 hint why SMP might be a factor here.
 
 This is a rather busy box also running Teredo relay (5-10kpps).  I get 
 hit by this crash in minutes or hours if SMP is enabled.
 
 =========================
 
 Fatal trap 12: page fault while in kernel mode
 cpuid = 0; apic id = 00
 fault virtual address   = 0x4c
 fault code              = supervisor read, page not present
 instruction pointer     = 0x20:0xc065427c
 stack pointer           = 0x28:0xe7781788
 frame pointer           = 0x28:0xe77817f8
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                           = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 929 (miredo)
 trap number             = 12
 panic: page fault
 cpuid = 0
 Uptime: 16m9s
 Physical memory: 2039 MB
 Dumping 176 MB: 161 145 129 113 97 81 65 49 33 17 1
 
 #0  doadump () at pcpu.h:195
 195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
 (kgdb) bt
 #0  doadump () at pcpu.h:195
 #1  0xc058bc37 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
 #2  0xc058bef9 in panic (fmt=Variable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:572
 #3  0xc073a48c in trap_fatal (frame=0xe7781748, eva=76)
       at /usr/src/sys/i386/i386/trap.c:899
 #4  0xc073a710 in trap_pfault (frame=0xe7781748, usermode=0, eva=76)
       at /usr/src/sys/i386/i386/trap.c:812
 #5  0xc073b08c in trap (frame=0xe7781748) at /usr/src/sys/i386/i386/trap.c:490
 #6  0xc0720b1b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
 #7  0xc065427c in ip_output (m=0xc51ef800, opt=0x0, ro=0xc50fec84, flags=0,
       imo=0x0, inp=0x0) at /usr/src/sys/netinet/ip_output.c:171
 #8  0xc0628e26 in stf_output (ifp=0xc4fd6c00, m=0xc51ef800, dst=0xe7781a00,
       rt=0xc51da8b8) at /usr/src/sys/net/if_stf.c:537
 #9  0xc068708d in nd6_output (ifp=0xc4fd6c00, origifp=0xc4fd6c00,
       m0=0xc51ef800, dst=0xe7781a00, rt0=0xc51da8b8)
       at /usr/src/sys/netinet6/nd6.c:2123
 #10 0xc0684342 in ip6_output (m0=0xc51ef800, opt=0x0, ro=0xe77819fc, flags=0,
       im6o=0x0, ifpp=0xe7781a80, inp=0xc52cb924)
       at /usr/src/sys/netinet6/ip6_output.c:944
 #11 0xc068f4cb in rip6_output (m=0xc51ef800)
       at /usr/src/sys/netinet6/raw_ip6.c:448
 #12 0xc068fad8 in rip6_send (so=0xc52d51a0, flags=0, m=0xc51ef800,
       nam=0xc5007960, control=0x0, td=0xc52ec000)
 ---Type <return> to continue, or q <return> to quit---
       at /usr/src/sys/netinet6/raw_ip6.c:790
 #13 0xc05e30a5 in sosend_generic (so=0xc52d51a0, addr=0xc5007960,
       uio=0xe7781be8, top=0xc51ef800, control=0x0, flags=0, td=0xc52ec000)
       at /usr/src/sys/kern/uipc_socket.c:1246
 #14 0xc05debbf in sosend (so=0xc52d51a0, addr=0xc5007960, uio=0xe7781be8,
       top=0x0, control=0x0, flags=0, td=0xc52ec000)
       at /usr/src/sys/kern/uipc_socket.c:1292
 #15 0xc05e5856 in kern_sendit (td=0xc52ec000, s=6, mp=0xe7781c64, flags=0,
       control=0x0, segflg=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:805
 #16 0xc05e81b2 in sendit (td=0xc52ec000, s=6, mp=0xe7781c64, flags=0)
       at /usr/src/sys/kern/uipc_syscalls.c:742
 #17 0xc05e83ef in sendto (td=0xc52ec000, uap=0xe7781cfc)
       at /usr/src/sys/kern/uipc_syscalls.c:857
 #18 0xc073aa49 in syscall (frame=0xe7781d38)
       at /usr/src/sys/i386/i386/trap.c:1035
 #19 0xc0720b80 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196
 #20 0x00000033 in ?? ()
 Previous frame inner to this frame (corrupt stack?)
 
 
 
 
 
 -- 
 Pekka Savola                 "You each name yourselves king, yet the
 Netcore Oy                    kingdom bleeds."
 Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
 
State-Changed-From-To: feedback->patched 
State-Changed-By: dwmalone 
State-Changed-When: Thu Sep 25 12:35:12 UTC 2008 
State-Changed-Why:  
I've committed some changes to -current that I think should help 
with this. I'll try to MFC them before 7.1. 

David. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=122283 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/122283: commit references a PR
Date: Thu, 25 Sep 2008 12:35:18 +0000 (UTC)

 dwmalone    2008-09-25 12:35:01 UTC
 
   FreeBSD src repository
 
   Modified files:
     sys/net              if_stf.c 
   Log:
   SVN rev 183351 on 2008-09-25 12:35:01Z by dwmalone
   
   Some people's 6to4 routers seem to have been blowing up because of
   the unlocked route caching in if_stf. Add a mutex that protects
   access to cached route. This seemed to fix problems for Pekka Savola.
   
   Nick Sayer had similar problems, and in his case completly disabling
   the route cache seemed to help. Add a sysctl net.link.stf.route_cache
   that can be used to turn off route caching in if_stf.
   
   PR:             122283
   MFC after:      2 weeks
   Tested by:      Pekka Savola, Nick Sayer.
   
   Revision  Changes    Path
   1.64      +30 -6     src/sys/net/if_stf.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: vwe 
State-Changed-When: Mon Oct 27 23:14:02 UTC 2008 
State-Changed-Why:  
patched and MFC'd to RELENG_7: rev 183893 

http://www.freebsd.org/cgi/query-pr.cgi?pr=122283 
>Unformatted:
