From nobody@FreeBSD.org  Mon Aug 23 20:03:06 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 9BA66106566C
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 23 Aug 2010 20:03:06 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 8AB668FC14
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 23 Aug 2010 20:03:06 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o7NK35Pr046354
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 23 Aug 2010 20:03:05 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o7NK35Du046353;
	Mon, 23 Aug 2010 20:03:05 GMT
	(envelope-from nobody)
Message-Id: <201008232003.o7NK35Du046353@www.freebsd.org>
Date: Mon, 23 Aug 2010 20:03:05 GMT
From: Ingo Flaschberger <if@xip.at>
To: freebsd-gnats-submit@FreeBSD.org
Subject: freebsd 8.1 crash with ECMP
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         149917
>Category:       kern
>Synopsis:       [net] [patch] freebsd 8.1 crash with ECMP
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    qingli
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 23 20:10:01 UTC 2010
>Closed-Date:    
>Last-Modified:  Mon Sep  6 12:00:17 UTC 2010
>Originator:     Ingo Flaschberger
>Release:        8.1 stable
>Organization:
crossip communications gmbh
>Environment:
FreeBSD  8.1-STABLE FreeBSD 8.1-STABLE #16: Mon Aug 23 19:51:34 UTC 2010     root@jail.xip.at:/usr/obj/usr/src/sys/ROUTER  amd64
>Description:
system crash when adding 2nd route with other weight

Dump:
#0  doadump () at pcpu.h:224
224             __asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:224
#1  0xffffffff802b0e25 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
#2  0xffffffff802b126d in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:590
#3  0xffffffff80464b65 in trap_fatal (frame=0xffffff00017ed460, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:777
#4  0xffffffff80464f06 in trap_pfault (frame=0xffffff803e6394a0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:693
#5  0xffffffff8046553c in trap (frame=0xffffff803e6394a0) at /usr/src/sys/amd64/amd64/trap.c:451
#6  0xffffffff8044b984 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224
#7  0xffffffff80363cd3 in rtalloc_mpath_fib (ro=0xffffff803e639620, hash=192947925, fibnum=Variable "fibnum" is not available.
) at /usr/src/sys/net/radix_mpath.c:307
#8  0xffffffff8035016f in flowtable_lookup (ft=0xffffff80005a7000, ssa=0xffffff803e6396a0, dsa=0xffffff803e639720, fibnum=0, flags=2) at /usr/src/sys/net/flowtable.c:1229
#9  0xffffffff80350a67 in flowtable_lookup_mbuf (ft=0xffffff80005a7000, m=0xffffff00014e2100, af=Variable "af" is not available.
) at /usr/src/sys/net/flowtable.c:607
#10 0xffffffff80392f69 in ip_output (m=0xffffff00014e2100, opt=0x0, ro=0x0, flags=32, imo=0x0, inp=0xffffff00167c5a80) at /usr/src/sys/netinet/ip_output.c:164
#11 0xffffffff80393f07 in rip_output (m=0xffffff00014e2100, so=Variable "so" is not available.
) at /usr/src/sys/netinet/raw_ip.c:507
#12 0xffffffff80314b70 in sosend_generic (so=0xffffff00166e3550, addr=0xffffff00167b6160, uio=0xffffff803e639a40, top=0xffffff00014e2100, control=0x0, flags=0, td=0xffffff00164303e0)
    at /usr/src/sys/kern/uipc_socket.c:1260
#13 0xffffffff80316878 in kern_sendit (td=0xffffff00164303e0, s=3, mp=0xffffff803e639b10, flags=0, control=0x0, segflg=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:788
#14 0xffffffff80316aa8 in sendit (td=0xffffff00164303e0, s=3, mp=0xffffff803e639b10, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:724
#15 0xffffffff80316b90 in sendto (td=Variable "td" is not available.
) at /usr/src/sys/kern/uipc_syscalls.c:840
#16 0xffffffff804650f5 in syscall (frame=0xffffff803e639c80) at /usr/src/sys/amd64/amd64/trap.c:945
#17 0xffffffff8044bc62 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:377
#18 0x0000000800940e5c in ?? ()
>How-To-Repeat:
ifconfig em0 10.11.11.175/24
ifconfig em0 alias 10.20.20.1/24
route add -net 10.20.20.0/24 10.11.11.1 -weight 2

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x98
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80363cd3
stack pointer           = 0x28:0xffffff803e5da550
frame pointer           = 0x28:0xffffff803e5da580
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1767 (ping)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 3m46s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
>Fix:


Patch attached with submission follows:

--- /usr/src/sys/net/radix_mpath.c_org	2010-08-23 16:36:57.000000000 +0000
+++ /usr/src/sys/net/radix_mpath.c	2010-08-23 16:38:18.000000000 +0000
@@ -294,6 +294,9 @@
 	     weight >= rt->rt_rmx.rmx_weight && rn; 
 	     weight -= rt->rt_rmx.rmx_weight) {
 		
+		/* check if next key exists */
+		if (rn->rn_dupedkey == NULL)
+			break;
 		/* stay within the multipath routes */
 		if (rn->rn_dupedkey && rn->rn_mask != rn->rn_dupedkey->rn_mask)
 			break;


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Aug 24 04:13:05 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=149917 
Responsible-Changed-From-To: freebsd-net->qingli 
Responsible-Changed-By: andre 
Responsible-Changed-When: Tue Aug 24 08:25:53 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=149917 

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Ingo Flaschberger <if@xip.at>
Cc: bug-followup@FreeBSD.org
Subject: kern/149917
Date: Tue, 31 Aug 2010 15:49:19 +0400

   I can't reproduce this on 9.0-CURRENT, where radix_mpath
 code is almost the same as in 8.
 
   Can you please provide more information: full ifconfig output,
 more data in routing table, may be a more precise reproduction
 recipe?
 
 -- 
 Totus tuus, Glebius.

From: Ingo Flaschberger <if@xip.at>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/149917: freebsd 8.1 crash with ECMP
Date: Mon, 6 Sep 2010 13:27:40 +0200 (CEST)

 Try this:
 sysctl -w net.inet.flowtable.enable=0
 ifconfig em0 10.11.11.175/24
 ifconfig em0 alias 10.20.20.1/24
 route add -net 10.20.20.0/24 10.11.11.1 -weight 2
 ping -c1 10.20.20.100
 
 Kind regards,
  	Ingo Flaschberger
 
>Unformatted:
