From lazykang@hotmail.com  Sun Mar 23 10:06:11 2003
Return-Path: <lazykang@hotmail.com>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 74C3A37B401
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 23 Mar 2003 10:06:11 -0800 (PST)
Received: from hotmail.com (f103.pav2.hotmail.com [64.4.37.103])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 058E943F85
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 23 Mar 2003 10:06:11 -0800 (PST)
	(envelope-from lazykang@hotmail.com)
Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
	 Sun, 23 Mar 2003 10:06:10 -0800
Received: from 218.246.96.6 by pv2fd.pav2.hotmail.msn.com with HTTP;
	Sun, 23 Mar 2003 18:06:10 GMT
Message-Id: <F103XXUv1MJ3G9Vbj2l000025c4@hotmail.com>
Date: Mon, 24 Mar 2003 02:06:10 +0800
From: "Kang Liu" <lazykang@hotmail.com>
To: FreeBSD-gnats-submit@freebsd.org
Subject: kernel panic on 5.0-current when use ipfw2 with limit rules

>Number:         50216
>Category:       kern
>Synopsis:       kernel panic on 5.0-current when use ipfw2 with dynamic rules
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    andre
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Mar 23 10:10:14 PST 2003
>Closed-Date:    Mon Dec 22 12:09:56 PST 2003
>Last-Modified:  Mon Dec 22 12:10:21 PST 2003
>Originator:     Kang Liu
>Release:        FreeBSD 5.0-CURRENT i386
>Organization:
Beijing University of Technology
>Environment:
System: FreeBSD cnproxy.bjpu.edu.cn 5.0-CURRENT FreeBSD 5.0-CURRENT #2: Sun 
Mar 23 21:35:41 CST 2003 
root@cnproxy.bjpu.edu.cn:/usr/obj/usr/src/sys/CNPROXY i386
DELL poweredge2650 CPU: 2*Intel(R) Xeon(TM) CPU 2.00GHz (1993.54-MHz 
686-class CPU) (SMP and HyperThreading are both enabled in kenrel 
configuration file)
>Description:
  I tried to use ipfw2 with dynamic rules by commands shown below:
ipfw add allow tcp from any to any established
ipfw add allow tcp from 192.168.0.0/16 to server_ip some_ports limit 
src-addr 20 setup
ipfw add allow udp from 192.168.0.0/16 to server_ip some_ports
ipfw add allow tcp from some_ip to server_ip some_ports limit src-addr 80 
setup
... and so on
The kernel will panic immediately while network connection is active. If I 
use static rules instead of those dynamic rules or disable network 
connection by use "ifconfig bge0 down", nothing happens.
I've add "options DDB" and some other debug options into my kernel configure 
file, I get the following message when kernel panic:
----Start of message---
Memory modified after free 0xc9471f0 (124)
panic: Most recently used by IpFw/IpAcct
cpuid=3;lapic.id=0300000
Stack bactrace:
backtrace(c0349879,3000000,C035a79a,e231fabe,1) at backtrace+0x17
panic(c035a79a,c03524d7,7c,c082ab64,c082ab40) at panic+0x10a
mtrash_ctor(c9471f00,80,0,54d,3) at mtrash_ctor+0x5d
uma_zalloc_arg(c082ab40,0,101,c034bc1,1be) at uma_zalloc_arg+0x17f
malloc(48,c037ef20,101,2700001b,e231fc70) at malloc+0xdc
add_dyn_rule(e231fc90,27,c9472180,c9472180,0)at add_dyn_rule+0x7b
install_state(c8fdd70,c8fdd764,e231fc70,e231fbf4,c01f1954)at 
install_state+0x1fd
ipfw_chk(e231fc70,c01cc3ad,c03a96a0,1,c0348bc1)at ipfw_chk+0x9a1
ip_input(c3b52000,0,c0351940,e9,c8b6a240)at ip_input+0x2c3
swi_net(0,0,c0347569,217,c3b18000)at swi_net+0x112
ithread_loop(c3b17080,e231fd48,c0347300,363,0)at ithread_loop+0x182
fork_exit(c01c3190,c3b17080,e231fd48)at fork_exit+0xc4
fork_trampoline()at fork_trampoline+0x1a
--trap 0x1,eip=0,esp=0xe231fd7c,ebp=0--
Debygger("panic")
Stopped at Debugger+0xff:xchgl %ebx,in_Debugger,0
----End of Message---
I copy the message above from screen,I'm not sure whether I've typed it 
exactly as displayed on the screen.I hope it is helpful.
>How-To-Repeat:
use dynamic rules with ipfw2.
(I do not have a machine with only a single CPU to run freebsd-current, the 
problem might be related to SMP or HTT).
>Fix:
Sorry, I can not give and patch now.
The only way I found to get rid of this problem is run ipfw2 with static 
rules instead of dynamic rules.

_________________________________________________________________
The new MSN 8: smart spam protection and 2 months FREE*  
http://join.msn.com/?page=features/junkmail

>Release-Note:
>Audit-Trail:

From: "Kang Liu" <lazykang@hotmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with dynamic rules
Date: Mon, 24 Mar 2003 10:39:02 +0800

 There is the same problem in stable-branch. I've tested ipfw2 with dynamic 
 rules on the latest 4.8-RC,while ipfw(I mean ipfw-1) works on well with 
 dynamic rule.
 
 _________________________________________________________________
 Add photos to your messages with MSN 8. Get 2 months FREE*. 
 http://join.msn.com/?page=features/featuredemail
 

From: Coleman Kane <cokane@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org, lazyking@hotmail.com
Cc:  
Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with dynamic rules
Date: Mon, 24 Mar 2003 12:52:04 -0500

 I have experienced this problem as well, on three 5.0-R boxes (one an
 alpha). It seems to manifest itself when using limit rules in the
 firewall. I can successfully use keep-state/setup rules on all of
 these machines. I have not had time to look into the source due to
 work/school constraints. I figured I would put in my experience to
 help out luigi or whomever else decides to hash this out.
 
 --
 coleman kane

From: Coleman Kane <cokane@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org, lazyking@hotmail.com
Cc:  
Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with dynamic rules
Date: Mon, 24 Mar 2003 13:58:46 -0500

 More info:
 Using the following rule:
 allow tcp from any to me port 25 limit src-addr 10
 
 The system will panic as stated, though having multiple clients hammer
 the smtp port will cause this panic in about 5 minutes or so. Only
 connecting repeatedly from one client doesn't seem to have this effect.
 
 On Mon, Mar 24, 2003 at 12:52:04PM -0500, Coleman Kane wrote:
 > I have experienced this problem as well, on three 5.0-R boxes (one an
 > alpha). It seems to manifest itself when using limit rules in the
 > firewall. I can successfully use keep-state/setup rules on all of
 > these machines. I have not had time to look into the source due to
 > work/school constraints. I figured I would put in my experience to
 > help out luigi or whomever else decides to hash this out.
 > 
 > --
 > coleman kane

From: Maxim Konovalov <maxim@macomnet.ru>
To: Coleman Kane <cokane@FreeBSD.ORG>
Cc: bug-followup@FreeBSD.ORG, lazykang@hotmail.com
Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with
 dynamic rules
Date: Mon, 24 Mar 2003 22:10:24 +0300 (MSK)

 On 11:00-0800, Mar 24, 2003, Coleman Kane wrote:
 
 > The following reply was made to PR kern/50216; it has been noted by GNATS.
 >
 > From: Coleman Kane <cokane@FreeBSD.org>
 > To: freebsd-gnats-submit@FreeBSD.org, lazyking@hotmail.com
 > Cc:
 > Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with dynamic rules
 > Date: Mon, 24 Mar 2003 13:58:46 -0500
 >
 >  More info:
 >  Using the following rule:
 >  allow tcp from any to me port 25 limit src-addr 10
 
 Could you please show the whole ipfw ruleset?  Is your box UP or MP?
 Can you obtain a kernel coredump?
Responsible-Changed-From-To: freebsd-bugs->ipfw 
Responsible-Changed-By: johan 
Responsible-Changed-When: Tue May 6 12:35:46 PDT 2003 
Responsible-Changed-Why:  
Over to maintainer group. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=50216 

From: "Kang Liu" <liukang@bjpu.edu.cn>
To: <freebsd-gnats-submit@FreeBSD.org>
Cc: <freebsd-ipfw@freebsd.org>, <freebsd-hackers@freebsd.org>,
	"'Xin LI/??'" <delphij@frontfree.net>, <maxim@macomnet.ru>,
	<cokane@FreeBSD.ORG>
Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with dynamic rules
Date: Wed, 1 Oct 2003 00:47:17 +0800

 I reproduced it on the latest 5.1current.
 
 Here is the backtrace:
 #####
 GNU gdb 5.2.1 (FreeBSD)
 Copyright 2002 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-undermydesk-freebsd"...
 panic: Most recently used by cred
 
 panic messages:
 ---
 panic: Most recently used by cred
 
 Stack backtrace:
 
 syncing disks, buffers remaining... 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 628 
 giving up on 520 buffers
 Uptime: 16m0s
 Dumping 255 MB
  16 32 48 64 80 96 112 128 144 160 176 192 208 224 240
 ---
 Reading symbols from /usr/obj/usr/src/sys/IPFW/modules/usr/src/sys/modules/acpi/acpi.ko.debug...done.
 Loaded symbols for /usr/obj/usr/src/sys/IPFW/modules/usr/src/sys/modules/acpi/acpi.ko.debug
 Reading symbols from /boot/kernel/daemon_saver.ko...done.
 Loaded symbols for /boot/kernel/daemon_saver.ko
 #0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
 240		dumping++;
 (kgdb) bt
 #0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
 #1  0xc01a0221 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:372
 #2  0xc01a05b7 in panic () at /usr/src/sys/kern/kern_shutdown.c:550
 #3  0xc02817f7 in mtrash_ctor (mem=0xc29f8a80, size=0, arg=0x0)
     at /usr/src/sys/vm/uma_dbg.c:137
 #4  0xc028002e in uma_zalloc_arg (zone=0xc083ab40, udata=0x0, flags=257)
     at /usr/src/sys/vm/uma_core.c:1413
 #5  0xc0194a23 in malloc (size=3229854528, type=0xc03020c0, flags=257)
     at /usr/src/sys/vm/uma.h:234
 #6  0xc021e03f in add_dyn_rule (id=0xcd7bfc90, dyn_type=39 '\'', 
     rule=0xc2815e00) at /usr/src/sys/netinet/ip_fw2.c:976
 #7  0xc021e43e in install_state (rule=0xc28f3a80, cmd=0xc28f3ac0, 
     args=0xcd7bfc70) at /usr/src/sys/netinet/ip_fw2.c:1140
 #8  0xc021f4dc in ipfw_chk (args=0xcd7bfc70)
     at /usr/src/sys/netinet/ip_fw2.c:1942
 #9  0xc0221dd7 in ip_input (m=0xc0ed9800)
     at /usr/src/sys/netinet/ip_input.c:489
 #10 0xc0211a82 in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:236
 #11 0xc018c762 in ithread_loop (arg=0xc0ebec80)
     at /usr/src/sys/kern/kern_intr.c:534
 #12 0xc018b76f in fork_exit (callout=0xc018c5e0 <ithread_loop>, arg=0x0, 
     frame=0x0) at /usr/src/sys/kern/kern_fork.c:796
 (kgdb) 
 
 #####
 Here is my full ipfw rule set script:
 # cat ./ipfwpanic.sh 
 dumpon -v /dev/ad0s1b
 /sbin/ipfw add allow tcp from any to any established
 /sbin/ipfw add allow ip from a.b.c.0 to a.b.c.d
 /sbin/ipfw add allow tcp from any to a.b.c.d 80 limit src-addr 20 setup
 /sbin/ipfw add allow ip from a.b.c.d to any
 
 And I added "IPFIREWALL_DEFAULT_TO_ACCEPT" into kernel configure file.
 
 #####
 Here is my test script. I installed an apache on that machine, and use ab to connect 80 port.
 cat panicstart.sh 
 #!/bin/sh
 number=0
 while (test $number -lt 10000)
 do
 echo "$number"
 ab -c 100 http://a.b.c.d/
 number=`expr $number + 1`
 done
 
 #####
 This problem can be reproduced on both MP and UP machine.
 I've tested it on a dell poweredge2650(with 2 P4xeon, HTT enabled/disabled) and a DIY PC(1 PIII CPU).
 The backtrace I post above is produced on PC(1CPU).
 

From: Sam Leffler <sam@errno.com>
To: "Kang Liu" <liukang@bjpu.edu.cn>,
	<freebsd-gnats-submit@freebsd.org>
Cc: freebsd-ipfw@freebsd.org
Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with dynamic rules
Date: Tue, 30 Sep 2003 10:28:19 -0700

 On Tuesday 30 September 2003 09:47 am, Kang Liu wrote:
 > I reproduced it on the latest 5.1current.
 
 Thanks.  I'll look at it.
 
 	Sam
 

From: "Kang Liu" <liukang@bjpu.edu.cn>
To: <freebsd-gnats-submit@FreeBSD.org>
Cc:  
Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with dynamic rules
Date: Wed, 29 Oct 2003 23:24:12 +0800

 it seems that the stable branch does not affected by this problem after
 Mckusick's commit. :-) 
 (src/sys/netinet/ip_fw2.c Revision 1.6.2.18)
 testing 5.1current...
 
State-Changed-From-To: open->closed 
State-Changed-By: andre 
State-Changed-When: Mon Dec 22 12:08:41 PST 2003 
State-Changed-Why:  
Revision 1.40 of netinet/ip_fw2.c fixes this for -CURRENT too. Closing case. 


Responsible-Changed-From-To: ipfw->andre 
Responsible-Changed-By: andre 
Responsible-Changed-When: Mon Dec 22 12:08:41 PST 2003 
Responsible-Changed-Why:  
Revision 1.40 of netinet/ip_fw2.c fixes this for -CURRENT too. Closing case. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=50216 

From: Andre Oppermann <andre@freebsd.org>
To: freebsd-gnats-submit@FreeBSD.org, lazykang@hotmail.com
Cc:  
Subject: Re: kern/50216: kernel panic on 5.0-current when use ipfw2 with dynamic
 rules
Date: Mon, 22 Dec 2003 21:02:58 +0100

 This bug has been fixed in -CURRENT netinet/ip_fw2.c rev. 1.40 too.
 
 -- 
 Andre
>Unformatted:
 >rules
