From nobody@FreeBSD.org  Sun Feb 21 19:01:15 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 09AF9106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 21 Feb 2010 19:01:14 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id B66F98FC14
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 21 Feb 2010 19:01:14 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o1LJ1EVd047221
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 21 Feb 2010 19:01:14 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o1LJ1EZW047220;
	Sun, 21 Feb 2010 19:01:14 GMT
	(envelope-from nobody)
Message-Id: <201002211901.o1LJ1EZW047220@www.freebsd.org>
Date: Sun, 21 Feb 2010 19:01:14 GMT
From: Dmitriy Demidov <dima_bsd@inbox.lv>
To: freebsd-gnats-submit@FreeBSD.org
Subject: deadlock using multiple ipfw nat and multiple limit statements
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         144187
>Category:       kern
>Synopsis:       [ipfw] deadlock using multiple ipfw nat and multiple limit statements
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-ipfw
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Feb 21 19:10:01 UTC 2010
>Closed-Date:    Sat Jul 02 19:43:50 UTC 2011
>Last-Modified:  Sat Jul 02 19:43:50 UTC 2011
>Originator:     Dmitriy Demidov
>Release:        7.3-PRERELEASE
>Organization:
>Environment:
FreeBSD evo.local.home 7.3-PRERELEASE FreeBSD 7.3-PRERELEASE #0: Sat Feb 20 22:57:09 EET 2010     root@evo.local.home:/usr/obj/usr/src/sys/STABLE  i386
>Description:
I met one problem with multiple ipfw nat instances and multiple limit rules. If I configure my ipfw as showed below and make a try to connect to 192.168.1.2 from 192.168.1.1 using ssh, then system just die without any error messages or panics. It do not reboots but just stay in deadlock forever - I am unable to use console or ssh/ping it anymore, and only way to get it back is a reset or power buttons.

Problem disappears only then I remove "limit src-addr 80" from first, or from second nat instances.

System is a nat router what have 3 network interfaces - fxp0 is connected to inner LAN (172.16.1.0/24) and each of rl adapters is connected to different upstreem ISP. There is no any setfib at this moment. Default gateway (192.168.1.1) is on rl0 side.


Customized GENERIC kernel:
===
#cpu            I486_CPU
#cpu            I586_CPU
cpu             I686_CPU
ident           STABLE

options         IPFIREWALL
options         IPFIREWALL_NAT
options         LIBALIAS
options         ROUTETABLES=2
options         DUMMYNET
options         HZ=1000

makeoptions     DEBUG=-g

options         KDB
options         DDB
options         GDB
options         INVARIANTS
options         INVARIANT_SUPPORT
options         WITNESS
options         DEBUG_LOCKS
options         DEBUG_VFS_LOCKS
options         DIAGNOSTIC
##
===

sysctl:
===
sysctl -a | grep one_pass
net.inet.ip.fw.one_pass: 0
===

ipfw configuration:
===
add 1040 allow ip from any to any via fxp0

nat 1 config if rl0 reset same_ports deny_in redirect_port tcp 192.168.1.2:22 22
nat 2 config if rl1 reset same_ports deny_in

add 10130 nat 1 tcp from any to any out xmit rl0 limit src-addr 80
add 10131 allow tcp from any to any out xmit rl0
add 10140 nat 1 ip from any to any out xmit rl0
add 10141 allow ip from any to any out xmit rl0

add 20130 nat 2 tcp from any to any out xmit rl1 limit src-addr 80
add 20131 allow tcp from any to any out xmit rl1
add 20140 nat 2 ip from any to any out xmit rl1
add 20141 allow ip from any to any out xmit rl1

add 20150 nat 1 ip from any to any in recv rl0
add 20151 nat 2 ip from any to any in recv rl1

add 30160 allow ip from any to any
add 65534 deny ip from any to any
===

ipfw show:
===
00100 0  0 allow ip from any to any via lo0
00200 0  0 deny ip from any to 127.0.0.0/8
00300 0  0 deny ip from 127.0.0.0/8 to any
01040 0  0 allow ip from any to any via fxp0
10130 1 40 nat 1 tcp from any to any out xmit rl0 limit src-addr 80
10131 1 40 allow tcp from any to any out xmit rl0
10140 0  0 nat 1 ip from any to any out xmit rl0
10141 0  0 allow ip from any to any out xmit rl0
20130 0  0 nat 2 tcp from any to any out xmit rl1 limit src-addr 80
20131 0  0 allow tcp from any to any out xmit rl1
20140 0  0 nat 2 ip from any to any out xmit rl1
20141 0  0 allow ip from any to any out xmit rl1
20150 1 52 nat 1 ip from any to any in recv rl0
20151 0  0 nat 2 ip from any to any in recv rl1
30160 1 52 allow ip from any to any
65534 0  0 deny ip from any to any
65535 0  0 deny ip from any to any
===

ifconfig:
===
evo# ifconfig
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:30:4f:11:11:11
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2009<RXCSUM,VLAN_MTU,WOL_MAGIC>
        ether 00:0b:cd:22:22:22
        inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255
        media: Ethernet autoselect (none)
        status: no carrier
rl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:0c:46:33:33:33
        inet 192.168.2.2 netmask 0xffffff00 broadcast 192.168.2.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000

===

===
evo# setfib 0 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.1.1        UGS         0        0    rl0
127.0.0.1          127.0.0.1          UH          0        0    lo0
172.16.1.0/24      link#2             UC          0        0   fxp0
192.168.1.0/24     link#1             UC          0        0    rl0
192.168.1.1        00:90:27:2c:12:12  UHLW        1       20    rl0   1193
192.168.2.0/24     link#3             UC          0        0    rl1

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UHL         lo0
fe80::%lo0/64                     fe80::1%lo0                   U           lo0
fe80::1%lo0                       link#5                        UHL         lo0
ff01:5::/32                       fe80::1%lo0                   UC          lo0
ff02::%lo0/32                     fe80::1%lo0                   UC          lo0

evo# setfib 1 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
127.0.0.1          127.0.0.1          UH          0        0    lo0
172.16.1.0/24      link#2             UC          0        0   fxp0
192.168.1.0/24     link#1             UC          0        0    rl0
192.168.1.1        00:90:27:2c:12:12  UHLW        1        0    rl0   1190
192.168.2.0/24     link#3             UC          0        0    rl1

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UHL         lo0
fe80::%lo0/64                     fe80::1%lo0                   U           lo0
fe80::1%lo0                       link#5                        UHL         lo0
ff01:5::/32                       fe80::1%lo0                   UC          lo0
ff02::%lo0/32                     fe80::1%lo0                   UC          lo0

===
>How-To-Repeat:
Configure kernel options and ipfw rules as it mentioned in Description and try to connect to system via ssh (or just telnet to port 22).
>Fix:
Do not use multiple limit statements.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-ipfw 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Feb 22 05:28:06 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144187 
State-Changed-From-To: open->feedback 
State-Changed-By: ae 
State-Changed-When: Tue Jun 28 05:29:45 UTC 2011 
State-Changed-Why:  
Can you still reproduce this on a supported release?  

Or maybe you can test your rules on head/ branch? 
There were some changes related to ipfw_nat. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144187 
State-Changed-From-To: feedback->closed 
State-Changed-By: ae 
State-Changed-When: Sat Jul 2 19:42:05 UTC 2011 
State-Changed-Why:  
The submitter has reported that the problem is already fixed. Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144187 
>Unformatted:
