From nobody@FreeBSD.org  Wed Oct 24 04:11:50 2001
Return-Path: <nobody@FreeBSD.org>
Received: from freefall.freebsd.org (freefall.FreeBSD.org [216.136.204.21])
	by hub.freebsd.org (Postfix) with ESMTP id 212F837B405
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 24 Oct 2001 04:11:50 -0700 (PDT)
Received: (from nobody@localhost)
	by freefall.freebsd.org (8.11.4/8.11.4) id f9OBBoY72859;
	Wed, 24 Oct 2001 04:11:50 -0700 (PDT)
	(envelope-from nobody)
Message-Id: <200110241111.f9OBBoY72859@freefall.freebsd.org>
Date: Wed, 24 Oct 2001 04:11:50 -0700 (PDT)
From: Ivan Mikhnevich <ivan@interVelopers.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Specific IPFW's FWD rule crashes the kernel (panic fatal double fault)
X-Send-Pr-Version: www-1.0

>Number:         31471
>Category:       kern
>Synopsis:       Specific IPFW's FWD rule crashes the kernel (panic fatal double fault)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 24 04:20:01 PDT 2001
>Closed-Date:    Sun Aug 25 20:30:55 PDT 2002
>Last-Modified:  Sun Aug 25 20:30:55 PDT 2002
>Originator:     Ivan Mikhnevich
>Release:        4.3
>Organization:
interVelopers.com
>Environment:
FreeBSD dbaol.com 4.3-RELEASE FreeBSD 4.3-RELEASE #0: Wed Oct 10 00:32:23 EEST 2001     root@dbaol.com:/usr/src/sys/compile/DBAOL  i386

>Description:
This report is linked to Problem Report kern/31147 (http://www.freebsd.org/cgi/query-pr.cgi?pr=31147).

The problem is in frequent kernel panic (fatal double fault). It occurs every 2 days on the average. It happens in some network functions, but most frequently in ip_output(). Moreover, kernel overwlows the stack in ip_output() at the command "push %edi":
c01f46b0 <ip_output> 55                   	push   %ebp
c01f46b1 <ip_output+0x1> 89 e5                	mov    %esp,%ebp
c01f46b3 <ip_output+0x3> 83 ec 48             	sub    $0x48,%esp
c01f46b6 <ip_output+0x6> 57                   	push   %edi
c01f46b7 <ip_output+0x7> 56                   	push   %esi
c01f46b8 <ip_output+0x8> 53                   	push   %ebx

>How-To-Repeat:
FreeBSD 4.1-RELEASE or 4.3-RELEASE with kernel that differs from GENERIC only by the following options:
options 	IPFIREWALL
options 	IPFIREWALL_FORWARD
options 	IPFIREWALL_VERBOSE_LIMIT=100
options 	IPFIREWALL_DEFAULT_TO_ACCEPT
options 	IPDIVERT
options 	IPFILTER
options 	IPFILTER_LOG
options 	IPSTEALTH

/etc/firewall.rules
add deny icmp from any to any frag
add pass icmp from any to any
add pass udp from any to any 53,161,514
add pass udp from any 53,161,514 to any
add fwd 216.55.6.182,8080 tcp from any to 216.55.15.17 80
add fwd 216.55.6.182,25 tcp from any to any 2525
add pass tcp from any to any smtp,http,ftp,ftp\-data,pop3,https,telnet,ssh
add pass tcp from any smtp,http,ftp,ftp\-data,pop3,https,telnet,ssh to any
add pass tcp from any to any 2525,3128,3514,8080,40202
add pass tcp from any 2525,3128,3514,8080,40202 to any
add pass all from any to any via lo0
add deny all from any to 127.0.0.0/8
add deny tcp from any to any 3306 via fxp0
add 65000 deny all from any to any

>Fix:
The problem is with the rule:
add fwd 216.55.6.182,8080 tcp from any to 216.55.15.17 80
Since the follwing change the server has been running OK (over 8 days already and still running OK). The above line was changed to this one so that both IP adresses are equal:
add fwd 216.55.15.17,8080 tcp from any to 216.55.15.17 80

So, the problem is with IPFW's FWD rules which forward packets from one port of one IP address to another port of another IP address, but both IP addresses are on the same machine. It happens when both IPs are bound to a single Network Card, but I suspect that the problem would occur even if the IP addresses were bound to 2 different Network Cards in the same server.

To avoid such problem I suggest that the following IPFW rules should not be used on FreeBSD:
fwd IP_address_1,port_1 tcp from any to IP_address_2 port_2
where both IP adresses are on the same machine, especially when they are IP aliases.

Also I recommend FreeBSD developers to revise the sources of ip_output() in netinet/ip_output.c and div_output() in netintet/ip_divert.c functions. Please, pay attention to loopback avoidance (ip_divert_ignore variable) because the kernel panics due to endless recursion in ip_output() function, which is called from div_output() that is run when this very specific IPFW rule is used.
>Release-Note:
>Audit-Trail:

From: "Yar Tikhiy" <yar@FreeBSD.org>
To: <freebsd-gnats-submit@FreeBSD.org>, <ivan@interVelopers.com>
Cc:  
Subject: Re: kern/31471: Specific IPFW's FWD rule crashes the kernel (panic fatal double fault)
Date: Wed, 19 Dec 2001 17:34:51 +0300

 Ivan,
 
 First of all, do you realize that IPFIREWALL and IPFILTER are alternative
 packet filter facilities?
 
 Second, would you mind following the FreeBSD Developers Handbook
 instructions on how to inspect kernel crashes with ``gdb'':
 
 http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneld
 ebug.html
 
 and sending a post-mortem kernel stack trace?  Without it, it's rather hard
 to guess a sequence of calls that leads to the particular crash.  Thank you.
 
 --Yar
 
State-Changed-From-To: open->feedback 
State-Changed-By: sheldonh 
State-Changed-When: Sun Dec 30 06:02:14 PST 2001 
State-Changed-Why:  
Crashdump post mortem requested. 


http://www.FreeBSD.org/cgi/query-pr.cgi?pr=31471 
State-Changed-From-To: feedback->closed 
State-Changed-By: keramida 
State-Changed-When: Sun Aug 25 20:30:45 PDT 2002 
State-Changed-Why:  
Feedback timeout. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=31471 
>Unformatted:
