From nobody@FreeBSD.org  Sun Jun 12 16:09:47 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 208BA1065673
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 12 Jun 2011 16:09:47 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 105398FC13
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 12 Jun 2011 16:09:47 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p5CG9khN035167
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 12 Jun 2011 16:09:46 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p5CG9kjW035166;
	Sun, 12 Jun 2011 16:09:46 GMT
	(envelope-from nobody)
Message-Id: <201106121609.p5CG9kjW035166@red.freebsd.org>
Date: Sun, 12 Jun 2011 16:09:46 GMT
From: Ozkan KIRIK <ozkan.kirik@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: IPFW in-kernel NAT nat loopback / Default Router Changes Unexpectedly
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         157796
>Category:       kern
>Synopsis:       [ipfw] IPFW in-kernel NAT nat loopback / Default Router Changes Unexpectedly
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    rrs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jun 12 16:10:14 UTC 2011
>Closed-Date:    Mon Jun 24 09:50:42 UTC 2013
>Last-Modified:  Mon Jun 24 09:50:42 UTC 2013
>Originator:     Ozkan KIRIK
>Release:        FreeBSD 8.2-RELEASE
>Organization:
Mersin University @ Turkey
>Environment:
FreeBSD ipfw.ipfw 8.2-RELEASE FreeBSD 8.2-RELEASE #9: Wed Mar  2 11:40:32 EET 2011     root@ipfw.ipfw:/usr/obj/usr/src/sys/ipfw amd64
>Description:
When a packet injects into ipfw in-kernel nat more then once,
operating systems behave abnormally.

Let's inspect the ruleset below:

ipfw table 12 add 10.0.14.1/32 33
ipfw table 13 add 193.X.128.30/32 33

ipfw nat 33 config redirect_addr 10.0.14.1 193.X.128.30
ipfw nat 799 config ip 3.3.3.3 reverse

55000 nat tablearg ip from table(12) to not 3.3.3.3 via em3
55000 nat tablearg ip from any to table(13) via em3
55000 nat 799 ip from any to table(13) not via em3
55000 nat tablearg ip from 3.3.3.3 to table(13)
55000 nat tablearg ip from table(12) to 3.3.3.3
55000 nat 799 ip from table(13) to 3.3.3.3


This ruleset, makes static nat.
To access from a client ( i.e. 10.0.14.5 ) to 193.X.128.30, 
I decided to make source address translation to incoming requests.

All incoming static nat requests is to be exposed to reverse nat. 
( source address translation as 3.3.3.3 )
All of this ruleset works, but under 5-6 Mbps static nat traffic load
( and total throughput about 150-200 Mbps ), operating system's
default router changes unexpectedly. When i try to monitor changes by "route -n monitor" command, I cannot see any events about default gateway.

New default gateway shown as the local ip address of static nat ( 10.0.14.1 ).
If you apply bandwidth limiting with dummynet, this problem occurs
more frequently. Dummynet catalyze the problem.

when i remove the reverse nat rules as :
55000 nat tablearg ip from table(12) to any
55000 nat tablearg ip from any to table(13)

everything is fine. ( default gateway doesnt change. ). Altough
dummynet is active, problem doesnt seen.
I think there is no problem with dummynet.

I tried different rulesets for different aims, and then i understood
that if you inject packets into in-kernel nat more then once system
behaves unexpectedly.


Another example attached below:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
       ether 00:23:8b:89:e4:9c
       inet6 fe80::223:8bff:fe89:e49c%em0 prefixlen 64 scopeid 0x1
       inet 193.X.128.1 netmask 0xffffff00 broadcast 193.X.128.255
       inet 193.X.128.4 netmask 0xffffffff broadcast 193.X.128.4
       nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
       ether 00:23:8b:89:e4:9d
       inet6 fe80::223:8bff:fe89:e49d%em1 prefixlen 64 scopeid 0x2
       inet 192.168.254.254 netmask 0xffffff00 broadcast 192.168.254.255
       nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
       ether 00:23:8b:89:e4:9e
       inet6 fe80::223:8bff:fe89:e49e%em2 prefixlen 64 scopeid 0x4
       inet 10.200.202.254 netmask 0xffffff00 broadcast 10.200.202.255
       inet 1.1.184.254 netmask 0xffffff00 broadcast 1.1.184.255
       nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

bce0.14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=103<RXCSUM,TXCSUM,TSO4>
       ether 00:1b:78:38:55:f8
       inet6 fe80::223:8bff:fe89:e49c%bce0.14 prefixlen 64 scopeid 0xb
       inet 10.0.14.254 netmask 0xffffff00 broadcast 10.0.14.255
       inet 1.1.3.254 netmask 0xffffff00 broadcast 1.1.3.255
       nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
       media: Ethernet autoselect (1000baseSX <full-duplex>)
       status: active
       vlan: 14 parent interface: bce0

ipfw nat 800 config same_ports reset redirect_addr 192.168.254.4 193.X.128.4

45000 nat 800 ip from 192.168.254.4 to any not via em2 // DMZ: Web_Server
45000 nat 800 ip from any to 193.X.128.4 not via em2 // DMZ: Web_Server

With this configuration, when you try to access from a client (
10.0.14.5 ) to 193.X.128.4 the system hangs immediately.
>How-To-Repeat:
ipfw table 12 add 10.0.14.1/32 33
ipfw table 13 add 193.X.128.30/32 33

ipfw nat 33 config redirect_addr 10.0.14.1 193.X.128.30
ipfw nat 799 config ip 3.3.3.3 reverse

55000 nat tablearg ip from table(12) to not 3.3.3.3 via em3
55000 nat tablearg ip from any to table(13) via em3
55000 nat 799 ip from any to table(13) not via em3
55000 nat tablearg ip from 3.3.3.3 to table(13)
55000 nat tablearg ip from table(12) to 3.3.3.3
55000 nat 799 ip from table(13) to 3.3.3.3


make a request from a client who is at same network with the static natted server about 4-5 Mbit/s. ( ie: client: 10.0.14.5 )

default router changes unexpectedly ( frequency : twice per minute )
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-ipfw 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Jun 13 02:15:48 UTC 2011 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=157796 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/157796: commit references a PR
Date: Wed, 24 Apr 2013 18:30:40 +0000 (UTC)

 Author: rrs
 Date: Wed Apr 24 18:30:32 2013
 New Revision: 249848
 URL: http://svnweb.freebsd.org/changeset/base/249848
 
 Log:
   This fixes the issue with the "randomly changing" default
   route. What it was is there are two places in ip_output.c
   where we do a goto again. One place was fine, it
   copies out the new address and then resets dst = ro->rt_dst;
   But the other place does *not* do that, which means earlier
   when we found the gateway, we have dst pointing there
   aka dst = ro->rt_gateway is done.. then we do a
   goto again.. bam now we clobber the default route.
   
   The fix is just to move the again so we are always
   doing dst = &ro->rt_dst; in the again loop.
   
   PR:	 174749,157796
   MFC after:	1 week
 
 Modified:
   head/sys/netinet/ip_output.c
 
 Modified: head/sys/netinet/ip_output.c
 ==============================================================================
 --- head/sys/netinet/ip_output.c	Wed Apr 24 18:00:28 2013	(r249847)
 +++ head/sys/netinet/ip_output.c	Wed Apr 24 18:30:32 2013	(r249848)
 @@ -196,8 +196,8 @@ ip_output(struct mbuf *m, struct mbuf *o
  		hlen = ip->ip_hl << 2;
  	}
  
 -	dst = (struct sockaddr_in *)&ro->ro_dst;
  again:
 +	dst = (struct sockaddr_in *)&ro->ro_dst;
  	ia = NULL;
  	/*
  	 * If there is a cached route,
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/157796: commit references a PR
Date: Thu, 25 Apr 2013 11:24:53 +0000 (UTC)

 Author: rrs
 Date: Thu Apr 25 11:24:40 2013
 New Revision: 249891
 URL: http://svnweb.freebsd.org/changeset/base/249891
 
 Log:
   MFC of PR r249848.
   
   PR:		174749, 157796
 
 Modified:
   stable/8/sys/netinet/ip_output.c
 Directory Properties:
   stable/8/sys/   (props changed)
   stable/8/sys/netinet/   (props changed)
 
 Modified: stable/8/sys/netinet/ip_output.c
 ==============================================================================
 --- stable/8/sys/netinet/ip_output.c	Thu Apr 25 08:57:15 2013	(r249890)
 +++ stable/8/sys/netinet/ip_output.c	Thu Apr 25 11:24:40 2013	(r249891)
 @@ -197,8 +197,8 @@ ip_output(struct mbuf *m, struct mbuf *o
  		hlen = ip->ip_hl << 2;
  	}
  
 -	dst = (struct sockaddr_in *)&ro->ro_dst;
  again:
 +	dst = (struct sockaddr_in *)&ro->ro_dst;
  	/*
  	 * If there is a cached route,
  	 * check that it is to the same destination
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/157796: commit references a PR
Date: Thu, 25 Apr 2013 11:25:42 +0000 (UTC)

 Author: rrs
 Date: Thu Apr 25 11:25:24 2013
 New Revision: 249892
 URL: http://svnweb.freebsd.org/changeset/base/249892
 
 Log:
   MFC of r249848
   
   PR:	174749, 157796
 
 Modified:
   stable/9/sys/netinet/ip_output.c
 Directory Properties:
   stable/9/sys/   (props changed)
 
 Modified: stable/9/sys/netinet/ip_output.c
 ==============================================================================
 --- stable/9/sys/netinet/ip_output.c	Thu Apr 25 11:24:40 2013	(r249891)
 +++ stable/9/sys/netinet/ip_output.c	Thu Apr 25 11:25:24 2013	(r249892)
 @@ -194,8 +194,8 @@ ip_output(struct mbuf *m, struct mbuf *o
  		hlen = ip->ip_hl << 2;
  	}
  
 -	dst = (struct sockaddr_in *)&ro->ro_dst;
  again:
 +	dst = (struct sockaddr_in *)&ro->ro_dst;
  	ia = NULL;
  	/*
  	 * If there is a cached route,
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
Responsible-Changed-From-To: freebsd-ipfw->rrs 
Responsible-Changed-By: glebius 
Responsible-Changed-When: Fri Apr 26 08:58:57 UTC 2013 
Responsible-Changed-Why:  
Randall fixed this recently. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=157796 
State-Changed-From-To: open->closed 
State-Changed-By: glebius 
State-Changed-When: Mon Jun 24 09:50:33 UTC 2013 
State-Changed-Why:  
Merged to stable/9 on April 24th as r249848. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=157796 
>Unformatted:
