From nobody@FreeBSD.org  Sun Jun 12 04:42:04 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 2F2E6106566C
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 12 Jun 2011 04:42:04 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 151DA8FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 12 Jun 2011 04:42:04 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p5C4g3bL089889
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 12 Jun 2011 04:42:03 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p5C4g3nN089888;
	Sun, 12 Jun 2011 04:42:03 GMT
	(envelope-from nobody)
Message-Id: <201106120442.p5C4g3nN089888@red.freebsd.org>
Date: Sun, 12 Jun 2011 04:42:03 GMT
From: Ben Baron <einsteinx2@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: amd64 + jail + ipfw + natd = very slow outbound traffic from jail (5KB/s)
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         157785
>Category:       kern
>Synopsis:       amd64 + jail + ipfw + natd = very slow outbound traffic from jail (5KB/s)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-net
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jun 12 04:50:00 UTC 2011
>Closed-Date:    
>Last-Modified:  Mon Jul 04 02:17:05 UTC 2011
>Originator:     Ben Baron
>Release:        8.2-RELEASE
>Organization:
Be Five Enterprises, Inc.
>Environment:
FreeBSD localhost 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Feb 17 02:41:51 UTC 2011     root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
I have a newly set up FreeBSD 8.2 server with one public network interface setup with a jail for housing Apache. I was using ipfw + natd as my solution to firewall and nat traffic between the machine and jail (actually had multiple jails for different processes). The setup was working fine, however transfers out from my Apache jail were limited to about 5 KB/sec (~40kbit/sec) (I didn't test other jails but I assume they were the same speed, also didn't test jail to jail). Traffic into the jail from the internet was full speed. Traffic out from the host was also full speed. Only traffic out from the jail was affected.

I researched the issue online and determined that this is a known issue with ipfw + natd and jails when using amd64 processors (in my case an Intel Core i7). I read conflicting statements on the FreeBSD forums about whether this also affects i386 (it looks like it doesn't, but one person did comment that they had the same issue on an i386 machine IIRC).
>How-To-Repeat:

1. Do a fresh install of FreeBSD 8.2

2. Create an lo1 interface with the IP 10.0.0.2/24

3. Use ezjail to create a jail with 10.0.0.2 IP address

4. Install Apache in jail

5. Setup ipfw with the following settings (edited from my original settings to remove some sensitive information):

# Interface: em1 
# List of Ports:
#   Host SSH: 22
#   JAIL WEB SERVER: 80, 443 

IPF="ipfw -q add"
ipfw -q -f flush

# Replace with your interface
PIF="em1"

# Host IP
inet="XXX.XXX.XXX.XXX"

# Jail IPs 
apache_jail="10.0.0.2"

# Loopback
$IPF 100 allow all from any to any via lo0 
$IPF 110 deny all from any to 127.0.0.0/8
$IPF 120 deny all from 127.0.0.0/8 to any 
$IPF 130 deny tcp from any to any frag

# Allow SSH to host
$IPF 300 allow tcp from any to me 22 established

# Allow out NATed traffic from Apache Jail. This is DNATed traffic.
$IPF 500 divert natd ip from $apache_jail to not me out via $PIF
$IPF 570 skipto 10000 ip from any to any diverted

# Allow remaining all outgoing traffic
$IPF 600 allow all from me to any out 

# Allow NATed traffic to Jail. This is SNATed traffic. natd (NAT daemon) will take care of these.
$IPF 700 divert natd ip from not me to any in via $PIF
$IPF 710 skipto 10000 ip from any to any diverted

# Deny ICMP
$IPF 800 allow icmp from any to any 

# Log an deny the traffic
$IPF 900 deny log logamount 10000 ip from any to any

# This is where all diverted NAT traffic are skipped to after being processed.
$IPF 10000 allow ip from any to any


6. Setup natd with the following settings:

# Apache Jail
redirect_port tcp 10.0.0.2:80 80
redirect_port tcp 10.0.0.2:443 443 

>Fix:
The only fix I was able to find was to switch to pf. After switching
to pf for my firewall and nat solution, I'm getting full speed traffic
both ways from the jail.

>Release-Note:
>Audit-Trail:
From: "Steven Hartland" <killing@multiplay.co.uk>
To: "Ben Baron" <einsteinx2@gmail.com>,
	<freebsd-gnats-submit@FreeBSD.org>
Subject: Re: amd64/157785: amd64 + jail + ipfw + natd = very slow outboundtraffic from jail (5KB/s)
Date: Sun, 12 Jun 2011 14:20:58 +0100
MIME-Version: 1.0
Content-Type: text/plain;
	format=flowed;
	charset="Windows-1252";
	reply-type=original
Content-Transfer-Encoding: 7bit
 
 Have you tried disabling hw checksums and framing i.e. RXCSUM, TXCSUM & TSO4?

From: Ben Baron <einsteinx2@gmail.com>
To: Steven Hartland <killing@multiplay.co.uk>, freebsd-gnats-submit@freebsd.org
Cc:  
Subject: Re: amd64/157785: amd64 + jail + ipfw + natd = very slow
 outboundtraffic from jail (5KB/s)
Date: Sun, 12 Jun 2011 13:42:51 -0400

 --20cf30564309533c6d04a5875782
 Content-Type: text/plain; charset=ISO-8859-1
 
 Yes I did try disabling rx and tx checksums, but that did not affect the
 speed at all. I did not come across any suggestions about TSO4 however, so I
 didn't try anything with that.
 
 How would I go about disabling TSO4? While I have pf setup now, I still have
 the ipfw and natd configurations on the machine so I can switch back to test
 any suggestions until the box goes into production in about a week (though I
 have to say I'm thoroughly impressed with pf and much prefer it's
 configuration style, so I'll probably be sticking with it regardless of
 whether this ipfw bug is fixed).
 
 I also tried setting sysctl net.inet.tcp.rfc1323=0 as well, which I believe
 has to do with window sizing (it was recommended in a mailing list reply to
 someone with a slow NAT issue), but it only decreased speed drastically
 (transfer would start at 5KB/sec then quickly drop below 1KB/sec) rather
 than increasing it.
 
 Also, I noticed that when doing a simple transfer from my Apache jail out, I
 was hitting as high as 6% CPU on a 3.8 GHz core i7. Obviously that's
 unacceptable performance. There is no high load like that with the pf
 configuration.
 _______________________________________________________
 *Ben Baron*
 
 --20cf30564309533c6d04a5875782--

From: "Steven Hartland" <killing@multiplay.co.uk>
To: "Ben Baron" <einsteinx2@gmail.com>,
	<freebsd-gnats-submit@freebsd.org>
References: <201106120442.p5C4g3nN089888@red.freebsd.org> <CE1EEF4A3C4D42B1B79DD314379898A1@multiplay.co.uk> <BANLkTinOvit+s5etM1WNQ62HSZDVPcuk-w@mail.gmail.com>
Subject: Re: amd64/157785: amd64 + jail + ipfw + natd = very slow outboundtraffic from jail (5KB/s)
Date: Sun, 12 Jun 2011 18:59:18 +0100
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0202_01CC2932.D4FF5A70"

This is a multi-part message in MIME format.

------=_NextPart_000_0202_01CC2932.D4FF5A70
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

ifconfig <interface> -tso
if your nic also supports lso you will also want to also try -lso

I know there's are or where an issue on igb when running pfsense not sure if em is also effected

Some useful links might be:-

http://forum.pfsense.org/index.php/topic,20797.0.html
http://forum.pfsense.org/index.php/topic,16236.0.html

    Regards
    Steve
------=_NextPart_000_0202_01CC2932.D4FF5A70--

From: "Steven Hartland" <killing@multiplay.co.uk>
To: "Ben Baron" <einsteinx2@gmail.com>,
	<freebsd-gnats-submit@freebsd.org>
References: <201106120442.p5C4g3nN089888@red.freebsd.org> <CE1EEF4A3C4D42B1B79DD314379898A1@multiplay.co.uk> <BANLkTinOvit+s5etM1WNQ62HSZDVPcuk-w@mail.gmail.com>
Subject: Re: amd64/157785: amd64 + jail + ipfw + natd = very slow outboundtraffic from jail (5KB/s)
Date: Sun, 12 Jun 2011 19:00:52 +0100
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0208_01CC2933.0D3638F0"

This is a multi-part message in MIME format.

------=_NextPart_000_0208_01CC2933.0D3638F0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Oh forgot to mention, iirc there was something about it not taking effect with ifconfig and the sysctl and a reboot was required but cant find the link to that atm, so worth keeping that in mind if the ifconfig emX -tso -lro doesn't have the desired effect.

    Regards
    Steve

------=_NextPart_000_0208_01CC2933.0D3638F0--
From: Ben Baron <einsteinx2@gmail.com>
To: Steven Hartland <killing@multiplay.co.uk>, freebsd-gnats-submit@freebsd.org
Cc:  
Subject: Re: amd64/157785: amd64 + jail + ipfw + natd = very slow
 outboundtraffic from jail (5KB/s)
Date: Sun, 12 Jun 2011 14:13:42 -0400

 --20cf3010e901a9743004a587c50f
 Content-Type: text/plain; charset=ISO-8859-1
 
 Thanks for the suggestions. I'll try this later today sometime and report
 back.
 
 So that ifconfig command is persistent between reboots? I figured that would
 be reset. How do I go about re-enabling those options after the test?
 _______________________________________________________
 *Ben Baron*
 
 --20cf3010e901a9743004a587c50f--
Responsible-Changed-From-To: freebsd-amd64->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Jul 4 02:16:25 UTC 2011 
Responsible-Changed-Why:  
reclassify. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=157785 
>Unformatted:
