From nobody@FreeBSD.org  Sat Mar  1 10:05:03 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D678F1065671
	for <freebsd-gnats-submit@FreeBSD.org>; Sat,  1 Mar 2008 10:05:03 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id C2A3C8FC18
	for <freebsd-gnats-submit@FreeBSD.org>; Sat,  1 Mar 2008 10:05:03 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m21A2AF8078075
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 1 Mar 2008 10:02:10 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.2/8.14.1/Submit) id m21A2ACM078074;
	Sat, 1 Mar 2008 10:02:10 GMT
	(envelope-from nobody)
Message-Id: <200803011002.m21A2ACM078074@www.freebsd.org>
Date: Sat, 1 Mar 2008 10:02:10 GMT
From: Vitezslav Novy <vnovy@vnovy.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: TSO + natd  -> slow outgoing tcp traffic
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         121257
>Category:       kern
>Synopsis:       [tcp] TSO + natd  -> slow outgoing tcp traffic
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-net
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Mar 01 10:10:01 UTC 2008
>Closed-Date:    
>Last-Modified:  Thu Jan 30 03:50:00 UTC 2014
>Originator:     Vitezslav Novy
>Release:        7.0-RELEASE
>Organization:
>Environment:
FreeBSD vn.chello.upc.cz 7.0-RELEASE FreeBSD 7.0-RELEASE #2: Fri Feb 29 21:37:33 CET 2008     rumik@vn.chello.upc.cz:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
If TSO flag is set on interface and packets are diverted to natd by ipfw on 
same interface, tcp traffic going out through this interface is very slow.

It's because tcp layer sends big packet with  TSO flag set, packet is       
diverted by ipfw and tcp layer gets OK return value.

Then packet is reinjected to ip stack by natd, but TSO flag is lost. Packet 
is dropped by ip_output and ERR return value is delivered to natd, which can 
do nothing with it.

Because tcp layer has no info about problem, packet is resend after tcp     
retransmit timeout. retransmited packets are sent without TSO flag, so it is 
sent successfully.
>How-To-Repeat:
Just look at outgoing tcp traffic on interface with tso flag set and natd   
running on it.
>Fix:
Ad hoc:
Disable TSO flag on interface OR
if possible, change ipfw rules to not divert all traffic on interface

OR

in natd rc script
clear tso flag on interface when natd is started on it

OR
 
in kernel
during divert, reject packet with tso flag set. It gives tcp layer feedback 
and connection tso flag will be cleared.
It's easy to make this change in ipfw code. Maybe packet diverted by divert 
rule should be rejected, and packet diverted by tee rule should be accepted 
(they are not reinjected to ip stack usually).

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sat Mar 1 23:21:15 UTC 2008 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121257 

From: Alex Samorukov <samm@os2.kiev.ua>
To: bug-followup@FreeBSD.org, vnovy@vnovy.net
Cc:  
Subject: Re: kern/121257: [tcp] TSO + natd  -&gt; slow outgoing tcp traffic
Date: Fri, 27 Jun 2008 16:48:15 +0200

 I can approve the problem. I found VERY slow outgoing speed on my new 
 server with natd, and the problem was with TSO flag on public interface.
 Freebsd 7.0/i386, em network driver

From: Takefu <takefu@airport.fm>
To: bug-followup@FreeBSD.org, vnovy@vnovy.net
Cc:  
Subject: Re: kern/121257: [tcp] TSO + natd  -> slow outgoing tcp traffic
Date: Wed, 10 Jun 2009 14:14:42 +0900

 > Dropped Receive Packets on Half-duplex 10/100 Networks
 >   ------------------------------------------------------
 >   If you have an Intel PCI Express adapter installed, running at 10 or 
 >   100 Mbps, half-duplex, with TCP Segment Offload (TSO) enabled, you may 
 >   observe occasional dropped receive packets.  To work around this 
 >   problem, disable TSO, or update the network to operate in full-duplex 
 >   and/or 1 Gbps.
 
 http://downloadmirror.intel.com/14614/eng/rel_notes_v14_0.txt
 
 :-)
 
 

From: Nick Christenson <npc@GangOfOne.com>
To: bug-followup@FreeBSD.org, vnovy@vnovy.net
Cc:  
Subject: Re: kern/121257: [tcp] TSO + natd -&gt; slow outgoing tcp
 traffic
Date: Sat, 30 Oct 2010 14:03:13 -0700 (PDT)

 I notice that not much has been done on this bug in the last 
 couple of years, so here's a bump.
 
 I have reproduced this bug on an amd64 machine running FreeBSD 
 7.3-RELEASE-p3 with network cards using the msk driver.  I'm
 running natd and pf.  I'm not the only one who has encountered 
 this.  Other references to what looks like the same problem:
 
 http://lists.freebsd.org/pipermail/freebsd-net/2010-July/025731.html
 http://forums.freebsd.org/showthread.php?t=13900
 http://osdir.com/ml/freebsd.bugs/2002-03/msg00051.html
 http://unix.derkeiler.com/Mailing-Lists/FreeBSD/questions/2010-01/msg00674.html
 http://comments.gmane.org/gmane.os.freebsd.current/127669
 	(subject "significantly slow IPFW + NATD + amd64)
 
 Many of these don't include resolutions, so if this can't be
 fixed, at least publicizing the work around would seem to be 
 well advised.
 
 It sounds from the bug like the developers have a good handle
 on the problem, but if I can do anything to help test, let me 
 know.
 
 -- 
 Nick Christenson
 npc@gangofone.com

From: Takefu <takefu@airport.fm>
To: bug-followup@FreeBSD.org
Cc: vnovy@vnovy.net
Subject: Re: kern/121257: [tcp] TSO + natd  -> slow outgoing tcp traffic
Date: Thu, 30 Jan 2014 12:42:33 +0900

 Limited improvement method
 
 8.4-RELEASE
 9.2-RELEASE
 10.0-RELEASE
 
 --- /usr/src/etc/rc.d/natd	2013-07-01 15:47:09.000000000 +0900
 +++ /etc/rc.d/natd	2014-01-30 12:26:43.000000000 +0900
 @@ -36,6 +36,7 @@
  		fi
  	fi
 
 +	sysctl net.inet.tcp.tso=0 > /dev/null
  	return 0
  }
 
 
>Unformatted:
