From nobody@FreeBSD.org  Thu Aug 16 20:23:03 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 3877816A419
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 16 Aug 2007 20:23:03 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 2594713C4A3
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 16 Aug 2007 20:23:03 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.1/8.14.1) with ESMTP id l7GKN2bC088101
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 16 Aug 2007 20:23:02 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.1/8.14.1/Submit) id l7GKN2kM088100;
	Thu, 16 Aug 2007 20:23:02 GMT
	(envelope-from nobody)
Message-Id: <200708162023.l7GKN2kM088100@www.freebsd.org>
Date: Thu, 16 Aug 2007 20:23:02 GMT
From: Andrew Gallatin <gallatin@cs.duke.edu>
To: freebsd-gnats-submit@FreeBSD.org
Subject: options IPSEC disables TSO 
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         115586
>Category:       kern
>Synopsis:       options IPSEC disables TSO
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    bz
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Aug 16 20:30:02 GMT 2007
>Closed-Date:    Wed Nov 21 22:37:01 UTC 2007
>Last-Modified:  Wed Nov 21 22:40:01 UTC 2007
>Originator:     Andrew Gallatin
>Release:        7.0-CURRENT
>Organization:
Myricom
>Environment:
FreeBSD venice 7.0-CURRENT FreeBSD 7.0-CURRENT #4: Thu Aug 16 15:46:45 EDT 2007     gallatin@venice:/usr/src/sys/amd64/compile/IPSEC  amd64

>Description:
Simply compiling IPSEC into the kernel causes TCP Segmentation Offload
to be silently disabled for all connections, not just connections using IPSEC.
This causes a severe performance loss, especially on 10GbE networks.
>How-To-Repeat:
- Compile a kernel with 'options IPSEC' and device crypto
- boot the kernel
- notice that TSO stops working (can be verified by watching
the number of packets sent via systat -tcp 1)
>Fix:
A *hack*, not a fix, is attached.  The hack enables normal connections to do TSO even when IPSEC is compiled in.  Essentially, the TSO code needs to be a bit smarter, and check for more than just a null tp->t_inpcb->inp_sp.  I know nothing about IPSEC, so I'm not able to propose a real fix.

Patch attached with submission follows:

Index: netinet/tcp_output.c
===================================================================
RCS file: /home/ncvs/src/sys/netinet/tcp_output.c,v
retrieving revision 1.140
diff -u -r1.140 tcp_output.c
--- netinet/tcp_output.c	3 Jul 2007 12:13:43 -0000	1.140
+++ netinet/tcp_output.c	16 Aug 2007 19:46:09 -0000
@@ -459,8 +459,8 @@
 		    ((tp->t_flags & TF_SIGNATURE) == 0) &&
 		    tp->rcv_numsacks == 0 && sack_rxmit == 0 &&
 		    tp->t_inpcb->inp_options == NULL &&
-		    tp->t_inpcb->in6p_options == NULL &&
-		    tp->t_inpcb->inp_sp == NULL) {
+		    tp->t_inpcb->in6p_options == NULL /* &&
+		    tp->t_inpcb->inp_sp == NULL*/) {
 			tso = 1;
 		} else {
 			len = tp->t_maxseg;


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->bz 
Responsible-Changed-By: bz 
Responsible-Changed-When: Thu Aug 16 20:44:06 UTC 2007 
Responsible-Changed-Why:  
Let me see if I can come up with a proper check tomorrow. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=115586 

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, gallatin@cs.duke.edu
Cc:  
Subject: Re: kern/115586: options IPSEC disables TSO
Date: Thu, 16 Aug 2007 21:22:30 +0000 (UTC)

 Hi,
 
 it's late already but something like the following should do the trick
 IMHO. That will disable TSO if there is an outgoing IPsec policy
 attached to this pcb.
 That does not mean that anything would be done to the packet (and the
 further down mentioned testes should not) but I think a more complete
 check would be too cost intensive?
 
 Is there a way to test that? I think I don't have TSO aware hardware
 around. Could you
 a) check that this works (and compiles might need an #ifdef
     IPSEC and another include maybe?) and does not panic.
 b) without IPSec compiled in connections are unaffected
 c) with ipsec compiled in the connections are unaffected
 d) add a policy (that does nothing) and see that the
     connection is affected?
     Try any of those (setkey -PF will delete it again) and
     should return to a state as seen in c).
     setkey spdadd <your IP> <dest IP> any -P out none;
     setkey spdadd <your IP> <dest IP> tcp -P out none;
     setkey spdadd <your IP> <dest IP> udp -P out none;
 
 I have not thoroughly thought things through so in case something does
 not work as expected let me know.
 
 
 Index: sys/netinet/tcp_output.c
 ===================================================================
 RCS file: /shared/mirror/FreeBSD/r/ncvs/src/sys/netinet/tcp_output.c,v
 retrieving revision 1.140
 diff -u -p -1 -r1.140 tcp_output.c
 --- sys/netinet/tcp_output.c    3 Jul 2007 12:13:43 -0000       1.140
 +++ sys/netinet/tcp_output.c    16 Aug 2007 21:04:39 -0000
 @@ -462,3 +462,4 @@ after_sack_rexmit:
                      tp->t_inpcb->in6p_options == NULL &&
 -                   tp->t_inpcb->inp_sp == NULL) {
 +                   (tp->t_inpcb->inp_sp == NULL ||
 +                       tp->t_inpcb->inp_sp->sp_out->req == NULL)) {
                          tso = 1;
 
 
 -- 
 Bjoern A. Zeeb                                 bzeeb at Zabbadoz dot NeT
 Software is harder than hardware  so better get it right the first time.
State-Changed-From-To: open->feedback 
State-Changed-By: bz 
State-Changed-When: Thu Aug 16 21:57:30 UTC 2007 
State-Changed-Why:  
Let me know if the patch helps. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=115586 

From: Andrew Gallatin <gallatin@cs.duke.edu>
To: "Bjoern A. Zeeb" <bz@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/115586: options IPSEC disables TSO
Date: Thu, 16 Aug 2007 19:04:25 -0400 (EDT)

 Bjoern A. Zeeb writes:
  > Hi,
  > 
  > it's late already but something like the following should do the trick
  > IMHO. That will disable TSO if there is an outgoing IPsec policy
  > attached to this pcb.
  > That does not mean that anything would be done to the packet (and the
  > further down mentioned testes should not) but I think a more complete
  > check would be too cost intensive?
  > 
  > Is there a way to test that? I think I don't have TSO aware hardware
  > around. Could you
 
 Thanks!  I will test this in the morning..
 
 BTW, some 1Gb hardware has TSO support in FreeBSD
 (if_em, if_bce, if_msk, if_nfe, if_re)..
 
 Drew

From: Andrew Gallatin <gallatin@cs.duke.edu>
To: "Bjoern A. Zeeb" <bz@FreeBSD.org>
Cc: bug-followup@FreeBSD.org, gallatin@myri.com
Subject: Re: kern/115586: options IPSEC disables TSO
Date: Fri, 17 Aug 2007 09:51:37 -0400 (EDT)

 Bjoern A. Zeeb writes:
  > Hi,
  > 
  > it's late already but something like the following should do the trick
  > IMHO. That will disable TSO if there is an outgoing IPsec policy
  > attached to this pcb.
  > That does not mean that anything would be done to the packet (and the
  > further down mentioned testes should not) but I think a more complete
  > check would be too cost intensive?
  > 
  > Is there a way to test that? I think I don't have TSO aware hardware
  > around. Could you
  > a) check that this works (and compiles might need an #ifdef
  >     IPSEC and another include maybe?) and does not panic.
 
 Yes, please see the attached patch.
 
  > b) without IPSec compiled in connections are unaffected
 
 This works.
 
  > c) with ipsec compiled in the connections are unaffected
 
 This works.
 
  > d) add a policy (that does nothing) and see that the
  >     connection is affected?
  >     Try any of those (setkey -PF will delete it again) and
  >     should return to a state as seen in c).
  >     setkey spdadd <your IP> <dest IP> any -P out none;
  >     setkey spdadd <your IP> <dest IP> tcp -P out none;
  >     setkey spdadd <your IP> <dest IP> udp -P out none;
 
 TSO was not disabled by any of these; should it have been?
 For the record, I made a file and put those commands in a file
 and ran with setkey -f.
 
 Thank you very much for the help!
 
 Drew
 
 Index: netinet/tcp_output.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/netinet/tcp_output.c,v
 retrieving revision 1.140
 diff -u -r1.140 tcp_output.c
 --- netinet/tcp_output.c	3 Jul 2007 12:13:43 -0000	1.140
 +++ netinet/tcp_output.c	17 Aug 2007 13:24:08 -0000
 @@ -459,8 +459,12 @@
  		    ((tp->t_flags & TF_SIGNATURE) == 0) &&
  		    tp->rcv_numsacks == 0 && sack_rxmit == 0 &&
  		    tp->t_inpcb->inp_options == NULL &&
 -		    tp->t_inpcb->in6p_options == NULL &&
 -		    tp->t_inpcb->inp_sp == NULL) {
 +		    tp->t_inpcb->in6p_options == NULL 
 +#ifdef IPSEC
 +		    && (tp->t_inpcb->inp_sp == NULL ||
 +			tp->t_inpcb->inp_sp->sp_out->req == NULL)
 +#endif
 +		    ) {
  			tso = 1;
  		} else {
  			len = tp->t_maxseg;
 

Why has this change not gone in yet?

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: Andrew Gallatin <gallatin@cs.duke.edu>
Cc: bug-followup@FreeBSD.org, gallatin@myri.com, 
    "George V. Neville-Neil" <gnn@freebsd.org>, 
    Robert Watson <rwatson@FreeBSD.org>, Kip Macy <kmacy@freebsd.org>, 
    Sam Leffler <sam@freebsd.org>
Subject: Re: kern/115586: options IPSEC disables TSO
Date: Sun, 18 Nov 2007 21:39:20 +0000 (UTC)

 Hi,
 
 I have a new patch here which would need more review,
 
 http://sources.zabbadoz.net/freebsd/patchset/tcp-ipsec-tso-20071118-01.diff
 
 which seems to work at least  but still pessimizes all connections once
 IPSEC is compiled in - not more than the current code so far would have
 done but still way too much.
 
 It allows TSO to work again though if IPSEC is compiled in. No idea if
 it's worth with the pessimization at all but...
 
 I have not found a working shortcut but Sam says there used to be one
 if he remembers correctly.
 
 It seems there is more work to be done in the IPsec field in the
 future also with allocation on pcb initialization, etc. but though
 related might be beyond the scpoe of this PR;)
 
 -- 
 Bjoern A. Zeeb                                 bzeeb at Zabbadoz dot NeT
 Software is harder than hardware  so better get it right the first time.

From: Andrew Gallatin <gallatin@cs.duke.edu>
To: "Bjoern A. Zeeb" <bz@FreeBSD.org>
Cc: bug-followup@FreeBSD.org, gallatin@myri.com,
        "George V. Neville-Neil" <gnn@FreeBSD.org>,
        Robert Watson <rwatson@FreeBSD.org>, Kip Macy <kmacy@FreeBSD.org>,
        Sam Leffler <sam@FreeBSD.org>
Subject: Re: kern/115586: options IPSEC disables TSO
Date: Tue, 20 Nov 2007 10:54:26 -0500 (EST)

 Bjoern A. Zeeb writes:
  > Hi,
  > 
  > I have a new patch here which would need more review,
  > 
  > http://sources.zabbadoz.net/freebsd/patchset/tcp-ipsec-tso-20071118-01.diff
  > 
  > which seems to work at least  but still pessimizes all connections once
  > IPSEC is compiled in - not more than the current code so far would have
  > done but still way too much.
  > 
  > It allows TSO to work again though if IPSEC is compiled in. No idea if
  > it's worth with the pessimization at all but...
  
 The patch indeed works, and it increases bandwidth dramatically,
 allowing my wimpy machines to fill 10GbE link even with IPSEC compiled
 in.  The nice thing about TSO is it dramatically reduces the number of
 "packets" that TCP sends, so it also reduces the number of expensive
 ipsec_hdrsiz_tcp() calls.  Now having ipsec compiled in does not hurt
 b/w at all on non-ipsec connections, but just increases CPU usage by
 5-10%.
  
  > I have not found a working shortcut but Sam says there used to be one
  > if he remembers correctly.
 
 I think that this should be committed in the meantime, though a
 shortcut would be nice.
 
 Thank you for this!
 
 Drew
 
State-Changed-From-To: feedback->closed 
State-Changed-By: bz 
State-Changed-When: Wed Nov 21 22:34:15 UTC 2007 
State-Changed-Why:  
Thanks for reporting and testing. 

I'll MFC the patch after 7.0R. IPSEC is not shipped by default so 
anyone who wants it would have to compile a kernel anyway. 
The point is to not find a missed edge case and introduce a TCP problem 
that late in the release cycle. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=115586 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/115586: commit references a PR
Date: Wed, 21 Nov 2007 22:30:22 +0000 (UTC)

 bz          2007-11-21 22:30:14 UTC
 
   FreeBSD src repository
 
   Modified files:
     sys/netinet          tcp_output.c 
   Log:
   Make TSO work with IPSEC compiled into the kernel.
   
   The lookup hurts a bit for connections but had been there anyway
   if IPSEC was compiled in. So moving the lookup up a bit gives us
   TSO support at not extra cost.
   
   PR:             kern/115586
   Tested by:      gallatin
   Discussed with: kmacy
   MFC after:      2 months
   
   Revision  Changes    Path
   1.143     +16 -3     src/sys/netinet/tcp_output.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
>Unformatted:
