From nobody@FreeBSD.org  Thu Apr 26 13:56:21 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 3AEAB1065670
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 26 Apr 2012 13:56:21 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 0CBA38FC18
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 26 Apr 2012 13:56:21 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q3QDuKsf024135
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 26 Apr 2012 13:56:20 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q3QDuKLq024134;
	Thu, 26 Apr 2012 13:56:20 GMT
	(envelope-from nobody)
Message-Id: <201204261356.q3QDuKLq024134@red.freebsd.org>
Date: Thu, 26 Apr 2012 13:56:20 GMT
From: Greg Becker <greg@codeconcepts.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: sosend sometimes return EINVAL with TSO and VLAN on 82599 NIC
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         167325
>Category:       kern
>Synopsis:       [netinet] [patch] sosend sometimes return EINVAL with TSO and VLAN on 82599 NIC
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-net
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Apr 26 14:00:22 UTC 2012
>Closed-Date:    
>Last-Modified:  Fri Apr 27 06:07:29 UTC 2012
>Originator:     Greg Becker
>Release:        FreeBSD-8.2
>Organization:
CacheIQ
>Environment:
FreeBSD perf-node-1.i.cacheiq.com 8.2-RELEASE FreeBSD 8.2-RELEASE #84: Wed Apr 25 20:34:52 CDT 2012     greg@groover.i.cacheiq.com:/build/trunk/out/PHOENIX/usr/obj/build/trunk/freebsd/RELENG_8/src/sys/PHOENIX  amd64

>Description:
At CacheIQ we discovered a problem where sosend() would occassionally return EINVAL.  This is pretty bad, as it leads our software to believe something was wrong with the calling arguments or perhaps the socket is in a problematic state.

Digging into it we found that every now and then requests larger than dmat->maxsize are rejected by bus_dmamap_load_mbuf_sg().

It's important to note that in our environment we are using the Intel 82599 10Gb/s adapter with TSO enabled and VLAN tagging (QINQ).

The problem only occurs for TSO packets generated by tcp_output().  When the failure occurs, we see that tcp_ouput() submits an mbuf chain to ip_output() that is 65520 bytes long.  ip_output() adds on the link header bringing it to 65534 bytes.  Finally, vlan_tag() adds 4 more bytes bringing it to 65538 bytes, and thus failing the lower level checks.

As a quick fix we factor in max_linkhdr into the TSO length adjustment in tcp_output().

>How-To-Repeat:
Enable TSO and VLAN tagging for 82599 NIC, and drive a lot of data through the system.  We use a custom tool, but any networking tool that can keep the link saturated should illustrate the problem.

Note, our software is based on FreeBSD-8.2, but includes a number of fixes and select updates.
>Fix:
Factor max_linkhdr into the TSO length adjustment in tcp_output().



Patch attached with submission follows:

Index: freebsd/RELENG_8/src/sys/netinet/tcp_output.c
===================================================================
--- freebsd/RELENG_8/src/sys/netinet/tcp_output.c	(revision 5803)
+++ freebsd/RELENG_8/src/sys/netinet/tcp_output.c	(working copy)
@@ -747,9 +747,9 @@
 	if (len + optlen + ipoptlen > tp->t_maxopd) {
 		flags &= ~TH_FIN;
 		if (tso) {
-			if (len > TCP_MAXWIN - hdrlen - optlen) {
-				len = TCP_MAXWIN - hdrlen - optlen;
-				len = len - (len % (tp->t_maxopd - optlen));
+			if (len > TCP_MAXWIN - hdrlen - optlen - max_linkhdr) {
+				len = TCP_MAXWIN - hdrlen - optlen - max_linkhdr;
+				len = len - (len % (tp->t_maxopd - optlen - max_linkhdr));
 				sendalot = 1;
 			} else if (tp->t_flags & TF_NEEDFIN)
 				sendalot = 1;


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Apr 27 06:07:00 UTC 2012 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=167325 
>Unformatted:
