From nobody@FreeBSD.org  Mon Mar  8 19:41:22 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D8D0E106567A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  8 Mar 2010 19:41:22 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id C79218FC12
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  8 Mar 2010 19:41:22 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o28JfMnW010985
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 8 Mar 2010 19:41:22 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o28JfM2p010984;
	Mon, 8 Mar 2010 19:41:22 GMT
	(envelope-from nobody)
Message-Id: <201003081941.o28JfM2p010984@www.freebsd.org>
Date: Mon, 8 Mar 2010 19:41:22 GMT
From: Tom Hicks <thicks@averesystems.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: ixgbe driver errors
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         144561
>Category:       kern
>Synopsis:       [ixgbe] [patch] ixgbe driver errors
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    jfv
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 08 19:50:01 UTC 2010
>Closed-Date:    
>Last-Modified:  Mon Aug 23 14:35:39 UTC 2010
>Originator:     Tom Hicks
>Release:        7.1-RELEASE-p3
>Organization:
Avere Systems
>Environment:
FreeBSD <obscured> 7.1-RELEASE-p3 FreeBSD 7.1-RELEASE-p3 #35: Mon Mar  8 10:32:29 EST 2010     <obscured>@<obscured>/amd64/compile/GENERIC  amd64
>Description:
I have encountered the following problems in the HEAD version of the driver (but compiled to run on FreeBSD 7.1).  All of the problems identified should exist in any version of the kernel though.

1.  memory corruption of mbuf headers.

2.  TCP segmentation offload not usable with checksum offload enabled

3.  Add-in cards without SFP+ modules plugged in will fail ixgbe_attach (regression from 7.1 version of the driver)

4.  Interfaces will be reset when an AF_INET address is added (via SIOCSIFADDR) (regression from 7.1 version of the driver)

5.  Requested alignment (1 byte alignment) of the receive/transmit descriptor rings does not match the hardware alignment.  This is probably not a real problem today, as the returned memory always appears to be 4K aligned anyhow, but could become a problem on new architectures.

6.  Errored frames (crc errors, length errors) cause kernel panics on next use of the receive ring entry.  This problem exists in all versions of the ixgbe driver (including the 7.1 version).

>How-To-Repeat:
For 1, try to use the driver with either an 82598 or 82599 based card.  The kernel will panic quickly

For 2, try to enable TSO, send large segments by enqueuing 4K+ data in repeated send calls using a TCP socket, and monitor the tso counts in the driver.

For 3, don't plug in an SFP+ module.  A corresponding interface (ie. ix0) will only show up when an SFP+ module is inserted.

For 4:
ifconfig ix0 (verify that status is active)
ifconfig ix0 192.168.1.2 netmask 255.255.255.0 (add address)
ifconfig ix0 (status will show no carrier for 5-10 seconds, then active again)

For 5:  not really repeatable, this was found while looking for the memory corruption in 1 as part of a code review.

For 6, cause an errored packet to be injected (via a raw socket), or put in a fail point that causes a packet properly received to be rejected in ixgbe_rxeof.

>Fix:


Patch attached with submission follows:

Index: sys/dev/ixgbe/ixgbe.c
===================================================================
--- sys/dev/ixgbe/ixgbe.c	(revision 204874)
+++ sys/dev/ixgbe/ixgbe.c	(working copy)
@@ -576,7 +576,7 @@
 	} else if (error == IXGBE_ERR_SFP_NOT_SUPPORTED)
 		device_printf(dev,"Unsupported SFP+ Module\n");
 
-	if (error) {
+	if (error && error!=IXGBE_ERR_SFP_NOT_PRESENT) {
 		error = EIO;
 		device_printf(dev,"Hardware Initialization Failure\n");
 		goto err_late;
@@ -944,6 +944,44 @@
 
 	switch (command) {
 
+	case SIOCSIFADDR:
+		IOCTL_DEBUGOUT("ioctl: SIOCTSIFADDR (Set Interface ADDR)");
+		{
+			struct ifaddr *ifa = (struct ifaddr *)data;
+			int avoid_reset = 0;
+
+#if defined(INET) || __FreeBSD_version < 800000
+			if (ifa->ifa_addr->sa_family == AF_INET) {
+				avoid_reset=1;
+			}
+#endif /* defined(INET) */
+
+#if defined(INET6) || __FreeBSD_version < 800000
+			if (ifa->ifa_addr->sa_family == AF_INET6) {
+				avoid_reset=1;
+			}
+#endif /* defined(INET6) */
+
+			if (avoid_reset) {
+				/*
+				* Since resetting hardware takes a very long time
+				* and results in link renegotiation, only
+				* initialize the hardware only when it is absolutely
+				* required.
+				*/
+				ifp->if_flags |= IFF_UP;
+				if (!(ifp->if_drv_flags & IFF_DRV_RUNNING)) {
+					IXGBE_CORE_LOCK(adapter);
+					ixgbe_init_locked(adapter);
+					IXGBE_CORE_LOCK(adapter);
+				}
+				arp_ifinit(ifp, ifa);
+			} else {
+				error = ether_ioctl(ifp, command, data);
+			}
+		}
+		break;
+
 	case SIOCSIFMTU:
 		IOCTL_DEBUGOUT("ioctl: SIOCSIFMTU (Set Interface MTU)");
 		if (ifr->ifr_mtu > IXGBE_MAX_FRAME_SIZE - ETHER_HDR_LEN) {
@@ -1122,7 +1160,7 @@
 	if (ifp->if_capenable & IFCAP_TSO4)
 		ifp->if_hwassist |= CSUM_TSO;
 	if (ifp->if_capenable & IFCAP_TXCSUM)
-		ifp->if_hwassist = (CSUM_TCP | CSUM_UDP);
+		ifp->if_hwassist |= (CSUM_TCP | CSUM_UDP);
 
 	/* Set MTU size */
 	if (ifp->if_mtu > ETHERMTU) {
@@ -2614,7 +2652,7 @@
 	int             r;
 
 	r = bus_dma_tag_create(NULL,	/* parent */
-			       1, 0,	/* alignment, bounds */
+			       16, 0,	/* alignment, bounds */
 			       BUS_SPACE_MAXADDR,	/* lowaddr */
 			       BUS_SPACE_MAXADDR,	/* highaddr */
 			       NULL, NULL,	/* filter, filterarg */
@@ -4270,22 +4308,12 @@
 #endif
 			}
 		} else {
-			ifp->if_ierrors++;
-			/* Reuse loaded DMA map and just update mbuf chain */
-			mh->m_len = MHLEN;
-			mh->m_flags |= M_PKTHDR;
-			mh->m_next = NULL;
-			mp->m_len = mp->m_pkthdr.len = adapter->rx_mbuf_sz;
-			mp->m_data = mp->m_ext.ext_buf;
-			if (mp->m_next) { /* Free chain */
-				sendmp = mp->m_next;
-				m_free(sendmp);
+			++ifp->if_ierrors;
+			++processed;
+			if (sendmp) {
+			    m_freem(sendmp);
+			    sendmp = NULL;
 			}
-			mp->m_next = NULL;
-			if (adapter->max_frame_size <=
-			    (MCLBYTES - ETHER_ALIGN))
-				m_adj(mp, ETHER_ALIGN);
-			sendmp = NULL;
 		}
 		bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map,
 		    BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE);
Index: sys/dev/ixgbe/ixgbe.h
===================================================================
--- sys/dev/ixgbe/ixgbe.h	(revision 204874)
+++ sys/dev/ixgbe/ixgbe.h	(working copy)
@@ -176,7 +176,7 @@
 #define MSIX_82599_BAR			4
 #define IXGBE_TSO_SIZE			65535
 #define IXGBE_TX_BUFFER_SIZE		((u32) 1514)
-#define IXGBE_RX_HDR			256
+#define IXGBE_RX_HDR			128
 #define IXGBE_VFTA_SIZE			128
 #define IXGBE_BR_SIZE			4096
 #define CSUM_OFFLOAD			7	/* Bits in csum flags */
Index: sys/dev/ixgbe/ixgbe_82598.c
===================================================================
--- sys/dev/ixgbe/ixgbe_82598.c	(revision 204874)
+++ sys/dev/ixgbe/ixgbe_82598.c	(working copy)
@@ -255,10 +255,6 @@
 		ret_val = ixgbe_get_sfp_init_sequence_offsets(hw,
 		                                            &list_offset,
 		                                            &data_offset);
-		if (ret_val != IXGBE_SUCCESS) {
-			ret_val = IXGBE_ERR_SFP_NOT_SUPPORTED;
-			goto out;
-		}
 		break;
 	default:
 		break;


>Release-Note:
>Audit-Trail:
Class-Changed-From-To: change-request->sw-bug  
Class-Changed-By: brucec 
Class-Changed-When: Mon Mar 8 20:30:13 UTC 2010 
Class-Changed-Why:  
Bug. 


Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: brucec 
Responsible-Changed-When: Mon Mar 8 20:30:13 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144561 
Responsible-Changed-From-To: freebsd-net->jfv 
Responsible-Changed-By: andre 
Responsible-Changed-When: Mon Aug 23 14:35:23 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144561 
>Unformatted:
