From mm@mail2.vx.sk  Fri Feb 25 14:05:00 2011
Return-Path: <mm@mail2.vx.sk>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 1A05D106564A
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Feb 2011 14:05:00 +0000 (UTC)
	(envelope-from mm@mail2.vx.sk)
Received: from mail2.vx.sk (mail2.vx.sk [IPv6:2a01:4f8:100:84a3::3])
	by mx1.freebsd.org (Postfix) with ESMTP id A3D698FC18
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Feb 2011 14:04:59 +0000 (UTC)
Received: from neo.vx.sk (localhost [127.0.0.1])
	by mail2.vx.sk (Postfix) with ESMTP id CD2F7EFB46
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Feb 2011 15:04:58 +0100 (CET)
Received: from mail2.vx.sk ([127.0.0.1])
	by neo.vx.sk (mail.vx.sk [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id HZ0Blz9rNlOs for <FreeBSD-gnats-submit@freebsd.org>;
	Fri, 25 Feb 2011 15:04:47 +0100 (CET)
Received: by mail2.vx.sk (Postfix, from userid 1001)
	id 476DCEFB39; Fri, 25 Feb 2011 15:04:47 +0100 (CET)
Message-Id: <20110225140447.476DCEFB39@mail2.vx.sk>
Date: Fri, 25 Feb 2011 15:04:47 +0100 (CET)
From: Martin Matuska <mm@FreeBSD.org>
Reply-To: Martin Matuska <mm@FreeBSD.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: igb(4) DEVICE_POLLING does not work with carp(4)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         155030
>Category:       kern
>Synopsis:       [igb] igb(4) DEVICE_POLLING does not work with carp(4)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    jfv
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Feb 25 14:10:10 UTC 2011
>Closed-Date:    Sat Jul 06 03:13:31 UTC 2013
>Last-Modified:  Sat Jul 06 03:13:31 UTC 2013
>Originator:     Martin Matuska
>Release:        FreeBSD 8.2-RELEASE amd64
>Organization:
VX Solutions s. r. o.
>Environment:
>Description:
We are using two redundant load-balancing firewall servers with 4-core Xeon
processors and 2x 4-port igb(4) interfaces using carp(4) for IP failover. 

When DEVICE_POLLING is enabled on the igb(4) interfaces, traffic on these
interfaces stops. After disabling it again traffic continues normally as
before.

The traffic throughput on the desired interface is 100-400 MBit/s.

The CPU load is about 3 (=75%) because of network interrupts.

dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.7
dev.igb.0.%driver: igb
dev.igb.0.%location: slot=0 function=0
dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10e8 subvendor=0x8086 subdevice=0xa02c class=0x020000

A similar issue was already reported in freebsd-stable mailing list:
http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061259.html
>How-To-Repeat:
>Fix:

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sat Feb 26 09:17:18 UTC 2011 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=155030 

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Martin Matuska <mm@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: kern/155030: [igb] igb(4) DEVICE_POLLING does not work with carp(4)
Date: Sat, 15 Oct 2011 18:31:25 +0400

   Martin,
 
   it isn't clear from the PR text whether removing carp(4) fixes
 polling(4) on igb(4)? Can you confirm or disclaim that removing
 carp(4) address fixes polling operation?
 
   In the email that is pointed to, they have watchdog timeouts,
 and these timeouts bring interface down, and no suprise that
 CARP goes down to.
 
 -- 
 Totus tuus, Glebius.

From: Martin Matuska <mm@FreeBSD.org>
To: bug-followup@FreeBSD.org, mm@FreeBSD.org
Cc:  
Subject: Re: kern/155030: [igb] igb(4) DEVICE_POLLING does not work with carp(4)
Date: Fri, 20 Apr 2012 09:18:50 +0200

 The problem was actually in the configuration of the igb driver.
 Polling works only with hw.igb.num_queues=1 - and this is also described
 in code comments of if_igb.c:
 
  *  Legacy polling routine : if using this code you MUST be sure that
  *  multiqueue is not defined, ie, set igb_num_queues to 1.
 
 This should be:
 a) added to the manpage
 b) the driver should not attempt polling if hw.igb.num_queues > 1
 
 -- 
 Martin Matuska
 FreeBSD committer
 http://blog.vx.sk
 

From: Mark Johnston <markjdb@gmail.com>
To: bug-followup@FreeBSD.org, mm@FreeBSD.org
Cc:  
Subject: Re: kern/155030: [igb] igb(4) DEVICE_POLLING does not work with
 carp(4)
Date: Thu, 21 Jun 2012 21:38:36 -0400

 --+HP7ph2BbKc20aGI
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 Another way to go is just to fix igb(4)'s polling routine to work with
 multiple queues. That's what I've done in the src tree at my work; the
 patch is attached.
 
 Can you test it and let me know if it solves your problem? If so, I'll
 bug jfv@ to commit it. =)
 
 Thanks,
 -Mark
 
 --+HP7ph2BbKc20aGI
 Content-Type: text/x-diff; charset=us-ascii
 Content-Disposition: attachment; filename="igb_polling.patch"
 
 diff --git a/share/man/man4/polling.4 b/share/man/man4/polling.4
 index 2c711cc..6abc37c 100644
 --- a/share/man/man4/polling.4
 +++ b/share/man/man4/polling.4
 @@ -184,6 +184,7 @@ As of this writing, the
  .Xr fwe 4 ,
  .Xr fwip 4 ,
  .Xr fxp 4 ,
 +.Xr igb 4 ,
  .Xr ixgb 4 ,
  .Xr nfe 4 ,
  .Xr nge 4 ,
 diff --git a/sys/dev/e1000/if_igb.c b/sys/dev/e1000/if_igb.c
 index cb6c63f..7a92b9a 100644
 --- a/sys/dev/e1000/if_igb.c
 +++ b/sys/dev/e1000/if_igb.c
 @@ -1482,12 +1482,6 @@ igb_irq_fast(void *arg)
  }
  
  #ifdef DEVICE_POLLING
 -/*********************************************************************
 - *
 - *  Legacy polling routine : if using this code you MUST be sure that
 - *  multiqueue is not defined, ie, set igb_num_queues to 1.
 - *
 - *********************************************************************/
  #if __FreeBSD_version >= 800000
  #define POLL_RETURN_COUNT(a) (a)
  static int
 @@ -1498,11 +1492,12 @@ static void
  igb_poll(struct ifnet *ifp, enum poll_cmd cmd, int count)
  {
  	struct adapter		*adapter = ifp->if_softc;
 -	struct igb_queue	*que = adapter->queues;
 -	struct tx_ring		*txr = adapter->tx_rings;
 +	struct igb_queue	*que;
 +	struct tx_ring		*txr;
  	u32			reg_icr, rx_done = 0;
  	u32			loop = IGB_MAX_LOOP;
  	bool			more;
 +	int			i;
  
  	IGB_CORE_LOCK(adapter);
  	if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) {
 @@ -1521,20 +1516,26 @@ igb_poll(struct ifnet *ifp, enum poll_cmd cmd, int count)
  	}
  	IGB_CORE_UNLOCK(adapter);
  
 -	igb_rxeof(que, count, &rx_done);
 +	for (i = 0; i < adapter->num_queues; i++) {
 +		que = &adapter->queues[i];
 +		txr = que->txr;
 +
 +		igb_rxeof(que, count, &rx_done);
  
 -	IGB_TX_LOCK(txr);
 -	do {
 -		more = igb_txeof(txr);
 -	} while (loop-- && more);
 +		IGB_TX_LOCK(txr);
 +		do {
 +			more = igb_txeof(txr);
 +		} while (loop-- && more);
  #if __FreeBSD_version >= 800000
 -	if (!drbr_empty(ifp, txr->br))
 -		igb_mq_start_locked(ifp, txr, NULL);
 +		if (!drbr_empty(ifp, txr->br))
 +			igb_mq_start_locked(ifp, txr, NULL);
  #else
 -	if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd))
 -		igb_start_locked(txr, ifp);
 +		if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd))
 +			igb_start_locked(txr, ifp);
  #endif
 -	IGB_TX_UNLOCK(txr);
 +		IGB_TX_UNLOCK(txr);
 +	}
 +
  	return POLL_RETURN_COUNT(rx_done);
  }
  #endif /* DEVICE_POLLING */
 @@ -4846,7 +4847,7 @@ next_desc:
  	}
  
  	if (done != NULL)
 -		*done = rxdone;
 +		*done += rxdone;
  
  	IGB_RX_UNLOCK(rxr);
  	return ((staterr & E1000_RXD_STAT_DD) ? TRUE : FALSE);
 
 --+HP7ph2BbKc20aGI--

From: Mark Johnston <markjdb@gmail.com>
To: bug-followup@FreeBSD.org, mm@FreeBSD.org
Cc:  
Subject: Re: kern/155030: [igb] igb(4) DEVICE_POLLING does not work with
 carp(4)
Date: Mon, 6 Aug 2012 18:58:14 -0400

 The above patch was commited as r239109.
State-Changed-From-To: open->patched 
State-Changed-By: pluknet 
State-Changed-When: Tue Aug 7 05:52:41 UTC 2012 
State-Changed-Why:  
Reflect the recent PR state change as per Mark Johnston. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=155030 
State-Changed-From-To: patched->closed 
State-Changed-By: markj 
State-Changed-When: Sat Jul 6 03:13:23 UTC 2013 
State-Changed-Why:  
The fix was merged to stable/9 in r239109 and stable/8 in r247430. 


Responsible-Changed-From-To: freebsd-net->jfv 
Responsible-Changed-By: markj 
Responsible-Changed-When: Sat Jul 6 03:13:23 UTC 2013 
Responsible-Changed-Why:  
The fix was merged to stable/9 in r239109 and stable/8 in r247430. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=155030 
>Unformatted:
