From anarcat@rtr1.koumbit.net  Tue Jun 25 19:44:45 2013
Return-Path: <anarcat@rtr1.koumbit.net>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id 8F66CCAE
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 25 Jun 2013 19:44:45 +0000 (UTC)
	(envelope-from anarcat@rtr1.koumbit.net)
Received: from rtr1.koumbit.net (rtr1.core0-arin01.koumbit.net [199.58.81.2])
	by mx1.freebsd.org (Postfix) with ESMTP id 4B04410F7
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 25 Jun 2013 19:44:45 +0000 (UTC)
Received: from rtr1.koumbit.net (localhost [127.0.0.1])
	by rtr1.koumbit.net (Postfix) with ESMTP id 3449CFEBF4
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 25 Jun 2013 15:39:32 -0400 (EDT)
Received: (from root@localhost)
	by rtr1.koumbit.net (8.14.5/8.14.5/Submit) id r5PJdWKd003026;
	Tue, 25 Jun 2013 15:39:32 -0400 (EDT)
	(envelope-from anarcat)
Message-Id: <201306251939.r5PJdWKd003026@rtr1.koumbit.net>
Date: Tue, 25 Jun 2013 15:39:32 -0400 (EDT)
From: Antoine Beaupre <anarcat@koumbit.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: igb(4) fails to do polling(4)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         179975
>Category:       kern
>Synopsis:       [igb] igb(4) fails to do polling(4)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-net
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jun 25 19:50:00 UTC 2013
>Closed-Date:    Wed Aug 07 01:42:50 UTC 2013
>Last-Modified:  Wed Aug 07 01:42:50 UTC 2013
>Originator:     Antoine Beaupre
>Release:        FreeBSD 9.1-RELEASE-p3 amd64
>Organization:
Koumbit.org
>Environment:
System: FreeBSD rtr1.koumbit.net 9.1-RELEASE-p3 FreeBSD 9.1-RELEASE-p3 #0 r251605: Mon Jun 10 16:17:26 EDT 2013 root@rtr1.koumbit.net:/usr/obj/usr/src/sys/KOUMBIT1 amd64

>Description:

the igb(4) driver doesn't work properly in polling mode, despite advertising that capability.

This may be related to VLAN configuration here.

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        capabilities=505fb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,POLLING,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO>
        ether 90:e2:ba:39:d3:7c
        inet 199.58.81.2 netmask 0xffffffc0 broadcast 199.58.81.63
        inet6 fe80::92e2:baff:fe39:d37c%igb0 prefixlen 64 scopeid 0x1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 1000baseT
                media 1000baseT mediaopt full-duplex
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP

here's some bits from the dmesg:

igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.4> mem 0xf7b00000-0xf7b7ffff,0xf7b8c000-0xf7b8ffff irq 16 at
device 0.0 on pci1
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 90:e2:ba:39:d3:7c
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb1: <Intel(R) PRO/1000 Network Connection version - 2.3.4> mem 0xf7a00000-0xf7a7ffff,0xf7b88000-0xf7b8bfff irq 17 at device 0.1 on pci1
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 90:e2:ba:39:d3:7d
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
igb2: <Intel(R) PRO/1000 Network Connection version - 2.3.4> mem 0xf7980000-0xf79fffff,0xf7b84000-0xf7b87fff irq 18 at device 0.2 on pci1
igb2: Using MSIX interrupts with 5 vectors
igb2: Ethernet address: 90:e2:ba:39:d3:7e
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: Bound queue 2 to cpu 2
igb2: Bound queue 3 to cpu 3
igb3: <Intel(R) PRO/1000 Network Connection version - 2.3.4> mem 0xf7900000-0xf797ffff,0xf7b80000-0xf7b83fff irq 19 at device 0.3 on pci1
igb3: Using MSIX interrupts with 5 vectors
igb3: Ethernet address: 90:e2:ba:39:d3:7f
igb3: Bound queue 0 to cpu 0
igb3: Bound queue 1 to cpu 1
igb3: Bound queue 2 to cpu 2
igb3: Bound queue 3 to cpu 3

Note that we cannot reproduce the problem with the following interface:

em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xe000-0xe01f mem 0xf7d00000-0xf7d1ffff,0xf7d20000-0xf7d23fff ir
q 18 at device 0.0 on pci3
em0: Using MSIX interrupts with 3 vectors
em0: Ethernet address: 00:25:90:ae:dc:02

>How-To-Repeat:

Here's our kernel configuration:

include GENERIC
ident KOUMBIT0
device          pf
device          pflog
device          pfsync
options         ALTQ
options         ALTQ_CBQ
options         ALTQ_RED
options         ALTQ_RIO
options         ALTQ_HFSC
options         ALTQ_CDNR
options         ALTQ_PRIQ
options   IPSEC        #IP security
device    crypto
options         DEVICE_POLLING
device          carp

In our configuration, we are building a new router (rtr1 FreeBSD 9.1) to replace our old router (rtr0, FreeBSD 8.3).

Configure igb0 with an IP, enable polling and some VLANs:

ifconfig_igb0="inet 199.58.81.2 netmask 255.255.255.192 polling"
cloned_interfaces="vlan141 vlan60"
ifconfig_vlan141="inet 199.58.80.2 netmask 255.255.255.128 vlan 141 vlandev igb0"
ifconfig_vlan60="inet 199.58.82.2 netmask 255.255.255.192 vlan 60 vlandev igb0"
ifconfig_vlan60_alias0="inet 199.58.81.254 netmask 255.255.255.192"

Configure a switch with those VLANs. Make the VLANs 141 and 60 be tagged and share an untagged VLAN for the native interface.

Configure another router with a similar configuration.

Expected results:

 * all those IPs are pingable from the other router
 * the other router should also be pingable
 * adding the other router as a gateway should make it pingable

Actual results:

 * ICMP request packets go out of the interface, are received by the other router which responds
 * ICMP response packets are not picked up by the interface until POLLING is disabled

This may be related with http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/155030

>Fix:

No known fix. Workaround:

ifconfig igb0 -polling
ifconfig igb0 polling

It seems that cycling the polling configuration fixes the problem, at least temporarily.

We have yet to put this server in production and would really, really appreciate some help here. We are a small non-profit ISP, but we are skilled enough to test patches on the kernel tree, and would be ready to offer a bounty (100$?) to see this problem fixed quickly.

Our ETA for production is july 9th.
>Release-Note:
>Audit-Trail:

From: Mark Johnston <markj@freebsd.org>
To: Antoine Beaupre <anarcat@koumbit.org>
Cc: bug-followup@freebsd.org
Subject: Re: kern/179975: igb(4) fails to do polling(4)
Date: Tue, 25 Jun 2013 16:10:10 -0400

 > Our ETA for production is july 9th.
 
 Hello,
 
 Prior to the change in the above-mentioned PR, igb's polling mode only
 worked with a single queue enabled. I submitted a patch to get it to
 work with multiple queues, and it was committed in r239109. It didn't
 make it into 9.1 though, so igb's polling mode with multiple queues is
 broken.
 
 You have two options. The first is to apply the above revision to your
 driver and recompile the kernel. I can help you with this if you need
 it. The second is to tell igb to only use one queue, using the following
 lines in /boot/loader.conf:
 
 hw.igb.num_queues=1
 
 You'll need to reboot to apply it. I don't think there's really much
 reason to have multiple queues enabled in polling mode, I just had a
 special use case.
 
 HTH,
 -Mark

From: Antoine =?utf-8?Q?Beaupr=C3=A9?= <anarcat@koumbit.org>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc:  
Subject: Re: kern/179975: igb(4) fails to do polling(4)
Date: Tue, 25 Jun 2013 16:08:08 -0400

 Correction: cycling polling(4) doesn't actually work. The interface
 failed to route packets after a while.
 
 I'm trying the patch from kern/155030.
 
 A.
 
 --=20
 Ils versent un pauvre miel sur leurs mots pourris et te parlent de p=C3=A9n=
 urie
 Et sur ta faim, sur tes amis, ils aiguisent leur app=C3=A9tit
                         - Richard Desjardins, La maison est ouverte

From: Antoine =?utf-8?Q?Beaupr=C3=A9?= <anarcat@koumbit.org>
To: Mark Johnston <markj@freebsd.org>
Cc: bug-followup@freebsd.org
Subject: Re: kern/179975: igb(4) fails to do polling(4)
Date: Tue, 25 Jun 2013 16:20:05 -0400

 --=-=-=
 Content-Transfer-Encoding: quoted-printable
 
 Hi!
 
 Thank you very much for the fast response!
 
 I have recompiled with the patch and things seem to be holding. I will
 keep the loader.conf hack in mind for now.
 
 Will this patch make it to 9.2 or 9.1-p4?
 
 Thanks!
 
 =2D-=20
 The individual has always had to struggle to keep from being overwhelmed
 by the tribe. If you try it, you will be lonely often, and sometimes
 frightened. But no price is too high to pay for the privilege of owning
 yourself.                - Friedrich Nietzsche
 
 --=-=-=
 Content-Type: application/pgp-signature
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.12 (GNU/Linux)
 
 iQIcBAEBCAAGBQJRyft1AAoJEHkhUlJ7dZIeqkgP/1wap1S1kLlPHYP1SUxFgDXL
 gvPoeCZC3rdmPUWL6jWcR1G6G3oX5+SrYVMOotlZ7rsq7zbCxWwPvQFDEe9Tja2n
 1azv4TiqrM1lpUHR88bptch91mi1QwznSNfsIazF3bfWsuGCMSCmVrB3XXxweibN
 Yjvpoz7lVY1CpDhfdD5FC6dXb8ibSXv/VpNrwB83BLIl/wnetizwCuLZAOs8hG/6
 UF0nuOaLRTvqSKy0GrezUCchQDx9cy67TlpOOdi1L3187C9MXx2/hrkHD4NjC0T7
 wcl4OHtGjA7Q+mu/owZbrdzHVR9LRUXwQuQpRLWb5aEBCt/jXeSLwZ+K0kj5+3F8
 cVmtYNwnuKpa9k28rLQ7LU+9x2McIN5in53IfOH9vfPcWHKzP+hZxlANL9qugO+k
 ubQ00m38R61a4tC6HHzkJjWtMnBL8uXDJqsFt30owOE0oNgZKOieZ1AFdseR2IxY
 /0hxznFx+hk9ePbKysPFBQLIrFwzQR5R95gIcSNmQyRcrJ+NxdOPl2m3LLoxQ+n9
 jxkpOZAVm6UPymmErMHp/UWtVk5URRi4E4m8EpeGTetDqAFZReKclAUY4e+KhVuc
 BFzgU3n+InJXo3psDZNwhJ74V+ulQv4Nw2UewrtmHEhm2OJRgoU76rVKERbsAkdA
 oGGSDFVrBjAI9XbwXno2
 =2M7g
 -----END PGP SIGNATURE-----
 --=-=-=--

From: Mark Johnston <markj@freebsd.org>
To: Antoine =?iso-8859-1?Q?Beaupr=E9?= <anarcat@koumbit.org>
Cc: bug-followup@freebsd.org
Subject: Re: kern/179975: igb(4) fails to do polling(4)
Date: Tue, 25 Jun 2013 16:32:00 -0400

 It has already been merged to the stable/9 branch, so it'll be in 9.2.
 Existing releases only receive fixes for security-related issues, so
 this change will never become part of 9.1.
State-Changed-From-To: open->patched 
State-Changed-By: linimon 
State-Changed-When: Wed Jun 26 06:32:48 UTC 2013 
State-Changed-Why:  
apparently is already in 9-STABLE. 


Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Jun 26 06:32:48 UTC 2013 
Responsible-Changed-Why:  

http://www.freebsd.org/cgi/query-pr.cgi?pr=179975 
State-Changed-From-To: patched->closed 
State-Changed-By: markj 
State-Changed-When: Wed Aug 7 01:42:49 UTC 2013 
State-Changed-Why:  
Submitter reported that they haven't had any problems since applying the 
patch. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=179975 
>Unformatted:
