From nobody@FreeBSD.org  Wed Jun  3 18:25:47 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id AADAF1065689
	for <freebsd-gnats-submit@FreeBSD.org>; Wed,  3 Jun 2009 18:25:47 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 99A8D8FC1A
	for <freebsd-gnats-submit@FreeBSD.org>; Wed,  3 Jun 2009 18:25:47 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n53IPlOX081719
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 3 Jun 2009 18:25:47 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n53IPkIY081718;
	Wed, 3 Jun 2009 18:25:46 GMT
	(envelope-from nobody)
Message-Id: <200906031825.n53IPkIY081718@www.freebsd.org>
Date: Wed, 3 Jun 2009 18:25:46 GMT
From: Mishustin Andrew <1@hnt.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [igb] low speed routing between two igb interfaces
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         135222
>Category:       kern
>Synopsis:       [igb] low speed routing between two igb interfaces
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    jfv
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jun 03 18:30:01 UTC 2009
>Closed-Date:    
>Last-Modified:  Mon Aug 23 17:52:01 UTC 2010
>Originator:     Mishustin Andrew
>Release:        FreeBSD 7.1-RELEASE amd64, FreeBSD 7.2-RELEASE amd64
>Organization:
HNT
>Environment:
FreeBSD test.hnt 7.2-RELEASE FreeBSD 7.2-RELEASE #12: Thu Apr 30 18:28:15 MSD 20
09     admin@test.hnt:/usr/src/sys/amd64/compile/GENERIC  amd64
>Description:
I made a FreeBSD multiprocesor server to act as simple gateway.
It use onboard Intel 82575EB Dual-Port Gigabit Ethernet Controller.
I observe traffic speed near 400 Kbit/s.
I test both interfaces separately -
ftp client work at speed near 1 Gbit/s in both directions.
Then I change NIC to old Intel "em" NIC - gateway work at speed near 1 Gbit/s.

Looks like a bug in igb driver have an effect upon forwarded traffic.

If you try
hw.igb.enable_aim=0
The speed is near 1 Mbit/s

hw.igb.rxd, hw.igb.txd, "ifconfig -tso" has no effect.

Nothing in messages.log

netstat -m
516/1674/2190 mbufs in use (current/cache/total)
515/927/1442/66560 mbuf clusters in use (current/cache/total/max)
515/893 mbuf+clusters out of packet secondary zone in use (current/cache)
0/44/44/33280 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/16640 9k jumbo clusters in use (current/cache/total/max)
0/0/0/8320 16k jumbo clusters in use (current/cache/total/max)
1159K/2448K/3607K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

I use only IPv4 traffic.

>How-To-Repeat:
On machine with two igb interfaces
use rc.conf like this:

hostname="test.test"
gateway_enable="YES"
ifconfig_igb0="inet 10.10.10.1/24"
ifconfig_igb1="inet 10.10.11.1/24"

And try create heavy traffic between two networks.
>Fix:


>Release-Note:
>Audit-Trail:

From: Michael <freebsdusb@bindone.de>
To: Mishustin Andrew <1@hnt.ru>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces
Date: Thu, 04 Jun 2009 09:41:22 +0200

 Hello Andrew,
 
 look at my post "Bug in igb driver" from two days ago.
 We had exactly the same issues using this adapter (and presumambly the
 same board, your bug description matches 100% our setup :)
 
 The following should fix those issues:
 
 echo "dev.igb.0.enable_lro=0" >>/etc/sysctl.conf
 echo "dev.igb.1.enable_lro=0" >>/etc/sysctl.conf
 reboot
 
 (this disabled large receive offloading - tuning this while the machine
 is running has no effect).
 
 I suggested putting this into the release errata, but got no feedback on
 this request so far.
 
 cheers
 Michael
 
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Jun 12 05:46:45 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135222 

From: Michael <freebsdusb@bindone.de>
To: Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces
Date: Fri, 12 Jun 2009 11:45:47 +0200

 The original poster reported that the suggested fix works for him:
 ---
 Hello Michael,
 
 Thank you. It's working.
 
 I consider it necessary to put this into the release errata.

From: Michael <freebsdusb@bindone.de>
To: Barney Cordoba <barney_cordoba@yahoo.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces
Date: Thu, 18 Jun 2009 03:32:15 +0200

 Barney Cordoba wrote:
 > 
 > 
 > --- On Wed, 6/17/09, Michael <freebsdusb@bindone.de> wrote:
 > 
 >> From: Michael <freebsdusb@bindone.de>
 >> Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces
 >> To: "Barney Cordoba" <barney_cordoba@yahoo.com>
 >> Cc: freebsd-net@FreeBSD.org
 >> Date: Wednesday, June 17, 2009, 5:28 PM
 >> Barney Cordoba wrote:
 >>>
 >>> --- On Fri, 6/12/09, Michael <freebsdusb@bindone.de>
 >> wrote:
 >>>> From: Michael <freebsdusb@bindone.de>
 >>>> Subject: Re: kern/135222: [igb] low speed routing
 >> between two igb interfaces
 >>>> To: freebsd-net@FreeBSD.org
 >>>> Date: Friday, June 12, 2009, 5:50 AM
 >>>> The following reply was made to PR
 >>>> kern/135222; it has been noted by GNATS.
 >>>>
 >>>> From: Michael <freebsdusb@bindone.de>
 >>>> To: Cc: freebsd-gnats-submit@FreeBSD.org
 >>>> Subject: Re: kern/135222: [igb] low speed routing
 >> between
 >>>> two igb interfaces
 >>>> Date: Fri, 12 Jun 2009 11:45:47 +0200
 >>>>
 >>>>   The original poster reported that the
 >> suggested fix works
 >>>> for him:
 >>>>   ---
 >>>>   Hello Michael,
 >>>>   
 >>>>   Thank you. It's working.
 >>>>   
 >>>>   I consider it necessary to put this into the
 >> release
 >>>> errata.
 >>>>   
 >>>>   
 >>>>   Mishustin Andrew wrote:
 >>>>   >> Number:     
 >>>>     135222
 >>>>   >> Category:   
 >>    kern
 >>>>   >> Synopsis:   
 >>    [igb]
 >>>> low speed routing between two igb interfaces
 >>>>   >> Confidential:   no
 >>>>   >> Severity:   
 >>    serious
 >>>>   >> Priority:   
 >>    medium
 >>>>   >> Responsible:   
 >> freebsd-bugs
 >>>>   >> State:       
 >>   open
 >>>>   >> Quarter:       
 >>>>   >> Keywords:   
 >>    
 >>>>   >> Date-Required:
 >>>>   >> Class:       
 >>   sw-bug
 >>>>   >>
 >> Submitter-Id:   current-users
 >>>>   >> Arrival-Date:   Wed
 >> Jun 03
 >>>> 18:30:01 UTC 2009
 >>>>   >> Closed-Date:
 >>>>   >> Last-Modified:
 >>>>   >> Originator: 
 >>    Mishustin
 >>>> Andrew
 >>>>   >> Release:       
 >> FreeBSD
 >>>> 7.1-RELEASE amd64, FreeBSD 7.2-RELEASE amd64
 >>>>   >> Organization:
 >>>>   > HNT
 >>>>   >> Environment:
 >>>>   > FreeBSD test.hnt 7.2-RELEASE FreeBSD
 >> 7.2-RELEASE #12:
 >>>> Thu Apr 30 18:28:15 MSD 20
 >>>>   > 09     admin@test.hnt:/usr/src/sys/amd64/compile/GENERIC
 >>>> amd64
 >>>>   >> Description:
 >>>>   > I made a FreeBSD multiprocesor server
 >> to act as
 >>>> simple gateway.
 >>>>   > It use onboard Intel 82575EB Dual-Port
 >> Gigabit
 >>>> Ethernet Controller.
 >>>>   > I observe traffic speed near 400
 >> Kbit/s.
 >>>>   > I test both interfaces separately -
 >>>>   > ftp client work at speed near 1 Gbit/s
 >> in both
 >>>> directions.
 >>>>   > Then I change NIC to old Intel "em" NIC
 >> - gateway
 >>>> work at speed near 1 Gbit/s.
 >>>>   > 
 >>>>   > Looks like a bug in igb driver have an
 >> effect upon
 >>>> forwarded traffic.
 >>>>   > 
 >>>>   > If you try
 >>>>   > hw.igb.enable_aim=0
 >>>>   > The speed is near 1 Mbit/s
 >>>>   > 
 >>>>   > hw.igb.rxd, hw.igb.txd, "ifconfig -tso"
 >> has no
 >>>> effect.
 >>>>   > 
 >>>>   > Nothing in messages.log
 >>>>   > 
 >>>>   > netstat -m
 >>>>   > 516/1674/2190 mbufs in use
 >> (current/cache/total)
 >>>>   > 515/927/1442/66560 mbuf clusters in
 >> use
 >>>> (current/cache/total/max)
 >>>>   > 515/893 mbuf+clusters out of packet
 >> secondary zone in
 >>>> use (current/cache)
 >>>>   > 0/44/44/33280 4k (page size) jumbo
 >> clusters in use
 >>>> (current/cache/total/max)
 >>>>   > 0/0/0/16640 9k jumbo clusters in use
 >>>> (current/cache/total/max)
 >>>>   > 0/0/0/8320 16k jumbo clusters in use
 >>>> (current/cache/total/max)
 >>>>   > 1159K/2448K/3607K bytes allocated to
 >> network
 >>>> (current/cache/total)
 >>>>   > 0/0/0 requests for mbufs denied
 >>>> (mbufs/clusters/mbuf+clusters)
 >>>>   > 0/0/0 requests for jumbo clusters
 >> denied (4k/9k/16k)
 >>>>   > 0/0/0 sfbufs in use (current/peak/max)
 >>>>   > 0 requests for sfbufs denied
 >>>>   > 0 requests for sfbufs delayed
 >>>>   > 0 requests for I/O initiated by
 >> sendfile
 >>>>   > 0 calls to protocol drain routines
 >>>>   > 
 >>>>   > I use only IPv4 traffic.
 >>>>   > 
 >>>>   >> How-To-Repeat:
 >>>>   > On machine with two igb interfaces
 >>>>   > use rc.conf like this:
 >>>>   > 
 >>>>   > hostname="test.test"
 >>>>   > gateway_enable="YES"
 >>>>   > ifconfig_igb0="inet 10.10.10.1/24"
 >>>>   > ifconfig_igb1="inet 10.10.11.1/24"
 >>>>   > 
 >>>>   > And try create heavy traffic between
 >> two networks.
 >>>>   >> Fix:
 >>>>   > 
 >>>>   > 
 >>>>   >> Release-Note:
 >>>>   >> Audit-Trail:
 >>>>   >> Unformatted:
 >>>>   >
 >> _______________________________________________
 >>>>   > freebsd-bugs@freebsd.org
 >>>
 >>> This is not a bug. Unless you consider poorly written
 >> drivers to be bugs. You need to provide your tuning
 >> parameters for the card as well otherwise there's nothing to
 >> learn.
 >>> The issue is that the driver doesn't address the
 >> purpose of the controller; which is to utilize
 >> multiprocessor systems more effectively. The effect is that
 >> lock contention actually makes things worse than if you just
 >> use a single task as em does. Until the multiqueue drivers
 >> are re-written to manage locks properly you are best advised
 >> to save your money and stick with em.
 >>> You should get similar performance using 1 queue as
 >> with em. You could also force legacy configuration by
 >> forcing igb_setup_msix to return 0. Sadly, this is the best
 >> performance you will get from the stock driver.
 >>> Barney
 >>>
 >>> Barney
 >>>
 >>>
 >>>        
 >> I tried using 1 queue and it didn't make things any better
 >> (actually I'm
 >> not sure if that worked at all). If it is considered a bug
 >> or not
 >> doesn't really matter, what actually matters for users (who
 >> cannot
 >> always chose which network controller will be on-board) is
 >> that they get
 >> a least decent performance when doing IP forwarding (and
 >> not the
 >> 5-50kb/s I've seen). You can get this out of the
 >> controller, when
 >> disabling lro through the sysctl. That's why I've been
 >> asking to put
 >> this into the release errata section and/or at least the
 >> igb man page,
 >> because the sysctl isn't documented anywhere. Also the
 >> fact, that tuning
 >> the sysctl only affects the behaviour when it's set on boot
 >> might be
 >> considered problematic.
 >>
 >> So at the very least, I think the following should be
 >> done:
 >> 1. Document the sysctl in man igb(4)
 >> 2. Put a known issues paragraph to man igb(4) which
 >> explains the issue
 >> and what to put in sysctl.conf to stop this from happening
 >> 3. Add an entry to the release errata page about this issue
 >> (like I
 >> suggested in one of my earlier emails) and stating
 >> something like "see
 >> man igb(4) for details)
 >>
 >> This is not about using the controller to its full
 >> potential, but to
 >> safe Joe Admin from spending days on figuring out why the
 >> machine is
 >> forwarding packages slower than his BSD 2.x machine did in
 >> the 90s.
 >>
 >> cheers
 >> Michael
 > 
 > None of the offload crap should be enabled by default. 
 > 
 > The real point is that "Joe Admin" shouldn't be using controllers that have bad drivers at all. If you have to use whatever hardware you have laying around, and don't have enough flexibility to lay out $100 for a 2 port controller that works to use with your $2000 server, than you need to get your priorities in order. People go out and buy redundant power supplies, high GHZ quad core processors and gobs of memory and then they use whatever crappy onboard controller they get no matter how poorly its suppo rted. Its mindless.
 > 
 > Barney
 > 
 > 
 >       
 
 How should anybody know that the controller is poorly supported if there
 is nothing in the documentation, release notes, man pages or anywhere
 else about this?
 
 The fact of the matter is that "the offload crap" _is_ enabled by
 default. The release is out, it claims to support the controller. There
 _is_ a workaround and I'm asked if somebody could document this so users
 will have a chance. I'm also not convinced that it is a crappy
 controller per se, but just poorly supported. We used those a lot before
 without any issues, unfortunately now we had touse IP forwarding in a
 machine that has that controller (it has 6 interfaces in total, four em
 ports and two igb ports, all of them are in use and I don't feel like
 hooking up the sodering iron).
 
 So bottomline:
 I said, there is a problem with the driver, there is a workaround and it
 should be documented.
 
 You say, the driver is bad and nobody should use it and if they do it's
 their own damn fault. We won't do anything about it and refuse to tell
 anybody, because we are the only ones who should know. We don't care if
 people can actually use our software and still claim the hardware is
 actually supported.
 
 Your attitude is really contra productive (actually googling around I
 see  you made similar statements in the past about stupid people not
 willing to spend xxx$ on whatever piece of hardware, so maybe you're
 just trolling).
 
 Michael
 
 
 
 

From: Joshua Reynolds <josh@jtechcommunications.com>
To: bug-followup@FreeBSD.org,
 1@hnt.ru
Cc:  
Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces
Date: Mon, 13 Jul 2009 13:57:20 -0600

 --Apple-Mail-35-772384282
 Content-Type: text/plain;
 	charset=US-ASCII;
 	format=flowed;
 	delsp=yes
 Content-Transfer-Encoding: 7bit
 
 We had this same problem with our SuperMicro box and have been stumped  
 since switching our interface to igb.  Thanks Michael for the work  
 around... I hope they put this in the errata, it certainly would have  
 helped us a great deal.
 
 
 
 Best regards,
 
 Joshua Reynolds
 President
 J-Tech Communications
 
 jtechcommunications.com
 
 406-586-7100 (ph)
 406-586-1584 (fx)
 888-586-3000 (tf)
 
 
 --Apple-Mail-35-772384282
 Content-Type: text/html;
 	charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 
 <html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
 -webkit-line-break: after-white-space; ">We had this same problem with =
 our SuperMicro box and have been stumped since switching our interface =
 to igb. &nbsp;Thanks&nbsp;Michael for the work around... I hope they put =
 this in the errata, it certainly would have helped us a great =
 deal.<br><div> <span class=3D"Apple-style-span" style=3D"border-collapse: =
 separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; =
 font-style: normal; font-variant: normal; font-weight: normal; =
 letter-spacing: normal; line-height: normal; orphans: 2; text-align: =
 auto; text-indent: 0px; text-transform: none; white-space: normal; =
 widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; =
 -webkit-border-vertical-spacing: 0px; =
 -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
 auto; -webkit-text-stroke-width: 0px; "><div><div><div =
 style=3D"margin-top: 0in; margin-right: 0in; margin-left: 0in; =
 margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New =
 Roman', serif; "><font class=3D"Apple-style-span" face=3D"Helvetica" =
 size=3D"3"><span class=3D"Apple-style-span" style=3D"font-size: 12px; =
 "><br class=3D"Apple-interchange-newline"><br =
 class=3D"Apple-interchange-newline"><br =
 class=3D"webkit-block-placeholder"></span></font></div><div =
 style=3D"margin-top: 0in; margin-right: 0in; margin-left: 0in; =
 margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New =
 Roman', serif; "><span style=3D"font-size: 9pt; font-family: Helvetica, =
 sans-serif; color: black; ">Best regards,</span><span style=3D"color: =
 black; "><o:p></o:p></span></div></div></div><div><div><div =
 style=3D"margin-top: 0in; margin-right: 0in; margin-left: 0in; =
 margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New =
 Roman', serif; "><span style=3D"font-size: 9pt; font-family: Helvetica, =
 sans-serif; color: black; ">&nbsp;</span><span style=3D"color: black; =
 "><o:p></o:p></span></div></div></div><div><div><div style=3D"margin-top: =
 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
 font-size: 12pt; font-family: 'Times New Roman', serif; "><span =
 style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: =
 black; ">Joshua Reynolds</span><span style=3D"color: black; =
 "><o:p></o:p></span></div></div></div><div><div><div style=3D"margin-top: =
 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
 font-size: 12pt; font-family: 'Times New Roman', serif; "><span =
 style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: =
 black; ">President</span><span style=3D"color: black; =
 "><o:p></o:p></span></div></div></div><div><div><div style=3D"margin-top: =
 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
 font-size: 12pt; font-family: 'Times New Roman', serif; "><span =
 style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: =
 black; ">J-Tech Communications</span><span style=3D"color: black; =
 "><o:p></o:p></span></div></div></div><div><div><div style=3D"margin-top: =
 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
 font-size: 12pt; font-family: 'Times New Roman', serif; "><span =
 style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: =
 black; ">&nbsp;</span><span style=3D"color: black; =
 "><o:p></o:p></span></div></div></div><div><div><div style=3D"margin-top: =
 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
 font-size: 12pt; font-family: 'Times New Roman', serif; "><span =
 style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: =
 black; ">jtechcommunications.com</span><span style=3D"color: black; =
 "><o:p></o:p></span></div></div></div><div><div><div style=3D"margin-top: =
 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
 font-size: 12pt; font-family: 'Times New Roman', serif; "><span =
 style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: =
 black; ">&nbsp;</span><span style=3D"color: black; =
 "><o:p></o:p></span></div></div></div><div><div><div style=3D"margin-top: =
 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
 font-size: 12pt; font-family: 'Times New Roman', serif; "><span =
 style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: =
 black; ">406-586-7100 (ph)</span><span style=3D"color: black; =
 "><o:p></o:p></span></div></div></div><div><div><div style=3D"margin-top: =
 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
 font-size: 12pt; font-family: 'Times New Roman', serif; "><span =
 style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: =
 black; ">406-586-1584 (fx)</span><span style=3D"color: black; =
 "><o:p></o:p></span></div></div></div><div><div><div style=3D"margin-top: =
 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
 font-size: 12pt; font-family: 'Times New Roman', serif; "><span =
 style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: =
 black; ">888-586-3000 (tf)</span></div></div></div></span> =
 </div><br></body></html>=
 
 --Apple-Mail-35-772384282--
Responsible-Changed-From-To: freebsd-net->jfv 
Responsible-Changed-By: andre 
Responsible-Changed-When: Mon Aug 23 17:51:40 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135222 
>Unformatted:
