From nobody@FreeBSD.org  Wed Mar 19 00:44:05 2014
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTPS id E906CC5E
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 19 Mar 2014 00:44:05 +0000 (UTC)
Received: from cgiserv.freebsd.org (cgiserv.freebsd.org [IPv6:2001:1900:2254:206a::50:4])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id C8CBD791
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 19 Mar 2014 00:44:05 +0000 (UTC)
Received: from cgiserv.freebsd.org ([127.0.1.6])
	by cgiserv.freebsd.org (8.14.8/8.14.8) with ESMTP id s2J0i5RO018944
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 19 Mar 2014 00:44:05 GMT
	(envelope-from nobody@cgiserv.freebsd.org)
Received: (from nobody@localhost)
	by cgiserv.freebsd.org (8.14.8/8.14.8/Submit) id s2J0i5H9018941;
	Wed, 19 Mar 2014 00:44:05 GMT
	(envelope-from nobody)
Message-Id: <201403190044.s2J0i5H9018941@cgiserv.freebsd.org>
Date: Wed, 19 Mar 2014 00:44:05 GMT
From: Olivier Cochard-Labbe <olivier@cochatrd.me>
To: freebsd-gnats-submit@FreeBSD.org
Subject: UDP bad performance and out-of-order packet with iperf3 and igb(4) drivers
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         187718
>Category:       kern
>Synopsis:       [igb] UDP bad performance and out-of-order packet with iperf3 and igb(4) drivers
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-net
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Mar 19 00:50:00 UTC 2014
>Closed-Date:    
>Last-Modified:  Wed Apr 16 01:17:49 UTC 2014
>Originator:     Olivier Cochard-Labbe
>Release:        -current
>Organization:
BSD Router project
>Environment:
FreeBSD R1 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r263118M: Fri Mar 14 00:02:53 CET 2014     root@orange.bsdrp.net:/usr/obj/BSDRPcur.amd64/usr/local/BSDRP/BSDRPcur/FreeBSD/src/sys/amd64  amd64

>Description:
Testing UDP throughput with iperf and iperf3 give totally different value:

With iperf:

[root@R1]~# iperf -u -c 99.99.99.100 -w65536 -l1460 -t30 -i4 -b1000m -P1                                              
------------------------------------------------------------
Client connecting to 99.99.99.100, UDP port 5001
Sending 1460 byte datagrams
UDP buffer size: 64.0 KByte
------------------------------------------------------------
[  3] local 99.99.99.99 port 64273 connected with 99.99.99.100 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 4.0 sec   373 MBytes   783 Mbits/sec
[  3]  4.0- 8.0 sec   374 MBytes   784 Mbits/sec
[  3]  8.0-12.0 sec   372 MBytes   780 Mbits/sec
[  3] 12.0-16.0 sec   373 MBytes   783 Mbits/sec
[  3] 16.0-20.0 sec   373 MBytes   783 Mbits/sec
[  3] 20.0-24.0 sec   373 MBytes   782 Mbits/sec
[  3] 24.0-28.0 sec   373 MBytes   783 Mbits/sec
[  3]  0.0-30.0 sec  2.73 GBytes   783 Mbits/sec
[  3] Sent 2010212 datagrams
[  3] Server Report:
[  3]  0.0-30.0 sec  2.73 GBytes   781 Mbits/sec   0.010 ms 4953/2010211 (0.25%)
[  3]  0.0-30.0 sec  954425 datagrams received out-of-order


with iperf3.

[root@R1]~# iperf3 -u -c 99.99.99.100 -w65536 -l1460 -t30 -i4 -b1000m -P1                                             
Connecting to host 99.99.99.100, port 5201
[  4] local 99.99.99.99 port 43694 connected to 99.99.99.100 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-4.00   sec  48.7 MBytes   102 Mbits/sec  1295380  
[  4]   4.00-8.00   sec  28.2 MBytes  59.2 Mbits/sec  1334970  
[  4]   8.00-12.00  sec  31.6 MBytes  66.4 Mbits/sec  1334995  
[  4]  12.00-16.00  sec  32.0 MBytes  67.1 Mbits/sec  1330605  
[  4]  16.00-20.00  sec  47.3 MBytes  99.2 Mbits/sec  1335097  
[  4]  20.00-24.00  sec  30.6 MBytes  64.1 Mbits/sec  1335121  
[  4]  24.00-28.00  sec  35.3 MBytes  74.0 Mbits/sec  1331542  
[  4]  28.00-30.00  sec  19.3 MBytes  81.0 Mbits/sec  663836  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-30.00  sec   273 MBytes  76.4 Mbits/sec  7.755 ms  9381619/9954141 (94%)  
[  4] Sent 9954141 datagrams
>How-To-Repeat:
Load on the generator during the iperf3 test is:
[root@R1]~# top -nCHSIzs1
last pid: 14453;  load averages:  0.37,  0.19,  0.09  up 4+17:17:33    01:26:07
160 processes: 6 running, 104 sleeping, 50 waiting

Mem: 4192K Active, 26M Inact, 751M Wired, 27M Buf, 15G Free
Swap: 


  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME     CPU COMMAND
14452 root     100    0 14300K  1880K CPU1    1   0:26  96.39% iperf3
   11 root     -92    -     0K   800K WAIT    1   0:07   0.88% intr{irq279: igb2:que}
   11 root     -92    -     0K   800K WAIT    2   0:13   0.78% intr{irq280: igb2:que}
   11 root     -92    -     0K   800K WAIT    3   0:05   0.68% intr{irq281: igb2:que}
   11 root     -92    -     0K   800K WAIT    0   0:11   0.49% intr{irq278: igb2:que}


and an extract of the iperf receiver during this test:

iperf3: OUT OF ORDER - incoming packet = 9972872 and received packet = 9983051 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950852 and received packet = 9983052 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947731 and received packet = 9983052 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972873 and received packet = 9983052 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950853 and received packet = 9983053 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947732 and received packet = 9983053 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972874 and received packet = 9983053 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950854 and received packet = 9983054 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947733 and received packet = 9983054 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972875 and received packet = 9983054 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950855 and received packet = 9983055 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947734 and received packet = 9983055 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972876 and received packet = 9983055 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950856 and received packet = 9983056 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9947735 and received packet = 9983056 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9972877 and received packet = 9983056 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 9950857 and received packet = 9983057 AND SP = 5
[  5]  30.00-30.22  sec  25.6 MBytes   957 Mbits/sec  55.317 ms  41962/46554 (90%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-30.22  sec   318 MBytes  88.3 Mbits/sec  55.317 ms  9425274/9983057 (94%)  
[SUM]  0.0-30.2 sec  9425274 datagrams received out-of-order
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Apr 16 01:17:25 UTC 2014 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=187718 
>Unformatted:
