From nobody@FreeBSD.org  Tue Feb 28 17:31:40 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id AF5FC106566C
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 28 Feb 2012 17:31:40 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 9F2708FC16
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 28 Feb 2012 17:31:40 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q1SHVe5O060458
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 28 Feb 2012 17:31:40 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q1SHVe8c060456;
	Tue, 28 Feb 2012 17:31:40 GMT
	(envelope-from nobody)
Message-Id: <201202281731.q1SHVe8c060456@red.freebsd.org>
Date: Tue, 28 Feb 2012 17:31:40 GMT
From: Eugen Konkov <kes-kes@yandex.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: netisr bad distribution between CPUs
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         165536
>Category:       kern
>Synopsis:       [netisr] netisr bad distribution between CPUs
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Feb 28 17:40:11 UTC 2012
>Closed-Date:    
>Last-Modified:  Tue Feb 28 18:45:16 UTC 2012
>Originator:     Eugen Konkov
>Release:        9.0-CURRENT
>Organization:
ISP FreeLine
>Environment:
FreeBSD flux2 9.0-CURRENT FreeBSD 9.0-CURRENT #4: Fri Jun 10 01:30:12 UTC 2011
>Description:
As you see two of four threads get about 100% cpu usage and other two only 30%

# top -SIHP
last pid: 43373;  load averages:  5.14,  5.58,  5.79                        up 0+06:25:33  19:15:20
293 processes: 6 running, 268 sleeping, 17 waiting, 2 lock
CPU 0:  3.6% user,  0.0% nice, 24.6% system, 42.0% interrupt, 29.7% idle
CPU 1:  3.6% user,  0.0% nice, 14.5% system, 64.5% interrupt, 17.4% idle
CPU 2:  0.0% user,  0.0% nice, 10.9% system, 77.5% interrupt, 11.6% idle
CPU 3:  0.7% user,  0.0% nice, 10.9% system, 70.3% interrupt, 18.1% idle
Mem: 515M Active, 1309M Inact, 230M Wired, 260K Cache, 112M Buf, 1844M Free
Swap: 32G Total, 32G Free

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   12 root       -72    -     0K   160K *per-i  2 223:06 99.61% {swi1: netisr 0}
   12 root       -72    -     0K   160K CPU1    1 180:02 88.43% {swi1: netisr 2}
   12 root       -72    -     0K   160K *per-i  0  65:57 34.72% {swi1: netisr 1}
   12 root       -72    -     0K   160K WAIT    3  65:10 32.52% {swi1: netisr 3}
   11 root       155 ki31     0K    32K CPU0    0 199:44 31.35% {idle: cpu0}
   11 root       155 ki31     0K    32K RUN     1 181:36 18.02% {idle: cpu1}
   11 root       155 ki31     0K    32K RUN     3 174:51 14.31% {idle: cpu3}
   11 root       155 ki31     0K    32K CPU2    2 174:50 13.43% {idle: cpu2}
   12 root       -92    -     0K   160K WAIT    0  45:35 12.89% {irq256: re0}
   13 root       -16    -     0K    32K sleep   0  17:41  4.93% {ng_queue2}
   13 root       -16    -     0K    32K sleep   2  17:40  4.74% {ng_queue1}
   13 root       -16    -     0K    32K sleep   0  17:44  4.69% {ng_queue3}
   13 root       -16    -     0K    32K sleep   0  17:41  4.59% {ng_queue0}
# netstat -Q
Configuration:
Setting                        Current        Limit
Thread count                         4            4
Default queue limit                256        10240
Direct dispatch               disabled          n/a
Forced direct dispatch        disabled          n/a
Threads bound to CPUs         disabled          n/a

Protocols:
Name   Proto QLimit Policy Flags
ip         1   1024   flow   ---
igmp       2    256 source   ---
rtsock     3    256 source   ---
arp        7    256 source   ---
ip6       10    256   flow   ---

Workstreams:
WSID CPU   Name     Len WMark   Disp'd  HDisp'd   QDrops   Queued  Handled
   0   0   ip       493  1024        0        0  7217665 377961834 377960765
   0   0   igmp       0     0        0        0        0        0        0
   0   0   rtsock     0   256        0        0      113    27218    27218
   0   0   arp        0    12        0        0        0    36476    36476
   0   0   ip6        0     5        0        0        0      191      191
   1   1   ip         0   786        0        0        0 103683774 103683774
   1   1   igmp       0     0        0        0        0        0        0
   1   1   rtsock     0     0        0        0        0        0        0
   1   1   arp        0     3        0        0        0    16403    16403
   1   1   ip6        0     1        0        0        0      165      165
   2   2   ip        33  1024        0        0    20062 406492881 406492832
   2   2   igmp       0     0        0        0        0        0        0
   2   2   rtsock     0     0        0        0        0        0        0
   2   2   arp        0     9        0        0        0    58454    58454
   2   2   ip6        0     1        0        0        0      191      191
   3   3   ip         1  1024        0        0     8312 267733875 267733873
   3   3   igmp       0     0        0        0        0        0        0
   3   3   rtsock     0     0        0        0        0        0        0
   3   3   arp        0    10        0        0        0   135230   135230
   3   3   ip6        0     3        0        0        0     3107     3107


>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted:
