From nobody@FreeBSD.org  Thu Jun  6 19:23:53 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id 3581B2F6
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  6 Jun 2013 19:23:53 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from oldred.freebsd.org (oldred.freebsd.org [8.8.178.121])
	by mx1.freebsd.org (Postfix) with ESMTP id 0CE1E134E
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  6 Jun 2013 19:23:53 +0000 (UTC)
Received: from oldred.freebsd.org ([127.0.1.6])
	by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id r56JNpXx021790
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 6 Jun 2013 19:23:51 GMT
	(envelope-from nobody@oldred.freebsd.org)
Received: (from nobody@localhost)
	by oldred.freebsd.org (8.14.5/8.14.5/Submit) id r56JNpS2021789;
	Thu, 6 Jun 2013 19:23:51 GMT
	(envelope-from nobody)
Message-Id: <201306061923.r56JNpS2021789@oldred.freebsd.org>
Date: Thu, 6 Jun 2013 19:23:51 GMT
From: Michal Sviba <michal@sviba.cz>
To: freebsd-gnats-submit@FreeBSD.org
Subject: xhci ehci irq storm
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         179376
>Category:       amd64
>Synopsis:       xhci ehci irq storm
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jun 06 19:30:00 UTC 2013
>Closed-Date:    
>Last-Modified:  Mon Oct 21 12:50:00 UTC 2013
>Originator:     Michal Sviba
>Release:        9.1
>Organization:
>Environment:
9.1-RELEASE-p2 #0: Mon Apr 29 13:25:59 CEST 2013
>Description:
CPU load is high because of to much interrupts.

#vmstat -i
interrupt                          total       rate
irq16: xhci0 ehci0           20060263963      14114
irq23: ehci1                     2844944          2
cpu0:timer                    1227368754        863
irq264: em0                     39704787         27
irq265: hdac0                         77          0
irq266: ahci0                    5757249          4
cpu1:timer                      30460523         21
cpu3:timer                      26554702         18
cpu2:timer                      35402259         24
Total                        21428357258      15077

#sysctl hw.usb.ehci.debug=1
#sysctl hw.usb.xhci.debug=1

#tail -f /var/log/messages
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Async is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Async is active
Jun  6 21:22:25 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:29 ms10 kernel: xhci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Async is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Periodic is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power:
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Async is active
Jun  6 21:22:29 ms10 kernel: ehci_set_hw_power: Periodic is active
>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:

From: Michal Sviba <michal@sviba.cz>
To: bug-followup@FreeBSD.org, michal@sviba.cz
Cc:  
Subject: Re: amd64/179376: xhci ehci irq storm
Date: Tue, 11 Jun 2013 11:12:39 +0200

 no device attached
 
 
 CPU: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz (3392.37-MHz K8-class CPU)
    Origin = "GenuineIntel"  Id = 0x306a9  Family = 6  Model = 3a 
 Stepping = 9
 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
    AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
    AMD Features2=0x1<LAHF>
    TSC: P-state invariant, performance statistics
 real memory  = 17179869184 (16384 MB)
 avail memory = 16401600512 (15641 MB)
 Event timer "LAPIC" quality 600
 ACPI APIC Table: <HPQOEM SLIC-BPC>
 FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 FreeBSD/SMP: 1 package(s) x 4 core(s)
   cpu0 (BSP): APIC ID:  0
   cpu1 (AP): APIC ID:  2
   cpu2 (AP): APIC ID:  4
   cpu3 (AP): APIC ID:  6
 ACPI Warning: FADT (revision 5) is longer than ACPI 2.0 version, 
 truncating length 268 to 244 (20110527/tbfadt-320)
 ioapic0 <Version 2.0> irqs 0-23 on motherboard
 kbd1 at kbdmux0
 ctl: CAM Target Layer loaded
 cryptosoft0: <software crypto> on motherboard
 acpi0: <HPQOEM SLIC-BPC> on motherboard
 acpi0: Power Button (fixed)
 acpi0: reservation of 67, 1 (4) failed
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 cpu2: <ACPI CPU> on acpi0
 cpu3: <ACPI CPU> on acpi0
 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
 Timecounter "HPET" frequency 14318180 Hz quality 950
 Event timer "HPET" frequency 14318180 Hz quality 550
 Event timer "HPET1" frequency 14318180 Hz quality 440
 Event timer "HPET2" frequency 14318180 Hz quality 440
 Event timer "HPET3" frequency 14318180 Hz quality 440
 Event timer "HPET4" frequency 14318180 Hz quality 440
 atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
 atrtc0: Warning: Couldn't map I/O.
 Event timer "RTC" frequency 32768 Hz quality 0
 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
 Timecounter "i8254" frequency 1193182 Hz quality 0
 Event timer "i8254" frequency 1193182 Hz quality 100
 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 pci0: <ACPI PCI bus> on pcib0
 vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 
 0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
 agp0: <IvyBridge desktop GT1 IG> on vgapci0
 agp0: aperture size is 256M, detected 65532k stolen memory
 xhci0: <Intel Panther Point USB 3.0 controller> mem 
 0xf7c20000-0xf7c2ffff irq 16 at device 20.0 on pci0
 xhci0: 32 byte context size.
 usbus0 on xhci0
 pci0: <simple comms> at device 22.0 (no driver attached)
 pci0: <simple comms, UART> at device 22.3 (no driver attached)
 em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xf080-0xf09f mem 
 0xf7c00000-0xf7c1ffff,0xf7c39000-0xf7c39fff irq 20 at device 25.0 on pci0
 em0: Using an MSI interrupt
 em0: Ethernet address: 6c:3b:e5:38:ca:00
 ehci0: <Intel Panther Point USB 2.0 controller> mem 
 0xf7c38000-0xf7c383ff irq 16 at device 26.0 on pci0
 usbus1: EHCI version 1.0
 usbus1 on ehci0
 hdac0: <Intel Panther Point HDA Controller> mem 0xf7c30000-0xf7c33fff 
 irq 22 at device 27.0 on pci0
 ehci1: <Intel Panther Point USB 2.0 controller> mem 
 0xf7c37000-0xf7c373ff irq 23 at device 29.0 on pci0
 usbus2: EHCI version 1.0
 usbus2 on ehci1
 pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0
 pci1: <ACPI PCI bus> on pcib1
 isab0: <PCI-ISA bridge> at device 31.0 on pci0
 isa0: <ISA bus> on isab0
 ahci0: <Intel Panther Point AHCI SATA controller> port 
 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f 
 mem 0xf7c36000-0xf7c367ff irq 19 at device 31.2 on pci0
 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
 ahcich0: <AHCI channel> at channel 0 on ahci0
 ahcich1: <AHCI channel> at channel 1 on ahci0
 pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
 acpi_button0: <Power Button> on acpi0
 acpi_tz0: <Thermal Zone> on acpi0
 acpi_tz1: <Thermal Zone> on acpi0
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 driver bug: Unable to set devclass (devname: (null))
 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
 orm0: <ISA Option ROM> at iomem 0xce800-0xcf7ff on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=0x300>
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
 ppc0: cannot reserve I/O port range
 est0: <Enhanced SpeedStep Frequency Control> on cpu0
 p4tcc0: <CPU Frequency Thermal Control> on cpu0
 est1: <Enhanced SpeedStep Frequency Control> on cpu1
 p4tcc1: <CPU Frequency Thermal Control> on cpu1
 est2: <Enhanced SpeedStep Frequency Control> on cpu2
 p4tcc2: <CPU Frequency Thermal Control> on cpu2
 est3: <Enhanced SpeedStep Frequency Control> on cpu3
 p4tcc3: <CPU Frequency Thermal Control> on cpu3
 Timecounters tick every 1.000 msec
 
 
 -- 
 Michal Sviba
 

From: Michal Sviba <michal@myserver.cz>
To: bug-followup@FreeBSD.org, michal@sviba.cz
Cc:  
Subject: Re: amd64/179376: xhci ehci irq storm
Date: Tue, 25 Jun 2013 12:20:05 +0200

 20days without restart made 10times higher irq16 rate
 
 
 
 #uptime
 12:10PM  up 35 days,  1:36, 1 user, load averages: 0.35, 0.41, 0.41
 
 #vmstat -i
 interrupt                          total       rate
 irq16: xhci0 ehci0          312902807569     103275
 irq23: ehci1                     6065098          2
 cpu0:timer                    3038336324       1002
 irq264: em0                     86937072         28
 irq265: hdac0                         77          0
 irq266: ahci0                   12706295          4
 cpu1:timer                      88134649         29
 cpu3:timer                      80433524         26
 cpu2:timer                     103143958         34
 Total                       316318564566     104402
 
 -- 
 Michal
 

From: michal_sjx <michal_sjx@seznam.cz>
To: bug-followup@FreeBSD.org, michal@sviba.cz
Cc: Konstantin Belousov <kostikbel@gmail.com>
Subject: Re: amd64/179376: xhci ehci irq storm
Date: Wed, 26 Jun 2013 10:28:08 +0200

 I tried remove xhci support from kernel. But without success.
 Maybe the problem isn't in xhci.
 
 ms10~#uptime
 10:13AM  up 12:46, 1 user, load averages: 0.20, 0.34, 0.32
  
  
                  ms10~#vmstat -i
 interrupt                          total       rate
 irq1: atkbd0                          18          0
 irq16: ehci0                  9855800830     214485
 irq23: ehci1                       91988          2
 cpu0:timer                      51247683       1115
 irq264: em0                      1095725         23
 irq265: hdac0                         74          0
 irq266: ahci0                     261277          5
 cpu2:timer                       1566559         34
 cpu3:timer                       1232863         26
 cpu1:timer                       1234862         26
 Total                         9912531879     215719

From: Michal Sviba <michal@sviba.cz>
To: bug-followup@FreeBSD.org, michal@sviba.cz
Cc: Konstantin Belousov <kostikbel@gmail.com>
Subject: Re: amd64/179376: xhci ehci irq storm
Date: Wed, 26 Jun 2013 10:32:46 +0200

 I tried remove xhci support from kernel. But without success.
 Maybe the problem isn't in xhci.
 
 ms10~#uptime
 10:13AM  up 12:46, 1 user, load averages: 0.20, 0.34, 0.32
 
 
                  ms10~#vmstat -i
 interrupt                          total       rate
 irq1: atkbd0                          18          0
 irq16: ehci0                  9855800830     214485
 irq23: ehci1                       91988          2
 cpu0:timer                      51247683       1115
 irq264: em0                      1095725         23
 irq265: hdac0                         74          0
 irq266: ahci0                     261277          5
 cpu2:timer                       1566559         34
 cpu3:timer                       1232863         26
 cpu1:timer                       1234862         26
 Total                         9912531879     215719

From: tin <tin@tin.org.ua>
To: bug-followup@FreeBSD.org,michal@sviba.cz
Cc:  
Subject: Re: amd64/179376: xhci ehci irq storm
Date: Thu, 22 Aug 2013 19:33:30 +0300

 --Sig_/V=ZCWjGkaLzA_z+AnM5xQ+n
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 
 The same issue on FreeBSD 9.1-RELEASE GENERIC amd64:
 
 # uname -rsmi
 FreeBSD 9.1-RELEASE amd64 GENERIC
 
 #sysctl hw.usb.xhci.debug=3D16 ; sleep 1; sysctl hw.usb.xhci.debug=3D0
 Aug 22 19:00:00 pedpresa-NEW kernel: interrupt storm detected on "irq16:"; =
 throttling interrupt source
 Aug 22 19:00:31 pedpresa-NEW last message repeated 31 times
 Aug 22 19:02:32 pedpresa-NEW last message repeated 121 times
 Aug 22 19:12:33 pedpresa-NEW last message repeated 601 times
 Aug 22 19:16:53 pedpresa-NEW last message repeated 260 times
 Aug 22 19:16:54 pedpresa-NEW kernel: xhci_interrupt: real interrupt (sts=3D=
 0x00000000, iman=3D0x00000002)
 Aug 22 19:16:54 pedpresa-NEW last message repeated 279 times
 Aug 22 19:16:54 pedpresa-NEW kernel: interrupt storm detected on "irq16:"; =
 throttling interrupt source
 Aug 22 19:16:54 pedpresa-NEW kernel: xhci_interrupt: real interrupt (sts=3D=
 0x00000000, iman=3D0x00000002)
 Aug 22 19:16:55 pedpresa-NEW last message repeated 311 times
 Aug 22 19:16:55 pedpresa-NEW kernel: interrupt storm detected on "irq16:"; =
 throttling interrupt source
 
 # vmstat -i
 interrupt                          total       rate
 irq16: xhci0 ehci0            5588674976      53577
 irq17: re0                       1247393         11
 irq23: ehci1                      157524          1
 cpu0:timer                      15492154        148
 irq264: hdac0                         93          0
 irq266: ahci1                     903811          8
 cpu1:timer                      10026802         96
 cpu7:timer                       9218769         88
 cpu4:timer                       9207822         88
 cpu2:timer                      10149700         97
 cpu6:timer                      47883543        459
 cpu5:timer                       8447886         80
 cpu3:timer                      10476273        100
 Total                         5711886746      54758
 
 # top -SHb 1000 | grep "intr{irq"
    12 root     -88    -     0K   368K istorm  3 363:26  0.88% intr{irq16: x=
 hci0 ehc}
    12 root     -92    -     0K   368K WAIT    5   0:06  0.00% intr{irq17: r=
 e0}
    12 root     -88    -     0K   368K WAIT    3   0:05  0.00% intr{irq266: =
 ahci1}
    12 root     -88    -     0K   368K WAIT    0   0:00  0.00% intr{irq23: e=
 hci1}
    12 root     -96    -     0K   368K WAIT    1   0:00  0.00% intr{irq264: =
 hdac0}
    12 root     -84    -     0K   368K WAIT    0   0:00  0.00% intr{irq1: at=
 kbd0}
    12 root     -84    -     0K   368K WAIT    0   0:00  0.00% intr{irq5: pp=
 c1}
    12 root     -88    -     0K   368K WAIT    0   0:00  0.00% intr{irq265: =
 ahci0}
 
 #pciconf -lvc
 xhci0@pci0:0:20:0:	class=3D0x0c0330 card=3D0x50071458 chip=3D0x1e318086 rev=
 =3D0x04 hdr=3D0x00
     vendor     =3D 'Intel Corporation'
     device     =3D 'Panther Point USB xHCI Host Controller'
     class      =3D serial bus
     subclass   =3D USB
     cap 01[70] =3D powerspec 2  supports D0 D3  current D0
     cap 05[80] =3D MSI supports 8 messages, 64 bit=20
 ---
 ehci0@pci0:0:26:0:	class=3D0x0c0320 card=3D0x50061458 chip=3D0x1e2d8086 rev=
 =3D0x04 hdr=3D0x00
     vendor     =3D 'Intel Corporation'
     device     =3D 'Panther Point USB Enhanced Host Controller'
     class      =3D serial bus
     subclass   =3D USB
     cap 01[50] =3D powerspec 2  supports D0 D3  current D0
     cap 0a[58] =3D EHCI Debug Port at offset 0xa0 in map 0x14
     cap 13[98] =3D PCI Advanced Features: FLR TP
 
 
 --=20
 #include <regards.h>
 /*    Pavel Shanaiev    */
 /* Mob. +38(050)7777850 */
 
 --Sig_/V=ZCWjGkaLzA_z+AnM5xQ+n
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Disposition: attachment; filename=signature.asc
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 
 iQEcBAEBAgAGBQJSFj1aAAoJEC7dEGKX1IjeuAsH/1peh33hK8gY2e2EGbsF1KWB
 MQ3wsQy46m0mL6ZXqKDdy1G4OQ8UDn6QOZLPDzYGE8fUTfeK9Z5K5dH4fRWWh+Bn
 MAL0iVsITpOyk/IOraLKRVhWe0td1Vz/EHxAdqFrvLbDqmBvj/w44LCsnYbxXf2A
 jS0B24GFEUFsAwfkCczEl6lIGjIK8GFtGh6btplNSBnllUpKJEflpZlHKpZRWpTh
 RBCui8SGAn3b9oQjnuR8GBlsMIohgM5qZOdIcfSewPjS75IrkBkGGBReiBN+DnSt
 Yhwju4Rb0Lgh9IIUl12EuJ3+5rA62vUr/ftGbUL9HlhIXGe5qG2AwvLPafo6yhc=
 =mGH6
 -----END PGP SIGNATURE-----
 
 --Sig_/V=ZCWjGkaLzA_z+AnM5xQ+n--

From: Raimundo Santos <raitech@gmail.com>
To: bug-followup@freebsd.org, michal@sviba.cz
Cc:  
Subject: Re: amd64/179376: xhci ehci irq storm
Date: Mon, 21 Oct 2013 10:45:34 -0200

 --001a1133de3837d7ab04e93faa05
 Content-Type: text/plain; charset=ISO-8859-1
 
 The same irq storm occurs here when I plug a monitor in my ISPs router,
 does not matter if the monitor is plugged via DVI, D-Sub (VGA) or HDMI.
 
 Because it is under production now, I can not test on it. I am using a
 custom kernel with ipfw fwd, verbosity and pass all, plus vimage enabled,
 but can confirm that an installation over a new machine came up with the
 same problem.
 
 Both machines are running 9.2-RELEASE, and both are over Gigabyte
 motherboards.
 
 To reproduce, does not need to go further: just boot an installation media,
 jump into shell and monitor irqs. Then, unplug the monitor from whatever
 kind of connector it is connected. Wait some seconds and plug it again. Now
 you got your own irq storrm!
 
 Make shure that the test machine has USB 3.0.
 
 Cheers,
 Raimundo Santos
 
 PS: my router was made over 9.1-RELEASE and I upgraded recently because of
 a bug with routes, aliases and multiple FIBs. But I have the same issue
 under 9.1-RELEASE.
 
 --001a1133de3837d7ab04e93faa05
 Content-Type: text/html; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 <div dir=3D"ltr"><div><div>The same irq storm occurs here when I plug a mon=
 itor in my ISPs router, does not matter if the monitor is plugged via DVI, =
 D-Sub (VGA) or HDMI.<br><br></div>Because it is under production now, I can=
  not test on it. I am using a custom kernel with ipfw fwd, verbosity and pa=
 ss all, plus vimage enabled, but can confirm that an installation over a ne=
 w machine came up with the same problem.<br>
 
 <br></div><div>Both machines are running 9.2-RELEASE, and both are over Gig=
 abyte motherboards.<br><br></div><div>To reproduce, does not need to go fur=
 ther: just boot an installation media, jump into shell and monitor irqs. Th=
 en, unplug the monitor from whatever kind of connector it is connected. Wai=
 t some seconds and plug it again. Now you got your own irq storrm!<br>
 
 <br>Make shure that the test machine has USB 3.0.<br><br></div><div>Cheers,=
 <br></div><div>Raimundo Santos<br><br></div><div>PS: my router was made ove=
 r 9.1-RELEASE and I upgraded recently because of a bug with routes, aliases=
  and multiple FIBs. But I have the same issue under 9.1-RELEASE.<br>
 
 </div></div>
 
 --001a1133de3837d7ab04e93faa05--
>Unformatted:
