From nobody@FreeBSD.org  Tue Feb 17 22:41:01 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 384EF1065678
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 17 Feb 2009 22:41:01 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 264A58FC1B
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 17 Feb 2009 22:41:01 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n1HMf0Ej089059
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 17 Feb 2009 22:41:00 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n1HMf0MW089058;
	Tue, 17 Feb 2009 22:41:00 GMT
	(envelope-from nobody)
Message-Id: <200902172241.n1HMf0MW089058@www.freebsd.org>
Date: Tue, 17 Feb 2009 22:41:00 GMT
From: Colin Faber <cfaber@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Introducing new USB or eSATA disks results in atapci1+ irq storm until reboot
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         131796
>Category:       kern
>Synopsis:       [ata] Introducing new USB or eSATA disks results in atapci1+ irq storm until reboot
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Feb 17 22:50:03 UTC 2009
>Closed-Date:    Sat Aug 28 09:25:03 UTC 2010
>Last-Modified:  Sat Aug 28 09:25:03 UTC 2010
>Originator:     Colin Faber
>Release:        7.1-RELENG-p2 amd64 kernel
>Organization:
n/a
>Environment:
System: FreeBSD hal.fpsn 7.1-RELEASE-p2 FreeBSD 7.1-RELEASE-p2 #2: Sun Feb 15 12:21:19 MST 2009 cfaber@hal.fpsn:/usr/obj/usr/src/sys/HAL amd64

>Description:
It appears that when plugging in any USB, or eSATA disk drive, atapci1+ gets stuck in a massive irq storm, running about 13k irqs a second. This results in about 90% interrupt load on the system and makes everything unstable and slow.

Note - this load appears even with ZERO activity on the device in question and little or no geom activity.

Also Note - this machine is running two geom_raid5 array's made of geom labeled providers


Non-storm:
cfaber@hal:~$ vmstat -i
interrupt                          total       rate
irq1: atkbd0                           6          0
irq14: ata0                         4316          0
irq16: mpt0                      1782986        243
irq21: ohci0                      213266         29
irq23: atapci1+                  2182108        298
cpu0: timer                     14628721       1999
irq256: nfe0                      215036         29
Total                           19026439       2601


Storm:
cfaber@hal:~$ vmstat -i
interrupt                          total       rate
irq1: atkbd0                           6          0
irq14: ata0                         1534          1
irq16: mpt0                          674          0
irq21: ohci0                        1169          0
irq22: ehci0                         193          0
irq23: atapci1+                 20420307      14230
cpu0: timer                      2868072       1998
irq256: nfe0                        1171          0
Total                           23293126      16232

>How-To-Repeat:
cvsup to RELENG_7_1

configure an amd64 kernel with the following parameters:

cpu		HAMMER
ident		HAL
options 	GEOM_JOURNAL
options 	SCHED_ULE		# ULE scheduler
options 	PREEMPTION		# Enable kernel thread preemption
options 	INET			# InterNETworking
options 	FFS			# Berkeley Fast Filesystem
options 	SOFTUPDATES		# Enable FFS soft updates support
options 	UFS_ACL			# Support for access control lists
options 	UFS_DIRHASH		# Improve performance on big directories
options 	UFS_GJOURNAL		# Enable gjournal-based UFS journaling
options 	MD_ROOT			# MD is a potential root device
options 	NFSCLIENT		# Network Filesystem Client
options 	NFSSERVER		# Network Filesystem Server
options 	NFSLOCKD		# Network Lock Manager
options 	NTFS			# NT File System
options 	MSDOSFS			# MSDOS Filesystem
options 	CD9660			# ISO 9660 Filesystem
options 	PROCFS			# Process filesystem (requires PSEUDOFS)
options 	PSEUDOFS		# Pseudo-filesystem framework
options 	GEOM_PART_GPT		# GUID Partition Tables.
options 	GEOM_LABEL		# Provides labelization
options 	COMPAT_43TTY		# BSD 4.3 TTY compat [KEEP THIS!]
options 	COMPAT_IA32		# Compatible with i386 binaries
options 	COMPAT_FREEBSD4		# Compatible with FreeBSD4
options 	COMPAT_FREEBSD5		# Compatible with FreeBSD5
options 	COMPAT_FREEBSD6		# Compatible with FreeBSD6
options 	SCSI_DELAY=5000		# Delay (in ms) before probing SCSI
options 	KTRACE			# ktrace(1) support
options 	STACK			# stack(9) support
options 	SYSVSHM			# SYSV-style shared memory
options 	SYSVMSG			# SYSV-style message queues
options 	SYSVSEM			# SYSV-style semaphores
options 	_KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
options 	ADAPTIVE_GIANT		# Giant mutex is adaptive.
options 	STOP_NMI		# Stop CPUS using NMI instead of IPI
options 	AUDIT			# Security event auditing
options         COMPAT_LINUX32
options         LINPROCFS
options 	SMP			# Symmetric MultiProcessor Kernel
device		cpufreq
device		acpi
device		pci
device		ata
device		atadisk		# ATA disk drives
device		ataraid		# ATA RAID drives
device		atapicd		# ATAPI CDROM drives
device		atapifd		# ATAPI floppy drives
device		atapist		# ATAPI tape drives
options 	ATA_STATIC_ID	# Static device numbering
device		mpt		# LSI-Logic MPT-Fusion
device		scbus		# SCSI bus (required for SCSI)
device		ch		# SCSI media changers
device		da		# Direct Access (disks)
device		sa		# Sequential Access (tape etc)
device		cd		# CD
device		pass		# Passthrough device (direct SCSI access)
device		ses		# SCSI Environmental Services (and SAF-TE)
device		atkbdc		# AT keyboard controller
device		atkbd		# AT keyboard
device		psm		# PS/2 mouse
device		kbdmux		# keyboard multiplexer
device		vga		# VGA video card driver
device		splash		# Splash screen and screen saver support
device		sc
device		miibus		# MII bus support
device		nfe		# nVidia nForce MCP on-board Ethernet
device		loop		# Network loopback
device		random		# Entropy device
device		ether		# Ethernet support
device		pty		# Pseudo-ttys (telnet etc)
device		md		# Memory "disks"
device		firmware	# firmware assist module
device		bpf		# Berkeley packet filter
device		uhci		# UHCI PCI->USB interface
device		ohci		# OHCI PCI->USB interface
device		ehci		# EHCI PCI->USB interface (USB 2.0)
device		usb		# USB Bus (required)
device		ugen		# Generic
device		uhid		# "Human Interface Devices"
device		ukbd		# Keyboard
device		ulpt		# Printer
device		umass		# Disks/Mass storage - Requires scbus and da
device		ums		# Mouse
device		uscanner	# Scanners
device		ucom		# Generic com ttys
device		sound		# Provide general sound support


Next, connect any umass device, or sata device. atapci1+ will go nuts the second the kernel sees the device online.
>Fix:
So far, the only way to get the problem to stop is to reboot the system. If the device is connected already, then there are no problems with irq storms on atapci1+.

I'm willing to try and debugging procedures provided. As well as provide additional information once requested.

>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: mav 
State-Changed-When: Thu May 7 18:17:41 UTC 2009 
State-Changed-Why:  
Additional information were requested. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=131796 

From: Alexander Motin <mav@FreeBSD.org>
To: bug-followup@FreeBSD.org, cfaber@gmail.com
Cc:  
Subject: Re: kern/131796: [ata] Introducing new USB or eSATA disks results
 in atapci1+ irq storm until reboot
Date: Thu, 07 May 2009 21:17:33 +0300

 This may be not an ATA related problem. Notation "atapci1+" means that 
 several devices sharing some interrupt and atapci1 is only the first of 
 them. Could you dig you boot messages looking fore devices using that 
 IRQ and somehow resolve the IRQ conflict? I just don't see here 
 relations between attaching USB drive and ATA subsystem.
 
 If it won't help, provide additional information about your ATA 
 controller and verbose boot messages and messages any verbose when 
 problem appears.
 
 -- 
 Alexander Motin
State-Changed-From-To: feedback->closed 
State-Changed-By: vwe 
State-Changed-When: Sat Aug 28 09:24:55 UTC 2010 
State-Changed-Why:  
no feedback received 

http://www.freebsd.org/cgi/query-pr.cgi?pr=131796 
>Unformatted:
