From vivek@lorax.kcilink.com  Thu Sep 16 17:40:44 2004
Return-Path: <vivek@lorax.kcilink.com>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 9600916A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 16 Sep 2004 17:40:44 +0000 (GMT)
Received: from lorax.kcilink.com (lorax.kciLink.com [206.112.95.1])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 3F5FA43D1D
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 16 Sep 2004 17:40:44 +0000 (GMT)
	(envelope-from vivek@lorax.kcilink.com)
Received: from localhost (localhost [127.0.0.1])
	by lorax.kcilink.com (Postfix) with ESMTP id 7123A4036
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 16 Sep 2004 13:40:43 -0400 (EDT)
Received: from lorax.kcilink.com ([127.0.0.1])
 by localhost (lorax.kcilink.com [127.0.0.1]) (amavisd-new, port 10024)
 with LMTP id 57293-03 for <FreeBSD-gnats-submit@freebsd.org>;
 Thu, 16 Sep 2004 13:40:42 -0400 (EDT)
Received: by lorax.kcilink.com (Postfix, from userid 120)
	id 7AAD44038; Thu, 16 Sep 2004 13:40:42 -0400 (EDT)
Message-Id: <20040916174042.7AAD44038@lorax.kcilink.com>
Date: Thu, 16 Sep 2004 13:40:42 -0400 (EDT)
From: Vivek Khera <vivek@khera.org>
Reply-To: Vivek Khera <vivek@khera.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: 5.3-BETA3 crash (infinite IRQ list dump)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         71800
>Category:       kern
>Synopsis:       5.3-RELEASE crash (infinite IRQ list dump) (SMP-related)
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Sep 16 17:50:28 GMT 2004
>Closed-Date:    Wed Nov 23 19:02:18 GMT 2005
>Last-Modified:  Wed Nov 23 19:02:18 GMT 2005
>Originator:     Vivek Khera
>Release:        FreeBSD 5.3-BETA3 i386
>Organization:
>Environment:
System: FreeBSD crash.int.kcilink.com 5.3-BETA3 FreeBSD 5.3-BETA3 #7: Thu Sep  9 13:58:20 EDT 2004     khera@crash.int.kcilink.com:/var/usr.obj/usr/src/sys/CRASH  i386

The cpu is AMD Duron.

	
>Description:
	

I was updating to BETA4 to test it out, on the reboot to single user after the
"make installkernel" the console went into the loop shown below.  The same
happened about 4 or 5 hours after BETA3 was first installed last thursday when
the machine was just idle.

I updated a 5.2.1 system via cvsup.  Only recourse is to hit the reset button
(serial port break to debugger is not responding).  The loop just keeps going
and going.


System shutdown time has arrived
Shutting down daemon processes:postfix/postfix-script: stopping the Postfix
mail system
 pgsqlpg_ctl: could not find /u/data/postgres/postmaster.pid
Is postmaster running?
.
Stopping cron.
Shutting down local daemons:.
Writing entropy file:.
Terminated
.
Sep SSWaiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...7 2 0 0 0 interrupt                   total
irq0: clk                       53320822
irq1: atkbd0                           6
irq4: sio0                          2982
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320318
interrupt                   total
irq0: clk                       53320823
irq1: atkbd0                           6
irq4: sio0                          2983
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320320
interrupt                   total
irq0: clk                       53320824
irq1: atkbd0                           6
irq4: sio0                          2984
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320322
interrupt                   total
irq0: clk                       53320825
irq1: atkbd0                           6
irq4: sio0                          2985
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320324
interrupt                   total
irq0: clk                       53320826
irq1: atkbd0                           6
irq4: sio0                          2986
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320326
interrupt                   total
irq0: clk                       53320827
irq1: atkbd0                           6
irq4: sio0                          2987
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320328
interrupt                   total
irq0: clk                       53320828
irq1: atkbd0                           6
irq4: sio0                          2988
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320330
interrupt                   total
irq0: clk                       53320829
irq1: atkbd0                           6
irq4: sio0                          2989
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320332
interrupt                   total
irq0: clk                       53320830
irq1: atkbd0                           6
irq4: sio0                          2990
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320334
interrupt                   total
irq0: clk                       53320831
irq1: atkbd0                           6
irq4: sio0                          2991
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320336
interrupt                   total
irq0: clk                       53320832
irq1: atkbd0                           6
irq4: sio0                          2992
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320338
interrupt                   total
irq0: clk                       53320833
irq1: atkbd0                           6
irq4: sio0                          2993
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320340
interrupt                   total
irq0: clk                       53320834
irq1: atkbd0                           6
irq4: sio0                          2994
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320342
interrupt                   total
irq0: clk                       53320835
irq1: atkbd0                           6
irq4: sio0                          2995
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320344
interrupt                   total
irq0: clk                       53320836
irq1: atkbd0                           6
irq4: sio0                          2996
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320346
interrupt                   total
irq0: clk                       53320837
irq1: atkbd0                           6
irq4: sio0                          2997
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320348
interrupt                   total
irq0: clk                       53320838
irq1: atkbd0                           6
irq4: sio0                          2998
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320350
interrupt                   total
irq0: clk                       53320839
irq1: atkbd0                           6
irq4: sio0                          2999
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320352
interrupt                   total
irq0: clk                       53320840
irq1: atkbd0                           6
irq4: sio0                          3000
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320354
interrupt                   total
irq0: clk                       53320841
irq1: atkbd0                           6
irq4: sio0                          3001
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320356
interrupt                   total
irq0: clk                       53320842
irq1: atkbd0                           6
irq4: sio0                          3002
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320358
interrupt                   total
irq0: clk                       53320843
irq1: atkbd0                           6
irq4: sio0                          3003
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320360
interrupt                   total
irq0: clk                       53320844
irq1: atkbd0                           6
irq4: sio0                          3004
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320362
interrupt                   total
irq0: clk                       53320845
irq1: atkbd0                           6
irq4: sio0                          3005
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320364
interrupt                   total
irq0: clk                       53320846
irq1: atkbd0                           6
irq4: sio0                          3006
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320366
interrupt                   total
irq0: clk                       53320847
irq1: atkbd0                           6
irq4: sio0                          3007
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320368
interrupt                   total
irq0: clk                       53320848
irq1: atkbd0                           6
irq4: sio0                          3008
irq6: fdc0                             4
irq8: rtc                       68240433
irq10: fxp0                       184517
irq11: atapci1                  23942596
irq13: npx0                            2
irq14: ata0                      3628908
irq15: ata1                           48
Total                   149320370


>How-To-Repeat:
	
Don't know.  It just happend twice.  FreeBSD 5.2.1 has been otherwise rock
solid stable on this machine under pretty stressful postgres database
pre-production testing.

>Fix:

	
Don't know.  Hopefully BETA4 won't crash thusly.

>Release-Note:
>Audit-Trail:

From: Vivek Khera <vivek@khera.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/71800: 5.3-BETA3 crash (infinite IRQ list dump)
Date: Thu, 23 Sep 2004 13:49:48 -0400

 FWIW, this exact same thing happened again when rebooting to update to 
 BETA5.  Otherwise it was pretty stable during some pretty heavy 
 database testing.
 

From: Vivek Khera <vivek@khera.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/71800: 5.3-BETA3 crash (infinite IRQ list dump)
Date: Mon, 27 Sep 2004 14:08:14 -0400

 I think I may have more of a clue now.  If I unmount my ccd partition 
 (2 SATA disks in a stripe) before shutdown, it tends not to happen.  It 
 did happen on BETA6 once, and once did not when I did the unmount 
 first.
 
 my ccd.conf is this:
 
 ccd0            126     0       /dev/ad4s1a /dev/ad6s1a
 

From: Vivek Khera <vivek@khera.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/71800: 5.3-BETA3 crash (infinite IRQ list dump) while using ccd
Date: Tue, 21 Dec 2004 11:44:10 -0500

 Unfortunately, I am observing this with 5-STABLE (as of December 19, 
 2004) on an Intel Xeon processor running FreeBSD-amd and a custom 
 kernel.  The 5.3-RELEASE generic kernel did not exhibit this the few 
 times I rebooted, but it also didn't support the NIC on the 
 motherboard, so I had to upgrade.  This is on a brand new Dell PE800 
 tower (Dell's diags report no errors on extended testing).  This box 
 has a SATA RAID (aac device) and no ccd.
 
 Essentially, I cannot reboot these boxes without having to manually hit 
 the reset/power button after shutdown.
 

From: Vivek Khera <vivek@khera.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/71800: 5.3-BETA3 crash (infinite IRQ list dump) while using ccd
Date: Mon, 17 Jan 2005 23:01:26 -0500

 FWIW, I just put 5.3-STABLE as of Jan 17 on a dual Opteron system with 
 4GB ram, and it never exhibits this problem.  It always reboots 
 successfully.  It has an amr RAID controller device and 8 disks.
 
Added to audit trail by linimon from personal email:

The Synopsis has been adjusted to reflect the fact that a) this problem
is still observed in 5.3-RELEASE, but b) only on certain boxes, and c)
is now not believed to be ccd-related.  e.g.:

  It still occurs every time I reboot the original machine, even with
  5.3-RELEASE.  It happens on my intel emt64 Dell PE800 with 5-STABLE on
  every reboot.  I have a Dell PE1750 server on which it never happens,
  and a custom-built Opteron system on which it never happens.  I prefer
  to keep it open since it affects two machines of mine.

From: John Baldwin <jhb@freebsd.org>
To: bug-followup@freebsd.org, vivek@khera.org
Cc:  
Subject: Re: kern/71800: 5.3-RELEASE crash (infinite IRQ list dump) (SMP-related)
Date: Wed, 23 Nov 2005 09:26:55 -0500

 Can you test 5.4 and 6.0 to see if they exhibit the same problem?  The=20
 interrupt output you are seeing is from the DDB command 'show intrcnt', and=
  I=20
 have no idea why that function would be called during shutdown.
 
 =2D-=20
 John Baldwin <jhb@FreeBSD.org> =A0<>< =A0http://www.FreeBSD.org/~jhb/
 "Power Users Use the Power to Serve" =A0=3D =A0http://www.FreeBSD.org

From: Vivek Khera <vivek@khera.org>
To: John Baldwin <jhb@freebsd.org>
Cc: bug-followup@freebsd.org
Subject: Re: kern/71800: 5.3-RELEASE crash (infinite IRQ list dump) (SMP-related)
Date: Wed, 23 Nov 2005 10:23:24 -0500

 On Nov 23, 2005, at 9:26 AM, John Baldwin wrote:
 
 > Can you test 5.4 and 6.0 to see if they exhibit the same problem?  The
 > interrupt output you are seeing is from the DDB command 'show  
 > intrcnt', and I
 > have no idea why that function would be called during shutdown.
 >
 > -- 
 > John Baldwin <jhb@FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
 > "Power Users Use the Power to Serve"  =  http://www.FreeBSD.org
 
 man, I forgot all about this PR....
 
 apparently 5.4 and/or 6.0 solve the problem since I no longer see  
 this happen.
 
 I suppose we close this PR now.
 
State-Changed-From-To: open->closed 
State-Changed-By: jhb 
State-Changed-When: Wed Nov 23 19:01:25 GMT 2005 
State-Changed-Why:  
Submitter reports it is fixed in 5.4 and 6.0. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=71800 
>Unformatted:
