From nobody@FreeBSD.org  Thu Feb  8 10:16:49 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 4B52216A40E
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  8 Feb 2007 10:16:40 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id 0DD4E13C494
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  8 Feb 2007 10:16:40 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l18AGdQ3049158
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 8 Feb 2007 10:16:39 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id l18AGdWE049157;
	Thu, 8 Feb 2007 10:16:39 GMT
	(envelope-from nobody)
Message-Id: <200702081016.l18AGdWE049157@www.freebsd.org>
Date: Thu, 8 Feb 2007 10:16:39 GMT
From: Taras Savchuk<taras@elantech.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Panics when Intel MatrixRAID RAID1 is degraded 
X-Send-Pr-Version: www-3.0

>Number:         108924
>Category:       kern
>Synopsis:       [ar] Panics when Intel MatrixRAID RAID1 is degraded
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Feb 08 10:20:15 GMT 2007
>Closed-Date:    Sun Aug 07 11:31:45 UTC 2011
>Last-Modified:  Sun Aug 07 11:31:45 UTC 2011
>Originator:     Taras Savchuk
>Release:        6.2-RELEASE/i386
>Organization:
>Environment:
FreeBSD eee.local 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jan 12 11:05:30 UTC 2007   root@dessler.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP  i386
>Description:
FreeBSD 6.2 RELEASE sucessfully installed on RAID-1 volume (HP ProLiant
ML110G4, Intel MatrixRAID), but while I'm trying to boot without one of
two RAID-1 HDDs system panics. atacontrol detach works well, but after
reboot system panics too.

dmesg.boot
----------
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE #0: Fri Jan 12 11:05:30 UTC 2007
    root@dessler.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU            3040  @ 1.86GHz (1862.01-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f6  Stepping = 6
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15>>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 2
real memory  = 535363584 (510 MB)
avail memory = 514240512 (490 MB)
ACPI APIC Table: <HP ML110 G4>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <HP> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of fed13000, 1000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib3
pci3: <display, VGA> at device 0.0 (no driver attached)
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci4: <ACPI PCI bus> on pcib4
bge0: <Broadcom BCM5750 C1, ASIC rev. 0x4201> mem 0xef900000-0xef90ffff irq 17 at device 0.0 on pci4
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
bge0: Ethernet address: 00:18:71:77:fe:11
uhci0: <UHCI (generic) USB controller> port 0x3000-0x301f irq 23 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x3020-0x303f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x3040-0x305f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x3060-0x307f irq 16 at device 29.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xefd00000-0xefd003ff irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci10: <ACPI PCI bus> on pcib5
vr0: <VIA VT6105 Rhine III 10/100BaseTX> port 0x4000-0x40ff mem 0xefa00000-0xefa000ff irq 16 at device 0.0 on pci10
miibus1: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> on miibus1
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:17:9a:bf:9c:97
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3080-0x308f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
atapci1: <Intel ICH7 SATA300 controller> port 0x30c8-0x30cf,0x30bc-0x30bf,0x30c0-0x30c7,0x30b8-0x30bb,0x3090-0x309f mem 0xefd00400-0xefd007ff irq 19 at device 31.2 on pci0
atapci1: AHCI Version 01.10 controller with 4 ports detected
ata2: <ATA channel 0> on atapci1
ata3: <ATA channel 1> on atapci1
ata4: <ATA channel 2> on atapci1
ata5: <ATA channel 3> on atapci1
acpi_button0: <Power Button> on acpi0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xdc000-0xdffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ukbd0: ServerEngines SE USB Device, rev 1.10/0.01, addr 2, iclass 3/1
kbd2 at ukbd0
ums0: ServerEngines SE USB Device, rev 1.10/0.01, addr 2, iclass 3/1
ums0: 8 buttons and Z dir.
Timecounters tick every 1.000 msec
acd0: CDRW <TSSTcorp CDW/DVD TS-H492C/TB01> at ata0-master UDMA33
ad4: 152627MB <Seagate ST3160812AS 3.AJJ> at ata2-master SATA150
ad6: 152627MB <Seagate ST3160827AS 3.42> at ata3-master SATA150
ar0: 152625MB <Intel MatrixRAID RAID1> status: READY
ar0: disk0 READY (master) using ad4 at ata2-master
ar0: disk1 READY (mirror) using ad6 at ata3-master
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ar0s1a
>How-To-Repeat:
Remove one of RAID-1 HDDs and reboot.
>Fix:

>Release-Note:
>Audit-Trail:

From: Taras Savchuk <taras@elantech.ru>
To: bug-followup@FreeBSD.org,  taras@elantech.ru
Cc:  
Subject: Re: i386/108924: Panics when Intel MatrixRAID RAID1 is degraded
Date: Thu, 08 Feb 2007 17:23:33 +0300

 In fresh install of FreBSD 6.1-RELEASE I have the same problem.
 
 -- 
  ,  
  "" :  , WEB-
 http://www.elantech.ru
 +7 (495) 589 68 81
 +7 (926) 575 22 11
Responsible-Changed-From-To: freebsd-i386->freebsd-bugs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Feb 13 00:50:18 UTC 2007 
Responsible-Changed-Why:  
This does not sound i386-specific. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=108924 

From: "Alex Wang" <alexwang@synology.com>
To: <bug-followup@FreeBSD.org>, <taras@elantech.ru>
Cc:  
Subject: Re: kern/108924: Panics when Intel MatrixRAID RAID1 is degraded
Date: Wed, 28 Mar 2007 15:38:25 +0800

 I have the same problem when using ICH8, RAID 5
 Unplug 1 disk from 4 disks RAID 5, after boot, the system hangs in the 
 detecting ata raid
 
 Motherboard: ASUS P5B-E
 I am running FreeBSD 6.2 Release 
 

From: "Claudio Ferronato" <claiudio@libero.it>
To: "bug-followup" <bug-followup@FreeBSD.org>,
	"taras" <taras@elantech.ru>
Cc:  
Subject: Re: kern/108924: Panics when Intel MatrixRAID RAID1 is degraded
Date: Mon,  9 Apr 2007 01:56:47 +0200

 I have an ASUS p5b deluxe. In raid 5, with 4 identical disks, if I do
 # atacontrol detach ata8
 when freebsd is up and running, after 1-2 minutes, system can't access
 the disks. Is needed a reset, but at the boot, the system reboots, in loop.
 The only thing I can do is to boot from the installation CD, run a shell
 and type this command
 # ln -sf /mnt2/usr/bin /usr/bin
 # ln -sf /mnt2/bin/dd /stand/dd
 then run atacontrol:
 atacontrol detach ata8
 atacontrol attach ata8
 atacontrol addspare ar0 ad16
 atacontrol rebuild ar0
 
 but output of # atacontrol status ar0 is always
 ar0: ATA RAID5 subdisks: ad10 ad12 ad14 ad16 status: REBUILDING 0% completed
 
 Also if I make a RAID 1, rebuilding don't start.
 Motherboard: Asus p5b deluxe
 disks (from dmesg):
 ad10: 381554MB <WDC WD4000KS-00MNB0 07.02E07> at ata5-master SATA300
 ad12: 381554MB <WDC WD4000KS-00MNB0 07.02E07> at ata6-master SATA300
 ad14: 381554MB <WDC WD4000KS-00MNB0 07.02E07> at ata7-master SATA300
 ad16: 381554MB <WDC WD4000KS-00MNB0 07.02E07> at ata8-master SATA300
 ar0: 1144655MB <Intel MatrixRAID RAID5 (stripe 64 KB)> status: READY
 
 OS: FreeBSD 6.2 Release
 ------------------------------------------------------
 Passa a Infostrada. ADSL e Telefono senza limiti e senza canone Telecom
 http://click.libero.it/infostrada
 

From: Frederic Gargula <fred@gargula.net>
To: bug-followup@FreeBSD.org,  taras@elantech.ru
Cc:  
Subject: Re: kern/108924: [ar] Panics when Intel MatrixRAID RAID1 is degraded
Date: Thu, 16 Aug 2007 23:57:17 +0200

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Hi,
 
 I've a RAID5 array (on an Intel MatrixRAID), on which I had a failed
 disk. I decided to replace it, but....
 I followed the usual detach/attach/add spare/rebuild method, but
 "atacontrol status ar0" keeps giving me:
 
 ar0: ATA RAID5 stripesize=128 subdisks: ad10 ad14 ad16 ad12 status:
 REBUILDING 0% completed
 
 as mentioned in kern/110962 and kern/110960 the rebuild process
 doesn't work on my Intel MatrixRAID (ICH8) controller.
 
 Is there any news on those bugs ?
 As you imagine, I'm really hoping to recover my data, but I'm quite
 desperate right now..
 
 I hope there's something I can do ;-)
 
 Best Regards,
 
 Fred
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.6 (Darwin)
 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
 
 iD8DBQFGxMg9CPtPpQxaVX8RAix6AKDCVvXtngIZcINzs8qPZkkfXLtpnQCgtUTk
 UzBxaC2MCBCt2zKyfY5YQNo=
 =Mr25
 -----END PGP SIGNATURE-----
 

From: Jeremy Chadwick <koitsu@freebsd.org>
To: bug-followup@FreeBSD.org, taras@elantech.ru
Cc: sos@freebsd.org, delphij@FreeBSD.org
Subject: Re: kern/108924: [ar] Panics when Intel MatrixRAID RAID1 is
	degraded
Date: Mon, 4 Feb 2008 13:55:01 -0800

 Wow, this is a fairly old problem with no solution in over a year?
 
 Here's some additional details from my testing.  This is easily
 reproducable.  I'll work on getting a kernel with DDB/KDB so one can do
 backtraces via serial console; I can provide access to this if need be.
 
 Details:
 
 * FreeBSD 7.0-RC1 (and previous 7.0 releases)
 * Supermicro SuperServer 5015M-T  (Supermicro PDSMI+ motherboard)
 * Built-in Intel ICH7 controller
 * Hot-swap backplane (which works when disks are JBOD and not using
   MatrixRAID)
 
 Installed i386 FreeBSD on ar0 without a problem:
 
   ad4: 190782MB <WDC WD2000JD-00HBB0 08.02D08> at ata2-master SATA150
   ad6: 190782MB <WDC WD2000JD-00HBB0 08.02D08> at ata3-master SATA150
   ar0: 190779MB <Intel MatrixRAID RAID1> status: READY
   ar0: disk0 READY (master) using ad4 at ata2-master
   ar0: disk1 READY (mirror) using ad6 at ata3-master
 
 But I attempted a hard failure of a disk, and reattachment of that disk,
 FreeBSD eventually made the entire mirror unusable.
 
 Here's the steps I took:
 
 1) Removed ad4 disk
      - Kernel said:
          ar0: WARNING - mirror protection lost. RAID1 array in DEGRADED mode
          subdisk4: detached
          ad4: detached
 
 2) atacontrol list
      - no sign of ad4 on ATA channel 2
 
 3) atacontrol status ar0
      ar0: ATA RAID1 status: DEGRADED
       subdisks:
         0 ---- MISSING
         1 ad6  ONLINE
 
 4) I then decided to copy some data to the array while degraded, just to
 make sure data got re-mirrored after bringing ad4 back online.
 
 5) cp /boot/kernel/kernel /usr/test
 
 6) Plugged ad4 disk back in
      - Disk LED came on for a second, then went off
      - No messages from kernel
 
 7) atacontrol list
      - no sign of ad4 on ATA channel 2
 
 8) atacontrol attach ata2
      atacontrol: ioctl(IOCATAATTACH): File exists
      - LED on ad4 disk suddenly turns on and is lit constantly
      - gstat showed no activity on ad4
 
 9) atacontrol status ar0
      - same as previous run
 
 10) atacontrol reinit ata2
       no device present
       - LED on ad4 disk shut off
 
 11) atacontrol status ar0
       - same as previous run
 
 12) atacontrol reinit ata2
       - same as previous run
 
 13) atacontrol detach ata2
 
 14) atacontrol attach ata2
       no device present
       - Kernel said:
           ata2: [ITHREAD]
 
 15) atacontrol detach ata2
 
 16) atacontrol attach ata2
       no device present
       - Kernel said:
           ata2: [ITHREAD]
 
 17) atacontrol reinit ata2
       no device present
 
 18) atacontrol list
       - no sign of ad4 on ATA channel 2
 
 19) atacontrol detach ata2
 
 20) atacontrol reinit ata2
       - Kernel immediately paniced, and machine rebooted.
       - Intel RAID BIOS showed disk 0 (ad4) as "Offline Member", but
         disk statistics (size) were available, meaning the disk was
         visible and accessible
       - Array labelled as "Degraded" in BIOS
 
 21) Booted into FreeBSD
       - Kernel started, and said:
           ad4: 190782MB <WDC WD2000JD-00HBB0 08.02D08> at ata2-master SATA150
           ad6: 190782MB <WDC WD2000JD-00HBB0 08.02D08> at ata3-master SATA150
       - Kernel immediately paniced; ar0 is never shown.
       - Process which paniced is 0 (swapper)
       - Single-user mode crashes at same point
       - Power-cycling doesn't help
 
 This thread also complains about similar issues:
 
 http://lists.freebsd.org/pipermail/freebsd-questions/2006-February/114274.html
 
 This really needs some focus.  I'd be more than happy to purchase and
 donate new hardware for testing if required.
 
 -- 
 | Jeremy Chadwick                                    jdc at parodius.com |
 | Parodius Networking                           http://www.parodius.com/ |
 | UNIX Systems Administrator                      Mountain View, CA, USA |
 | Making life hard for others since 1977.                  PGP: 4BD6C0CB |
 

From: Jeremy Chadwick <koitsu@freebsd.org>
To: bug-followup@FreeBSD.org, taras@elantech.ru
Cc: sos@freebsd.org, delphij@FreeBSD.org
Subject: Re: kern/108924: [ar] Panics when Intel MatrixRAID RAID1 is
	degraded
Date: Mon, 4 Feb 2008 15:20:11 -0800

 I've completed setting up a testbox with serial console access if anyone
 wants to poke at this.  I can leave the box like this for a few weeks,
 but after that it needs to go into our datacenter.  So "time is of the
 essence".  (Note that I didn't enable drop-to-DDB-on-serial-break, so if
 you need that feature, let me know).
 
 I'll also start digging around for details on MatrixRAID; my hope is
 that something in the MatrixRAID metadata is getting corrupted (for
 whatever reason), and having before-and-after metadata dumps might prove
 useful.
 
 -- 
 | Jeremy Chadwick                                    jdc at parodius.com |
 | Parodius Networking                           http://www.parodius.com/ |
 | UNIX Systems Administrator                      Mountain View, CA, USA |
 | Making life hard for others since 1977.                  PGP: 4BD6C0CB |
 

From: Ted Mittelstaedt <tedm@mittelstaedt.us>
To: bug-followup@FreeBSD.org, taras@elantech.ru
Cc:  
Subject: Re: kern/108924: [ar] Panics when Intel MatrixRAID RAID1 is degraded
Date: Sat, 06 Aug 2011 11:49:20 -0700

 This is the same bug as kern/102211 and was fixed years ago.
State-Changed-From-To: open->closed 
State-Changed-By: jh 
State-Changed-When: Sun Aug 7 11:31:43 UTC 2011 
State-Changed-Why:  
Duplicate of kern/102211. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=108924 
>Unformatted:
