From nobody@FreeBSD.org  Wed Dec 29 10:38:03 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 719351065672
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 29 Dec 2010 10:38:03 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (unknown [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 5FB428FC19
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 29 Dec 2010 10:38:03 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id oBTAc3O3030584
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 29 Dec 2010 10:38:03 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id oBTAc3vf030583;
	Wed, 29 Dec 2010 10:38:03 GMT
	(envelope-from nobody)
Message-Id: <201012291038.oBTAc3vf030583@red.freebsd.org>
Date: Wed, 29 Dec 2010 10:38:03 GMT
From: "Vladimir I. Pushkar" <vladimir.pushkar@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Boot from GPT ZFS root on HP BL460c G1 unstable
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         153520
>Category:       kern
>Synopsis:       [zfs] Boot from GPT ZFS root on HP BL460c G1 unstable
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    avg
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Dec 29 10:40:06 UTC 2010
>Closed-Date:    Thu Nov 22 15:03:15 UTC 2012
>Last-Modified:  Thu Nov 22 15:03:15 UTC 2012
>Originator:     Vladimir I. Pushkar
>Release:        8.2-PRERELEASE
>Organization:
Digital Ventures LLC
>Environment:
FreeBSD temp.digital-ventures.net 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0: Wed Dec 22 09:51:31 UTC 2010     root@temp.dvdev.org.ua:/usr/obj/usr/src/sys/GENERIC  amd64

>Description:
FreeBSD 8.2 installed using Fixit# environment on several HP ProLiant BL460c G1 servers with HP SmartArray mirror sometimes does not boot from ZFS pool. Installation done according to http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot with swap volume located on ZFS.
There are 2 disks configured on HP SmartArray as RAID1 array. Thus there is one logical device da0 and two GPT partitions:
# gpart show
=>       34  143305853  da0  GPT  (68G)
         34        128    1  freebsd-boot  (64K)
        162  143305725    2  freebsd-zfs  (68G)

# dmesg
ciss0: <HP Smart Array E200i> port 0x4000-0x40ff mem 0xfde80000-0xfdefffff,0xfde70000-0xfde77fff irq 18 at device 8.0 on pci11
ciss0: PERFORMANT Transport
ciss0: got 2 MSI messages]
ciss0: [ITHREAD]
da0 at ciss0 bus 0 scbus0 target 0 lun 0
da0: <COMPAQ RAID 1(1VOLUME OK> Fixed Direct Access SCSI-5 device 
da0: 135.168MB/s transfers
da0: Command Queueing enabled
da0: 69973MB (143305920 512 byte sectors: 255H 32S/T 17562C)
# diskinfo -v da0
da0
        512             # sectorsize
        73372631040     # mediasize in bytes (68G)
        143305920       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        17562           # Cylinders according to firmware.
        255             # Heads according to firmware.
        32              # Sectors according to firmware.
        P688MQ0598      # Disk ident.
# pciconf -lv
ciss0@pci0:11:8:0:      class=0x010400 card=0x3211103c chip=0x3238103c rev=0x00 hdr=0x00
    vendor     = 'Hewlett-Packard Company'
    device     = 'Smart Array E200/E200i SAS/SATA Controller'
    class      = mass storage
    subclass   = RAID
# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
zroot                      12,0G  55,0G   548M  legacy
zroot/home                   56K  55,0G    56K  /home
zroot/storage                21K  30,0G    21K  /storage
zroot/swap                   10G  65,0G    16K  -
zroot/usr                  1,31G  55,0G   760M  /usr
zroot/usr/obj                20K  55,0G    20K  /usr/obj
zroot/usr/ports             267M  55,0G   267M  /usr/ports
zroot/usr/ports/distfiles    20K  55,0G    20K  /usr/ports/distfiles
zroot/usr/ports/packages     21K  55,0G    21K  /usr/ports/packages
zroot/usr/src               311M  55,0G   311M  /usr/src
zroot/var                   125M  55,0G   247K  /var
zroot/var/db                125M  55,0G   121M  /var/db
zroot/var/db/pkg           3,94M  55,0G  3,94M  /var/db/pkg
zroot/var/log               215K  55,0G   215K  /var/log
zroot/var/mail               20K  55,0G    20K  /var/mail

# md5 /boot/gptzfsboot 
MD5 (/boot/gptzfsboot) = fd31f87e1c8e2bb4fa06b89658c63d06

# zdb -uuu zroot
Uberblock

        magic = 0000000000bab10c
        version = 15
        txg = 20726
        guid_sum = 3529677120116096247
        timestamp = 1293618442 UTC = Wed Dec 29 12:27:22 2010
        rootbp = [L0 DMU objset] 800L/200P DVA[0]=<0:1200ee400:200> DVA[1]=<0:3803ec200:200> DVA[2]=<0:5c0005c00:200> fletcher4 lzjb LE contiguous birth=20726 fill=245 cksum=c37f85b6c:4b1d45d93d8:eb0f06388e48:1f34f5d644dceb


Sometimes, after high disk load (e.g. buildworld) followed by reboot we got:
error 1 lba 32
error 1 lba 1
No ZFS pools located, can't boot

But after some numbers of reboots we can normally boot...

>How-To-Repeat:
1. Install 8.2-RC1 amd64 according to http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
2. reboot
3. rebuild/install world && kernel
4. reboot

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Jan 3 20:56:16 UTC 2011 
Responsible-Changed-Why:  
Hard to tell if this is a ZFS problem or a controller problem, but 
take a guess. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=153520 

From: "Vladislav V. Prodan" <universite@ukr.net>
To: bug-followup@FreeBSD.org, vladimir.pushkar@gmail.com
Cc:  
Subject: Re: kern/153520: [zfs] Boot from GPT ZFS root on HP BL460c G1 unstable
Date: Mon, 03 Jan 2011 23:17:08 +0200

 Acknowledge flawed work zdb -uuu
 
 # zdb -uuu tank
 Assertion failed: (rwlp->rw_count == 0), file
 /usr/src/cddl/lib/libzpool/../../../cddl/contrib/opensolaris/lib/libzpool/common/kernel.c,
 line 203.
 Аварийное завершение(core dumped)
 
 # uname -a
 FreeBSD mary-teresa.XXX.ua 9.0-CURRENT FreeBSD 9.0-CURRENT #0: Sun Aug
 29 19:00:25 EEST 2010
 vlad11@mary-teresa.XXX.ua:/usr/obj/usr/src/sys/mary-teresa.24  amd64
 
 # zpool status
   pool: tank
  state: ONLINE
  scrub: scrub completed after 20h41m with 0 errors on Sun Jan  2
 22:39:05 2011
 config:
 
         NAME        STATE     READ WRITE CKSUM
         tank        ONLINE       0     0     0
           raidz2    ONLINE       0     0     0
             ad8     ONLINE       0     0     0
             ad10    ONLINE       0     0     0
             ad12    ONLINE       0     0     0
             ad14    ONLINE       0     0     0
             ad16    ONLINE       0     0     0
             ad18    ONLINE       0     0     0  224K repaired
 
 # zfs list
 NAME   USED  AVAIL  REFER  MOUNTPOINT
 tank  2,48T   185G  2,48T  /tank
 
 
 # zdb
 tank
     version=14
     name='tank'
     state=0
     txg=5960112
     pool_guid=17628573572433446879
     hostid=2180312168
     hostname='mary-teresa.XXX.ua'
     vdev_tree
         type='root'
         id=0
         guid=17628573572433446879
         children[0]
                 type='raidz'
                 id=0
                 guid=17179795338638175685
                 nparity=2
                 metaslab_array=14
                 metaslab_shift=35
                 ashift=9
                 asize=4500909195264
                 is_log=0
                 children[0]
                         type='disk'
                         id=0
                         guid=8139272885001209353
                         path='/dev/ad8'
                         whole_disk=0
                         DTL=18
                 children[1]
                         type='disk'
                         id=1
                         guid=11968336865843447944
                         path='/dev/ad10'
                         whole_disk=0
                         DTL=23
                 children[2]
                         type='disk'
                         id=2
                         guid=9145116577681738566
                         path='/dev/ad12'
                         whole_disk=0
                         DTL=22
                 children[3]
                         type='disk'
                         id=3
                         guid=1210172854298467640
                         path='/dev/ad14'
                         whole_disk=0
                         DTL=21
                 children[4]
                         type='disk'
                         id=4
                         guid=13774245294784048495
                         path='/dev/ad16'
                         whole_disk=0
                         DTL=20
                 children[5]
                         type='disk'
                         id=5
                         guid=14124887767830680597
                         path='/dev/ad18'
                         whole_disk=0
                         DTL=19
 
 
 -- 
 Vladislav V. Prodan
 VVP24-UANIC
 +38[067]4584408
 +38[099]4060508
 vlad11@jabber.ru

From: Peter Wullinger <peter.wullinger+freebsd@googlemail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/153520: [zfs] Boot from GPT ZFS root on HP BL460c G1
 unstable.
Date: Sun, 18 Nov 2012 23:10:42 +0100

 I see this too on two identical HP Proliant 320 DL G6 with 9-STABLE.
 
 The machine usually needs some nudging in the form of warm restarts
 to boot the operating system.
 
 =3D=3D bootloader output =3D=3D
 error 1 lba 32
 error 1 lba 1
 No ZFS pools located, can't boot
 =3D=3D bootloader output =3D=3D
 
 =3D=3D gpart show =3D=3D
 =3D>       34  488326973  da0  GPT  (232G)
          34        128    1  freebsd-boot  (64k)
         162   67108864    2  freebsd-swap  (32G)
    67109026  421217981    3  freebsd-zfs  [bootme]  (200G)
 =3D=3D gpart show =3D=3D
 
 Root is on a hardware-mirror provided by the machine's HPQ Smart Array P410i
 
 The other 6 drives are configured as JBOD with a raidz2 data pool
 (not shown here)
 
 =3D=3D zpool status rpool =3D=3D
 zpool status rpool
   pool: rpool
  state: ONLINE
   scan: scrub repaired 0 in 0h12m with 0 errors on Wed Nov  7 04:47:21 2012
 config:
 
 	NAME        STATE     READ WRITE CKSUM
 	rpool       ONLINE       0     0     0
 	  da0p3     ONLINE       0     0     0
 
 errors: No known data errors
 =3D=3D zpool status rpool =3D=3D
 
 =3D=3D zdb =3D=3D
 rpool:
     version: 28
     name: 'rpool'
     state: 0
     txg: 22695230816
     pool_guid: 16981309850887562666
     hostid: 1031366389
     hostname: '...'
     vdev_children: 1
     vdev_tree:
         type: 'root'
         id: 0
         guid: 16981309850887562666
         children[0]:
             type: 'disk'
             id: 0
             guid: 6716485486554561877
             path: '/dev/da0p3'
             phys_path: '/dev/da0p3'
             whole_disk: 0
             metaslab_array: 23
             metaslab_shift: 31
             ashift: 9
             asize: 215658790912
             is_log: 0
             DTL: 65
 =3D=3D zdb =3D=3D
 
 =3D=3D dmesg =3D=3D
 Copyright (c) 1992-2012 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 9.1-PRERELEASE #3 r242394: Wed Oct 31 13:15:48 CET 2012
     src@...:/usr/obj/usr/src/sys/ML350 amd64
 CPU: Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz (2000.12-MHz K8-class =
 CPU)
   Origin =3D "GenuineIntel"  Id =3D 0x106a5  Family =3D 0x6  Model =3D 0x1a=
   Stepping =3D 5
   Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PG=
 E,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=3D0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,P=
 DCM,DCA,SSE4.1,SSE4.2,POPCNT>
   AMD Features=3D0x28100800<SYSCALL,NX,RDTSCP,LM>
   AMD Features2=3D0x1<LAHF>
   TSC: P-state invariant, performance statistics
 real memory  =3D 6442450944 (6144 MB)
 avail memory =3D 6177263616 (5891 MB)
 Event timer "LAPIC" quality 400
 ACPI APIC Table: <HP     ProLiant>
 FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 FreeBSD/SMP: 1 package(s) x 4 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  2
  cpu2 (AP): APIC ID:  4
  cpu3 (AP): APIC ID:  6
 ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20=
 110527/tbfadt-638)
 ACPI Warning: Invalid length for Pm2ControlBlock: 32, using default 8 (2011=
 0527/tbfadt-638)
 ioapic1 <Version 2.0> irqs 24-47 on motherboard
 ioapic0 <Version 2.0> irqs 0-23 on motherboard
 kbd1 at kbdmux0
 cryptosoft0: <software crypto> on motherboard
 acpi0: <HP ProLiant> on motherboard
 acpi0: Power Button (fixed)
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 cpu2: <ACPI CPU> on acpi0
 cpu3: <ACPI CPU> on acpi0
 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
 Timecounter "i8254" frequency 1193182 Hz quality 0
 Event timer "i8254" frequency 1193182 Hz quality 100
 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
 Timecounter "HPET" frequency 14318180 Hz quality 950
 Event timer "HPET" frequency 14318180 Hz quality 450
 Event timer "HPET1" frequency 14318180 Hz quality 440
 Event timer "HPET2" frequency 14318180 Hz quality 440
 Event timer "HPET3" frequency 14318180 Hz quality 440
 atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
 Event timer "RTC" frequency 32768 Hz quality 0
 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
 pcib0: <ACPI Host-PCI bridge> on acpi0
 pci0: <ACPI PCI bus> on pcib0
 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
 pci4: <ACPI PCI bus> on pcib1
 ciss0: <HP Smart Array P410i> port 0x4000-0x40ff mem 0xfb000000-0xfb3fffff,=
 0xfaff0000-0xfaff0fff irq 28 at device 0.0 on pci4
 ciss0: PERFORMANT Transport
 pcib2: <ACPI PCI-PCI bridge> at device 2.0 on pci0
 pci23: <ACPI PCI bus> on pcib2
 pcib3: <ACPI PCI-PCI bridge> at device 3.0 on pci0
 pci5: <ACPI PCI bus> on pcib3
 pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
 pci8: <ACPI PCI bus> on pcib4
 pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
 pci11: <ACPI PCI bus> on pcib5
 pcib6: <ACPI PCI-PCI bridge> at device 6.0 on pci0
 pci24: <ACPI PCI bus> on pcib6
 pcib7: <ACPI PCI-PCI bridge> at device 7.0 on pci0
 pci14: <ACPI PCI bus> on pcib7
 ciss1: <HP Smart Array P212> port 0x5000-0x50ff mem 0xfb800000-0xfbbfffff,0=
 xfb7f0000-0xfb7f0fff irq 30 at device 0.0 on pci14
 ciss1: PERFORMANT Transport
 pcib8: <ACPI PCI-PCI bridge> at device 8.0 on pci0
 pci25: <ACPI PCI bus> on pcib8
 pcib9: <ACPI PCI-PCI bridge> at device 9.0 on pci0
 pci17: <ACPI PCI bus> on pcib9
 pcib10: <ACPI PCI-PCI bridge> at device 10.0 on pci0
 pci20: <ACPI PCI bus> on pcib10
 mpt0: <LSILogic SAS/SATA Adapter> port 0x6000-0x60ff mem 0xfbff0000-0xfbff3=
 fff,0xfbfe0000-0xfbfeffff irq 33 at device 0.0 on pci20
 mpt0: MPI Version=3D1.5.16.0
 mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
 mpt0: 0 Active Volumes (2 Max)
 mpt0: 0 Hidden Drive Members (10 Max)
 pci0: <base peripheral, interrupt controller> at device 20.0 (no driver att=
 ached)
 pci0: <base peripheral, interrupt controller> at device 20.1 (no driver att=
 ached)
 pci0: <base peripheral, interrupt controller> at device 20.2 (no driver att=
 ached)
 pcib11: <ACPI PCI-PCI bridge> at device 28.0 on pci0
 pci2: <ACPI PCI bus> on pcib11
 pcib12: <ACPI PCI-PCI bridge> at device 0.0 on pci2
 pci3: <ACPI PCI bus> on pcib12
 bge0: <HP NC326i Dual Port PCI-E Gigabit Adapter, ASIC rev. 0x009003> mem 0=
 xfaef0000-0xfaefffff,0xfaee0000-0xfaeeffff irq 16 at device 4.0 on pci3
 bge0: CHIP ID 0x00009003; ASIC REV 0x09; CHIP REV 0x90; PCI-X 133 MHz
 miibus0: <MII bus> on bge0
 brgphy0: <BCM5714 1000BASE-T media interface> PHY 1 on miibus0
 brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000ba=
 seT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
 bge0: Ethernet address: 00:25:b3:ad:a5:8c
 bge1: <HP NC326i Dual Port PCI-E Gigabit Adapter, ASIC rev. 0x009003> mem 0=
 xfaed0000-0xfaedffff,0xfaec0000-0xfaecffff irq 17 at device 4.1 on pci3
 bge1: CHIP ID 0x00009003; ASIC REV 0x09; CHIP REV 0x90; PCI-X 133 MHz
 miibus1: <MII bus> on bge1
 brgphy1: <BCM5714 1000BASE-T media interface> PHY 1 on miibus1
 brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000ba=
 seT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
 bge1: Ethernet address: 00:25:b3:ad:a5:8d
 uhci0: <Intel 82801JI (ICH10) USB controller USB-A> port 0x1000-0x101f irq =
 20 at device 29.0 on pci0
 usbus0 on uhci0
 uhci1: <Intel 82801JI (ICH10) USB controller USB-B> port 0x1020-0x103f irq =
 23 at device 29.1 on pci0
 usbus1 on uhci1
 uhci2: <Intel 82801JI (ICH10) USB controller USB-C> port 0x1040-0x105f irq =
 22 at device 29.2 on pci0
 usbus2 on uhci2
 uhci3: <Intel 82801JI (ICH10) USB controller USB-F> port 0x1060-0x107f irq =
 23 at device 29.3 on pci0
 usbus3 on uhci3
 ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xfabf0000-0xfa=
 bf03ff irq 20 at device 29.7 on pci0
 usbus4: EHCI version 1.0
 usbus4 on ehci0
 pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
 pci1: <ACPI PCI bus> on pcib13
 vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem 0xf0000000-0xf7fff=
 fff,0xfadf0000-0xfadfffff irq 23 at device 3.0 on pci1
 pci1: <base peripheral> at device 4.0 (no driver attached)
 pci1: <base peripheral> at device 4.2 (no driver attached)
 uhci4: <UHCI (generic) USB controller> port 0x3800-0x381f irq 22 at device =
 4.4 on pci1
 usbus5 on uhci4
 ipmi0: <IPMI System Interface> mem 0xfacf0000-0xfacf00ff irq 21 at device 4=
 =2E6 on pci1
 ipmi0: using KSC interface
 isab0: <PCI-ISA bridge> at device 31.0 on pci0
 isa0: <ISA bus> on isab0
 atapci0: <Intel ICH10 SATA300 controller> port 0x10c0-0x10c7,0x10c8-0x10cb,=
 0x10d0-0x10d7,0x10d8-0x10db,0x10e0-0x10ef,0x10f0-0x10ff irq 17 at device 31=
 =2E5 on pci0
 ata2: <ATA channel> at channel 0 on atapci0
 ata3: <ATA channel> at channel 1 on atapci0
 acpi_tz0: <Thermal Zone> on acpi0
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 qpi0: <QPI system bus> on motherboard
 ipmi1: <IPMI System Interface> on isa0
 device_attach: ipmi1 attach returned 16
 ipmi1: <IPMI System Interface> on isa0
 device_attach: ipmi1 attach returned 16
 orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=3D0x300>
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
 coretemp0: <CPU On-Die Thermal Sensors> on cpu0
 est0: <Enhanced SpeedStep Frequency Control> on cpu0
 est: CPU supports Enhanced Speedstep, but is not recognized.
 est: cpu_vendor GenuineIntel, msr f
 device_attach: est0 attach returned 6
 p4tcc0: <CPU Frequency Thermal Control> on cpu0
 coretemp1: <CPU On-Die Thermal Sensors> on cpu1
 est1: <Enhanced SpeedStep Frequency Control> on cpu1
 est: CPU supports Enhanced Speedstep, but is not recognized.
 est: cpu_vendor GenuineIntel, msr f
 device_attach: est1 attach returned 6
 p4tcc1: <CPU Frequency Thermal Control> on cpu1
 coretemp2: <CPU On-Die Thermal Sensors> on cpu2
 est2: <Enhanced SpeedStep Frequency Control> on cpu2
 est: CPU supports Enhanced Speedstep, but is not recognized.
 est: cpu_vendor GenuineIntel, msr f
 device_attach: est2 attach returned 6
 p4tcc2: <CPU Frequency Thermal Control> on cpu2
 coretemp3: <CPU On-Die Thermal Sensors> on cpu3
 est3: <Enhanced SpeedStep Frequency Control> on cpu3
 est: CPU supports Enhanced Speedstep, but is not recognized.
 est: cpu_vendor GenuineIntel, msr f
 device_attach: est3 attach returned 6
 p4tcc3: <CPU Frequency Thermal Control> on cpu3
 ZFS filesystem version 5
 ZFS storage pool version 28
 Timecounters tick every 10.000 msec
 usbus0: 12Mbps Full Speed USB v1.0
 usbus1: 12Mbps Full Speed USB v1.0
 usbus2: 12Mbps Full Speed USB v1.0
 usbus3: 12Mbps Full Speed USB v1.0
 usbus4: 480Mbps High Speed USB v2.0
 usbus5: 12Mbps Full Speed USB v1.0
 ugen0.1: <Intel> at usbus0
 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
 ugen1.1: <Intel> at usbus1
 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
 (probe0:ciss0:0:0:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0=20
 (probe0:ciss0:0:0:0): CAM status: SCSI Status Error
 (probe0:ciss0:0:0:0): SCSI status: Check Condition
 (probe0:ciss0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command=
  operation code)
 (probe0:ciss0:0:0:0): Error 22, Unretryable error
 ugen2.1: <Intel> at usbus2
 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
 ugen3.1: <Intel> at usbus3
 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
 (probe1:ciss0:0:1:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0=20
 (probe1:ciss0:0:1:0): CAM status: SCSI Status Error
 (probe1:ciss0:0:1:0): SCSI status: Check Condition
 (probe1:ciss0:0:1:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command=
  operation code)
 (probe1:ciss0:0:1:0): Error 22, Unretryable error
 ugen4.1: <Intel> at usbus4
 uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
 ugen5.1: <0x103c> at usbus5
 uhub5: <0x103c UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
 (probe2:ciss0:0:2:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0=20
 (probe2:ciss0:0:2:0): CAM status: SCSI Status Error
 (probe2:ciss0:0:2:0): SCSI status: Check Condition
 (probe2:ciss0:0:2:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command=
  operation code)
 (probe2:ciss0:0:2:0): Error 22, Unretryable error
 ipmi0: IPMI device rev. 1, firmware rev. 2.07, version 2.0
 unknown: FAILURE - INQUIRY ILLEGAL REQUEST asc=3D0x24 ascq=3D0x00=20
 ipmi0: Number of channels 0
 (probe3:ciss0:0:3:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0=20
 (probe3:ciss0:0:3:0): CAM status: SCSI Status Error
 (probe3:ciss0:0:3:0): SCSI status: Check Condition
 (probe3:ciss0:0:3:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command=
  operation code)
 (probe3:ciss0:0:3:0): Error 22, Unretryable error
 ipmi0: Attached watchdog
 (probe4:ciss0:0:4:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0=20
 (probe4:ciss0:0:4:0): CAM status: SCSI Status Error
 (probe4:ciss0:0:4:0): SCSI status: Check Condition
 (probe4:ciss0:0:4:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command=
  operation code)
 (probe4:ciss0:0:4:0): Error 22, Unretryable error
 (probe5:ciss0:0:5:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0=20
 (probe5:ciss0:0:5:0): CAM status: SCSI Status Error
 (probe5:ciss0:0:5:0): SCSI status: Check Condition
 (probe5:ciss0:0:5:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command=
  operation code)
 (probe5:ciss0:0:5:0): Error 22, Unretryable error
 (probe6:ciss0:0:6:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0=20
 (probe6:ciss0:0:6:0): CAM status: SCSI Status Error
 (probe6:ciss0:0:6:0): SCSI status: Check Condition
 (probe6:ciss0:0:6:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command=
  operation code)
 (probe6:ciss0:0:6:0): Error 22, Unretryable error
 da0 at ciss0 bus 0 scbus0 target 0 lun 0
 da0: <COMPAQ RAID 1(1VOLUME OK> Fixed Direct Access SCSI-5 device=20
 da0: 135.168MB/s transfers
 da0: Command Queueing enabled
 da0: 238440MB (488327040 512 byte sectors: 255H 32S/T 59844C)
 da1 at ciss0 bus 0 scbus0 target 1 lun 0
 da1: <COMPAQ RAID 0  VOLUME OK> Fixed Direct Access SCSI-5 device=20
 da1: 135.168MB/s transfers
 da1: Command Queueing enabled
 da1: 953837MB (1953459632 512 byte sectors: 255H 32S/T 65535C)
 da2 at ciss0 bus 0 scbus0 target 2 lun 0
 da2: <COMPAQ RAID 0  VOLUME OK> Fixed Direct Access SCSI-5 device=20
 da2: 135.168MB/s transfers
 da2: Command Queueing enabled
 da2: 953837MB (1953459632 512 byte sectors: 255H 32S/T 65535C)
 da3 at ciss0 bus 0 scbus0 target 3 lun 0
 da3: <COMPAQ RAID 0  VOLUME OK> Fixed Direct Access SCSI-5 device=20
 da3: 135.168MB/s transfers
 da3: Command Queueing enabled
 da3: 953837MB (1953459632 512 byte sectors: 255H 32S/T 65535C)
 da4 at ciss0 bus 0 scbus0 target 4 lun 0
 da4: <COMPAQ RAID 0  VOLUME OK> Fixed Direct Access SCSI-5 device=20
 da4: 135.168MB/s transfers
 da4: Command Queueing enabled
 da4: 953837MB (1953459632 512 byte sectors: 255H 32S/T 65535C)
 da5 at ciss0 bus 0 scbus0 target 5 lun 0
 da5: <COMPAQ RAID 0  VOLUME OK> Fixed Direct Access SCSI-5 device=20
 da5: 135.168MB/s transfers
 da5: Command Queueing enabled
 da5: 953837MB (1953459632 512 byte sectors: 255H 32S/T 65535C)
 da6 at ciss0 bus 0 scbus0 target 6 lun 0
 da6: <COMPAQ RAID 0  VOLUME OK> Fixed Direct Access SCSI-5 device=20
 da6: 135.168MB/s transfers
 da6: Command Queueing enabled
 da6: 953837MB (1953459632 512 byte sectors: 255H 32S/T 65535C)
 sa0 at ciss1 bus 32 scbus3 target 3 lun 0
 sa0: <HP Ultrium 3-SCSI Q24D> Removable Sequential Access SCSI-5 device=20
 sa0: 135.168MB/s transfers
 sa0: Command Queueing enabled
 sa1 at mpt0 bus 0 scbus4 target 5 lun 0
 sa1: <HP Ultrium 3-SCSI Q25W> Removable Sequential Access SCSI-5 device=20
 sa1: 300.000MB/s transfers
 sa1: Command Queueing enabled
 cd0 at ata3 bus 0 scbus6 target 0 lun 0
 cd0: <HL-DT-ST DVD-ROM DH10N DAE1> Removable CD-ROM SCSI-0 device=20
 cd0: 3.300MB/s transfers
 cd0: Attempt to query device size failed: NOT READY, Medium not present - t=
 ray closed
 ch0 at mpt0 bus 0 scbus4 target 5 lun 1
 ch0: <HP 1x8 G2 AUTOLDR 3.80> Removable Changer SCSI-5 device=20
 ch0: 300.000MB/s transfers
 ch0: Command Queueing enabled
 SMP: AP CPU #3 Launched!
 ch0: 8 slots, 1 drive, 1 picker, 0 portals
 SMP: AP CPU #2 Launched!
 SMP: AP CPU #1 Launched!
 Timecounter "TSC-low" frequency 15625916 Hz quality 1000
 GEOM: da3: the primary GPT table is corrupt or invalid.
 GEOM: da3: using the secondary instead -- recovery strongly advised.
 GEOM: da4: the primary GPT table is corrupt or invalid.
 GEOM: da4: using the secondary instead -- recovery strongly advised.
 GEOM: da5: the primary GPT table is corrupt or invalid.
 GEOM: da5: using the secondary instead -- recovery strongly advised.
 GEOM: da6: the primary GPT table is corrupt or invalid.
 GEOM: da6: using the secondary instead -- recovery strongly advised.
 Root mount waiting for: usbus5 usbus4 usbus3 usbus2 usbus1 usbus0
 uhub0: 2 ports with 2 removable, self powered
 uhub1: 2 ports with 2 removable, self powered
 uhub2: 2 ports with 2 removable, self powered
 uhub5: 2 ports with 2 removable, self powered
 uhub3: 2 ports with 2 removable, self powered
 Root mount waiting for: usbus5 usbus4
 ugen5.2: <HP> at usbus5
 ukbd0: <Virtual Keyboard> on usbus5
 kbd2 at ukbd0
 ums0: <Virtual Mouse> on usbus5
 ums0: 3 buttons and [XY] coordinates ID=3D0
 Root mount waiting for: usbus4
 Root mount waiting for: usbus4
 uhub4: 8 ports with 8 removable, self powered
 Trying to mount root from zfs:rpool []...
 ugen3.2: <EATON Powerware> at usbus3
 =3D=3D dmesg =3D=3D
 
 --=20
 Erfahrung hei=DFt gar nichts. Man kann seine Sache auch 35 Jahre schlecht
 machen. =20
 	-- Kurt Tucholsky
Responsible-Changed-From-To: freebsd-fs->avg 
Responsible-Changed-By: avg 
Responsible-Changed-When: Wed Nov 21 16:51:30 UTC 2012 
Responsible-Changed-Why:  
Watch.  This problem is potentially fixed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=153520 
State-Changed-From-To: open->closed 
State-Changed-By: avg 
State-Changed-When: Thu Nov 22 14:42:07 UTC 2012 
State-Changed-Why:  
I believe that this problem has been fixed in r243025.  Please let me 
know if you still experience it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=153520 
>Unformatted:
