From nobody@FreeBSD.org  Wed Nov  3 09:37:50 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 64226106566C
	for <freebsd-gnats-submit@FreeBSD.org>; Wed,  3 Nov 2010 09:37:50 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 523B58FC14
	for <freebsd-gnats-submit@FreeBSD.org>; Wed,  3 Nov 2010 09:37:50 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id oA39boNQ045529
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 3 Nov 2010 09:37:50 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id oA39bo3F045528;
	Wed, 3 Nov 2010 09:37:50 GMT
	(envelope-from nobody)
Message-Id: <201011030937.oA39bo3F045528@www.freebsd.org>
Date: Wed, 3 Nov 2010 09:37:50 GMT
From: Emil Smolenski <am@raisa.eu.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [zfs] booting from raidz/raidz2 on ciss(4) doesn't work
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         151910
>Category:       kern
>Synopsis:       [zfs] booting from raidz/raidz2 on ciss(4) doesn't work
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    pjd
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Nov 03 09:40:10 UTC 2010
>Closed-Date:    
>Last-Modified:  Tue Nov 22 16:10:03 UTC 2011
>Originator:     Emil Smolenski
>Release:        FreeBSD 8.1-RELEASE
>Organization:
>Environment:
FreeBSD  8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC 2010     root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
FreeBSD 8.1 installed using Fixit# environment on HP Proliant 185 G5 with HP SmartArray does not boot from ZFS pool in raidz or raidz2 setup. Mirror-based configurations work as expected.

There are 6 disks, each configured on HP SmartArray as single disk RAID0 array. Thus there are 6 logical devices (da[0-5]).

System information gathered from Fixit# environment:

# dmesg
(...)
ciss0: <HP Smart Array P400> port 0xe800-0xe8ff mem 0xdef00000-0xdeffffff,0xdeeff000-0xdeefffff irq 35 at device 0.0 on pci4
ciss0: PERFORMANT Transport
ciss0: [ITHREAD]
(...)
da0 at ciss0 bus 0 scbus0 target 0 lun 0
da0: <COMPAQ RAID 0  VOLUME OK> Fixed Direct Access SCSI-5 device
da0: 135.168MB/s transfers
da0: Command Queueing enabled
da0: 1430767MB (2930211632 512 byte sectors: 255H 32S/T 65535C)
(...)
da1 at ciss0 bus 0 scbus0 target 1 lun 0
(...)
da2 at ciss0 bus 0 scbus0 target 2 lun 0
(...)
da3 at ciss0 bus 0 scbus0 target 3 lun 0
(...)
da4 at ciss0 bus 0 scbus0 target 4 lun 0
(...)
da5 at ciss0 bus 0 scbus0 target 5 lun 0
(...)

# diskinfo -v da0
da0
        512             # sectorsize
        1500268355584   # mediasize in bytes (1.4T)
        2930211632      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        359094          # Cylinders according to firmware.
        255             # Heads according to firmware.
        32              # Sectors according to firmware.
        PAFGL0T9SXH13E  # Disk ident.
(...)

# camcontrol devlist -v
scbus0 on ciss0 bus 0:
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 0 lun 0 (da0,pass0)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 1 lun 0 (da1,pass1)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 2 lun 0 (da2,pass2)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 3 lun 0 (da3,pass3)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 4 lun 0 (da4,pass4)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 5 lun 0 (da5,pass5)
scbus1 on ciss0 bus 32:
scbus-1 on xpt0 bus 0:
<>                                 at scbus-1 target -1 lun -1 (xpt0)

# pciconf -lv
(...)
ciss0@pci0:4:0:0:       class=0x010400 card=0x3234103c chip=0x3230103c rev=0x04 hdr=0x00
    class      = mass storage
    subclass   = RAID
(...)

I've done several tests with different configurations. Common setup:

Each logical device (array) has GPT scheme with following partitions:

=>        34  2930211565  da0  GPT  (1.4T)
          34         128    1  freebsd-boot  (64K)
         162     2097152    2  freebsd-swap  (1.0G)
     2097314     4194304    3  freebsd-zfs  (2.0G)
     6291618  2923919981       - free -  (1.4T)
(...)

Each freebsd-zfs partition has GPT label: gpt/test0, gpt/test1, etc. ZFS pool is created with following default options: canmount=off, checksum=fletcher4, atime=off, setuid=off. There are some datasets with compress=lzjb or compress=gzip option set.

Boot code is initialized on each disk this way:
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

Results from tests:
1) + One logical device -> works
2) + Mirror built on 2 logical devices -> works
3) + Mirror built on 6 LDs -> works
4) - RAIDZ2 built on 3 LDs -> doesn't work
5) - RAIDZ2 built on 6 LDs -> doesn't work
6) - RAIDZ built on 3 LDs -> doesn't work
7) - RAIDZ built on 6 LDs -> doesn't work

8) + RAIDZ built on 3 USB sticks on the same machine -> works
9) + RAIDZ built on 3 LDs on another machine (aacdX devices) -> works

There are three different error messages I encountered. It depends on where bootcodes were built.

a) Error message encountered when using bootcodes from Fixit# media:

error 1 lba 32
error 1 lba 1
error 1 lba 32
error 1 lba 1
No ZFS pools located, can't boot

b) Error message encountered when using bootcodes that I built myself:

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS object directory
Cant' find root filesystem - giving up
ZFS: unexpected object set type 0 
ZFS: unexpected object set type 0

FreeBSD/x86 boot
Default: test:/boot/kernel/kernel
boot:
ZFS: unexpected object set type 0

c) I also saw message similar to shown above, but with "ZFS: can't read MOS" message.

Example output from "status" command at "boot:" prompt (when error b appears):

boot: status pool: test
config:
          NAME STATE
          test ONLINE
            raidz1 ONLINE
              /dev/gpt/test0 ONLINE
              /dev/gpt/test1 ONLINE
              /dev/gpt/test2 ONLINE

So, the bootcode sees healthy ZFS pool with all devs available, but it can't boot from it.

More details on _working_ configuration (2) (mirror on 2 LDs):

works# zpool list test
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
test  1.98G   434M  1.56G    21%  ONLINE  -

works# zpool status test
  pool: test
 state: ONLINE
 scrub: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        test           ONLINE       0     0     0
          mirror       ONLINE       0     0     0
            gpt/test0  ONLINE       0     0     0
            gpt/test1  ONLINE       0     0     0

errors: No known data errors

works# zdb -uuu test
Uberblock

        magic = 0000000000bab10c
        version = 14
        txg = 77
        guid_sum = 2401146990467298568
        timestamp = 1287650011 UTC = Thu Oct 21 08:33:31 2010
        rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:e90ca00:200> DVA[1]=<0:26037600:200> DVA[2]=<0:3e00ce00:200> fletcher4 lzjb LE contiguous birth=77 fill=169 cksum=b02658433:48b259d7157:f39e911b0c82:2286d07e38a6a3

More details on _NOT_working_ configuration (6) (raidz on 3 LDs):

doesntwork# zpool list test
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
test  5.97G   655M  5.33G    10%  ONLINE  -

doesntwork# zpool status test
  pool: test
 state: ONLINE
 scrub: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        test           ONLINE       0     0     0
          raidz1       ONLINE       0     0     0
            gpt/test0  ONLINE       0     0     0
            gpt/test1  ONLINE       0     0     0
            gpt/test2  ONLINE       0     0     0

errors: No known data errors

doesntwork# zdb -uuu test
Uberblock

        magic = 0000000000bab10c
        version = 14
        txg = 78
        guid_sum = 8302404134133891378
        timestamp = 1287656704 UTC = Thu Oct 21 10:25:04 2010
        rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:5b9a7000:400> DVA[1]=<0:a205d800:400> DVA[2]=<0:ea016000:400> fletcher4 lzjb LE contiguous birth=78 fill=159 cksum=a13038304:4232b1013de:dc9ba0c2751d:1f1144c425d7a4

Please see also possibly the same issue reported on mailing list: http://lists.freebsd.org/pipermail/freebsd-stable/2010-October/059610.html

I also tried mm's mfsBSD with ZFSv15. System installed from it also doesn't boot. Same with FreeBSD 8.0-RELEASE.

I can provide more details on this issue and test patches.
>How-To-Repeat:
1. Install ZFS-only FreeBSD 8.1-RELEASE on HP SmartArray. Use raidz or raidz2.
2. Reboot.
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Nov 5 07:53:13 UTC 2010 
Responsible-Changed-Why:  
not sure if this is a zfs problem or a ciss problem, but make a guess 
and assign.  Committers, feel free to reassign it if wrong. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=151910 

From: Andriy Gapon <avg@icyb.net.ua>
To: bug-followup@FreeBSD.org, am@raisa.eu.org
Cc:  
Subject: Re: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesn't
 work
Date: Fri, 05 Nov 2010 15:17:05 +0200

 8.1 release had a bug which prevented booting from raidz.
 Please see http://www.freebsd.org/cgi/query-pr.cgi?pr=148655
 
 -- 
 Andriy Gapon

From: "Emil Smolenski" <am@raisa.eu.org>
To: bug-followup@freebsd.org, am@raisa.eu.org
Cc:  
Subject: Re: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesn't
 work
Date: Fri, 05 Nov 2010 15:45:39 +0100

 Please see at "Originator" field in both PRs. I'm familiar with PR 148655  
 very well.
 
 Problem in PR 148655 is related to booting from _degraded_  
 raidz/raidz2/mirror. In this PR we have problem with booting from  
 _non_degraded_ raidz/raidz2 and only when ciss(4) is used. Mirror-based  
 configurations, SATA disks, USB sticks, other RAID controllers are all OK.
 
 Of course I tested 8.1-STABLE (from Oct) and 8.0-RELEASE (where PR 148655  
 is not applicable) but there were no difference -- still can't boot from  
 raidz or raidz2.
 
 Thanks.
 
 -- 
 am

From: Guido Falsi <mad@madpilot.net>
To: bug-followup@FreeBSD.org, am@raisa.eu.org
Cc:  
Subject: Re: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesn't
 work
Date: Fri, 5 Nov 2010 16:18:48 +0100

 I can confirm this same problem with similar hardware.
 
 At present I'm booting this machine from an USB stick.
 
 Hope this helps give this PR some traction.
 
 -- 
 Guido Falsi <mad@madpilot.net>

From: Andriy Gapon <avg@icyb.net.ua>
To: Emil Smolenski <am@raisa.eu.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesn't
 work
Date: Fri, 05 Nov 2010 19:38:26 +0200

 on 05/11/2010 17:10 Emil Smolenski said the following:
 > The following reply was made to PR kern/151910; it has been noted by GNATS.
 > 
 > From: "Emil Smolenski" <am@raisa.eu.org>
 > To: bug-followup@freebsd.org, am@raisa.eu.org
 > Cc:  
 > Subject: Re: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesn't
 >  work
 > Date: Fri, 05 Nov 2010 15:45:39 +0100
 > 
 >  Please see at "Originator" field in both PRs. I'm familiar with PR 148655  
 >  very well.
 >  
 >  Problem in PR 148655 is related to booting from _degraded_  
 >  raidz/raidz2/mirror. In this PR we have problem with booting from  
 >  _non_degraded_ raidz/raidz2 and only when ciss(4) is used. Mirror-based  
 >  configurations, SATA disks, USB sticks, other RAID controllers are all OK.
 >  
 >  Of course I tested 8.1-STABLE (from Oct) and 8.0-RELEASE (where PR 148655  
 >  is not applicable) but there were no difference -- still can't boot from  
 >  raidz or raidz2.
 
 Apologies for missing that important information.
 It looks like this might be some exotic issue about ciss, BIOS (and ciss's option
 ROM) and how our boot code accesses/enumerates BIOS-visible disks.
 
 -- 
 Andriy Gapon
State-Changed-From-To: open->feedback 
State-Changed-By: pjd 
State-Changed-When: pon 8 lis 2010 15:08:24 UTC 
State-Changed-Why:  
Could you take a look at two files in FreeBSD HEAD: 

sys/boot/i386/zfsboot/zfsboot.c 
sys/boot/i386/libi386/biosdisk.c 

Look for VIRTUALBOX in there and apply the same changes to your stable/8 code 
or just modify the code to use code that is compiled with VIRTUALBOX defined. 
There is a bug in VirtualBox that the BIOS reports only one disk available, 
but if you ignore that and just look for more, you will find them. 
Maybe there is a similar bug in your BIOS? 
Please try it out and let me know. If it won't work we ca add more debug to 
see where and why it fails exactly. 


Responsible-Changed-From-To: freebsd-fs->pjd 
Responsible-Changed-By: pjd 
Responsible-Changed-When: pon 8 lis 2010 15:08:24 UTC 
Responsible-Changed-Why:  
I'll take this one. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=151910 

From: "Emil Smolenski" <am@raisa.eu.org>
To: bug-followup@freebsd.org, am@raisa.eu.org
Cc:  
Subject: Re: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesn't
 work
Date: Tue, 09 Nov 2010 15:27:02 +0100

 Thanks!
 
 Unfortunately it didn't help. This is what I've done:
 
 1. Apply patches [1]
 2. Full [build|install][world|kernel]
 3. Copy /boot/gptzfsboot and /boot/pmbr to target machine
 4. On target machine:
 # gpart bootcode -b pmbr -p gptzfsboot -i 1 da0
 # gpart bootcode -b pmbr -p gptzfsboot -i 1 da1
 # gpart bootcode -b pmbr -p gptzfsboot -i 1 da2
 5. Copy /boot/zfsloader to /boot on target machine
 6. Reboot
 7. Type 'status' command when error message appears:
 
 ZFS: i/o error - all block copies unavailable
 ZFS: can't read MOS object directory
 Cant' find root filesystem - giving up
 ZFS: unexpected object set type 0
 ZFS: unexpected object set type 0
 
 FreeBSD/x86 boot
 Default: test:/boot/kernel/kernel
 boot:
 ZFS: unexpected object set type 0
 
 FreeBSD/x86 boot
 Default: test:/boot/kernel/kernel
 boot: status pool: test
 config:
            NAME STATE
            test ONLINE
              raidz1 ONLINE
                /dev/gpt/test0 ONLINE
                /dev/gpt/test1 ONLINE
                /dev/gpt/test2 ONLINE
 
 
 
 [1]
 
 --- biosdisk.c.orig     2010-11-09 10:22:33.311797575 +0100
 +++ biosdisk.c  2010-11-09 10:23:42.471306832 +0100
 @@ -214,11 +214,6 @@
       /* sequence 0, 0x80 */
       for (base = 0; base <= 0x80; base += 0x80) {
          for (unit = base; (nbdinfo < MAXBDDEV); unit++) {
 -           /* check the BIOS equipment list for number of fixed disks */
 -           if((base == 0x80) &&
 -              (nfd >= *(unsigned char *)PTOV(BIOS_NUMDRIVES)))
 -               break;
 -
              bdinfo[nbdinfo].bd_unit = unit;
              bdinfo[nbdinfo].bd_flags = (unit < 0x80) ? BD_FLOPPY : 0;
 
 --- zfsboot.c.orig      2010-11-09 10:15:39.893495888 +0100
 +++ zfsboot.c   2010-11-09 10:21:07.376447165 +0100
 @@ -495,7 +495,7 @@
        * will find any other available pools and it may fill in missing
        * vdevs for the boot pool.
        */
 -    for (i = 0; i < *(unsigned char *)PTOV(BIOS_NUMDRIVES); i++) {
 +    for (i = 0; i < MAXBDDEV; i++) {
          if ((i | DRV_HARD) == *(uint8_t *)PTOV(ARGS))
              continue;

From: George Barnett <george@alink.co.za>
To: bug-followup@FreeBSD.org,
 am@raisa.eu.org
Cc:  
Subject: Re: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesn't work
Date: Sat, 13 Nov 2010 16:14:12 +1100

 --Apple-Mail-1--67076389
 Content-Transfer-Encoding: quoted-printable
 Content-Type: text/plain;
 	charset=us-ascii
 
 Hello,
 
 I see the same on 8.1-RELEASE.  I use raidz2 and a 'status' shows the =
 pool is there just fine, but trying to boot results in the same error =
 message:
 
 ZFS: i/o error - all block copies unavailable
 ZFS: can't read MOS object directory
 Cant' find root filesystem - giving up
 ZFS: unexpected object set type 0
 ZFS: unexpected object set type 0
 
 I do not have any disks on a CISS device though - I have standard ata =
 disks on a gigabyte sata controller.
 
 George=
 
 --Apple-Mail-1--67076389
 Content-Transfer-Encoding: quoted-printable
 Content-Type: text/html;
 	charset=us-ascii
 
 <html><head></head><body style=3D"word-wrap: break-word; =
 -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
 ">Hello,<div><br></div><div>I see the same on 8.1-RELEASE. &nbsp;I use =
 raidz2 and a 'status' shows the pool is there just fine, but trying to =
 boot results in the same error message:</div><div><br></div><div><meta =
 charset=3D"utf-8"><span class=3D"Apple-style-span" style=3D"font-family: =
 verdana, sans-serif; font-size: 11px; ">ZFS: i/o error - all block =
 copies unavailable<br>ZFS: can't read MOS object directory<br>Cant' find =
 root filesystem - giving up<br>ZFS: unexpected object set type 0<br>ZFS: =
 unexpected object set type 0</span></div><div><span =
 class=3D"Apple-style-span" style=3D"font-family: verdana, sans-serif; =
 font-size: 11px; "><br></span></div><div><span class=3D"Apple-style-span" =
 style=3D"font-family: verdana, sans-serif; font-size: 11px; ">I do not =
 have any disks on a CISS device though - I have standard ata disks on a =
 gigabyte sata controller.</span></div><div><span =
 class=3D"Apple-style-span" style=3D"font-family: verdana, sans-serif; =
 font-size: 11px; "><br></span></div><div><span class=3D"Apple-style-span" =
 style=3D"font-family: verdana, sans-serif; font-size: 11px; =
 ">George</span></div></body></html>=
 
 --Apple-Mail-1--67076389--

From: Vladimir Pushkar <vladimir.pushkar@gmail.com>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesn't work
Date: Mon, 10 Jan 2011 12:28:14 +0200

 There is another problem with ciss and ZFS boot:
 http://www.freebsd.org/cgi/query-pr.cgi?pr=153520

From: Palle Girgensohn <girgen@FreeBSD.org>
To: bug-followup@FreeBSD.org, am@raisa.eu.org
Cc: Julian Akehurst <julian@pingpong.net>
Subject: Re: kern/151910: [zfs] booting from raidz/raidz2 on ciss(4) doesnt
 work
Date: Tue, 22 Nov 2011 16:52:36 +0100

 Hi,
 
 First, it not a very good idea to run raidz on ciss, since ciss cannot 
 supply you with JBODs. Instead you have to set up a bunch of RAID-0 volumes 
 in the ciss controller, pack them together in a raidz, and see your 
 performance drop to the bottom of the ocean. Just don't is my advice, and 
 believe me, I did, and barely lived to tell the story... ;-)
 
 Seriously, we have a couple of idle machines with ciss(4) and an iLO (for 
 remote connections). If someone has the knowledge and time to try and fix 
 the problems with ciss and ZFS boot, we have the equipment for it.
 
 It is not just raidz. We tried with a standard vanilla zpool, no mirror or 
 raid at all, on top of a ciss raid-5, and it failed with RC1. [trying RC2 
 now, but seems nothing is changed?]. We also tried gptboot for ufs. It 
 fails also, and I guess this is a bigger problem?
 
 With gptboot, it just goes into a bootloop. With gptzfsboot, it fails with 
 an error message that seem well known to google. I'll get back to that when 
 we've tested with RC2.
 
 Anyone up to the task of finding this culprit, we can let you into the 
 machine remotely through the iLO. Let me know.
 
 Best reagards
 Palle Girgensohn
 girgen@FreeBSD.org
 
>Unformatted:
