From nobody@FreeBSD.org  Tue Apr 24 18:42:56 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id DC234106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 24 Apr 2012 18:42:56 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id C6CDF8FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 24 Apr 2012 18:42:56 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q3OIguSC078664
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 24 Apr 2012 18:42:56 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q3OIgu6n078661;
	Tue, 24 Apr 2012 18:42:56 GMT
	(envelope-from nobody)
Message-Id: <201204241842.q3OIgu6n078661@red.freebsd.org>
Date: Tue, 24 Apr 2012 18:42:56 GMT
From: David Alves <david.alves@gmx.fr>
To: freebsd-gnats-submit@FreeBSD.org
Subject: ZFS Disks reordering causes ZFS to pick the wrong drive
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         167272
>Category:       kern
>Synopsis:       [zfs] ZFS Disks reordering causes ZFS to pick the wrong drive
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Tue Apr 24 18:50:10 UTC 2012
>Closed-Date:    
>Last-Modified:  Thu Apr 26 15:50:10 UTC 2012
>Originator:     David Alves
>Release:        8.2-RELEASE
>Organization:
>Environment:
FreeBSD xxxxxxxx 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Feb 17 02:41:51 UTC 2011     root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
Hello,

ZFS shows the disks labels when invoking zpool status, if a disk was removed (physically) and the server rebooted, a reordering of the disks is done, and it appears that the old label is used by a valid disk ( the slot of the removed disks don't contains any new disks)

ZFS reports it as follows :

	  raidz2       DEGRADED     0     0     0
	    da16       ONLINE       0     0     0
	    da17       ONLINE       0     0     0
	    da18       ONLINE       0     0     0
	    da19       ONLINE       0     0     0
	    da20       ONLINE       0     0     0
	    da21       OFFLINE      0     0     0
	    da21       ONLINE       0     0     0
	    da22       ONLINE       0     0     0
	  raidz2       DEGRADED     0     0     0
	    da23       ONLINE       0     0     0
	    da24       ONLINE       0     0     0
	    da25       ONLINE       0     0     0
	    da26       ONLINE       0     0     0
	    da27       ONLINE       0     0     0
	    da27       OFFLINE      0     0     0
	    da29       ONLINE       0     0     0
	    da30       ONLINE       0     0     0




Notice the da21 and da27 drives.
the old disks da21/da27 are shown offline (because they were offlined and removed) but the reordering as assigned those labels to others running drives.

The problem is when performing a "zpool replace", "zpool replace" will pick the first label when attempting to replace a disk

example when replacing da21:

It picked up the da21 offline drive to replace because it was the first on the list.

	  raidz2       DEGRADED     0     0     0
	    da16       ONLINE       0     0     0
	    da17       ONLINE       0     0     0
	    da18       ONLINE       0     0     0
	    da19       ONLINE       0     0     0
	    da20       ONLINE       0     0     0
	    replacing  DEGRADED     0     0     0
	      da21     OFFLINE      0     0     0
	      da31     ONLINE       0     0     0  37.1G resilvered
	    da21       ONLINE       0     0     0
	    da22       ONLINE       0     0     1  512 resilvered
	  raidz2       DEGRADED     0     0     0
	    da23       ONLINE       0     0     0
	    da24       ONLINE       0     0     0
	    da25       ONLINE       0     0     0
	    da26       ONLINE       0     0     0
	    da27       ONLINE       0     0     0
	    da27       OFFLINE      0     0     0
	    da29       ONLINE       0     0     0
	    da30       ONLINE       0     0     0

example when replacing da27:

It picked up the da27 online drive to replace because it was the first on the list.

	  raidz2       ONLINE       0     0     0
	    da16       ONLINE       0     0     0
	    da17       ONLINE       0     0     0
	    da18       ONLINE       0     0     0
	    da19       ONLINE       0     0     0
	    da20       ONLINE       0     0     0
	    da31       ONLINE       0     0     0
	    da21       ONLINE       0     0     0
	    da22       ONLINE       0     0     1
	  raidz2       DEGRADED     0     0     0
	    da23       ONLINE       0     0     0
	    da24       ONLINE       0     0     0
	    da25       ONLINE       0     0     0
	    da26       ONLINE       0     0     0
	    replacing  ONLINE       0     0     0
	      da27     ONLINE       0     0     0
	      da28     ONLINE       0     0     0  80.5G resilvered
	    da27       OFFLINE      0     0     0
	    da29       ONLINE       0     0     0
	    da30       ONLINE       0     0     0


That would be nice if we can choose exactly what drive from the pool we are going to replace.

Thanks you.
>How-To-Repeat:
To repeat the problem:

offline a drive
remove the drive
reboot

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Apr 25 05:00:27 UTC 2012 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=167272 

From: Anna-Maria <ichhasseesnamenauszuwaehlen@yahoo.ca>
To: bug-followup@FreeBSD.org, david.alves@gmx.fr
Cc:  
Subject: Re: kern/167272: [zfs] ZFS Disks reordering causes ZFS to pick the
 wrong drive
Date: Thu, 26 Apr 2012 17:40:19 +0200

 This is a multi-part message in MIME format.
 --------------060700060302040105000702
 Content-Type: text/plain; charset=ISO-8859-15
 Content-Transfer-Encoding: 7bit
 
 You can use the "zdb" command to see the guid of the disk, and use that
 to replace it.
 
 eg.
 
         children[2]:
             type: 'disk'
             id: 2
             guid: *713716085525288922*
             path: '/dev/*da21*'
             phys_path: '/dev/*da21*'
             whole_disk: 1
             *not_present: 1*
             metaslab_array: 13154506
             metaslab_shift: 25
             ashift: 9
             asize: 4289200128
             is_log: 1
             DTL: 13647709
             create_txg: 2302081
 
 zpool replace 713716085525288922 da31
 
 Also, I had a barely related problem with 8.2-RELEASE (or may have been
 the 8-STABLE from April which is in an iso on the ftp site), where
 rebooting would make ZFS use the wrong disk in the pool. (eg. my hot
 spare might end up being part of the raidz2, and so constantly create
 checksum errors until I scrub it or move it back). And then I tried
 warning others about it, and they would always say that I make no sense,
 and zfs uses the guid so can't get confused, etc. and only shows the
 label/device on the command output. So I tried reproducing it, and it
 was impossible for me to reproduce with 8-STABLE (October 2011). Your
 problem isn't quite the same, but may be related.
 
 Also in 8-STABLE, any time I do weird stuff like you did, the disk shows
 the guid on the next reboot instead of the old device name, and on the
 right it will add a comment saying "was da21". (but mostly I use gpt
 slices, so I might just not have enough experience with that particular
 problem).
 
 So I highly recommend you try 8.3-RELEASE (or 8-STABLE).
 
 And one more warning... if you ugrade to newer than 8.2-RELEAE, and have
 an old pool from 8.2-RELEASE and upgrade it to v28, you still can't
 remove logs. It is bugged. You need to destroy and recreate.
 
 --------------060700060302040105000702
 Content-Type: text/html; charset=ISO-8859-15
 Content-Transfer-Encoding: 8bit
 
 <html>
   <head>
 
     <meta http-equiv="content-type" content="text/html; charset=ISO-8859-15">
   </head>
   <body bgcolor="#FFFFFF" text="#000000">
     You can use the "zdb" command to see the guid of the disk, and use
     that to replace it.<br>
     <br>
     eg. <br>
     <br>
      children[2]:<br>
      type: 'disk'<br>
      id: 2<br>
      guid: <b>713716085525288922</b><br>
      path: '/dev/<b>da21</b>'<br>
      phys_path: '/dev/<b>da21</b>'<br>
      whole_disk: 1<br>
      <b>not_present: 1</b><br>
      metaslab_array: 13154506<br>
      metaslab_shift: 25<br>
      ashift: 9<br>
      asize: 4289200128<br>
      is_log: 1<br>
      DTL: 13647709<br>
      create_txg: 2302081<br>
     <br>
     zpool replace 713716085525288922 da31<br>
     <br>
     Also, I had a barely related problem with 8.2-RELEASE (or may have
     been the 8-STABLE from April which is in an iso on the ftp site),
     where rebooting would make ZFS use the wrong disk in the pool. (eg.
     my hot spare might end up being part of the raidz2, and so
     constantly create checksum errors until I scrub it or move it back).
     And then I tried warning others about it, and they would always say
     that I make no sense, and zfs uses the guid so can't get confused,
     etc. and only shows the label/device on the command output. So I
     tried reproducing it, and it was impossible for me to reproduce with
     8-STABLE (October 2011). Your problem isn't quite the same, but may
     be related.<br>
     <br>
     Also in 8-STABLE, any time I do weird stuff like you did, the disk
     shows the guid on the next reboot instead of the old device name,
     and on the right it will add a comment saying "was da21". (but
     mostly I use gpt slices, so I might just not have enough experience
     with that particular problem).<br>
     <br>
     So I highly recommend you try 8.3-RELEASE (or 8-STABLE).<br>
     <br>
     And one more warning... if you ugrade to newer than 8.2-RELEAE, and
     have an old pool from 8.2-RELEASE and upgrade it to v28, you still
     can't remove logs. It is bugged. You need to destroy and recreate.<br>
   </body>
 </html>
 
 --------------060700060302040105000702--
>Unformatted:
