From nobody@FreeBSD.org  Mon Aug 27 09:41:44 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id C2DE516A418
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 27 Aug 2007 09:41:44 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id A6E0B13C428
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 27 Aug 2007 09:41:44 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.1/8.14.1) with ESMTP id l7R9fiBq050781
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 27 Aug 2007 09:41:44 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.1/8.14.1/Submit) id l7R9fiwu050780;
	Mon, 27 Aug 2007 09:41:44 GMT
	(envelope-from nobody)
Message-Id: <200708270941.l7R9fiwu050780@www.freebsd.org>
Date: Mon, 27 Aug 2007 09:41:44 GMT
From: Troy Ablan <tablan@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: ZFS thought it was degraded when it should have been faulted
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         115856
>Category:       kern
>Synopsis:       [geli] ZFS thought it was degraded when it should have been faulted
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-geom
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 27 09:50:01 GMT 2007
>Closed-Date:    
>Last-Modified:  Thu May 28 22:15:54 UTC 2009
>Originator:     Troy Ablan
>Release:        7.0-CURRENT-200708
>Organization:
>Environment:
>Description:
For some reason, zfs thought a zpool was degraded when it should have
been faulted.  The kernel panicked at this point.  The devs are geli-
encrypted.  I was bringing them online one-by-one.  There was a double
failure during a replace of one of the devices on the previous boot.

[root@fbsd ~]# zpool status
  pool: whomp
 state: FAULTED
status: One or more devices could not be opened.  There are insufficient
        replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: none requested
config:

        NAME                  STATE     READ WRITE CKSUM
        whomp                 FAULTED      6     0     0  corrupted data
          raidz2              DEGRADED     6     0     0
            ad0s2.eli         ONLINE       0     0     0
            ad1s2             UNAVAIL      0     0     0  corrupted data
            ad2s2             UNAVAIL      0     0     0  corrupted data
            ad3s2             UNAVAIL      0     0     0  corrupted data
            ad12s2.eli        UNAVAIL      0     0     0  cannot open
            replacing         UNAVAIL      0     0     0  insufficient replicas
              ad13s2.eli/old  UNAVAIL      0     0     0  cannot open
              ad13s2.eli      UNAVAIL      0     0     0  cannot open
            ad14s2            UNAVAIL      0     0     0  corrupted data
            ad15s2            UNAVAIL      0     0     0  corrupted data
[root@fbsd ~]# zpool status
  pool: whomp
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: none requested
config:

        NAME                  STATE     READ WRITE CKSUM
        whomp                 DEGRADED     0     0     0
          raidz2              DEGRADED     0     0     0
            ad0s2.eli         ONLINE       0     0     0
            ad1s2.eli         ONLINE       0     0     0
            ad2s2.eli         ONLINE       0     0     0
            ad3s2.eli         ONLINE       0     0     0
            ad12s2.eli        UNAVAIL      0     0     0  cannot open
            replacing         UNAVAIL      0     0     0  insufficient replicas
              ad13s2.eli/old  UNAVAIL      0     0     0  cannot open
              ad13s2.eli      UNAVAIL      0     0     0  cannot open
            ad14s2.eli        ONLINE       0     0     0
            ad15s2            UNAVAIL      0     0     0  corrupted data

errors: No known data errors

>How-To-Repeat:
create a raidz2 of 8 geli devices (or devices that wouldn't be available
at boot)

geli attach them one by one

offline one of the devices, and zpool replace it (in place)

hard-reboot before the replace finishes

geli attach the devices one by one, doing a zpool status in between each
one.  zfs will think the pool is degraded while it's still missing three.

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->pjd 
Responsible-Changed-By: remko 
Responsible-Changed-When: Mon Aug 27 09:53:09 UTC 2007 
Responsible-Changed-Why:  
Over to maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=115856 
Responsible-Changed-From-To: pjd->freebsd-geom 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Thu May 28 22:15:43 UTC 2009 
Responsible-Changed-Why:  
pjd is not actively working on GEOM at the moment. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=115856 
>Unformatted:
