From nobody@FreeBSD.org  Tue Jul  8 19:37:10 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 0B1FC106567F
	for <freebsd-gnats-submit@FreeBSD.org>; Tue,  8 Jul 2008 19:37:10 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id ECE628FC13
	for <freebsd-gnats-submit@FreeBSD.org>; Tue,  8 Jul 2008 19:37:09 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m68Jb9dv092575
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 8 Jul 2008 19:37:09 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.2/8.14.1/Submit) id m68Jb9sR092574;
	Tue, 8 Jul 2008 19:37:09 GMT
	(envelope-from nobody)
Message-Id: <200807081937.m68Jb9sR092574@www.freebsd.org>
Date: Tue, 8 Jul 2008 19:37:09 GMT
From: Javier Martn Rueda <jmrueda@diatel.upm.es>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Panic when doing zfs raidz with gmirror and ggate
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         125413
>Category:       kern
>Synopsis:       [zfs] [panic] Panic when doing zfs raidz with gmirror and ggate
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    pjd
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jul 08 19:40:06 UTC 2008
>Closed-Date:    Wed Sep 09 20:33:32 UTC 2009
>Last-Modified:  Wed Sep 09 20:33:32 UTC 2009
>Originator:     Javier Martn Rueda
>Release:        FreeBSD 7.0-STABLE
>Organization:
DIATEL - UPM
>Environment:
FreeBSD fuego2.pruebas.local 7.0-STABLE FreeBSD 7.0-STABLE #0: Thu Jul  3 17:21:29 CEST 2008     root@fuego2.pruebas.local:/usr/src/sys/i386/compile/REPLICACION  i386

>Description:
I have two FreeBSD machines with 8 disks each. I am trying to create a replicated raidz ZFS pool using gmirror and ggate. I export the disks on one of the machines with ggate, and then create 8 gmirrors on the other one, each with two providers (the local disk, and the correspondig remote ggate disk). To clarify, this is the output of gmirror status:

# gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  ggate0
                      da0
mirror/gm1  COMPLETE  ggate1
                      da1
mirror/gm2  COMPLETE  ggate2
                      da2
mirror/gm3  COMPLETE  ggate3
                      da3
mirror/gm4  COMPLETE  ggate4
                      da4
mirror/gm5  COMPLETE  ggate5
                      da5
mirror/gm6  COMPLETE  ggate6
                      da6
mirror/gm7  COMPLETE  ggate7
                      da7

Now, if I create a non-raidz zpool, everything is fine:

# zpool create z1 mirror/gm0 mirror/gm1 mirror/gm2 mirror/gm3 mirror/gm4 mirror/gm5 mirror/gm6 mirror/gm7
# zpool status
  pool: z1
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        z1            ONLINE       0     0     0
          mirror/gm0  ONLINE       0     0     0
          mirror/gm1  ONLINE       0     0     0
          mirror/gm2  ONLINE       0     0     0
          mirror/gm3  ONLINE       0     0     0
          mirror/gm4  ONLINE       0     0     0
          mirror/gm5  ONLINE       0     0     0
          mirror/gm6  ONLINE       0     0     0
          mirror/gm7  ONLINE       0     0     0

errors: No known data errors

However, if I try to create a pool with raidz or raidz2, I get a panic. The sentence that causes the page fault is vdev_geom.c:420, when it tries to access a null pointer.

If I create the gmirrors with just the local disk as provider, there  is no panic in any case (raidz or raidz2). So, it seems that ggate has something to do with it, or more likely it unhides a problem somewhere else.

All this happened also with 7.0-RELEASE.

I have been looking at the code for a while, and the sequence of function calls that triggers the panic is this:

1) For some reason, zio_vdev_io_assess() tells SPA to reopen the vdev

2) vdev_reopen() calls vdev_close(), and then it will call vdev_open()

2.1) vdev_close() queues several events to close the 8 devices, but returns before they have been completely closed. The subsequent call to vdev_open() finds the devices still there and reuses them. However, eventually the events from vdev_close will dettach them, and that's when the problem comes, because suddenly a provider that was there vanishes. It looks like a race condition.

>How-To-Repeat:
It is explained in detail above. Summarizing:

The servers are fuego1 and fuego2. Both have 8 data disks (da0 - da7), appart from the system disks. Execute the following on fuego1:

ggatec create -u 0 fuego2 /dev/da0
ggatec create -u 1 fuego2 /dev/da1
ggatec create -u 2 fuego2 /dev/da2
ggatec create -u 3 fuego2 /dev/da3
ggatec create -u 4 fuego2 /dev/da4
ggatec create -u 5 fuego2 /dev/da5
ggatec create -u 6 fuego2 /dev/da6
ggatec create -u 7 fuego2 /dev/da7
gmirror label -h -b prefer gm0 da0 ggate0
gmirror label -h -b prefer gm1 da1 ggate1
gmirror label -h -b prefer gm2 da2 ggate2
gmirror label -h -b prefer gm3 da3 ggate3
gmirror label -h -b prefer gm4 da4 ggate4
gmirror label -h -b prefer gm5 da5 ggate5
gmirror label -h -b prefer gm6 da6 ggate6
gmirror label -h -b prefer gm7 da7 ggate7
zpool create z1 raidz2 mirror/gm0 mirror/gm1 mirror/gm2 mirror/gm3 mirror/gm4 mirror/gm5 mirror/gm6 mirror/gm7

And you'll get a panic.

>Fix:
I don't know a good fix, but I attach a shoddy patch that seems to work (and reinforces my belief that it is a race condition). Basically, I insert a delay between vdev_close and vdev_open in vdev_reopen, so that by the time vdev_open gets called all the closes have been completely finished.


Patch attached with submission follows:

--- vdev.c      2008-04-17 03:23:33.000000000 +0200
+++ /tmp/vdev.c 2008-07-08 21:27:35.000000000 +0200
@@ -1023,6 +1023,7 @@
        ASSERT(spa_config_held(spa, RW_WRITER));

        vdev_close(vd);
+       pause("chapuza", 2000);
        (void) vdev_open(vd);

        /*


>Release-Note:
>Audit-Trail:

From: Kris Kennaway <kris@FreeBSD.org>
To: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= <jmrueda@diatel.upm.es>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/125413: Panic when doing zfs raidz with gmirror and ggate
Date: Tue, 08 Jul 2008 22:43:58 +0200

 Javier Martn Rueda wrote:
 
 > And you'll get a panic.
 
 You forgot to show us the panic ;)
 
 Kris
 

From: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= <jmrueda@diatel.upm.es>
To: bug-followup@FreeBSD.org, jmrueda@diatel.upm.es
Cc:  
Subject: Re: kern/125413: Panic when doing zfs raidz with gmirror and ggate
Date: Wed, 09 Jul 2008 02:52:39 +0200

 This is a multi-part message in MIME format.
 --------------080503070700060808010208
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit
 
 I attach a console dump of the panic.
 
 
 
 --------------080503070700060808010208
 Content-Type: text/plain;
  name="p2bis.txt"
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline;
  filename="p2bis.txt"
 
 NOTE: in this dump, I was using da0s1, da1s1, etc. instead of da0, da1...
 
 # sysctl kern.geom.debugflags=5
 # sysctl kern.geom.mirror.debug=2
 # sysctl vfs.zfs.debug=1
 # zpool create z1 raidz2 mirror/gm0 mirror/gm1 mirror/gm2 mirror/gm3 mirror/gm4 mirror/gm5 mirror/gm6 mirror/gm7
 
 g_dev_open(mirror/gm0, 1, 8192, 0xc422d880)
 g_access(0xc3fb8d40(mirror/gm0), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Access request for mirror/gm0: r1w0e0.
 g_dev_close(mirror/gm0, 1, 8192, 0xc422d880)
 g_access(0xc3fb8d40(mirror/gm0), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Accest for mirror/gm0: r-1w0e0.
 g_dev_open(mirror/gm0, 1, 8192, 0xc422d880)
 g_access(0xc3fb8d40(mirror/gm0), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Access request for mirror/gm0: r1w0e0.
 g_dev_close(mirror/gm0, 1, 8192, 0xc422d880)
 g_access(0xc3fb8d40(mirror/gm0), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Access request for mirror/gm0: r-1w0e0.
 g_dev_open(mirror/gm1, 1, 8192, 0xc422d880)
 g_access(0xc419c100(mirror/gm1), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3b57700(mirror/gm1)
 GEOM_MIRROR[2]: Access request for mirror/gm1: r1w0e0.
 g_dev_close(mirror/gm1, 1, 8192, 0xc422d880)
 g_access(0xc419c100(mirror/gm1), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3b57700(mirror/gm1)
 GEOM_MIRROR[2]: Access request for mirror/gm1: r-1w0e0.
 g_dev_open(mirror/gm1, 1, 8192, 0xc422d880)
 g_access(0xc419c100(mirror/gm1), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3b57700(mirror/gm1)
 GEOM_MIRROR[2]: Access request for mirror/gm1: r1w0e0.
 g_dev_close(mirror/gm1, 1, 8192, 0xc422d880)
 g_access(0xc419c100(mirror/gm1), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3b57700(mirror/gm1)
 GEOM_MIRROR[2]: Access request for mirror/gm1: r-1w0e0.
 g_dev_open(mirror/gm2, 1, 8192, 0xc422d880)
 g_access(0xc419dbc0(mirror/gm2), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3f1d400(mirror/gm2)
 GEOM_MIRROR[2]: Access request for mirror/gm2: r1w0e0.
 g_dev_close(mirror/gm2, 1, 8192, 0xc422d880)
 g_access(0xc419dbc0(mirror/gm2), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3f1d400(mirror/gm2)
 GEOM_MIRROR[2]: Access request for mirror/gm2: r-1w0e0.
 g_dev_open(mirror/gm2, 1, 8192, 0xc422d880)
 g_access(0xc419dbc0(mirror/gm2), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3f1d400(mirror/gm2)
 GEOM_MIRROR[2]: Access request for mirror/gm2: r1w0e0.
 g_dev_close(mirror/gm2, 1, 8192, 0xc422d880)
 g_access(0xc419dbc0(mirror/gm2), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3f1d400(mirror/gm2)
 GEOM_MIRROR[2]: Access request for mirror/gm2: r-1w0e0.
 g_dev_open(mirror/gm3, 1, 8192, 0xc422d880)
 g_access(0xc4b60c00(mirror/gm3), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3f2a900(mirror/gm3)
 GEOM_MIRROR[2]: Access request for mirror/gm3: r1w0e0.
 g_dev_close(mirror/gm3, 1, 8192, 0xc422d880)
 g_access(0xc4b60c00(mirror/gm3), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3f2a900(mirror/gm3)
 GEOM_MIRROR[2]: Access request for mirror/gm3: r-1w0e0.
 g_dev_open(mirror/gm3, 1, 8192, 0xc422d880)
 g_access(0xc4b60c00(mirror/gm3), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3f2a900(mirror/gm3)
 GEOM_MIRROR[2]: Access request for mirror/gm3: r1w0e0.
 g_dev_close(mirror/gm3, 1, 8192, 0xc422d880)
 g_access(0xc4b60c00(mirror/gm3), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3f2a900(mirror/gm3)
 GEOM_MIRROR[2]: Access request for mirror/gm3: r-1w0e0.
 g_dev_open(mirror/gm4, 1, 8192, 0xc422d880)
 g_access(0xc3fb8dc0(mirror/gm4), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3ecf000(mirror/gm4)
 GEOM_MIRROR[2]: Access request for mirror/gm4: r1w0e0.
 g_dev_close(mirror/gm4, 1, 8192, 0xc422d880)
 g_access(0xc3fb8dc0(mi0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3ecf000(mirror/gm4)
 GEOM_MIRROR[2]: Access request for mirror/gm4: r-1w0e0.
 g_dev_open(mirror/gm4, 1, 8192, 0xc422d880)
 g_access(0xc3fb8dc0(mirror/gm4), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3ecf000(mirror/gm4)
 GEOM_MIRROR[2]: Access request for mirror/gm4: r1w0e0.
 g_dev_close(mirror/gm4, 1, 8192, 0xc422d880)
 g_access(0xc3fb8dc0(mirror/gm4), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3ecf000(mirror/gm4)
 GEOM_MIRROR[2]: Access request for mirror/gm4: r-1w0e0.
 g_dev_open(mirror/gm5, 1, 8192, 0xc422d880)
 g_access(0xc3fb94c0(mirror/gm5), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc417e400(mirror/gm5)
 GEOM_MIRROR[2]: Access request for mirror/gm5: r1w0e0.
 g_dev_close(mirror/gm5, 1, 8192, 0xc422d880)
 g_access(0xc3fb94c0(mirror/gm5), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc417e400(mirror/gm5)
 GEOM_MIRROR[2]: Access request for mirror/gm5: r-1w0e0.
 g_dev_open(mirror/gm5, 1, 8192, 0xc422d880)
 g_access(0xc3fb94c0(mirror/gm5), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc417e400(mirror/gm5)
 GEOM_MIRROR[2]: Access request for mirror/gm5: r1w0e0.
 g_dev_close(mirror/gm5, 1, 8192, 0xc422d880)
 g_access(0xc3fb94c0(mirror/gm5), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc417e400(mirror/gm5)
 GEOM_MIRROR[2]: Access request for mirror/gm5: r-1w0e0.
 g_dev_open(mirror/gm6, 1, 8192, 0xc422d880)
 g_access(0xc4b60c40(mirror/gm6), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3ed5000(mirror/gm6)
 GEOM_MIRROR[2]: Access request for mirror/gm6: r1w0e0.
 g_dev_close(mirror/gm6, 1, 8192, 0xc422d880)
 g_access(0xc4b60c40(mirror/gm6), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3ed5000(mirror/gm6)
 GEOM_MIRROR[2]: Access request for mirror/gm6: r-1w0e0.
 g_dev_open(mirror/gm6, 1, 8192, 0xc422d880)
 g_access(0xc4b60c40(mirror/gm6), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3ed5000(mirror/gm6)
 GEOM_MIRROR[2]: Access request for mirror/gm6: r1w0e0.
 g_dev_close(mirror/gm6, 1, 8192, 0xc422d880)
 g_access(0xc4b60c40(mirror/gm6), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3ed5000(mirror/gm6)
 GEOM_MIRROR[2]: Access request for mirror/gm6: r-1w0e0.
 g_dev_open(mirror/gm7, 1, 8192, 0xc422d880)
 g_access(0xc419c780(mirror/gm7), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc417e180(mirror/gm7)
 GEOM_MIRROR[2]: Access request for mirror/gm7: r1w0e0.
 g_dev_close(mirror/gm7, 1, 8192, 0xc422d880)
 g_access(0xc419c780(mirror/gm7), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc417e180(mirror/gm7)
 GEOM_MIRROR[2]: Access request for mirror/gm7: r-1w0e0.
 g_dev_open(mirror/gm7, 1, 8192, 0xc422d880)
 g_access(0xc419c780(mirror/gm7), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc417e180(mirror/gm7)
 GEOM_MIRROR[2]: Access request for mirror/gm7: r1w0e0.
 g_dev_close(mirror/gm7, 1, 8192, 0xc422d880)
 g_access(0xc419c780(mirror/gm7), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc417e180(mirror/gm7)
 GEOM_MIRROR[2]: Access request for mirror/gm7: r-1w0e0.
 g_post_t_x(0xc0720910, 0xc4a69560, 2, 262144)
 g_post_event_x(0xc0720910, 0xc48c9280, 2, 262144)
 g_post_event_x(0xc0720910, 0xc44c1080, 2, 262144)
 g_post_event_x(0xc0720910, 0xc496d120, 2, 262144)
 g_post_event_x(0xc0720910, 0xc4abd0a0, 2, 262144)
 g_post_event_x(0xc0720910, 0xc4abd0e0, 2, 262144)
 g_post_event_x(0xc0720910, 0xc4abd120, 2, 262144)
 g_post_event_x(0xc0720910, 0xc4abd160, 2, 262144)
 g_dev_open(mirror/gm0, 1, 8192, 0xc422d880)
 g_access(0xc3fb8d40(mirror/gm0), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Access request for mirror/gm0: r1w0e0.
 g_dev_close(mirror/gm0, 1, 8192, 0xc422d880)
 g_access(0xc3fb8d40(mirror/gm0), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Access request for mirror/gm0: r-1w0e0.
 g_dev_open(mirror/gm1, 1, 8192, 0xc422d880)
 g_access(0xc419c100(mirror/gm1), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3b57700(mirror/gm1)
 GEOM_MIRROR[2]: Access request for mirror/gm1: r1w0e0.
 g_dev_close(mirror/gm1, 1, 8192, 0xc422d880)
 g_access(0xc419c100(mirror/gm1), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3b57700(mirror/gm1)
 GEOM_MIRROR[2]: Access request for mirror/gm1: r-1w0e0.
 g_dev_open(mirror/gm2, 1, 8192, 0xc422d880)
 g_access(0xc419dbc0(mirror/gm2), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3f1d400(mirror/gm2)
 GEOM_MIRROR[2]: Access request for mirror/gm2: r1w0e0.
 g_dev_close(mirror/gm2, 1, 8192, 0xc422d880)
 g_access(0xc419dbc0(mirror/gm2), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3f1d400(mirror/gm2)
 GEOM_MIRROR[2]: Access request for mirror/gm2: r-1w0e0.
 g_dev_open(mirror/gm3, 1, 8192, 0xc422d880)
 g_access(0xc4b60c00(mirror/gm3), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3f2a900(mirror/gm3)
 GEOM_MIRROR[2]: Access request for mirror/gm3: r1w0e0.
 g_dev_close(mirror/gm3, 1, 8192, 0xc422d880)
 g_access(0xc4b60c00(mirror/gm3), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3f2a900(mirror/gm3)
 GEOM_MIRROR[2]: Access request for mirror/gm3: r-1w0e0.
 g_dev_open(mirror/gm4, 1, 8192, 0xc422d880)
 g_access(0xc3fb8dc0(mirror/gm4), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3ecf000(mirror/gm4)
 GEOM_MIRROR[2]: Access request for mirror/gm4: r1w0e0.
 g_dev_close(mirror/gm4, 1, 8192, 0xc422d880)
 g_access(0xc3fb8dc0(mirror/gm4), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3ecf000(mirror/gm4)
 GEOM_MIRROR[2]: Access request for mirror/gm4: r-1w0e0.
 g_dev_open(mirror/gm5, 1, 8192, 0xc422d880)
 g_access(0xc3fb94c0(mirror/gm5), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc417e400(mirror/gm5)
 GEOM_MIRROR[2]: Access request for mirror/gm5: r1w0e0.
 g_dev_close(mirror/gm5, 1, 8192, 0xc422d880)
 g_access(0xc3fb94c0(mirror/gm5), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc417e400(mirror/gm5)
 GEOM_MIRROR[2]: Access request for mirror/gm5: r-1w0e0.
 g_dev_open(mirror/gm6, 1, 8192, 0xc422d880)
 g_access(0xc4b60c40(mirror/gm6), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc3ed5000(mirror/gm6)
 GEOM_MIRROR[2]: Access request for mirror/gm6: r1w0e0.
 g_dev_close(mirror/gm6, 1, 8192, 0xc422d880)
 g_access(0xc4b60c40(mirror/gm6), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc3ed5000(mirror/gm6)
 GEOM_MIRROR[2]: Access request for mirror/gm6: r-1w0e0.
 g_dev_open(mirror/gm7, 1, 8192, 0xc422d880)
 g_access(0xc419c780(mirror/gm7), 1, 0, 0)
 open delta:[r1w0e0] old:[r0w0e0] provider:[r0w0e0] 0xc417e180(mirror/gm7)
 GEOM_MIRROR[2]: Access request for mirror/gm7: r1w0e0.
 g_dev_close(mirror/gm7, 1, 8192, 0xc422d880)
 g_access(0xc419c780(mirror/gm7), -1, 0, 0)
 open delta:[r-1w0e0] old:[r1w0e0] provider:[r1w0e0] 0xc417e180(mirror/gm7)
 GEOM_MIRROR[2]: Access request for mirror/gm7: r-1w0e0.
 vdev_geom_open:371[1]: Found provider by name /dev/mirror/gm0.
 vdev_geom_attach:116[1]: Attaching to mirror/gm0.
 g_access(0xc41a4800(mirror/gm0), 1, 1, 1)
 open delta:[r1w1e1] old:[r0w0e0] provider:[r0w0e0] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Access request for mirror/gm0: r1w1e1.
 g_post_event_x(0xc0724990, 0xc4184980, 2, 0)
   ref 0xc4184980
 vdev_geom_attach:137[1]: Created geom and consumer for mirror/gm0.
 vdev_geom_open:371[1]: Found provider by name /dev/mirror/gm1.
 vdev_geom_attach:116[1]: Attaching to mirror/gm1.
 g_access(0xc41a49c0(mirror/gm1), 1, 1, 1)
 open delta:[r1w1e1] old:[r0w0e0] provider:[r0w0e0] 0xc3b57700(mirror/gm1)
 GEOM_MIRROR[2]: Access request for mirror/gm1: r1w1e1.
 g_post_event_x(0xc0724990, 0xc3b57700, 2, 0)
   ref 0xc3b57700
 vdev_geom_attach:157[1]: Created consumer for mirror/gm1.
 vdev_geom_open:371[1]: Found provider by name /dev/mirror/gm2.
 vdev_geom_attach:116[1]: Attaching to mirror/gm2.
 g_access(0xc419da80(mirror/gm2), 1, 1, 1)
 open delta:[r1w1e1] old:[r0w0e0] provider:[r0w0e0] 0xc3f1d400(mirror/gm2)
 GEOM_MIRROR[2]: Access request for mirror/gm2: r1w1e1.
 g_post_event_x(0xc0724990, 0xc3f1d400, 2, 0)
   ref 0xc3f1d400
 vdev_geom_attach:157[1]: Created consumer for mirror/gm2.
 vdev_geom_open:371[1]: Found provider by name /dev/mirror/gm3.
 vdev_geom_attach:116[1]: Attaching to mirror/gm3.
 g_access(0xc4b60940(mirror/gm3), 1, 1, 1)
 open delta:[r1w1e1] old:[r0w0e0] provider:[r0w0e0] 0xc3f2a900(mirror/gm3)
 GEOM_MIRROR[2]: Access request for mirror/gm3: r1w1e1.
 g_post_event_x(0xc0724990, 0xc3f2a900, 2, 0)
   ref 0xc3f2a900
 vdev_geom_attach:157[1]: Created consumer for mirror/gm3.
 vdev_geom_open:371[1]: Found provider by name /dev/mirror/gm4.
 vdev_geom_attach:116[1]: Attaching to mirror/gm4.
 g_access(0xc4b61140(mirror/gm4), 1, 1, 1)
 open delta:[r1w1e1] old:[r0w0e0] provider:[r0w0e0] 0xc3ecf000(mirror/gm4)
 GEOM_MIRROR[2]: Access request for mirror/gm4: r1w1e1.
 g_post_event_x(0xc0724990, 0xc3ecf000, 2, 0)
   ref 0xc3ecf000
 vdev_geom_attach:157[1]: Created consumer for mirror/gm4.
 vdev_geom_open:371[1]: Found provider by name /dev/mirror/gm5.
 vdev_geom_attach:116[1]: Attaching to mirror/gm5.
 g_access(0xc4b60800(mirror/gm5), 1, 1, 1)
 open delta:[r1w1e1] old:[r0w0e0] provider:[r0w0e0] 0xc417e400(mirror/gm5)
 GEOM_MIRROR[2]: Access request for mirror/gm5: r1w1e1.
 g_post_event_x(0xc0724990, 0xc417e400, 2, 0)
   ref 0xc417e400
 vdev_geom_attach:157[1]: Created consumer for mirror/gm5.
 vdev_geom_open:371[1]: Found provider by name /dev/mirror/gm6.
 vdev_geom_attach:116[1]: Attaching to mirror/gm6.
 g_access(0xc4b60c80(mirror/gm6), 1, 1, 1)
 open delta:[r1w1e1] old:[r0w0e0] provider:[r0w0e0] 0xc3ed5000(mirror/gm6)
 GEOM_MIRROR[2]: Access request for mirror/gm6: r1w1e1.
 g_post_event_x(0xc0724990, 0xc3ed5000, 2, 0)
   ref 0xc3ed5000
 vdev_geom_attach:157[1]: Created consumer for mirror/gm6.
 vdev_geom_open:371[1]: Found provider by name /dev/mirror/gm7.
 vdev_geom_attach:116[1]: Attaching to mirror/gm7.
 g_access(0xc419d240(mirror/gm7), 1, 1, 1)
 open delta:[r1w1e1] old:[r0w0e0] provider:[r0w0e0] 0xc417e180(mirror/gm7)
 GEOM_MIRROR[2]: Access request for mirror/gm7: r1w1e1.
 g_post_event_x(0xc0724990, 0xc417e180, 2, 0)
   ref 0xc417e180
 vdev_geom_attach:157[1]: Created consumer for mirror/gm7.
 GEOM_MIRROR[1]: Disk da0s1 (device gm0) marked as dirty.
 GEOM_MIRROR[2]: Metadata on da0s1 updated.
 GEOM_MIRROR[1]: Disk ggate0 (device gm0) marked as dirty.
 GEOM_MIRROR[2]: Metadata on ggate0 updated.
 GEOM_MIRROR[1]: Disk da1s1 (device gm1) marked as dirty.
 GEOM_MIRROR[2]: Metadata on da1s1 updated.
 GEOM_MIRROR[1]: Disk ggate1 (device gm1) marked as dirty.
 GEOM_MIRROR[2]: Metadata on ggate1 updated.
 GEOM_MIRROR[1]: Disk da2s1 (device gm2) marked as dirty.
 GEOM_MIRROR[2]: Metadata on da2s1 updated.
 GEOM_MIRROR[1]: Disk ggate2 (device gm2) marked as dirty.
 GEOM_MIRROR[2]: Metadata on ggate2 updated.
 GEOM_MIRROR[1]: Disk da3s1 (device gm3) marked as dirty.
 GEOM_MIRROR[2]: Metadata on da3s1 updated.
 GEOM_MIRROR[1]: Disk ggate3 (device gm3) marked as dirty.
 GEOM_MIRROR[2]: Metadata on ggate3 updated.
 GEOM_MIRROR[1]: Disk da4s1 (device gm4) marked as dirty.
 GEOM_MIRROR[2]: Metadata on da4s1 updated.
 GEOM_MIRROR[1]: Disk ggate4 (device gm4) marked as dirty.
 GEOM_MIRROR[2]: Metadata on ggate4 updated.
 GEOM_MIRROR[1]: Disk da5s1 (device gm5) marked as dirty.
 GEOM_MIRROR[2]: Metadata on da5s1 updated.
 GEOM_MIRROR[1]: Disk ggate5 (device gm5) marked as dirty.
 GEOM_MIRROR[2]: Metadata on ggate5 updated.
 GEOM_MIRROR[1]: Disk da6s1 (device gm6) marked as dirty.
 GEOM_MIRROR[2]: Metadata on da6s1 updated.
 GEOM_MIRROR[1]: Disk ggate6 (device gm6) marked as dirty.
 GEOM_MIRROR[2]: Metadata on ggate6 updated.
 GEOM_MIRROR[1]: Disk da7s1 (device gm7) marked as dirty.
 GEOM_MIRROR[2]: Metadata on da7s1 updated.
 GEOM_MIRROR[1]: Disk ggate7 (device gm7) marked as dirty.
 GEOM_MIRROR[2]: Metadata on ggate7 updated.
 g_disk_flushcache(da0)
 g_disk_flushcache(da1)
 g_disk_flushcache(da2)
 g_disk_flushcache(da3)
 g_disk_flushcache(da4)
 g_disk_flushcache(da5)
 g_disk_flushcache(da6)
 g_disk_flushcache(da7)
 
 NOTE: this is when vdev_reopen gets called.
 
 g_post_event_x(0xc0e185a0, 0xc41a4800, 2, 0)
 vdev_geom_detach:177[1]: Closing access to mirror/gm0.
 g_access(0xc41a4800(mirror/gm0), -1, 0, -1)
 open delta:[r-1w0e-1] old:[r1w1e1] provider:[r1w1e1] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Access request for mirror/gm0: r-1w0e-1.
 vdev_geom_detach:181[1]: Destroyed consumer to mirror/gm0.
 g_access(0xc41a4800(mirror/gm0), 0, -1, 0)
 open delta:[r0w-1e0] old:[r0w1e0] provider:[r0w1e0] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Access request for mirror/gm0: r0w-1e0.
 GEOM_MIRROR[1]: Disk da0s1 (device gm0) marked as clean.
 g_poGEOM_MIRROR[2]: Metadata on da0s1 updated.
 GEOM_MIRROR[1]: Disk ggate0 (device gm0) marked as clean.
 st_event_x(0xc0e185a0, 0xc41a49c0, 2, 0)
 g_post_event_x(0xc0e185a0, 0xc419da80, 2, 0)
 g_post_event_x(0xc0e185a0, 0xc4b60940, 2, 0)
 g_post_event_x(0xc0e185a0, 0xc4b61140, 2, 0)
 g_post_event_x(0xc0e185a0, 0xc4b60800, 2, 0)
 g_post_event_x(0xc0e185a0, 0xc4b60c80, 2, 0)
 g_post_event_x(0xc0e185a0, 0xc419d240, 2, 0)
 
 NOTE: the above geom events are the vdev_geom_detach of vdev_close for 8 devices. The first one is processed, but the others remain in the queue when vdev_close returns.
 
 At this moment, vdev_reopen resumes and invokes vdev_open. The consumer for gm0 has been destroyed already, but the zfs::vdev geom still exists, as 7 of the consumers still appear as active. vdev_open reuses the consumer for gm0, but now it has no provider (it is a null pointer), and as soon as the code attempts to access the provider, it faults. See below.
 
 vdev_geom_open:371[1]: Found provider by name /dev/mirror/gm0.
 vdev_geom_attach:116[1]: Attaching to mirror/gm0.
 vdev_geom_attach:142[1]: Found consumer for mirror/gm0.
 g_access(0xc41a4800(mirror/gm0), 1, 0, 1)
 open delta:[r1w0e1] old:[r0w1e0] provider:[r0w1e0] 0xc4184980(mirror/gm0)
 GEOM_MIRROR[2]: Access request for mirror/gm0: r1w0e1.
 GEOM_MIRROR[2]: Metadata on ggate0 updated.
 g_post_event_x(0xc0724790, 0xc4184980, 2, 0)
   ref 0xc4184980
 g_detach(0xc41a4800)
 g_destroy_consumer(0xc41a4800)
 vdev_geom_detach:177[1]: Closing access to mirror/gm1.
 g_access(0xc41a49c0(mirror/gm1), -1, 0, -1)
 open delta:[r-1w0e-1] old:[r1w1e1] provider:[r1w1e1] 0xc3b57700(mirror/gm1)
 GEOM_MIRROR[2]: Access request for mirror/gm1: r-1w0e-1.
 vdev_geom_detach:181[1]: Destroyed consumer to mirror/gm1.
 g_access(0xc41a49c0(mirror/gm1), 0, -1, 0)
 open delta:[r0w-1e0] old:[r0w1e0] provider:[r0w1e0] 0xc3b57700(mirror/gm1)
 GEOM_MIRROR[2]: Access request for mirror/gm1: r0w-1e0.
 GEOM_MIRROR[1]: Disk da1s1 (device gm1) marked as clean.
 vdev_geom_attach:161[1]: Used existing consumer for mirror/gm0.
 Fatal trap GEOM_MIRROR[2]: Metadata on da1s1 updated.
 GEOM_MIRROR[1]: Disk ggate1 (device gm1) marked as clean.
 
 
 12: page fault while in kernel mode
 cpuid = 0; apic id = 00
 fault virtual address	= 0x30
 fault code		= supervisor read, page not present
 instruction pointer	= 0x20:0xc0e193d6
 stack pointer	        = 0x28:0xe0489aa0
 frame pointer	        = 0x28:0xe0489ae4
 code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, def32 1, gran 1
 processor eflags	= interrupt enabled, resume, IOPL = 0
 current process		= 1053 (solthread 0xc0dcfa7)
 [thread pid 1053 tid 100113 ]
 Stopped at      vdev_geom_open+0x6d6:   movl    0x30(%eax),%edx
 
 As I said in the bug report, the above instruction corresponds to vdev_geom.c:420.
 
 
 
 --------------080503070700060808010208--

From: Kris Kennaway <kris@FreeBSD.org>
To: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= <jmrueda@diatel.upm.es>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/125413: Panic when doing zfs raidz with gmirror and ggate
Date: Wed, 09 Jul 2008 14:40:17 +0200

 Javier Martn Rueda wrote:
 > The following reply was made to PR kern/125413; it has been noted by GNATS.
 > 
 > From: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= <jmrueda@diatel.upm.es>
 > To: bug-followup@FreeBSD.org, jmrueda@diatel.upm.es
 > Cc:  
 > Subject: Re: kern/125413: Panic when doing zfs raidz with gmirror and ggate
 > Date: Wed, 09 Jul 2008 02:52:39 +0200
 > 
 >  This is a multi-part message in MIME format.
 >  --------------080503070700060808010208
 >  Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 >  Content-Transfer-Encoding: 7bit
 >  
 >  I attach a console dump of the panic.
 
 >  12: page fault while in kernel mode
 >  cpuid = 0; apic id = 00
 >  fault virtual address	= 0x30
 >  fault code		= supervisor read, page not present
 >  instruction pointer	= 0x20:0xc0e193d6
 >  stack pointer	        = 0x28:0xe0489aa0
 >  frame pointer	        = 0x28:0xe0489ae4
 >  code segment		= base 0x0, limit 0xfffff, type 0x1b
 >  			= DPL 0, pres 1, def32 1, gran 1
 >  processor eflags	= interrupt enabled, resume, IOPL = 0
 >  current process		= 1053 (solthread 0xc0dcfa7)
 >  [thread pid 1053 tid 100113 ]
 >  Stopped at      vdev_geom_open+0x6d6:   movl    0x30(%eax),%edx
 >  
 >  As I said in the bug report, the above instruction corresponds to vdev_geom.c:420.
 
 OK, we're getting there, but you forgot the stack trace still :)
 
 Kris
 

From: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= <jmrueda@diatel.upm.es>
To: Kris Kennaway <kris@FreeBSD.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/125413: Panic when doing zfs raidz with gmirror and ggate
Date: Wed, 09 Jul 2008 16:17:48 +0200

 Kris Kennaway wrote:
 > Javier Martn Rueda wrote:
 >> The following reply was made to PR kern/125413; it has been noted by 
 >> GNATS.
 >>
 >> From: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= <jmrueda@diatel.upm.es>
 >> To: bug-followup@FreeBSD.org, jmrueda@diatel.upm.es
 >> Cc:  Subject: Re: kern/125413: Panic when doing zfs raidz with 
 >> gmirror and ggate
 >> Date: Wed, 09 Jul 2008 02:52:39 +0200
 >>
 >>  This is a multi-part message in MIME format.
 >>  --------------080503070700060808010208
 >>  Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 >>  Content-Transfer-Encoding: 7bit
 >>  
 >>  I attach a console dump of the panic.
 >
 >>  12: page fault while in kernel mode
 >>  cpuid = 0; apic id = 00
 >>  fault virtual address    = 0x30
 >>  fault code        = supervisor read, page not present
 >>  instruction pointer    = 0x20:0xc0e193d6
 >>  stack pointer            = 0x28:0xe0489aa0
 >>  frame pointer            = 0x28:0xe0489ae4
 >>  code segment        = base 0x0, limit 0xfffff, type 0x1b
 >>              = DPL 0, pres 1, def32 1, gran 1
 >>  processor eflags    = interrupt enabled, resume, IOPL = 0
 >>  current process        = 1053 (solthread 0xc0dcfa7)
 >>  [thread pid 1053 tid 100113 ]
 >>  Stopped at      vdev_geom_open+0x6d6:   movl    0x30(%eax),%edx
 >>  
 >>  As I said in the bug report, the above instruction corresponds to 
 >> vdev_geom.c:420.
 >
 > OK, we're getting there, but you forgot the stack trace still :)
 >
 > Kris
 >
 Sorry, here it is:
 
 Program received signal SIGSEGV, Segmentation fault.
 0xc0e193d6 in vdev_geom_open (vd=0xc3c74400, psize=0xe047db58, 
 ashift=0xe047db50)
     at 
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c:420
 420             *psize = pp->mediasize;
 (kgdb) where
 #0  0xc0e193d6 in vdev_geom_open (vd=0xc3c74400, psize=0xe047db58, 
 ashift=0xe047db50)
     at 
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c:420
 #1  0xc0ddc05a in vdev_open (vd=0xc3c74400) at 
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:803
 #2  0xc0de5735 in vdev_raidz_open (vd=0xc3c74c00, asize=0xe047dc70, 
 ashift=0xe047dc68)
     at 
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:581
 #3  0xc0ddc05a in vdev_open (vd=0xc3c74c00) at 
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:803
 #4  0xc0ddc7e5 in vdev_reopen (vd=0xc3c74c00) at 
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:1026
 #5  0xc0dcfa1d in spa_async_reopen (spa=0xc4192000) at 
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2659
 #6  0xc0dcfb52 in spa_async_thread (arg=0xc4192000) at 
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2692
 #7  0xc0753609 in fork_exit (callout=0xc0dcfa70 <spa_async_thread>, 
 arg=0xc4192000, frame=0xe047dd38) at ../../../kern/kern_fork.c:783
 #8  0xc0a6f530 in fork_trampoline () at ../../../i386/i386/exception.s:205
 
 
Responsible-Changed-From-To: freebsd-bugs->pjd 
Responsible-Changed-By: kris 
Responsible-Changed-When: Wed Jul 9 14:32:42 UTC 2008 
Responsible-Changed-Why:  
Seems to be pjd's area 

http://www.freebsd.org/cgi/query-pr.cgi?pr=125413 
State-Changed-From-To: open->feedback 
State-Changed-By: kmacy 
State-Changed-When: Sun May 17 06:06:00 UTC 2009 
State-Changed-Why:  


does this happen post MFC? 
http://svn.freebsd.org/base/user/kmacy/ZFS_MFC/  


Responsible-Changed-From-To: pjd->kmacy  
Responsible-Changed-By: kmacy 
Responsible-Changed-When: Sun May 17 06:06:00 UTC 2009 
Responsible-Changed-Why:  


Does this happen post-MFC? 
try: 
http://svn.freebsd.org/base/user/kmacy/ZFS_MFC/  

http://www.freebsd.org/cgi/query-pr.cgi?pr=125413 
State-Changed-From-To: feedback->closed 
State-Changed-By: pjd 
State-Changed-When: ro 9 wrz 2009 20:29:32 UTC 
State-Changed-Why:  
This can no longer happen. Both vdev_geom_close() and vdev_geom_open() use 
the GEOM event thread to post events. Even if vdev_geom_close() returns 
before consumers are actually closed, that's fine, because vdev_geom_open() 
also opens providers using events and open's events will be placed after 
close's events. Also vdev_geom_open() waits for the events to complete, so 
when vdev_geom_open() returns we can be sure that all earlier closes are 
finished now and opens are done too. 

Very good problem analysis. Thank you for the report! 


Responsible-Changed-From-To: kmacy->pjd 
Responsible-Changed-By: pjd 
Responsible-Changed-When: ro 9 wrz 2009 20:29:32 UTC 
Responsible-Changed-Why:  
I'll take this one. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=125413 
>Unformatted:
