From nobody@FreeBSD.org  Tue Jan 15 11:56:50 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id EFCDC9F2
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 15 Jan 2013 11:56:50 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id C9309E1
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 15 Jan 2013 11:56:50 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r0FBuoBg000268
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 15 Jan 2013 11:56:50 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r0FBuouL000267;
	Tue, 15 Jan 2013 11:56:50 GMT
	(envelope-from nobody)
Message-Id: <201301151156.r0FBuouL000267@red.freebsd.org>
Date: Tue, 15 Jan 2013 11:56:50 GMT
From: Alexei Volkov <softkot@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Fail to use ZVOL as a gmirror component
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         175323
>Category:       kern
>Synopsis:       Fail to use ZVOL as a gmirror component
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    avg
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jan 15 12:00:00 UTC 2013
>Closed-Date:    Sat Feb 02 12:46:13 UTC 2013
>Last-Modified:  Sat Feb 02 12:46:13 UTC 2013
>Originator:     Alexei Volkov
>Release:        9.1-RELEASE
>Organization:
SoftLynx
>Environment:
FreeBSD fresh-inst 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4 09:23:
10 UTC 2012     root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64

>Description:
kernel: GEOM_MIRROR (error=45) while trying to use ZFS volumes (ZVOL) geoms as a geom_mirror component.
>How-To-Repeat:
# zfs create -V10G tank0/vol0
# diskinfo -v /dev/zvol/tank0/vol0
/dev/zvol/tank0/vol0
        512             # sectorsize
        10737418240     # mediasize in bytes (10G)
        20971520        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset

# gmirror label mirror0 /dev/zvol/tank0/vol0
GEOM_MIRROR: Cannot add disk zvol/tank0/vol0 to mirror0 (error=45).
GEOM_MIRROR: Device mirror0 destroyed.

gmirror list is empty and gmirror list -a shows:

Geom name: mirror0
Consumers:
1. Name: zvol/tank0/vol0
   Mediasize: 10737418240 (10G)
   Sectorsize: 512
   Mode: r1w1e1

Right after the sequence above, the block device zvol/tank0/vol0 is not any more available for writing it (seems to be locked by geom_mirror)

# dd if=/dev/zero of=/dev/zvol/tank0/vol0
dd: /dev/zvol/tank0/vol0: Operation not permitted


And i was not able to release it.

# gmirror forget mirror0
gmirror: No such device: mirror0.

# gmirror clear /dev/zvol/tank0/vol0
Can't clear metadata on /dev/zvol/tank0/vol0: Operation not permitted.
gmirror: Not fully done.

# zfs destroy -f tank0/vol0
cannot destroy 'tank0/vol0': dataset is busy

The only way to destroy ZVOL is temporarily block geaom_morror loading during the boot process, reboot and destroy zvols.
>Fix:


>Release-Note:
>Audit-Trail:

From: =?UTF-8?B?0JDQu9C10LrRgdC10Lkg0JLQvtC70LrQvtCy?= <softkot@gmail.com>
To: bug-followup@freebsd.org, softkot@gmail.com
Cc:  
Subject: Re: kern/175323: Fail to use ZVOL as a gmirror component
Date: Tue, 15 Jan 2013 16:05:21 +0400

 --f46d040121939effa304d35294f6
 Content-Type: text/plain; charset=ISO-8859-1
 
 In advance, i was able to do that on 9.0-RELEASE and earlier ZFS aware
 kernels.
 
 --f46d040121939effa304d35294f6--

From: =?UTF-8?B?0JDQu9C10LrRgdC10Lkg0JLQvtC70LrQvtCy?= <softkot@gmail.com>
To: bug-followup@freebsd.org, 
	=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0JLQvtC70LrQvtCy?= <softkot@gmail.com>
Cc:  
Subject: Re: kern/175323: Fail to use ZVOL as a gmirror component
Date: Tue, 15 Jan 2013 17:22:22 +0400

 --bcaec54ee2900eeb4504d353a82e
 Content-Type: text/plain; charset=UTF-8
 Content-Transfer-Encoding: quoted-printable
 
 Here is the shell commands sequence and kernel log
 with kern.geom.mirror.debug=3D65535
 
 # zfs create -V1G tank0/vol0
 Jan 15 17:17:11 fresh-inst kernel: GEOM_MIRROR[2]: Tasting zvol/tank0/vol0.
 
 # gmirror label mirror0 /dev/zvol/tank0/vol0
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[2]: Tasting zvol/tank0/vol0.
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[2]: Tasting zvol/tank0/vol0.
 Jan 15 17:19:54 fresh-inst kernel: magic: GEOM::MIRROR
 Jan 15 17:19:54 fresh-inst kernel: version: 4
 Jan 15 17:19:54 fresh-inst kernel: name: mirror0
 Jan 15 17:19:54 fresh-inst kernel: mid: 2502644825
 Jan 15 17:19:54 fresh-inst kernel: did: 1157435422
 Jan 15 17:19:54 fresh-inst kernel: all: 1
 Jan 15 17:19:54 fresh-inst kernel: genid: 0
 Jan 15 17:19:54 fresh-inst kernel: syncid: 1
 Jan 15 17:19:54 fresh-inst kernel: priority: 0
 Jan 15 17:19:54 fresh-inst kernel: slice: 4096
 Jan 15 17:19:54 fresh-inst kernel: balance: load
 Jan 15 17:19:54 fresh-inst kernel: mediasize: 1073741312
 Jan 15 17:19:54 fresh-inst kernel: sectorsize: 512
 Jan 15 17:19:54 fresh-inst kernel: syncoffset: 0
 Jan 15 17:19:54 fresh-inst kernel: mflags: NONE
 Jan 15 17:19:54 fresh-inst kernel: dflags: NONE
 Jan 15 17:19:54 fresh-inst kernel: hcprovider:
 Jan 15 17:19:54 fresh-inst kernel: provsize: 1073741824
 Jan 15 17:19:54 fresh-inst kernel: MD5 hash:
 8af33e03530d3297e18ee99e9d402feb
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[1]: Creating device mirror0
 (id=3D2502644825).
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRRORGEOM_MIRROR[1][5]
 Jan 15 17:19:54 fresh-inst kernel: : :
 Jan 15 17:19:54 fresh-inst kernel: Device mirror0 created (1 components,
 id=3D2502644825).
 Jan 15 17:19:54 fresh-inst kernel: g_mirror_worker: Let's see...
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRRORGEOM_MIRROR[1][4]: :
 root_mount_hold 0
 Jan 15 17:19:54 fresh-inst kernel: g_mirror_worker: Sleeping
 0xfffffe000e918600.
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[1]: Adding disk
 zvol/tank0/vol0 to mirror0.
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[2]: Adding disk
 zvol/tank0/vol0.
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[2]: Disk zvol/tank0/vol0
 connected.
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[0]: Cannot add disk
 zvol/tank0/vol0 to mirror0 (error=3D45).
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRROR[4]: g_mirror_destroy: Waking
 up 0xfffffe000e918600.
 Jan 15 17:19:54 fresh-inst kernel: GEOM_MIRRORGEOM_MIRROR[4][4]: :
 g_mirror_destroy: Sleeping 0xfffffe000e9186a0.
 Jan 15 17:19:54 fresh-inst kernel: g_mirror_worker: Woken up
 0xfffffe000e918600.
 Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRROR[5]:
 Jan 15 17:19:55 fresh-inst kernel: g_mirror_worker: I'm here 4.
 Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRROR[5]: g_mirror_worker: Let's
 see...
 Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRROR[4]: g_mirror_try_destroy:
 Waking up 0xfffffe000e9186a0.
 Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRRORGEOM_MIRROR[1][4]: : Thread
 exiting.
 Jan 15 17:19:55 fresh-inst kernel: g_mirror_destroy: Woken up
 0xfffffe000e9186a0.
 Jan 15 17:19:55 fresh-inst kernel: GEOM_MIRROR[0]: Device mirror0 destroyed=
 .
 
 --bcaec54ee2900eeb4504d353a82e--

From: Andriy Gapon <avg@FreeBSD.org>
To: bug-followup@FreeBSD.org, softkot@gmail.com
Cc: freebsd-geom@FreeBSD.org
Subject: Re: kern/175323: Fail to use ZVOL as a gmirror component
Date: Mon, 21 Jan 2013 11:47:05 +0200

 This
 > GEOM_MIRROR: Cannot add disk zvol/tank0/vol0 to mirror0 (error=45).
 
 seems to be triggered by the following code in sys/geom/mirror/g_mirror.c:
 
 error = g_getattr("GEOM::candelete", disk->d_consumer, &i);
 if (error != 0)
         goto fail;
 
 plus the fact that ZFS zvol does the following:
 case BIO_GETATTR:
 case BIO_DELETE:
 default:
         g_io_deliver(bp, EOPNOTSUPP);
         break;
 
 Perhaps, the gmirror code should be more flexible with respect to EOPNOTSUPP
 from g_getattr?
 
 -- 
 Andriy Gapon

From: =?UTF-8?B?0JDQu9C10LrRgdC10Lkg0JLQvtC70LrQvtCy?=
 <Alexei.Volkov@softlynx.ru>
To: bug-followup@FreeBSD.org, Andriy Gapon <avg@FreeBSD.org>
Cc:  
Subject: Re: kern/175323: Fail to use ZVOL as a gmirror component
Date: Mon, 21 Jan 2013 15:47:32 +0400

 Andriy, many thanks for the suggestion. I am trying to rebuild and test
 the kernel with the following patch:
 
 --- sys/geom/mirror/g_mirror.c    (revision 245741)
 +++ sys/geom/mirror/g_mirror.c    (working copy)
 @@ -456,7 +456,7 @@
      disk->d_flags = md->md_dflags;
      error = g_getattr("GEOM::candelete", disk->d_consumer, &i);
      if (error != 0)
 -        goto fail;
 +        i=0;
      if (i)
          disk->d_flags |= G_MIRROR_DISK_FLAG_CANDELETE;
      if (md->md_provider[0] != '\0')
 
 
 It seems to correct if assume G_MIRROR_DISK_FLAG_CANDELETE is always off
 unless successfully reported by GEOM::candelete attribute.
 
 -- 
 Best regards
 
 
 21.01.2013 13:47, Andriy Gapon пишет:
 > This
 >> GEOM_MIRROR: Cannot add disk zvol/tank0/vol0 to mirror0 (error=45).
 > seems to be triggered by the following code in sys/geom/mirror/g_mirror.c:
 >
 > error = g_getattr("GEOM::candelete", disk->d_consumer, &i);
 > if (error != 0)
 >         goto fail;
 >
 > plus the fact that ZFS zvol does the following:
 > case BIO_GETATTR:
 > case BIO_DELETE:
 > default:
 >         g_io_deliver(bp, EOPNOTSUPP);
 >         break;
 >
 > Perhaps, the gmirror code should be more flexible with respect to EOPNOTSUPP
 > from g_getattr?
 >

From: =?UTF-8?B?0JDQu9C10LrRgdC10Lkg0JLQvtC70LrQvtCy?= <softkot@gmail.com>
To: Andriy Gapon <avg@freebsd.org>
Cc: bug-followup@freebsd.org, freebsd-geom@freebsd.org
Subject: Re: kern/175323: Fail to use ZVOL as a gmirror component
Date: Mon, 21 Jan 2013 16:39:57 +0400

 Eventually this patch works for me just fine.
 
 --- sys/geom/mirror/g_mirror.c    (revision 245741)
 +++ sys/geom/mirror/g_mirror.c    (working copy)
 @@ -456,7 +456,7 @@
      disk->d_flags = md->md_dflags;
      error = g_getattr("GEOM::candelete", disk->d_consumer, &i);
      if (error != 0)
 -        goto fail;
 +        i=0;
      if (i)
          disk->d_flags |= G_MIRROR_DISK_FLAG_CANDELETE;
      if (md->md_provider[0] != '\0')

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/175323: commit references a PR
Date: Sat, 26 Jan 2013 10:50:11 +0000 (UTC)

 Author: avg
 Date: Sat Jan 26 10:50:04 2013
 New Revision: 245946
 URL: http://svnweb.freebsd.org/changeset/base/245946
 
 Log:
   g_mirror: g_getattr() failure should not be fatal
   
   This allows to use gmirror e.g. on top of ZVOLs.
   
   PR:		kern/175323
   Submitted by:	Alexei.Volkov@softlynx.ru, mav
   Reported by:	Alexei.Volkov@softlynx.ru
   Tested by:	Alexei.Volkov@softlynx.ru
   Reviewed by:	ae, mav, pjd
   MFC after:	1 week
 
 Modified:
   head/sys/geom/mirror/g_mirror.c
 
 Modified: head/sys/geom/mirror/g_mirror.c
 ==============================================================================
 --- head/sys/geom/mirror/g_mirror.c	Sat Jan 26 10:34:17 2013	(r245945)
 +++ head/sys/geom/mirror/g_mirror.c	Sat Jan 26 10:50:04 2013	(r245946)
 @@ -457,9 +457,7 @@ g_mirror_init_disk(struct g_mirror_softc
  	disk->d_priority = md->md_priority;
  	disk->d_flags = md->md_dflags;
  	error = g_getattr("GEOM::candelete", disk->d_consumer, &i);
 -	if (error != 0)
 -		goto fail;
 -	if (i)
 +	if (error == 0 && i != 0)
  		disk->d_flags |= G_MIRROR_DISK_FLAG_CANDELETE;
  	if (md->md_provider[0] != '\0')
  		disk->d_flags |= G_MIRROR_DISK_FLAG_HARDCODED;
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
Responsible-Changed-From-To: freebsd-bugs->avg 
Responsible-Changed-By: avg 
Responsible-Changed-When: Sat Jan 26 12:18:50 UTC 2013 
Responsible-Changed-Why:  
I am handling this one. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175323 
State-Changed-From-To: open->patched 
State-Changed-By: avg 
State-Changed-When: Sat Jan 26 12:19:14 UTC 2013 
State-Changed-Why:  
Fixed in head.  This fix makes g_getattr("GEOM::candelete") failure 
non-fatal and that also fixes a GEOM access bits leak. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175323 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/175323: commit references a PR
Date: Sat,  2 Feb 2013 11:31:11 +0000 (UTC)

 Author: avg
 Date: Sat Feb  2 11:30:53 2013
 New Revision: 246241
 URL: http://svnweb.freebsd.org/changeset/base/246241
 
 Log:
   MFC r245946: g_mirror: g_getattr() failure should not be fatal
   
   PR:		kern/175323
 
 Modified:
   stable/9/sys/geom/mirror/g_mirror.c
 Directory Properties:
   stable/9/sys/   (props changed)
 
 Modified: stable/9/sys/geom/mirror/g_mirror.c
 ==============================================================================
 --- stable/9/sys/geom/mirror/g_mirror.c	Sat Feb  2 11:25:10 2013	(r246240)
 +++ stable/9/sys/geom/mirror/g_mirror.c	Sat Feb  2 11:30:53 2013	(r246241)
 @@ -456,9 +456,7 @@ g_mirror_init_disk(struct g_mirror_softc
  	disk->d_priority = md->md_priority;
  	disk->d_flags = md->md_dflags;
  	error = g_getattr("GEOM::candelete", disk->d_consumer, &i);
 -	if (error != 0)
 -		goto fail;
 -	if (i)
 +	if (error == 0 && i != 0)
  		disk->d_flags |= G_MIRROR_DISK_FLAG_CANDELETE;
  	if (md->md_provider[0] != '\0')
  		disk->d_flags |= G_MIRROR_DISK_FLAG_HARDCODED;
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: avg 
State-Changed-When: Sat Feb 2 12:46:12 UTC 2013 
State-Changed-Why:  
Fixed in stable/9. 8 is not affected. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175323 
>Unformatted:
