From nobody@FreeBSD.org  Mon Oct  1 23:35:50 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 93BCA106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  1 Oct 2012 23:35:50 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 658448FC15
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  1 Oct 2012 23:35:50 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id q91NZnux019643
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 1 Oct 2012 23:35:49 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id q91NZn8P019620;
	Mon, 1 Oct 2012 23:35:49 GMT
	(envelope-from nobody)
Message-Id: <201210012335.q91NZn8P019620@red.freebsd.org>
Date: Mon, 1 Oct 2012 23:35:49 GMT
From: Steven Hartland <steven.hartland@multiplay.co.uk>
To: freebsd-gnats-submit@FreeBSD.org
Subject: ZFS fails to receive valid snapshots (patch included)
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         172259
>Category:       kern
>Synopsis:       [zfs] [patch] ZFS fails to receive valid snapshots (patch included)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    smh
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Oct 01 23:40:02 UTC 2012
>Closed-Date:    Fri Jun 07 15:01:28 UTC 2013
>Last-Modified:  Fri Jun 07 15:01:28 UTC 2013
>Originator:     Steven Hartland
>Release:        8.3-RELEASE
>Organization:
Multiplay
>Environment:
FreeBSD dev 8.3-RELEASE-p4 FreeBSD 8.3-RELEASE-p4 #22: Mon Sep 17 17:18:32 UTC 2012     root@dev:/usr/obj/usr/src/sys/MULTIPLAY  amd64
>Description:
When using zfs receive to restore data saved with zfs send there is the possibility that the receive will fail due to replication of snapshots information being processed in a random order.

In addition to this when receiving snapshots where the parent from snapshot is replaced by another needless failures occur which can be seen clearly in verbose mode.
>How-To-Repeat:
1. First create some volumes & backups on a test pool (testtank)
 
zfs create testtank/test
zfs create testtank/test/1
zfs create testtank/test/2
zfs snapshot -r testtank@backup
zfs send -R testtank@backup > recvtest-1
zfs rename -r testtank@backup testtank@backup-last
zfs snapshot -r testtank@backup
zfs send -R -i testtank@backup-last testtank@backup > recvtest-2
zfs destroy -r testtank@backup-last
zfs rename -r testtank@backup testtank@backup-last
zfs snapshot -r testtank@backup
zfs send -R -i testtank@backup-last testtank@backup > recvtest-3

2. Clean out the pool
zfs destroy -r testtank

3. Restore the backups in order
zfs recv -duFv testtank < recvtest-1
zfs recv -duFv testtank < recvtest-2
zfs recv -duFv testtank < recvtest-3

At some point usually the third receive it will error and leave the filesystems in a random state. You'll see temporary named filesystems e.g.
zfs list -t all
testtank/recv-16556-1
testtank/recv-16556-1/1
testtank/recv-16556-1/2
>Fix:
The first problem is down to the fact that the replication of snapshots are processed in a random order resulting in snapshot deletes and renames happening in the wrong order.

The second problem is caused by deletes / renames effecting the parent snapshot of existing filesystems. The current process needlessly attempts to rename child filesystems of the deleted parents.

The attached patch fixes both issues.

The first by ensuring the list of snapshots used is sorted

The second by maintaining a list of deleted guid's which are used to check against before performing a parent guid based rename.

>Release-Note:
>Audit-Trail:

From: "Steven Hartland" <killing@multiplay.co.uk>
To: <bug-followup@freebsd.org>
Cc:  
Subject: Re: bin/172259: ZFS fails to receive valid snapshots (patch included)
Date: Tue, 2 Oct 2012 01:12:23 +0100

 This is a multi-part message in MIME format.
 
 ------=_NextPart_000_093F_01CDA03A.FA949040
 Content-Type: text/plain;
 	format=flowed;
 	charset="Windows-1252";
 	reply-type=original
 Content-Transfer-Encoding: 7bit
 
 Patch for this issue
 
 ================================================
 This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 
 
 In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
 or return the E.mail to postmaster@multiplay.co.uk.
 ------=_NextPart_000_093F_01CDA03A.FA949040
 Content-Type: text/plain;
 	format=flowed;
 	name="zfs-recv-sortfix.txt";
 	reply-type=original
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment;
 	filename="zfs-recv-sortfix.txt"
 
 Fixes zfs receive errors caused by snapshot replication being processed =
 in a=0A=
 random order instead of creation order.=0A=
 =0A=
 Eliminates needless filesystem renames caused by removed parent snapshots=0A=
 which subsequently causes many more errors.=0A=
 --- cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c.orig2	=
 2012-10-01 22:27:40.119762812 +0000=0A=
 +++ cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c	=
 2012-10-01 23:20:02.440540612 +0000=0A=
 @@ -718,7 +718,7 @@=0A=
  	sd->parent_fromsnap_guid =3D 0;=0A=
  	VERIFY(0 =3D=3D nvlist_alloc(&sd->parent_snaps, NV_UNIQUE_NAME, 0));=0A=
  	VERIFY(0 =3D=3D nvlist_alloc(&sd->snapprops, NV_UNIQUE_NAME, 0));=0A=
 -	(void) zfs_iter_snapshots(zhp, send_iterate_snap, sd);=0A=
 +	(void) zfs_iter_snapshots_sorted(zhp, send_iterate_snap, sd);=0A=
  	VERIFY(0 =3D=3D nvlist_add_nvlist(nvfs, "snaps", sd->parent_snaps));=0A=
  	VERIFY(0 =3D=3D nvlist_add_nvlist(nvfs, "snapprops", sd->snapprops));=0A=
  	nvlist_free(sd->parent_snaps);=0A=
 @@ -1859,11 +1859,12 @@=0A=
      recvflags_t *flags, nvlist_t *stream_nv, avl_tree_t *stream_avl,=0A=
      nvlist_t *renamed)=0A=
  {=0A=
 -	nvlist_t *local_nv;=0A=
 +	nvlist_t *local_nv, *deleted =3D NULL;=0A=
  	avl_tree_t *local_avl;=0A=
  	nvpair_t *fselem, *nextfselem;=0A=
  	char *fromsnap;=0A=
  	char newname[ZFS_MAXNAMELEN];=0A=
 +	char guidname[32];=0A=
  	int error;=0A=
  	boolean_t needagain, progress, recursive;=0A=
  	char *s1, *s2;=0A=
 @@ -1879,6 +1880,8 @@=0A=
  again:=0A=
  	needagain =3D progress =3D B_FALSE;=0A=
  =0A=
 +	VERIFY(0 =3D=3D nvlist_alloc(&deleted, NV_UNIQUE_NAME, 0));=0A=
 +=0A=
  	if ((error =3D gather_nvlist(hdl, tofs, fromsnap, NULL,=0A=
  	    recursive, &local_nv, &local_avl)) !=3D 0)=0A=
  		return (error);=0A=
 @@ -1999,6 +2002,8 @@=0A=
  					needagain =3D B_TRUE;=0A=
  				else=0A=
  					progress =3D B_TRUE;=0A=
 +				sprintf(guidname, "%lu", thisguid);=0A=
 +				nvlist_add_boolean(deleted, guidname);=0A=
  				continue;=0A=
  			}=0A=
  =0A=
 @@ -2054,6 +2059,8 @@=0A=
  				needagain =3D B_TRUE;=0A=
  			else=0A=
  				progress =3D B_TRUE;=0A=
 +			sprintf(guidname, "%lu", parent_fromsnap_guid);=0A=
 +			nvlist_add_boolean(deleted, guidname);=0A=
  			continue;=0A=
  		}=0A=
  =0A=
 @@ -2076,6 +2083,23 @@=0A=
  		s2 =3D strrchr(stream_fsname, '/');=0A=
  =0A=
  		/*=0A=
 +		 * Check if we're going to rename based on parent guid change=0A=
 +		 * and the current parent guid was also deleted. If it was then=0A=
 +		 * rename will fail and is likely unneeded, so avoid this and=0A=
 +		 * force an early retry to determine the new parent_fromsnap_guid=0A=
 +		 */=0A=
 +		if (stream_parent_fromsnap_guid !=3D 0 &&=0A=
 +                    parent_fromsnap_guid !=3D 0 &&=0A=
 +                    stream_parent_fromsnap_guid !=3D =
 parent_fromsnap_guid) {=0A=
 +			sprintf(guidname, "%lu", parent_fromsnap_guid);=0A=
 +			if (nvlist_exists(deleted, guidname)) {=0A=
 +				progress =3D B_TRUE;=0A=
 +				needagain =3D B_TRUE;=0A=
 +				goto doagain;=0A=
 +			}=0A=
 +		}=0A=
 +=0A=
 +		/*=0A=
  		 * Check for rename. If the exact receive path is specified, it=0A=
  		 * does not count as a rename, but we still need to check the=0A=
  		 * datasets beneath it.=0A=
 @@ -2129,8 +2153,10 @@=0A=
  		}=0A=
  	}=0A=
  =0A=
 +doagain:=0A=
  	fsavl_destroy(local_avl);=0A=
  	nvlist_free(local_nv);=0A=
 +	nvlist_free(deleted);=0A=
  =0A=
  	if (needagain && progress) {=0A=
  		/* do another pass to fix up temporary names */=0A=
 
 ------=_NextPart_000_093F_01CDA03A.FA949040--
 
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Oct 5 01:14:41 UTC 2012 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=172259 
Responsible-Changed-From-To: freebsd-fs->smh 
Responsible-Changed-By: smh 
Responsible-Changed-When: Tue Dec 11 14:09:31 UTC 2012 
Responsible-Changed-Why:  
I'll take it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=172259 
State-Changed-From-To: open->patched 
State-Changed-By: smh 
State-Changed-When: Thu Dec 13 22:05:35 UTC 2012 
State-Changed-Why:  
patched in head/ with r244194 - awaiting MFC 

http://www.freebsd.org/cgi/query-pr.cgi?pr=172259 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/172259: commit references a PR
Date: Wed,  5 Jun 2013 11:23:19 +0000 (UTC)

 Author: smh
 Date: Wed Jun  5 11:23:10 2013
 New Revision: 251411
 URL: http://svnweb.freebsd.org/changeset/base/251411
 
 Log:
   MFC r244194:
   Fixe zfs receive errors caused by snapshot replication being processed in
   random order.
   
   PR:		kern/172259
 
 Modified:
   stable/8/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c
 Directory Properties:
   stable/8/cddl/contrib/opensolaris/   (props changed)
   stable/8/cddl/contrib/opensolaris/lib/libzfs/   (props changed)
 
 Modified: stable/8/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c
 ==============================================================================
 --- stable/8/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c	Wed Jun  5 11:16:17 2013	(r251410)
 +++ stable/8/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c	Wed Jun  5 11:23:10 2013	(r251411)
 @@ -727,7 +727,7 @@ send_iterate_fs(zfs_handle_t *zhp, void 
  	sd->parent_fromsnap_guid = 0;
  	VERIFY(0 == nvlist_alloc(&sd->parent_snaps, NV_UNIQUE_NAME, 0));
  	VERIFY(0 == nvlist_alloc(&sd->snapprops, NV_UNIQUE_NAME, 0));
 -	(void) zfs_iter_snapshots(zhp, B_FALSE, send_iterate_snap, sd);
 +	(void) zfs_iter_snapshots_sorted(zhp, send_iterate_snap, sd);
  	VERIFY(0 == nvlist_add_nvlist(nvfs, "snaps", sd->parent_snaps));
  	VERIFY(0 == nvlist_add_nvlist(nvfs, "snapprops", sd->snapprops));
  	nvlist_free(sd->parent_snaps);
 @@ -1945,11 +1945,12 @@ recv_incremental_replication(libzfs_hand
      recvflags_t *flags, nvlist_t *stream_nv, avl_tree_t *stream_avl,
      nvlist_t *renamed)
  {
 -	nvlist_t *local_nv;
 +	nvlist_t *local_nv, *deleted = NULL;
  	avl_tree_t *local_avl;
  	nvpair_t *fselem, *nextfselem;
  	char *fromsnap;
  	char newname[ZFS_MAXNAMELEN];
 +	char guidname[32];
  	int error;
  	boolean_t needagain, progress, recursive;
  	char *s1, *s2;
 @@ -1965,6 +1966,8 @@ recv_incremental_replication(libzfs_hand
  again:
  	needagain = progress = B_FALSE;
  
 +	VERIFY(0 == nvlist_alloc(&deleted, NV_UNIQUE_NAME, 0));
 +
  	if ((error = gather_nvlist(hdl, tofs, fromsnap, NULL,
  	    recursive, &local_nv, &local_avl)) != 0)
  		return (error);
 @@ -2079,6 +2082,8 @@ again:
  					needagain = B_TRUE;
  				else
  					progress = B_TRUE;
 +				sprintf(guidname, "%lu", thisguid);
 +				nvlist_add_boolean(deleted, guidname);
  				continue;
  			}
  
 @@ -2134,6 +2139,8 @@ again:
  				needagain = B_TRUE;
  			else
  				progress = B_TRUE;
 +			sprintf(guidname, "%lu", parent_fromsnap_guid);
 +			nvlist_add_boolean(deleted, guidname);
  			continue;
  		}
  
 @@ -2156,6 +2163,24 @@ again:
  		s2 = strrchr(stream_fsname, '/');
  
  		/*
 +		 * Check if we're going to rename based on parent guid change
 +		 * and the current parent guid was also deleted. If it was then
 +		 * rename will fail and is likely unneeded, so avoid this and
 +		 * force an early retry to determine the new
 +		 * parent_fromsnap_guid.
 +		 */
 +		if (stream_parent_fromsnap_guid != 0 &&
 +                    parent_fromsnap_guid != 0 &&
 +                    stream_parent_fromsnap_guid != parent_fromsnap_guid) {
 +			sprintf(guidname, "%lu", parent_fromsnap_guid);
 +			if (nvlist_exists(deleted, guidname)) {
 +				progress = B_TRUE;
 +				needagain = B_TRUE;
 +				goto doagain;
 +			}
 +		}
 +
 +		/*
  		 * Check for rename. If the exact receive path is specified, it
  		 * does not count as a rename, but we still need to check the
  		 * datasets beneath it.
 @@ -2209,8 +2234,10 @@ again:
  		}
  	}
  
 +doagain:
  	fsavl_destroy(local_avl);
  	nvlist_free(local_nv);
 +	nvlist_free(deleted);
  
  	if (needagain && progress) {
  		/* do another pass to fix up temporary names */
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/172259: commit references a PR
Date: Wed,  5 Jun 2013 11:55:47 +0000 (UTC)

 Author: smh
 Date: Wed Jun  5 11:55:35 2013
 New Revision: 251417
 URL: http://svnweb.freebsd.org/changeset/base/251417
 
 Log:
   MFC r244194:
   Fix zfs receive errors caused by snapshot replication being processed in
   random order.
   
   PR:		kern/172259
 
 Modified:
   stable/9/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c
 Directory Properties:
   stable/9/cddl/contrib/opensolaris/   (props changed)
   stable/9/cddl/contrib/opensolaris/lib/libzfs/   (props changed)
 
 Modified: stable/9/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c
 ==============================================================================
 --- stable/9/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c	Wed Jun  5 11:46:43 2013	(r251416)
 +++ stable/9/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c	Wed Jun  5 11:55:35 2013	(r251417)
 @@ -731,7 +731,7 @@ send_iterate_fs(zfs_handle_t *zhp, void 
  	sd->parent_fromsnap_guid = 0;
  	VERIFY(0 == nvlist_alloc(&sd->parent_snaps, NV_UNIQUE_NAME, 0));
  	VERIFY(0 == nvlist_alloc(&sd->snapprops, NV_UNIQUE_NAME, 0));
 -	(void) zfs_iter_snapshots(zhp, B_FALSE, send_iterate_snap, sd);
 +	(void) zfs_iter_snapshots_sorted(zhp, send_iterate_snap, sd);
  	VERIFY(0 == nvlist_add_nvlist(nvfs, "snaps", sd->parent_snaps));
  	VERIFY(0 == nvlist_add_nvlist(nvfs, "snapprops", sd->snapprops));
  	nvlist_free(sd->parent_snaps);
 @@ -1946,11 +1946,12 @@ recv_incremental_replication(libzfs_hand
      recvflags_t *flags, nvlist_t *stream_nv, avl_tree_t *stream_avl,
      nvlist_t *renamed)
  {
 -	nvlist_t *local_nv;
 +	nvlist_t *local_nv, *deleted = NULL;
  	avl_tree_t *local_avl;
  	nvpair_t *fselem, *nextfselem;
  	char *fromsnap;
  	char newname[ZFS_MAXNAMELEN];
 +	char guidname[32];
  	int error;
  	boolean_t needagain, progress, recursive;
  	char *s1, *s2;
 @@ -1966,6 +1967,8 @@ recv_incremental_replication(libzfs_hand
  again:
  	needagain = progress = B_FALSE;
  
 +	VERIFY(0 == nvlist_alloc(&deleted, NV_UNIQUE_NAME, 0));
 +
  	if ((error = gather_nvlist(hdl, tofs, fromsnap, NULL,
  	    recursive, &local_nv, &local_avl)) != 0)
  		return (error);
 @@ -2080,6 +2083,8 @@ again:
  					needagain = B_TRUE;
  				else
  					progress = B_TRUE;
 +				sprintf(guidname, "%lu", thisguid);
 +				nvlist_add_boolean(deleted, guidname);
  				continue;
  			}
  
 @@ -2135,6 +2140,8 @@ again:
  				needagain = B_TRUE;
  			else
  				progress = B_TRUE;
 +			sprintf(guidname, "%lu", parent_fromsnap_guid);
 +			nvlist_add_boolean(deleted, guidname);
  			continue;
  		}
  
 @@ -2157,6 +2164,24 @@ again:
  		s2 = strrchr(stream_fsname, '/');
  
  		/*
 +		 * Check if we're going to rename based on parent guid change
 +		 * and the current parent guid was also deleted. If it was then
 +		 * rename will fail and is likely unneeded, so avoid this and
 +		 * force an early retry to determine the new
 +		 * parent_fromsnap_guid.
 +		 */
 +		if (stream_parent_fromsnap_guid != 0 &&
 +                    parent_fromsnap_guid != 0 &&
 +                    stream_parent_fromsnap_guid != parent_fromsnap_guid) {
 +			sprintf(guidname, "%lu", parent_fromsnap_guid);
 +			if (nvlist_exists(deleted, guidname)) {
 +				progress = B_TRUE;
 +				needagain = B_TRUE;
 +				goto doagain;
 +			}
 +		}
 +
 +		/*
  		 * Check for rename. If the exact receive path is specified, it
  		 * does not count as a rename, but we still need to check the
  		 * datasets beneath it.
 @@ -2210,8 +2235,10 @@ again:
  		}
  	}
  
 +doagain:
  	fsavl_destroy(local_avl);
  	nvlist_free(local_nv);
 +	nvlist_free(deleted);
  
  	if (needagain && progress) {
  		/* do another pass to fix up temporary names */
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: smh 
State-Changed-When: Fri Jun 7 15:01:28 UTC 2013 
State-Changed-Why:  
Committed. Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=172259 
>Unformatted:
