From madler@grumpy.tapil.com  Tue Jun 25 10:24:52 2002
Return-Path: <madler@grumpy.tapil.com>
Received: from tapil.com (dsl092-068-186.bos1.dsl.speakeasy.net [66.92.68.186])
	by hub.freebsd.org (Postfix) with ESMTP id 4376937B43E
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 25 Jun 2002 10:24:37 -0700 (PDT)
Received: from grumpy.tapil.com (localhost.tapil.com [127.0.0.1])
	by tapil.com (8.12.3/8.12.3) with ESMTP id g5PHOaGI010587
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 25 Jun 2002 13:24:36 -0400 (EDT)
	(envelope-from madler@grumpy.tapil.com)
Received: (from madler@localhost)
	by grumpy.tapil.com (8.12.3/8.12.3/Submit) id g5PHOVTg010586;
	Tue, 25 Jun 2002 13:24:31 -0400 (EDT)
Message-Id: <200206251724.g5PHOVTg010586@grumpy.tapil.com>
Date: Tue, 25 Jun 2002 13:24:31 -0400 (EDT)
From: "Michael C. Adler" <mad1@tapil.com>
Reply-To: "Michael C. Adler" <mad1@tapil.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: /sbin/restore fails to overwrite files with schg flag set
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         39849
>Category:       bin
>Synopsis:       restore(8) fails to overwrite files with schg flag set
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jun 25 10:30:01 PDT 2002
>Closed-Date:    
>Last-Modified:  Wed May 21 21:43:39 UTC 2008
>Originator:     Michael C. Adler
>Release:        FreeBSD 4.6-RELEASE i386
>Organization:
>Environment:
System: FreeBSD grumpy.tapil.com 4.6-RELEASE FreeBSD 4.6-RELEASE #0: Tue Jun 18 11:30:24 EDT 2002 madler@grumpy.tapil.com:/usr/obj/usr/src/sys/GRUMPY i386


	
>Description:

Incremental restore fails to overwrite an older file that has a flag
set making the file immutable.  (E.g. schg.)  This became painfully
obvious when I did a level 0 dump/restore.  Updated from 4.5 to 4.6
and then attempted a level 1 dump/restore to the same source/target
disk pair.

>How-To-Repeat:

Create a test tree with a file having the schg flag set.  dump/restore
to a new partition with restore having -r -u set.  Clear the schg flag
in the original file, modify the file and set the schg flag again.  dump
(level 1) and do another restore -r -u to the target.  Note that the
file does not get updated.

>Fix:

The following patch fixes the simplest cases.  It forces unlink() and
rmdir() to clear the flags first.  It forces rename() to clear all flags
and restore them after the file is renamed.

There is a remaining, more difficult, case that is not fixed.  Imagine
directory X is tagged schg.  If its contents are modified for the level
1 dump, restore would have to clear the schg flag on the directory before
restoring the files in X.  I have not fixed this.  It requires walking
up the path until you find a directory without an immutable flag, walking
back down to clear them and then resetting them all after the operation
is complete.

*** extern.h.~1~	Fri Aug 27 20:14:05 1999
--- extern.h	Tue Jun 25 12:33:32 2002
***************
*** 54,59 ****
--- 54,62 ----
  int		 extractfile __P((char *));
  void		 findunreflinks __P((void));
  char		*flagvalues __P((struct entry *));
+ int		 forcerename __P((char *, char *));
+ int		 forcermdir __P((char *));
+ int		 forceunlink __P((char *));
  void		 freeentry __P((struct entry *));
  void		 freename __P((char *));
  int	 	 genliteraldir __P((char *, ino_t));
*** tape.c.~1~	Tue Jun 18 09:09:34 2002
--- tape.c	Tue Jun 25 11:52:28 2002
***************
*** 600,606 ****
  			return (GOOD);
  		}
  		if (uflag && !Nflag)
! 			(void)unlink(name);
  		if (mkfifo(name, mode) < 0) {
  			fprintf(stderr, "%s: cannot create fifo: %s\n",
  			    name, strerror(errno));
--- 600,606 ----
  			return (GOOD);
  		}
  		if (uflag && !Nflag)
! 			(void)forceunlink(name);
  		if (mkfifo(name, mode) < 0) {
  			fprintf(stderr, "%s: cannot create fifo: %s\n",
  			    name, strerror(errno));
***************
*** 622,628 ****
  			return (GOOD);
  		}
  		if (uflag)
! 			(void)unlink(name);
  		if (mknod(name, mode, (int)curfile.dip->di_rdev) < 0) {
  			fprintf(stderr, "%s: cannot create special file: %s\n",
  			    name, strerror(errno));
--- 622,628 ----
  			return (GOOD);
  		}
  		if (uflag)
! 			(void)forceunlink(name);
  		if (mknod(name, mode, (int)curfile.dip->di_rdev) < 0) {
  			fprintf(stderr, "%s: cannot create special file: %s\n",
  			    name, strerror(errno));
***************
*** 643,649 ****
  			return (GOOD);
  		}
  		if (uflag)
! 			(void)unlink(name);
  		if ((ofile = open(name, O_WRONLY | O_CREAT | O_TRUNC,
  		    0666)) < 0) {
  			fprintf(stderr, "%s: cannot create file: %s\n",
--- 643,649 ----
  			return (GOOD);
  		}
  		if (uflag)
! 			(void)forceunlink(name);
  		if ((ofile = open(name, O_WRONLY | O_CREAT | O_TRUNC,
  		    0666)) < 0) {
  			fprintf(stderr, "%s: cannot create file: %s\n",
*** utilities.c.~1~	Fri Sep 21 16:05:22 2001
--- utilities.c	Tue Jun 25 12:34:14 2002
***************
*** 129,135 ****
  renameit(from, to)
  	char *from, *to;
  {
! 	if (!Nflag && rename(from, to) < 0) {
  		fprintf(stderr, "warning: cannot rename %s to %s: %s\n",
  		    from, to, strerror(errno));
  		return;
--- 129,135 ----
  renameit(from, to)
  	char *from, *to;
  {
! 	if (!Nflag && forcerename(from, to) < 0) {
  		fprintf(stderr, "warning: cannot rename %s to %s: %s\n",
  		    from, to, strerror(errno));
  		return;
***************
*** 173,179 ****
  	ep->e_flags |= REMOVED;
  	ep->e_flags &= ~TMPNAME;
  	cp = myname(ep);
! 	if (!Nflag && rmdir(cp) < 0) {
  		fprintf(stderr, "warning: %s: %s\n", cp, strerror(errno));
  		return;
  	}
--- 173,179 ----
  	ep->e_flags |= REMOVED;
  	ep->e_flags &= ~TMPNAME;
  	cp = myname(ep);
! 	if (!Nflag && forcermdir(cp) < 0) {
  		fprintf(stderr, "warning: %s: %s\n", cp, strerror(errno));
  		return;
  	}
***************
*** 194,200 ****
  	ep->e_flags |= REMOVED;
  	ep->e_flags &= ~TMPNAME;
  	cp = myname(ep);
! 	if (!Nflag && unlink(cp) < 0) {
  		fprintf(stderr, "warning: %s: %s\n", cp, strerror(errno));
  		return;
  	}
--- 194,200 ----
  	ep->e_flags |= REMOVED;
  	ep->e_flags &= ~TMPNAME;
  	cp = myname(ep);
! 	if (!Nflag && forceunlink(cp) < 0) {
  		fprintf(stderr, "warning: %s: %s\n", cp, strerror(errno));
  		return;
  	}
***************
*** 250,255 ****
--- 250,339 ----
  	vprintf(stdout, "Create %s link %s->%s\n",
  		type == SYMLINK ? "symbolic" : "hard", new, existing);
  	return (GOOD);
+ }
+ 
+ /*
+  * Make rename() work even if system flags prohibit it.  Don't print warning
+  * messages here since caller may be calling without testing if the
+  * file exists.
+  */
+ 
+ int
+ forcerename(from, to)
+ 	char *from, *to;
+ {
+ 	struct stat sb;
+ 
+ 	if (!Nflag) {
+ 		if (stat(from, &sb) < 0) {
+ 			return -1;
+ 		}
+ 		if (sb.st_flags) {
+ 			chflags(from, 0);
+ 		}
+ 		if (rename(from, to) < 0) {
+ 			return -1;
+ 		}
+ 		if (sb.st_flags) {
+ 			return chflags(to, sb.st_flags);
+ 		}
+ 	}
+ 
+ 	return 0;
+ }
+ 
+ /*
+  * Make rmdir() work even if system flags prohibit it.  Don't print warning
+  * messages here since caller may be calling without testing if the
+  * directory exists.
+  */
+ 
+ int
+ forcermdir(name)
+ 	char *name;
+ {
+ 	struct stat sb;
+ 
+ 	if (!Nflag) {
+ 		if (stat(name, &sb) < 0) {
+ 			return -1;
+ 		}
+ 		if (sb.st_flags) {
+ 			chflags(name, 0);
+ 		}
+ 		if (rmdir(name) < 0) {
+ 			return -1;
+ 		}
+ 	}
+ 
+ 	return 0;
+ }
+ 
+ /*
+  * Make unlink() work even if system flags prohibit it.  Don't print warning
+  * messages here since caller may be calling without testing if the
+  * file exists.
+  */
+ 
+ int
+ forceunlink(name)
+ 	char *name;
+ {
+ 	struct stat sb;
+ 
+ 	if (!Nflag) {
+ 		if (stat(name, &sb) < 0) {
+ 			return -1;
+ 		}
+ 		if (sb.st_flags) {
+ 			chflags(name, 0);
+ 		}
+ 		if (unlink(name) < 0) {
+ 			return -1;
+ 		}
+ 	}
+ 
+ 	return 0;
  }
  
  /*



>Release-Note:
>Audit-Trail:

From: "Crist J. Clark" <crist.clark@attbi.com>
To: "Michael C. Adler" <mad1@tapil.com>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: bin/39849: /sbin/restore fails to overwrite files with schg flag set
Date: Thu, 27 Jun 2002 00:15:41 -0700

 On Tue, Jun 25, 2002 at 01:24:31PM -0400, Michael C. Adler wrote:
 [snip]
 
 > Incremental restore fails to overwrite an older file that has a flag
 > set making the file immutable.
 
 I really think this is a feature and not a bug. However, I can see
 where one might want this. I think the right way would be to have a
 command line switch which enables this [su]chg-flag clobbering, and
 leave the default as-is.
 
 Note that there are other, tougher issues when doing restores on an
 existing filesystem when names collide (e.g. does restore(8)
 over-write an existing directory with a plain file from backup?).
 Handling this issue by running a,
 
   # chflags -R 0 /filesystem/path
 
 Before a restore(8) is fairly trivial.
 -- 
 Crist J. Clark                     |     cjclark@alum.mit.edu
                                    |     cjclark@jhu.edu
 http://people.freebsd.org/~cjc/    |     cjc@freebsd.org

From: Michael Adler <mad1@tapil.com>
To: "Crist J. Clark" <cjc@FreeBSD.ORG>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: bin/39849: /sbin/restore fails to overwrite files with
  schg flag set
Date: Thu, 27 Jun 2002 09:04:13 -0400

 At 03:15 AM 6/27/2002, Crist J. Clark wrote:
 >On Tue, Jun 25, 2002 at 01:24:31PM -0400, Michael C. Adler wrote:
 >[snip]
 >
 > > Incremental restore fails to overwrite an older file that has a flag
 > > set making the file immutable.
 >
 >I really think this is a feature and not a bug. However, I can see
 >where one might want this. I think the right way would be to have a
 >command line switch which enables this [su]chg-flag clobbering, and
 >leave the default as-is.
 
 I agree it is debatable.  It seems to me that restore is a special 
 case.  Its goal is to produce a target identical to the source.  Given that 
 the source changed, I thought the target should as well.  It could easily 
 be put on a switch or perhaps it should be turned on with the -r switch.
 
 >Note that there are other, tougher issues when doing restores on an
 >existing filesystem when names collide (e.g. does restore(8)
 >over-write an existing directory with a plain file from backup?).
 >Handling this issue by running a,
 >
 >   # chflags -R 0 /filesystem/path
 >
 >Before a restore(8) is fairly trivial.
 
 Of course this might not do what you want.  Does an incremental dump have 
 all the flags for all files in the target during restore -r?  If not, the 
 target files may lose their flags.
 
 -Michael
 

From: Nicolas Rachinsky <nicolas-2007@rachinsky.de>
To: bug-followup@FreeBSD.org, mad1@tapil.com, crist.clark@attbi.com
Cc:  
Subject: Re: bin/39849: /sbin/restore fails to overwrite files with schg
	flag set
Date: Thu, 24 May 2007 14:26:21 +0200

 Hallo,
 
 At 03:15 AM 6/27/2002, Crist J. Clark wrote:
 > # chflags -R 0 /filesystem/path
 >
 >Before a restore(8) is fairly trivial.
 
 But restore does not set all the flags again.
 
 So you either have to loose some flags or some file changes if you
 restore a multi level backup.
 
 Or you restore the flags manually.
 
 I can't believe this. What am I missing?
 
 Nicolas

From: "Crist J. Clark" <cristclark@comcast.net>
To: Nicolas Rachinsky <nicolas-2007@rachinsky.de>
Cc: bug-followup@FreeBSD.org, mad1@tapil.com
Subject: Re: bin/39849: /sbin/restore fails to overwrite files with schg flag set
Date: Thu, 24 May 2007 12:53:15 -0700

 On Thu, May 24, 2007 at 02:26:21PM +0200, Nicolas Rachinsky wrote:
 > Hallo,
 > 
 > At 03:15 AM 6/27/2002, Crist J. Clark wrote:
 > > # chflags -R 0 /filesystem/path
 > >
 > >Before a restore(8) is fairly trivial.
 > 
 > But restore does not set all the flags again.
 > 
 > So you either have to loose some flags or some file changes if you
 > restore a multi level backup.
 > 
 > Or you restore the flags manually.
 > 
 > I can't believe this. What am I missing?
 
 What flags does restore(8) not preserve?
 
 And I think only a,
 
 	# chflags -R noschg /filesystem/path
 
 Is really necessary.
 
 But there still can be a problem. If you do "chflags -R",
 you will lose the flag on any file that doesn't get touched
 by the restore(8).
 
 You could use find(1) to generate a list and then use the
 list to reset the flags after. But that is a kludge.
 
 An option on dump(8) to force rewriting on schg files (when
 securelevel(8) allows) is probably appropriate, but as
 discussed, when you start to consider directories, it becomes
 non-trivial to implement.
 
 Unfortunately, this PR is coming up on its fifth birthday
 and I don't do FreeBSD work anymore. Feel free to submit
 patches against a more modern CURRENT and STABLE.
 -- 
 Crist J. Clark                     |     cjclark@alum.mit.edu

From: Nicolas Rachinsky <nicolas-2007@rachinsky.de>
To: cjclark@alum.mit.edu
Cc: bug-followup@FreeBSD.org, mad1@tapil.com
Subject: Re: bin/39849: /sbin/restore fails to overwrite files with schg
	flag set
Date: Thu, 24 May 2007 22:23:56 +0200

 * "Crist J. Clark" <cristclark@comcast.net> [2007-05-24 12:53 -0700]:
 > On Thu, May 24, 2007 at 02:26:21PM +0200, Nicolas Rachinsky wrote:
 > > >Before a restore(8) is fairly trivial.
 > > 
 > > But restore does not set all the flags again.
 > > 
 > > So you either have to loose some flags or some file changes if you
 > > restore a multi level backup.
 > 
 > What flags does restore(8) not preserve?
 
 The flags of the files not touched by the following restore run.
 
 > But there still can be a problem. If you do "chflags -R",
 > you will lose the flag on any file that doesn't get touched
 > by the restore(8).
 
 These.
 
 > You could use find(1) to generate a list and then use the
 > list to reset the flags after. But that is a kludge.
 
 And you might preserve flags that shouldn't be preserved, since they
 were not during the backup (restored after the find).
 
 > An option on dump(8) to force rewriting on schg files (when
 > securelevel(8) allows) is probably appropriate, but as
 > discussed, when you start to consider directories, it becomes
 > non-trivial to implement.
 
 Yes.
 
 > Unfortunately, this PR is coming up on its fifth birthday
 > and I don't do FreeBSD work anymore. Feel free to submit
 > patches against a more modern CURRENT and STABLE.
 
 I noticed this problem today. And the (incomplete) patch in the PR
 was (kind of) rejected by you.
 
 I will try to find some time to look at this.
 
 Nicolas
>Unformatted:
