From nobody@FreeBSD.org  Wed Feb 10 19:23:56 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D093E106568F
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 10 Feb 2010 19:23:56 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id BE4588FC13
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 10 Feb 2010 19:23:56 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o1AJNuEm092762
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 10 Feb 2010 19:23:56 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o1AJNuTB092761;
	Wed, 10 Feb 2010 19:23:56 GMT
	(envelope-from nobody)
Message-Id: <201002101923.o1AJNuTB092761@www.freebsd.org>
Date: Wed, 10 Feb 2010 19:23:56 GMT
From: David Naylor <naylor.b.david@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [mtree] mtree does a full hierarchy walk when requested to just update structure (-u -e)
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         143732
>Category:       bin
>Synopsis:       [patch] mtree(8) does a full hierarchy walk when requested to just update structure (-u -e)
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 10 19:30:01 UTC 2010
>Closed-Date:    
>Last-Modified:  Wed Sep  1 07:50:01 UTC 2010
>Originator:     David Naylor
>Release:        FreeBSD 8
>Organization:
Private
>Environment:
FreeBSD dragon.dg 8.0-STABLE FreeBSD 8.0-STABLE #0: Wed Jan 20 20:06:26 SAST 2010     root@dragon.dg:/tmp/usr/src/sys/GENERIC  amd64

>Description:
mtree does a full hierarchy walk when requested to update a (directory) structure.  This is unnecessary as mtree need only visit the directories or files in question.  This can have particular performance impact if the path's structure is extensive (i.e. /usr/local with 1000+ ports installed) and the structure to update is relatively short or when IO is slow (i.e. stacked unionfs).  

This has real world impact as this mtree command is run every time a port is installed.  
>How-To-Repeat:
The following script simulates slow IO (using stacked unionfs).  Note that the time mtree takes is similar to find when unpatched but a fraction of the time (between 60% to 80% quicker [1 - new/old]).  I expect this performance to be better when the path is populated with extra content (e.g. installed ports).  

The script should be run as root in an empty root (the number of overlayed unionfs are specified as the first command, please see http://lists.freebsd.org/pipermail/freebsd-current/2010-January/015040.html for possible problems [system freeze, kstack overflow]).  

On my computer with 155 overlayed unionfs the performance I got:
# ./unionfs_mtree.sh 155 | tail -n 1
mtree 155:       123.77 real         0.03 user       112.67 sys (unpatched)
mtree 155:        29.94 real         0.00 user        29.81 sys (patched)

The script (unionfs_mtree.sh):

#!/bin/sh
set -e

mount_unionfs() {

  count=1
  while [ $count -lt $1 ]
  do
    mount -rt unionfs -o noatime $count top
    count=$(($count + 1))
  done

}

umount_unionfs() {

  count=$1
  while [ $count -gt 1 ]
  do
    count=$(($count - 1))
    umount top
  done

}

mkdir -p top

[ $# -eq 1 ] || (echo "Please specify directory count!!!"; false)

for i in `jot $1`
do
  mount_unionfs $i
  mkdir -p $i
  mount -t unionfs -o noatime $i top

  set +e
  trap "true" INT TERM EXIT
  echo -n "find $i:  "
  time find top > /dev/null
  status=$?

  if [ $status -eq 0 ]
  then
    echo -n "mtree $i: "
    time /usr/sbin/mtree -U -f /usr/ports/Templates/BSD.local.dist -d -e -p top
    status=$?
  fi
  trap - INT TERM EXIT
  set -e

  umount top
  umount_unionfs $i
  [ $status -eq 0 ] || (echo "Terminated"; false)
done

>Fix:
The attached patch prevents a hierarchy walk if only an update is requested.  It simulated the structure returned from fts_read as required by compare.  

An improvement in performance could be implemented using chdir (as fts_* does) however that would require a pervasive change to existing code.  

Patch attached with submission follows:

--- /usr/src/usr.sbin/mtree/verify.c	2010-02-07 15:07:28.000000000 +0200
+++ verify.c	2010-02-07 15:04:10.000000000 +0200
@@ -50,17 +50,23 @@
 static NODE *root;
 static char path[MAXPATHLEN];
 
-static void	miss(NODE *, char *);
+static int	miss(NODE *, char *);
+static int	check(NODE *, char *);
 static int	vwalk(void);
 
 int
 mtree_verifyspec(FILE *fi)
 {
-	int rval;
+	int rval = 0;
 
 	root = mtree_readspec(fi);
-	rval = vwalk();
-	miss(root, path);
+	/*
+	 * No need to walk entire tree if we are only updating the structure
+	 * and extra files are ignored.
+	 */
+	if (!(uflag && eflag))
+		rval = vwalk();
+	rval |= miss(root, path);
 	return (rval);
 }
 
@@ -155,15 +161,47 @@
 	return (rval);
 }
 
-static void
+static int
+check(NODE *p, char *tail)
+{
+	FTSENT fts;
+	struct stat fts_stat;
+
+	strcpy(tail, p->name);
+
+	/*
+	 * It is assumed that compare() only requires fts_accpath and fts_statp
+	 * fields in the FTSENT structure.
+	 */
+	fts.fts_accpath = path;
+	fts.fts_statp = &fts_stat;
+
+	if (stat(path, fts.fts_statp))
+		return (0);
+
+	p->flags |= F_VISIT;
+	if ((p->flags & F_NOCHANGE) == 0 && compare(p->name, p, &fts))
+		return (MISMATCHEXIT);
+	else
+		return (0);
+
+	/*
+	 * tail is not restored to '\0' as the next time tail (or path) is used
+	 * is with a strcpy (thus overriding the '\0').  See +19 lines below.
+	 */
+}
+
+static int
 miss(NODE *p, char *tail)
 {
 	int create;
 	char *tp;
 	const char *type, *what;
-	int serr;
+	int serr, rval = 0;
 
 	for (; p; p = p->next) {
+		if (uflag && eflag)
+			rval |= check(p, tail);
 		if (p->flags & F_OPT && !(p->flags & F_VISIT))
 			continue;
 		if (p->type != F_DIR && (dflag || p->flags & F_VISIT))
@@ -256,4 +294,5 @@
 			(void)printf("%s: file flags not set: %s\n",
 			    path, strerror(errno));
 	}
+	return (rval);
 }


>Release-Note:
>Audit-Trail:

From: David Naylor <naylor.b.david@gmail.com>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: bin/143732: [patch] mtree(1) does a full hierarchy walk when requested to just update structure (-u -e)
Date: Mon, 15 Feb 2010 18:34:46 +0200

 Using my "real" world benchmark (see http://unix.derkeiler.com/Mailing-
 Lists/FreeBSD/current/2010-01/msg00453.html) I have achieved a 20% speedup 
 using the previously attached patch.  My results:
 
 # time ./ports-union-builder.sh (old mtree)
      8123.25 real      2280.53 user      6319.77 sys
 
 # time ./ports-union-builder.sh (new mtree)
      6456.11 real      2272.07 user      5778.74 sys
 
 By my estimated the hierarchical walking of mtree resulted in an additional 28 
 minutes real time and 9 minutes system time.  

From: David Naylor <naylor.b.david@gmail.com>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: bin/143732: [patch] mtree(8) does a full hierarchy walk when requested to just update structure (-u -e)
Date: Wed, 1 Sep 2010 09:43:53 +0200

 --Boundary-00=_8QgfMw2WK9C4H4q
 Content-Type: Text/Plain;
   charset="us-ascii"
 Content-Transfer-Encoding: 7bit
 
 Thanks to the feedback from the hackers ML I've updated the patch.  It now is 
 more aggressive in using the optimisation and handles buffer overflows (that 
 previously went unhandled by mtree).  
 
 Please see attached for the patch.
 
 --Boundary-00=_8QgfMw2WK9C4H4q
 Content-Type: text/x-patch;
   charset="ISO-8859-1";
   name="mtree.diff"
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment;
 	filename="mtree.diff"
 
 --- /usr/src/usr.sbin/mtree/verify.c	2007-12-07 14:22:38.000000000 +0200
 +++ verify.c	2010-08-30 22:32:32.000000000 +0200
 @@ -47,20 +47,35 @@
  #include "mtree.h"
  #include "extern.h"
  
 +/*
 + * Error returned when buffer overflows, cannot be 2 as that is reserved for
 + * MISMATCHEXIT.
 + */
 +#define BUFFEROVERFLOWEXIT 3
 +
  static NODE *root;
  static char path[MAXPATHLEN];
  
 -static void	miss(NODE *, char *);
 +static int	miss(NODE *, char *, size_t);
 +static int	check(NODE *, char *, size_t);
  static int	vwalk(void);
  
  int
  mtree_verifyspec(FILE *fi)
  {
 -	int rval;
 +	int rval = 0;
  
  	root = mtree_readspec(fi);
 -	rval = vwalk();
 -	miss(root, path);
 +	/* No need to walk tree if we are ignoring extra files. */
 +	if (!eflag)
 +		rval = vwalk();
 +	*path = '\0';
 +	rval |= miss(root, path, MAXPATHLEN);
 +
 +	/* Called here to make sure vwalk() and check() have been called */
 +	if (sflag)
 +		warnx("%s checksum: %lu", fullpath, (unsigned long)crc_total);
 +
  	return (rval);
  }
  
 @@ -135,40 +150,96 @@
  		if (ep)
  			continue;
  extra:
 -		if (!eflag) {
 -			(void)printf("%s extra", RP(p));
 -			if (rflag) {
 -				if ((S_ISDIR(p->fts_statp->st_mode)
 -				    ? rmdir : unlink)(p->fts_accpath)) {
 -					(void)printf(", not removed: %s",
 -					    strerror(errno));
 -				} else
 -					(void)printf(", removed");
 -			}
 -			(void)putchar('\n');
 +		(void)printf("%s extra", RP(p));
 +		if (rflag) {
 +			if ((S_ISDIR(p->fts_statp->st_mode)
 +			    ? rmdir : unlink)(p->fts_accpath)) {
 +				(void)printf(", not removed: %s",
 +				    strerror(errno));
 +			} else
 +				(void)printf(", removed");
  		}
 +		(void)putchar('\n');
  		(void)fts_set(t, p, FTS_SKIP);
  	}
  	(void)fts_close(t);
 -	if (sflag)
 -		warnx("%s checksum: %lu", fullpath, (unsigned long)crc_total);
  	return (rval);
  }
  
 -static void
 -miss(NODE *p, char *tail)
 +static int
 +check(NODE *p, char *tail, size_t tail_len)
 +{
 +	FTSENT fts;
 +	struct stat fts_stat;
 +
 +	if (strlcpy(tail, p->name, tail_len) >= tail_len)
 +		return (BUFFEROVERFLOWEXIT);
 +
 +	/*
 +	 * It is assumed that compare() only requires fts_accpath and fts_statp
 +	 * fields in the FTSENT structure.
 +	 */
 +	fts.fts_accpath = path;
 +	fts.fts_statp = &fts_stat;
 +
 +	if (ftsoptions & FTS_LOGICAL) {
 +		if (stat(path, fts.fts_statp) || lstat(path, fts.fts_statp))
 +			return (0);
 +	} else if (lstat(path, fts.fts_statp))
 +		return (0);
 +
 +	p->flags |= F_VISIT;
 +	if ((p->flags & F_NOCHANGE) == 0 && compare(p->name, p, &fts))
 +		return (MISMATCHEXIT);
 +	else
 +		return (0);
 +
 +	/*
 +	 * tail is not restored to '\0' as the next time tail (or path) is used
 +	 * is with a strlcpy (thus overriding the '\0').
 +	 */
 +}
 +
 +static int
 +miss(NODE *p, char *tail, size_t tail_len)
  {
  	int create;
  	char *tp;
  	const char *type, *what;
 +	int rval = 0;
  	int serr;
 +	size_t name_len;
  
  	for (; p; p = p->next) {
 +		/*
 +		 * if check() needs to be called (eflag set) then directly
 +		 * update nodes if they are not directories and only
 +		 * directories are being checked otherwise check().
 +		 */
 +#if 1
 +		if (tail >= path + MAXPATHLEN)
 +			(void)printf("!!!max path len exceeded!!!");
 +#endif
 +		if (eflag) {
 +			if (dflag && (p->type != F_DIR))
 +				p->flags |= F_VISIT;
 +			else
 +				rval |= check(p, tail, tail_len);
 +		}
 +		/*
 +		 * if check() needs to be called and fails with buffer overflow
 +		 * it is assumed the node cannot be visited as it also cannot
 +		 * exist (due to MAXPATHLEN being exceeded).
 +		 */
  		if (p->flags & F_OPT && !(p->flags & F_VISIT))
  			continue;
  		if (p->type != F_DIR && (dflag || p->flags & F_VISIT))
  			continue;
 -		(void)strcpy(tail, p->name);
 +		if ((name_len = strlcpy(tail, p->name, tail_len)) >= tail_len) {
 +			(void)printf("%s%s (skipped, buffer overflow)\n", path, p->name);
 +			rval |= BUFFEROVERFLOWEXIT;
 +			continue;
 +		}
  		if (!(p->flags & F_VISIT)) {
  			/* Don't print missing message if file exists as a
  			   symbolic link and the -q flag is set. */
 @@ -228,10 +299,15 @@
  		if (!(p->flags & F_VISIT))
  			(void)putchar('\n');
  
 -		for (tp = tail; *tp; ++tp);
 -		*tp = '/';
 -		miss(p->child, tp + 1);
 -		*tp = '\0';
 +		if (tail_len - name_len > 1) {
 +			tp = tail + name_len;
 +			*tp = '/';
 +			rval |= miss(p->child, tp + 1, tail_len - name_len - 1);
 +			*tp = '\0';
 +		} else if (p->child) {
 +			(void)printf("%s (children skipped, buffer overflow)\n", path);
 +			rval |= BUFFEROVERFLOWEXIT;
 +		}
  
  		if (!create)
  			continue;
 @@ -256,4 +332,5 @@
  			(void)printf("%s: file flags not set: %s\n",
  			    path, strerror(errno));
  	}
 +	return (rval);
  }
 
 --Boundary-00=_8QgfMw2WK9C4H4q--
>Unformatted:
