From nobody@FreeBSD.org  Fri May  7 16:27:52 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 185B51065673
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  7 May 2010 16:27:52 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id 0822D8FC1A
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  7 May 2010 16:27:52 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o47GRpbw011797
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 7 May 2010 16:27:51 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o47GRpO1011796;
	Fri, 7 May 2010 16:27:51 GMT
	(envelope-from nobody)
Message-Id: <201005071627.o47GRpO1011796@www.freebsd.org>
Date: Fri, 7 May 2010 16:27:51 GMT
From: Peer Stritzinger <peer@stritzinger.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: sh(1) can't path expand  foo/*"/baz" patterns
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         146378
>Category:       bin
>Synopsis:       sh(1) can't path expand  foo/*"/baz" patterns
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    jilles
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri May 07 16:30:06 UTC 2010
>Closed-Date:    Sat Jul 03 21:15:10 UTC 2010
>Last-Modified:  Sat Jul 03 21:15:10 UTC 2010
>Originator:     Peer Stritzinger
>Release:        FreeBSD 8.0-RELEASE i386
>Organization:
Dipl.Phys. Peer Stritzinger GmbH
>Environment:
FreeBSD xxx.xxx.xxx 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:48:17 UTC 2009     root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
sh(1) does not path expand correctly if the combination *"/ appears in the path.

Paths like foo/*/"baz"  work and also path like fo*"bar".  Also "foo/"*/baz works.

But not paths like foo/*"/baz"

This occurs e.g. in the configure scripts of the latest rtems embedded operating system, this was seen "in the wild":

for bsp_cfgs in `ls "$srcdir/$RTEMS_TOPdir/c/src/lib/libbsp/$RTEMS_CPU/"*"/make/custom/$1.cfg" 2>/dev/null`; do

When the port in the ports collection is updated this problem might arise.

I tested this also under other BSDis systems especially an older version of FreeBSD:

FreeBSD 6.1-RELEASE-p3 i386 -> works
OpenBSD 4.2 i386 -> works


>How-To-Repeat:
in an empty directory:

Preparation:

$ mkdir -p foo/bar/baz

Test using /bin/sh:

$ echo foo/*"/baz"
foo/*/baz

-> correct would be the output foo/bar/baz


>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->jilles 
Responsible-Changed-By: jilles 
Responsible-Changed-When: Tue May 11 21:46:16 UTC 2010 
Responsible-Changed-Why:  
I'm working on this. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=146378 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/146378: commit references a PR
Date: Tue, 11 May 2010 23:19:41 +0000 (UTC)

 Author: jilles
 Date: Tue May 11 23:19:28 2010
 New Revision: 207944
 URL: http://svn.freebsd.org/changeset/base/207944
 
 Log:
   sh: Fix pathname expansion with quoted slashes like *\/.
   
   These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
   d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.
   
   Because this is the first code I'm importing from dash to expand.c, add the
   Herbert Xu copyright notice which is in dash's expand.c.
   
   When pathname expanding *\/, the CTLESC representing the quoted state was
   erroneously taken as part of the * pathname component. This CTLESC was then
   seen by the pattern matching code as escaping the '\0' terminating the
   string.
   
   The code is slightly different because dash converts the CTLESC characters
   to backslashes and removes all the other CTL* characters to allow
   substituting glob(3).
   
   The effect of the bug was also slightly different from dash (where nothing
   matched at all). Because a CTLESC can escape a '\0' in some way, whether
   files were included despite the bug depended on memory that should not be
   read. In particular, on many machines /*\/ expanded to a strict subset of
   what /*/ expanded to.
   
   Example:
     echo /*"/null"
   
   This should print /dev/null, not /*/null.
   
   PR:		bin/146378
   Obtained from:	dash
 
 Added:
   head/tools/regression/bin/sh/expansion/pathname2.0   (contents, props changed)
 Modified:
   head/bin/sh/expand.c
 
 Modified: head/bin/sh/expand.c
 ==============================================================================
 --- head/bin/sh/expand.c	Tue May 11 23:08:38 2010	(r207943)
 +++ head/bin/sh/expand.c	Tue May 11 23:19:28 2010	(r207944)
 @@ -1,6 +1,8 @@
  /*-
   * Copyright (c) 1991, 1993
   *	The Regents of the University of California.  All rights reserved.
 + * Copyright (c) 1997-2005
 + *	Herbert Xu <herbert@gondor.apana.org.au>.  All rights reserved.
   *
   * This code is derived from software contributed to Berkeley by
   * Kenneth Almquist.
 @@ -1150,10 +1152,11 @@ expmeta(char *enddir, char *name)
  	struct dirent *dp;
  	int atend;
  	int matchdot;
 +	int esc;
  
  	metaflag = 0;
  	start = name;
 -	for (p = name ; ; p++) {
 +	for (p = name; esc = 0, *p; p += esc + 1) {
  		if (*p == '*' || *p == '?')
  			metaflag = 1;
  		else if (*p == '[') {
 @@ -1178,12 +1181,14 @@ expmeta(char *enddir, char *name)
  			break;
  		else if (*p == CTLQUOTEMARK)
  			continue;
 -		else if (*p == CTLESC)
 -			p++;
 -		if (*p == '/') {
 -			if (metaflag)
 -				break;
 -			start = p + 1;
 +		else {
 +			if (*p == CTLESC)
 +				esc++;
 +			if (p[esc] == '/') {
 +				if (metaflag)
 +					break;
 +				start = p + esc + 1;
 +			}
  		}
  	}
  	if (metaflag == 0) {	/* we've reached the end of the file name */
 @@ -1229,7 +1234,8 @@ expmeta(char *enddir, char *name)
  		atend = 1;
  	} else {
  		atend = 0;
 -		*endname++ = '\0';
 +		*endname = '\0';
 +		endname += esc + 1;
  	}
  	matchdot = 0;
  	p = start;
 @@ -1257,7 +1263,7 @@ expmeta(char *enddir, char *name)
  	}
  	closedir(dirp);
  	if (! atend)
 -		endname[-1] = '/';
 +		endname[-esc - 1] = esc ? CTLESC : '/';
  }
  
  
 
 Added: head/tools/regression/bin/sh/expansion/pathname2.0
 ==============================================================================
 --- /dev/null	00:00:00 1970	(empty, because file is newly added)
 +++ head/tools/regression/bin/sh/expansion/pathname2.0	Tue May 11 23:19:28 2010	(r207944)
 @@ -0,0 +1,31 @@
 +# $FreeBSD$
 +
 +failures=0
 +
 +check() {
 +	testcase=$1
 +	expect=$2
 +	eval "set -- $testcase"
 +	actual="$*"
 +	if [ "$actual" != "$expect" ]; then
 +		failures=$((failures+1))
 +		printf '%s\n' "For $testcase, expected $expect actual $actual"
 +	fi
 +}
 +
 +set -e
 +T=$(mktemp -d ${TMPDIR:-/tmp}/sh-test.XXXXXX)
 +trap 'rm -rf $T' 0
 +cd -P $T
 +
 +mkdir testdir testdir2 'testdir/*' 'testdir/?' testdir/a testdir/b testdir2/b
 +mkdir testdir2/.c
 +touch testf 'testdir/*/1' 'testdir/?/1' testdir/a/1 testdir/b/1 testdir2/b/.a
 +
 +check '*\/' 'testdir/ testdir2/'
 +check '"testdir/"*"/1"' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +check '"testdir/"*"/"*' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +check '"testdir/"*\/*' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +check '"testdir"*"/"*"/"*' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +
 +exit $((failures != 0))
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: jilles 
State-Changed-When: Tue May 11 23:30:03 UTC 2010 
State-Changed-Why:  
Fixed in 9-CURRENT. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=146378 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/146378: commit references a PR
Date: Sun, 16 May 2010 22:53:07 +0000 (UTC)

 Author: jilles
 Date: Sun May 16 22:52:51 2010
 New Revision: 208174
 URL: http://svn.freebsd.org/changeset/base/208174
 
 Log:
   MFC r207944: sh: Fix pathname expansion with quoted slashes like *\/.
   
   These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
   d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.
   
   Because this is the first code I'm importing from dash to expand.c, add the
   Herbert Xu copyright notice which is in dash's expand.c.
   
   When pathname expanding *\/, the CTLESC representing the quoted state was
   erroneously taken as part of the * pathname component. This CTLESC was then
   seen by the pattern matching code as escaping the '\0' terminating the
   string.
   
   The code is slightly different because dash converts the CTLESC characters
   to backslashes and removes all the other CTL* characters to allow
   substituting glob(3).
   
   The effect of the bug was also slightly different from dash (where nothing
   matched at all). Because a CTLESC can escape a '\0' in some way, whether
   files were included despite the bug depended on memory that should not be
   read. In particular, on many machines /*\/ expanded to a strict subset of
   what /*/ expanded to.
   
   Example:
     echo /*"/null"
   
   This should print /dev/null, not /*/null.
   
   PR:		bin/146378
   Obtained from:	dash
 
 Added:
   stable/8/tools/regression/bin/sh/expansion/pathname2.0
      - copied unchanged from r207944, head/tools/regression/bin/sh/expansion/pathname2.0
 Modified:
   stable/8/bin/sh/expand.c
 Directory Properties:
   stable/8/bin/sh/   (props changed)
   stable/8/tools/regression/bin/sh/   (props changed)
 
 Modified: stable/8/bin/sh/expand.c
 ==============================================================================
 --- stable/8/bin/sh/expand.c	Sun May 16 22:21:33 2010	(r208173)
 +++ stable/8/bin/sh/expand.c	Sun May 16 22:52:51 2010	(r208174)
 @@ -1,6 +1,8 @@
  /*-
   * Copyright (c) 1991, 1993
   *	The Regents of the University of California.  All rights reserved.
 + * Copyright (c) 1997-2005
 + *	Herbert Xu <herbert@gondor.apana.org.au>.  All rights reserved.
   *
   * This code is derived from software contributed to Berkeley by
   * Kenneth Almquist.
 @@ -1142,10 +1144,11 @@ expmeta(char *enddir, char *name)
  	struct dirent *dp;
  	int atend;
  	int matchdot;
 +	int esc;
  
  	metaflag = 0;
  	start = name;
 -	for (p = name ; ; p++) {
 +	for (p = name; esc = 0, *p; p += esc + 1) {
  		if (*p == '*' || *p == '?')
  			metaflag = 1;
  		else if (*p == '[') {
 @@ -1170,12 +1173,14 @@ expmeta(char *enddir, char *name)
  			break;
  		else if (*p == CTLQUOTEMARK)
  			continue;
 -		else if (*p == CTLESC)
 -			p++;
 -		if (*p == '/') {
 -			if (metaflag)
 -				break;
 -			start = p + 1;
 +		else {
 +			if (*p == CTLESC)
 +				esc++;
 +			if (p[esc] == '/') {
 +				if (metaflag)
 +					break;
 +				start = p + esc + 1;
 +			}
  		}
  	}
  	if (metaflag == 0) {	/* we've reached the end of the file name */
 @@ -1221,7 +1226,8 @@ expmeta(char *enddir, char *name)
  		atend = 1;
  	} else {
  		atend = 0;
 -		*endname++ = '\0';
 +		*endname = '\0';
 +		endname += esc + 1;
  	}
  	matchdot = 0;
  	p = start;
 @@ -1249,7 +1255,7 @@ expmeta(char *enddir, char *name)
  	}
  	closedir(dirp);
  	if (! atend)
 -		endname[-1] = '/';
 +		endname[-esc - 1] = esc ? CTLESC : '/';
  }
  
  
 
 Copied: stable/8/tools/regression/bin/sh/expansion/pathname2.0 (from r207944, head/tools/regression/bin/sh/expansion/pathname2.0)
 ==============================================================================
 --- /dev/null	00:00:00 1970	(empty, because file is newly added)
 +++ stable/8/tools/regression/bin/sh/expansion/pathname2.0	Sun May 16 22:52:51 2010	(r208174, copy of r207944, head/tools/regression/bin/sh/expansion/pathname2.0)
 @@ -0,0 +1,31 @@
 +# $FreeBSD$
 +
 +failures=0
 +
 +check() {
 +	testcase=$1
 +	expect=$2
 +	eval "set -- $testcase"
 +	actual="$*"
 +	if [ "$actual" != "$expect" ]; then
 +		failures=$((failures+1))
 +		printf '%s\n' "For $testcase, expected $expect actual $actual"
 +	fi
 +}
 +
 +set -e
 +T=$(mktemp -d ${TMPDIR:-/tmp}/sh-test.XXXXXX)
 +trap 'rm -rf $T' 0
 +cd -P $T
 +
 +mkdir testdir testdir2 'testdir/*' 'testdir/?' testdir/a testdir/b testdir2/b
 +mkdir testdir2/.c
 +touch testf 'testdir/*/1' 'testdir/?/1' testdir/a/1 testdir/b/1 testdir2/b/.a
 +
 +check '*\/' 'testdir/ testdir2/'
 +check '"testdir/"*"/1"' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +check '"testdir/"*"/"*' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +check '"testdir/"*\/*' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +check '"testdir"*"/"*"/"*' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +
 +exit $((failures != 0))
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/146378: commit references a PR
Date: Sat,  3 Jul 2010 21:07:06 +0000 (UTC)

 Author: jilles
 Date: Sat Jul  3 21:06:48 2010
 New Revision: 209675
 URL: http://svn.freebsd.org/changeset/base/209675
 
 Log:
   MFC r207944: sh: Fix pathname expansion with quoted slashes like *\/.
   
   These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
   d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.
   
   Because this is the first code I'm importing from dash to expand.c, add the
   Herbert Xu copyright notice which is in dash's expand.c.
   
   When pathname expanding *\/, the CTLESC representing the quoted state was
   erroneously taken as part of the * pathname component. This CTLESC was then
   seen by the pattern matching code as escaping the '\0' terminating the
   string.
   
   The code is slightly different because dash converts the CTLESC characters
   to backslashes and removes all the other CTL* characters to allow
   substituting glob(3).
   
   The effect of the bug was also slightly different from dash (where nothing
   matched at all). Because a CTLESC can escape a '\0' in some way, whether
   files were included despite the bug depended on memory that should not be
   read. In particular, on many machines /*\/ expanded to a strict subset of
   what /*/ expanded to.
   
   Example:
     echo /*"/null"
   
   This should print /dev/null, not /*/null.
   
   PR:		bin/146378
   Obtained from:	dash
 
 Added:
   stable/7/tools/regression/bin/sh/expansion/
   stable/7/tools/regression/bin/sh/expansion/pathname2.0
      - copied unchanged from r207944, head/tools/regression/bin/sh/expansion/pathname2.0
 Modified:
   stable/7/bin/sh/expand.c
 Directory Properties:
   stable/7/bin/sh/   (props changed)
   stable/7/tools/regression/bin/sh/   (props changed)
 
 Modified: stable/7/bin/sh/expand.c
 ==============================================================================
 --- stable/7/bin/sh/expand.c	Sat Jul  3 21:02:11 2010	(r209674)
 +++ stable/7/bin/sh/expand.c	Sat Jul  3 21:06:48 2010	(r209675)
 @@ -1,6 +1,8 @@
  /*-
   * Copyright (c) 1991, 1993
   *	The Regents of the University of California.  All rights reserved.
 + * Copyright (c) 1997-2005
 + *	Herbert Xu <herbert@gondor.apana.org.au>.  All rights reserved.
   *
   * This code is derived from software contributed to Berkeley by
   * Kenneth Almquist.
 @@ -1122,10 +1124,11 @@ expmeta(char *enddir, char *name)
  	struct dirent *dp;
  	int atend;
  	int matchdot;
 +	int esc;
  
  	metaflag = 0;
  	start = name;
 -	for (p = name ; ; p++) {
 +	for (p = name; esc = 0, *p; p += esc + 1) {
  		if (*p == '*' || *p == '?')
  			metaflag = 1;
  		else if (*p == '[') {
 @@ -1150,12 +1153,14 @@ expmeta(char *enddir, char *name)
  			break;
  		else if (*p == CTLQUOTEMARK)
  			continue;
 -		else if (*p == CTLESC)
 -			p++;
 -		if (*p == '/') {
 -			if (metaflag)
 -				break;
 -			start = p + 1;
 +		else {
 +			if (*p == CTLESC)
 +				esc++;
 +			if (p[esc] == '/') {
 +				if (metaflag)
 +					break;
 +				start = p + esc + 1;
 +			}
  		}
  	}
  	if (metaflag == 0) {	/* we've reached the end of the file name */
 @@ -1201,7 +1206,8 @@ expmeta(char *enddir, char *name)
  		atend = 1;
  	} else {
  		atend = 0;
 -		*endname++ = '\0';
 +		*endname = '\0';
 +		endname += esc + 1;
  	}
  	matchdot = 0;
  	p = start;
 @@ -1229,7 +1235,7 @@ expmeta(char *enddir, char *name)
  	}
  	closedir(dirp);
  	if (! atend)
 -		endname[-1] = '/';
 +		endname[-esc - 1] = esc ? CTLESC : '/';
  }
  
  
 
 Copied: stable/7/tools/regression/bin/sh/expansion/pathname2.0 (from r207944, head/tools/regression/bin/sh/expansion/pathname2.0)
 ==============================================================================
 --- /dev/null	00:00:00 1970	(empty, because file is newly added)
 +++ stable/7/tools/regression/bin/sh/expansion/pathname2.0	Sat Jul  3 21:06:48 2010	(r209675, copy of r207944, head/tools/regression/bin/sh/expansion/pathname2.0)
 @@ -0,0 +1,31 @@
 +# $FreeBSD$
 +
 +failures=0
 +
 +check() {
 +	testcase=$1
 +	expect=$2
 +	eval "set -- $testcase"
 +	actual="$*"
 +	if [ "$actual" != "$expect" ]; then
 +		failures=$((failures+1))
 +		printf '%s\n' "For $testcase, expected $expect actual $actual"
 +	fi
 +}
 +
 +set -e
 +T=$(mktemp -d ${TMPDIR:-/tmp}/sh-test.XXXXXX)
 +trap 'rm -rf $T' 0
 +cd -P $T
 +
 +mkdir testdir testdir2 'testdir/*' 'testdir/?' testdir/a testdir/b testdir2/b
 +mkdir testdir2/.c
 +touch testf 'testdir/*/1' 'testdir/?/1' testdir/a/1 testdir/b/1 testdir2/b/.a
 +
 +check '*\/' 'testdir/ testdir2/'
 +check '"testdir/"*"/1"' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +check '"testdir/"*"/"*' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +check '"testdir/"*\/*' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +check '"testdir"*"/"*"/"*' 'testdir/*/1 testdir/?/1 testdir/a/1 testdir/b/1'
 +
 +exit $((failures != 0))
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: jilles 
State-Changed-When: Sat Jul 3 21:15:09 UTC 2010 
State-Changed-Why:  
Fixed in 9.x/8.x/7.x. Thanks for your report. (By the way, that it 
worked for you on 6.1 was just luck, and NetBSD's sh seems to have the 
bug as well. OpenBSD has a very different sh.) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=146378 
>Unformatted:
