From dima@burka.rdy.com  Fri May  8 14:58:57 1998
Received: from burka.rdy.com (dima@burka.rdy.com [205.149.163.30])
          by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id OAA23122
          for <FreeBSD-gnats-submit@freebsd.org>; Fri, 8 May 1998 14:58:55 -0700 (PDT)
          (envelope-from dima@burka.rdy.com)
Received: by burka.rdy.com id OAA01692;
  (8.8.8/RDY) Fri, 8 May 1998 14:58:55 -0700 (PDT)
Message-Id: <199805082158.OAA01692@burka.rdy.com>
Date: Fri, 8 May 1998 14:58:55 -0700 (PDT)
From: dima@best.net
Reply-To: dima@best.net
To: FreeBSD-gnats-submit@freebsd.org
Subject: /bin/sh && IFS
X-Send-Pr-Version: 3.2

>Number:         6557
>Category:       bin
>Synopsis:       /bin/sh is broken
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    cracauer
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri May  8 15:00:01 PDT 1998
>Closed-Date:    Wed Aug 16 12:50:56 MEST 2000
>Last-Modified:  Wed Aug 16 12:51:24 MEST 2000
>Originator:     Dima Ruban
>Release:        FreeBSD 2.2.6-STABLE i386
>Organization:
Best Internet Communications, Inc.
>Environment:

	Doesn't really matter. Any xBSD system.

>Description:

	/bin/sh doesn't work as it supposed to when dealing with $IFS
	variable.

>How-To-Repeat:

	Try to run this script:

	#!/bin/sh
	XXX=/1:/2:/3:
	IFS=:
	echo /0:$XXX

	The expected result should be: "/0 /1 /2 /3"
	However, with current /bin/sh it's: "/0:/1 /2 /3"

>Fix:
	
	Don't have one.

>Release-Note:
>Audit-Trail:

From: Jin Guojun (ITG staff) <jin@george.lbl.gov>
To: FreeBSD-gnats-submit@FreeBSD.ORG, dima@best.net
Cc:  Subject: Re: bin/6557: /bin/sh && IFS
Date: Fri, 8 May 1998 16:38:38 -0700 (PDT)

 >
 >>How-To-Repeat:
 >
 >        Try to run this script:
 >
 >        #!/bin/sh
 >        XXX=/1:/2:/3:
 >        IFS=:
 >        echo /0:$XXX
 >
 >        The expected result should be: "/0 /1 /2 /3"
 >        However, with current /bin/sh it's: "/0:/1 /2 /3"
 >
 >>Fix:
 >        
 >        Don't have one.
 
 It seems only IFS on variable, but no the argument. Need be fixed.
 
 This my temporary working around:
 
 #!/bin/sh
 XXX=/1:/2:/3:
 IFS=:
 tmp="/0:${XXX}"
 echo    $tmp
 
 
 This work-around solves many sh problems over different O.S.
 
 	-Jin
 

From: dima@best.net (Dima Ruban)
To: jin@george.lbl.gov (Jin Guojun)
Cc: FreeBSD-gnats-submit@FreeBSD.ORG, dima@best.net
Subject: Re: bin/6557: /bin/sh && IFS
Date: Fri, 8 May 1998 16:46:06 -0700 (PDT)

 Jin Guojun writes:
 > This my temporary working around:
 > 
 > #!/bin/sh
 > XXX=/1:/2:/3:
 > IFS=:
 > tmp="/0:${XXX}"
 > echo    $tmp
 > 
 > 
 > This work-around solves many sh problems over different O.S.
 
 It's not about work-around. For example, this problem breaks
 GNU configure, if it uses something like this:
 
 AC_PATH_PROG(PROG, prog, /usr/bin/prog, $PROGDIR/bin:$PATH)
 
 So it really needs to be fixed.
 
 (As you can imagine, if breaks number of ports)
 
 
 > 
 > 	-Jin
 > 
 
 -- dima

From: Martin Cracauer <cracauer@cons.org>
To: dima@best.net, FreeBSD-gnats-submit@FreeBSD.ORG
Cc:  Subject: Re: bin/6557: /bin/sh && IFS
Date: Tue, 12 May 1998 17:22:13 +0200

 In <199805082158.OAA01692@burka.rdy.com>, dima@best.net wrote: 
 > 	/bin/sh doesn't work as it supposed to when dealing with $IFS
 > 	variable.
 > 
 > >How-To-Repeat:
 > 
 > 	Try to run this script:
 > 
 > 	#!/bin/sh
 > 	XXX=/1:/2:/3:
 > 	IFS=:
 > 	echo /0:$XXX
 > 
 > 	The expected result should be: "/0 /1 /2 /3"
 > 	However, with current /bin/sh it's: "/0:/1 /2 /3"
 
 This has been discussed before, see
 http://www.freebsd.org/cgi/query-pr.cgi?pr=5263. My reply to this PR
 has a fix in it, which fixes both your case and the GNU autoconf case.
 
 However, it is my current opinion that my fix and bash are
 wrong. Hence I didn't commit it although you are invited to see if it
 fits your needs, it's in the url given above.
 
 I still don't have a POSIX reference, but I've been told that POSIX
 specifies that word splitting is to be done during evaluation, not
 during parsing. I think it's obvious that IFS split is part of word
 splitting.
 
 The question at hand is: How does IFS splitting fit into the picture?
 The value of $IFS is obviously a result of evaluation, but a result
 that affects parsing, at least this is what bash tries to do. This
 sounds very counter-intuitive, how can you define clean sematics for
 changing the lexical structure while parsing?
 
 Consider this:
 
   #! /bin/sh
   IFS="   :"
   var="bla:fasel:blubb"
   echo foo:bla:fasel:blubb
   echo foo:$var
 
 bash:
   foo:bla:fasel:blubb
   foo bla fasel blubb
 
 FreeBSD's unpatched sh and pdksh:
   foo:bla:fasel:blubb
   foo:bla fasel blubb
 
 My patched FreeBSD sh:
   foo bla fasel blubb
   foo bla fasel blubb
 
 zsh (doesn't really count):
   foo:bla:fasel:blubb
   foo:bla:fasel:blubb
 
 
 What bash and my fix do is that they treat IFS as something that is
 used "as soon as possible" (while in my fix it is possible even
 sooner as in bash). 
 
 My current conclusion is that this is wrong, that IFS *must* *not*
 affect the basic (top-level script file) paring, but that IFS
 splitting can happen when the parsed script again begins to parse
 something.
 
 Examples for such later parsing:
 - The contents of a string are parsed as in
     var="bla:fasel:blubb"
 - Some run-time input to the script.
 
 To make my point clearer: If bash is right in using IFS during the
 same parse pass, why doesn't it take it as far as my patch? How can
 you decide which of both is correct?
 
 For now, the only clean solution is not to use IFS for the top-level
 parse, hence I think FreeBSD's sh (ash) and pdksh are right and
 autoconf needs to be fixed (given that the fix is always a one-liner
 that can't be so difficult). 
 
 Suggestions and comments welcome!
 
 Martin
 -- 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 Martin Cracauer <cracauer@cons.org> http://www.cons.org/cracauer
 BSD User Group Hamburg, Germany     http://www.bsdhh.org/

From: Tor Egge <Tor.Egge@idi.ntnu.no>
To: dima@best.net
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: bin/6557: /bin/sh && IFS
Date: Tue, 12 May 1998 20:46:16 +0200

 > 	Try to run this script:
 > 
 > 	#!/bin/sh
 > 	XXX=/1:/2:/3:
 > 	IFS=:
 > 	echo /0:$XXX
 > 
 > 	The expected result should be: "/0 /1 /2 /3"
 > 	However, with current /bin/sh it's: "/0:/1 /2 /3"
 
 According to "X/Open Commands and Utilities Issue 4, Version 2"
 Section 2.6 "Word Expansion", field splitting is performed on the
 portions of the fields generated by tilde expansion, parameter
 expansion, command substitution and arithmetic expansion.
 
 i.e. your
                 echo /0:$XXX
 
 should give
 
         Before Word Expansion:          /0:$XXX 
         after tilde expansion, 
         parameter expansiion,
         command substitition and
         arithmetic expansion:           /0:/1:/2:/3:
                                            ^^^^^^^^^
                                            This is the portion of the field
                                            being the result of expansion. Any
                                            field splitting should only be
                                            performed on this portion.
 
         after field splitting:          /0:/1 /2 /3 
         after pathname expansion:       /0:/1 /2 /3 
         after quote removal:            /0:/1 /2 /3 
 
                 
 Correct field splitting results in 4 fields.
 
         `/0:/1'
         `/2'
         `3'
         `'
 
 The last (empty) field should not be removed, since it is not the complete
 expansion appropriate for a word (the word being `/0:$XXX' (without quotes)).
 
 ksh93 gives almost the same result as our /bin/sh (for this example).
 Unfortunately, it incorrectly removes the last field.
 
 bash incorrectly performs field splitting on portions of the fields that
 are not the result of expansion or substitution.
 
 bash also incorrectly removes the last field.
 
 - Tor Egge

From: woods@zeus.leitch.com (Greg A. Woods)
To: FreeBSD-gnats-submit@freebsd.org
Cc:  Subject: Re: bin/6557: /bin/sh && IFS
Date: Tue, 12 May 1998 15:51:52 -0400 (EDT)

 >        #!/bin/sh
 >        XXX=/1:/2:/3:
 >        IFS=:
 >        echo /0:$XXX
 >
 >        The expected result should be: "/0 /1 /2 /3"
 >        However, with current /bin/sh it's: "/0:/1 /2 /3"
 
 Oddly enough this same problem exists in PD KSH v5.2.13 97/10/27.
 
 This problem does not exist in any SysV derived Bourne shell, including
 the SunOS-{4,5} ones, nor in any Korn shell that I tried.
 
 I'm fairly sure POSIX.2 requires the SysV behaviour (though I haven't
 bothered to check).
 
 -- 
 							Greg A. Woods
 
 +1 416 443-1734      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

From: woods@zeus.leitch.com (Greg A. Woods)
To: Martin Cracauer <cracauer@cons.org>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/6557: /bin/sh && IFS
Date: Tue, 12 May 1998 16:07:30 -0400 (EDT)

 [ On Tue, May 12, 1998 at 08:20:01 (-0700), Martin Cracauer wrote: ]
 > Subject: Re: bin/6557: /bin/sh && IFS
 >
 >  However, it is my current opinion that my fix and bash are
 >  wrong. Hence I didn't commit it although you are invited to see if it
 >  fits your needs, it's in the url given above.
 
 It would seem Bash agrees with the real Korn Shell and the real Bourne
 Shell.  I agree with them too.
 
 >  I still don't have a POSIX reference, but I've been told that POSIX
 >  specifies that word splitting is to be done during evaluation, not
 >  during parsing. I think it's obvious that IFS split is part of word
 >  splitting.
 
 This is the relavent section from POSIX 1003.2 Draft 11.2:
 
 (the Open Group's "Single UNIX Specification" says essentially the same
 things too...)
 
 
                Copyright c 1991 IEEE.  All rights reserved.
       This is an unapproved IEEE Standards Draft, subject to change.
 
  Part 2: SHELL AND UTILITIES                                 P1003.2/D11.2
 
 . . .
 
  3.6.5  Field Splitting
 
  After parameter expansion (3.6.2), command substitution (3.6.3), and
  arithmetic expansion (3.6.4) the shell shall scan the results of
  expansions and substitutions that did not occur in double-quotes for
  field splitting and multiple fields can result.
 
  The shell shall treat each character of the IFS as a delimiter and use
  the delimiters to split the results of parameter expansion and command
  substitution into fields.
 
      (1)  If the value of IFS is <space>, <tab>, and <newline>, or if it
           is unset, any sequence of <space>, <tab>, or <newline>
           characters at the beginning or end of the input shall be ignored
           and any sequence of those characters within the input shall
           delimit a field.  (For example, the input
 
                 <newline><space><tab>foo<tab><tab>bar<space>
 
           yields two fields, foo and bar).
 
      (2)  If the value of IFS is null, no field splitting shall be
           performed.
 
      (3)  Otherwise, the following rules shall be applied in sequence.      1
           The term ``IFS white space'' is used to mean any sequence (zero   1
           or more instances) of white-space characters that are in the IFS  1
           value (e.g., if IFS contains <space><comma><tab>, any sequence    1
           of <space> and <tab> characters is considered IFS white space).   1
 
            (a)  IFS white space shall be ignored at the beginning and end   1
                 of the input.                                               1
 
            (b)  Each occurrence in the input of an IFS character that is    1
                 not IFS white space, along with any adjacent IFS white      1
                 space, shall delimit a field, as described previously.      1
 
            (c)  Nonzero-length IFS white space shall delimit a field.       1
 
  BEGIN_RATIONALE
 
  3.6.5.1  Field Splitting Rationale. (This subclause is not a part of
           P1003.2)
 
  The operation of field splitting using IFS as described in earlier drafts
  was based on the way the KornShell splits words, but is incompatible with
  other common versions of the shell.  However, each has merit, and so a
  decision was made to allow both.  If the IFS variable is unset, or is
  <space><tab><newline>, the operation is equivalent to the way the
  System V shell splits words.  Using characters outside the
  <space><tab><newline> set yields the KornShell behavior, where each of
  the non-<space><tab><newline> characters is significant.  This behavior,
  which affords the most flexibility, was taken from the way the original
  awk handled field splitting.
 
  The (3) rule can be summarized as a pseudo ERE:                            1
 
        (s*ns*|s+)                                                           1
 
  where s is an IFS white-space character and n is a character in the IFS    1
  that is not white space.  Any string matching that ERE delimits a field,   1
  except that the s+ form does not delimit fields at the beginning or the    1
  end of a line.  For example, if IFS is <space><comma>, the string          1
 
        <space><space>red<space><space>,<space>white<space>blue              1
 
  yields the three colors as the delimited fields.                           1
 
  END_RATIONALE
  
 
 -- 
 							Greg A. Woods
 
 +1 416 443-1734      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

From: woods@zeus.leitch.com (Greg A. Woods)
To: Tor Egge <Tor.Egge@idi.ntnu.no>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/6557: /bin/sh && IFS
Date: Tue, 12 May 1998 17:45:45 -0400 (EDT)

 [ On Tue, May 12, 1998 at 11:50:00 (-0700), Tor Egge wrote: ]
 > Subject: Re: bin/6557: /bin/sh && IFS
 >
 >  According to "X/Open Commands and Utilities Issue 4, Version 2"
 >  Section 2.6 "Word Expansion", field splitting is performed on the
 >  portions of the fields generated by tilde expansion, parameter
 >  expansion, command substitution and arithmetic expansion.
 
 That is much clearer language than in the POSIX 1003.2 draft.  In the
 posix language one could infer that the entire command line is the
 "result of expanding 
 
 I whould also note that the Korn shell I tested on SunOS-5 is "Version
 11/16/88i".
 
 However I don't think the XPG 4.2 wording makes any sense at all, nor do
 I agree that the behaviour of the current shell is "correct".
 
 Such field splitting is essentially useless since it requires silly work
 arounds that add unnecessary code, as revealed by the bugs it causes
 with GNU Autoconf generated scripts.  It is possible that there's
 something I'm missing here and that the current behaviour actually fills
 some imporant niche, but I don't see it just now.
 
 -- 
 							Greg A. Woods
 
 +1 416 443-1734      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

From: Martin Cracauer <cracauer@cons.org>
To: "Greg A. Woods" <woods@zeus.leitch.com>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/6557: /bin/sh && IFS
Date: Wed, 13 May 1998 10:57:42 +0200

 In <199805122007.QAA03553@brain.zeus.leitch.com>, Greg A. Woods wrote: 
 > [ On Tue, May 12, 1998 at 08:20:01 (-0700), Martin Cracauer wrote: ]
 > > Subject: Re: bin/6557: /bin/sh && IFS
 > >
 > >  However, it is my current opinion that my fix and bash are
 > >  wrong. Hence I didn't commit it although you are invited to see if it
 > >  fits your needs, it's in the url given above.
 > 
 > It would seem Bash agrees with the real Korn Shell and the real Bourne
 > Shell.  I agree with them too.
 
 Hm, Solaris' ksh and sh don't agree completely (Solaris 2.6/SPARC):
 
 #! /bin/sh
 IFS="   :"
 var="bla:fasel:blubb:"
 for i in foo:bla:fasel:blubb: ; do
     echo val: "'"$i"'"
 done
 echo
 for i in foo:$var ; do
     echo val: "'"$i"'"
 done
 
 
 ~(u2)6% ksh l2
 val: 'foo bla fasel blubb '
 
 val: 'foo'
 val: 'bla'
 val: 'fasel'
 val: 'blubb'
 
 ~(u2)7% sh l2
 val 'foo'
 val 'bla'
 val 'fasel'
 val 'blubb'
 
 val 'foo'
 val 'bla'
 val 'fasel'
 val 'blubb'
 
 But at least they agree that the `for` parameters are subject to IFS
 splitting.  
  
 > >  I still don't have a POSIX reference, but I've been told that POSIX
 > >  specifies that word splitting is to be done during evaluation, not
 > >  during parsing. I think it's obvious that IFS split is part of word
 > >  splitting.
 > 
 > This is the relavent section from POSIX 1003.2 Draft 11.2:
 
 Thanks!
 
 [...] 
 >  3.6.5  Field Splitting
 > 
 >  After parameter expansion (3.6.2), command substitution (3.6.3), and
 >  arithmetic expansion (3.6.4) the shell shall scan the results of
 >  expansions and substitutions that did not occur in double-quotes for
 >  field splitting and multiple fields can result.
 
 Hm, so what are the arguments to `for` (or to any command)?
 
 As far as I can tell, they are
 - not parameter expansion
 - not command substitution
 - not arithmetic expansion
 
 The paragraph above says that only results of these expansions and
 substitutions are subject to field splitting. What kind of
 substitution or expandsion are command arguments a result of?
 
 >  The shell shall treat each character of the IFS as a delimiter and use
 >  the delimiters to split the results of parameter expansion and command
 >  substitution into fields.
 > 
 >      (1)  If the value of IFS is <space>, <tab>, and <newline>, or if it
 >           is unset, any sequence of <space>, <tab>, or <newline>
 >           characters at the beginning or end of the input shall be ignored
 >           and any sequence of those characters within the input shall
 >           delimit a field.  (For example, the input
 > 
 >                 <newline><space><tab>foo<tab><tab>bar<space>
 > 
 >           yields two fields, foo and bar).
 > 
 >      (2)  If the value of IFS is null, no field splitting shall be
 >           performed.
 > 
 >      (3)  Otherwise, the following rules shall be applied in sequence.      1
 >           The term ``IFS white space'' is used to mean any sequence (zero   1
 >           or more instances) of white-space characters that are in the IFS  1
 >           value (e.g., if IFS contains <space><comma><tab>, any sequence    1
 >           of <space> and <tab> characters is considered IFS white space).   1
 > 
 >            (a)  IFS white space shall be ignored at the beginning and end   1
 >                 of the input.                                               1
 > 
 >            (b)  Each occurrence in the input of an IFS character that is    1
 >                 not IFS white space, along with any adjacent IFS white      1
 >                 space, shall delimit a field, as described previously.      1
 > 
 >            (c)  Nonzero-length IFS white space shall delimit a field.       1
 
 The test shell script I gave above contains an interesting matter
 pointed out to me in private email: What happens if a non-white IFS
 value is at the end of something that is subject to field
 splitting? pdksh generates an extra field with an empty value, which
 is IHMO required by these section.
 
 [...]
 
 Martin
 -- 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 Martin Cracauer <cracauer@cons.org> http://www.cons.org/cracauer
   cracauer@wavehh.hanse.de (batched, preferred for large mails)
   Tel.: (private) +4940 5221829 Fax.: (private) +4940 5228536
   Paper: (private) Waldstrasse 200, 22846 Norderstedt, Germany

From: woods@zeus.leitch.com (Greg A. Woods)
To: FreeBSD-gnats-submit@freebsd.org
Cc:  Subject: Re: bin/6557: /bin/sh && IFS
Date: Thu, 14 May 1998 14:00:38 -0400 (EDT)

 [ On Wed, May 13, 1998 at 02:00:02 (-0700), Martin Cracauer wrote: ]
 > Subject: Re: bin/6557: /bin/sh && IFS
 >
 >  Hm, Solaris' ksh and sh don't agree completely (Solaris 2.6/SPARC):
 
 Actually with your example the original Bourne Shell is the odd man
 out.  Ksh-88i, Ksh93, ash (both NetBSD & FreeBSD), and pdksh-5.2.13
 all behave similarly with your example (which is I think the one that
 gets right down to the meat of the problem).
 
 I've finally found the rationale in POSIX 1003.2 Draft 11.2 that talks
 about this, and it does seem to make a certain amount of sense, though
 it introduces strange magic that can lead to very unexpected results:
 
 
                Copyright c 1991 IEEE.  All rights reserved.
       This is an unapproved IEEE Standards Draft, subject to change.
 
  BEGIN_RATIONALE
 
  3.6.5.1  Field Splitting Rationale. (This subclause is not a part of
           P1003.2)
 
  The operation of field splitting using IFS as described in earlier drafts
  was based on the way the KornShell splits words, but is incompatible with
  other common versions of the shell.  However, each has merit, and so a
  decision was made to allow both.  If the IFS variable is unset, or is
  <space><tab><newline>, the operation is equivalent to the way the
  System V shell splits words.  Using characters outside the
  <space><tab><newline> set yields the KornShell behavior, where each of
  the non-<space><tab><newline> characters is significant.  This behavior,
  which affords the most flexibility, was taken from the way the original
  awk handled field splitting.
 
  The (3) rule can be summarized as a pseudo ERE:                            1
 
        (s*ns*|s+)                                                           1
 
  where s is an IFS white-space character and n is a character in the IFS    1
  that is not white space.  Any string matching that ERE delimits a field,   1
  except that the s+ form does not delimit fields at the beginning or the    1
  end of a line.  For example, if IFS is <space><comma>, the string          1
 
        <space><space>red<space><space>,<space>white<space>blue              1
 
  yields the three colors as the delimited fields.                           1
 
  END_RATIONALE                                                              1
 
 >  Hm, so what are the arguments to `for` (or to any command)?
 >  
 >  As far as I can tell, they are
 >  - not parameter expansion
 >  - not command substitution
 >  - not arithmetic expansion
 > 
 >  The paragraph above says that only results of these expansions and
 >  substitutions are subject to field splitting. What kind of
 >  substitution or expandsion are command arguments a result of?
 
 Command arguments are not a valid concept here at all.  A deep and dark
 alley full of many horrors awaits anyone trying to think of things in
 those terms.  Another section from P1003.2D11.2 may clear the fog (and
 also gives concrete reasons for siding with Korn on these mechanisms):
 
                Copyright c 1991 IEEE.  All rights reserved.
       This is an unapproved IEEE Standards Draft, subject to change.
 
  BEGIN_RATIONALE
 
  3.6.0.1  Word Expansions Rationale. (This subclause is not a part of
           P1003.2)
 
  IFS is used for performing field splitting on the results of parameter
  and command substitution; it is not used for splitting all fields.
  Previous versions of the shell used it for splitting all fields during
  field splitting, but this has severe problems because the shell can no
  longer parse its own script.  There are also important security
  implications caused by this behavior.  All useful applications of IFS use
  it for parsing input of the read utility and for splitting the results of
  parameter and command substitution.  New versions of the shell have fixed
  this bug, and POSIX.2 requires the corrected behavior.
 
  The rule concerning expansion to a single field requires that if foo=abc
  and bar=def, that
 
        "$foo""$bar"
 
  expands to the single field
 
        abcdef
 
  The rule concerning empty fields can be illustrated by:
 
        $ unset foo
        $ set $foo bar '' xyz "$foo" abc
        $ for i
        > do
        >       echo "-$i-"
        > done
        -bar-
        --
        -xyz-
        --
        -abc-
 
  Step (1) indicates that Tilde Expansion, Parameter Expansion, Command      1
  Substitution, and Arithmetic Expansion are all processed simultaneously
  as they are scanned.  For example, the following is valid arithmetic:
 
        x=1
        echo $(( $(echo 3)+$x ))
 
  An earlier draft stated that Tilde Expansion preceded the other steps,     1
  but this is not the case in known historical implementations; if it were,  1
  and a referenced home directory contained a $ character, expansions would  1
  result within the directory name.                                          1
 
  END_RATIONALE                                                              1
 
 If that didn't quite do it, then perhaps this will (the actual rules
 that appear before the above quoted rationale).  This next section
 also answers your last question about the empty field (i.e. pdksh is
 wrong):
 
                Copyright c 1991 IEEE.  All rights reserved.
       This is an unapproved IEEE Standards Draft, subject to change.
 
  3.6  Word Expansions
 
  This clause describes the various expansions that are performed on words.
  Not all expansions are performed on every word, as explained in the
  following subclauses.
 
  Tilde expansions, parameter expansions, command substitutions, arithmetic
  expansions, and quote removals that occur within a single word expand to
  a single field.  It is only field splitting or pathname expansion that
  can create multiple fields from a single word.  The single exception to
  this rule is the expansion of the special parameter @ within double-
  quotes, as is described in 3.5.2.
 
  The order of word expansion shall be as follows:
 
      (1)  Tilde Expansion (see 3.6.1), Parameter Expansion (see 3.6.2),     1
           Command Substitution (see 3.6.3), and Arithmetic Expansion (see
           3.6.4) shall be performed, beginning to end.  [See item (5) in
           3.3.]
 
      (2)  Field Splitting (see 3.6.5) shall be performed on fields
           generated by step (1) unless IFS is null.
 
 [[NOTE: there's a minor inconsistency in the above vs. the rationale
 quoted first in this message, specifically the earlier rationale stated
 "If the IFS variable is unset, or is <space><tab><newline>, the
 operation is equivalent to the way the System V shell splits words."
 which would imply more magic happens than the above actual rule allows.
 Hopefully nobody's implmented the extra magic given in the rationale.]]
 
      (3)  Pathname Expansion (see 3.6.6) shall be performed, unless set -f
           is in effect.
 
      (4)  Quote Removal (see 3.6.7) shall always be performed last.
 
  The expansions described in this clause shall occur in the same shell
  environment as that in which the command is executed.
 
  If the complete expansion appropriate for a word results in an empty
  field, that empty field shall be deleted from the list of fields that
  form the completely expanded command, unless the original word contained   1
  single-quote or double-quote characters.                                   1
 
  The $ character is used to introduce parameter expansion, command
  substitution, or arithmetic evaluation.  If an unquoted $ is followed by
  a character that is either not numeric, the name of one of the special
  parameters (see 3.5.2), a valid first character of a variable name, a
  left curly brace ({), or a left parenthesis, the result is unspecified.
 
 -- 
 							Greg A. Woods
 
 +1 416 443-1734      VE3TCP      <gwoods@acm.org>      <robohack!woods>
 Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

From: Garance A Drosihn <drosih@rpi.edu>
To: freebsd-gnats-submit@freebsd.org, dima@best.net
Cc: Martin Cracauer <cracauer@cons.org>, woods@zeus.leitch.com (Greg A. Woods)
Subject: Re: bin/6557: /bin/sh is broken
Date: Thu, 10 Sep 1998 04:33:30 -0400

 So, what (if anything) was decided after all this discussion about
 IFS handling in /bin/sh?
 
 I'm wandering into this topic now because someone used autoconfig
 to setup a program they are working on, and they wanted me to try
 this out on "any OS's" I had.  I noticed that the configure script
 got some syntax errors under freebsd, traced those down, and wrote
 a minimal script that showed the problem (of course I did this
 *before* checking the gnats database, thus wasting time that I
 could have easily avoided wasting...).
 
 If I understand the discussion in bin/6557 correctly, assorted
 standards imply that the script I wrote (based on what autoconf
 did for my friend) is *not* supposed to work.  At the same time,
 the script I have does in fact work on NeXTSTEP 3.3, AIX 4.1,
 AIX 4.3, Solaris 2.5, Solaris 2.6, IRIX 6.2, IRIX 6.3, and
 whatever versions of linux this guy is running.  It also works
 if I use bash under FreeBSD (I didn't check bash2...).
 
 So, either freebsd could change to work like every other /bin/sh
 I curently have access to, or all those /bin/sh's will change to
 follow these "standards" that everyone is quoting from.  The first
 question would be:  What is the probability that either one of
 these two things will happen?   Is it greater than zero?
 
 This would probably be a much less important issue if it wasn't
 for autoconf.  There are at least two other ways that autoconf
 could do it's checking which would work for freebsd's current
 /bin/sh and which would also be guareenteed to work on every
 operating system that their current strategy works on.  What
 is the probability that we could convince the autoconf guys to
 change the tactic some of their tests use?  (I have no experience
 using autoconf itself, just the scripts that it generates).
 
 I realize we could fix autoconf for freebsd, but is there any
 way to fix it on freebsd such that we "auto correct" for programs
 which were autoconfig-ed on some other platform?  (such as this
 tar file I picked up from my friend on his linux box).
 
 Note that I'm not pursuing the question of "what is the correct
 behavior of IFS in shells", I'm just at the level of the more
 pedestrian question of how to coexist gracefully with autoconf'ed
 programs.  My friend took a real mess of code and cleaned it up
 to use autoconf, which was a very good thing.  It'd be nice if
 the result worked correctly on freebsd without tripping into
 syntax errors.
 
 Admittedly, I'm not in a good mood about this at the moment, but
 I'm not upset with "freebsd" so much as "unix written standards",
 which are apparently totally disconnected from "unix reality".
 As odd as it sounds, I'd be much happier if this was just a bug
 in freebsd that no one had gotten around to fixing yet.
 
 ---
 Garance Alistair Drosehn           =   gad@eclipse.its.rpi.edu
 Senior Systems Programmer          or  drosih@rpi.edu
 Rensselaer Polytechnic Institute

From: Martin Cracauer <cracauer@cons.org>
To: Garance A Drosihn <drosih@rpi.edu>, freebsd-gnats-submit@freebsd.org,
        dima@best.net
Cc: Martin Cracauer <cracauer@cons.org>,
        "Greg A. Woods" <woods@zeus.leitch.com>
Subject: Re: bin/6557: /bin/sh is broken
Date: Thu, 10 Sep 1998 12:15:18 +0200

 In <v04011703b21d335451a4@[128.113.24.47]>, Garance A Drosihn wrote: 
 > So, what (if anything) was decided after all this discussion about
 > IFS handling in /bin/sh?
 
 There's no "FreeBSD" decision, but I can state my impressions: 
 
 1) The issue is utterly complex. Regarding to Posix, the semantics of
 $IFS change with the contents, to support every IFS construction that
 was formerly i use on different shells. Urgs...
 
 2) The construct autoconf is falling over is very simple to replace
 with one extra variable assignment and FreeBSD's ash isn't the only
 one that fails to supports this construct, so I really think the
 autoconf people should change it. After all, the whole point of
 autoconf is to determine differences in systems, it shouldn't rely on
 non-portable behaviour, at least not when not neccessary.
 
 3) I just ordered my Posix documents (expensive!), until they arrive,
 I can't really act.
 
 Said that, Thor Egge commited some syntax fixes to sh a few days
 ago. He did not change the basic IFS behaviour, but fixed some real
 bugs related to list handling. It is possible that the code you have
 problems with is affected by this. Could you try your example with the
 newest sh from -current and give us feedback (or post the code)?
 
 Martin
 -- 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 Martin Cracauer <cracauer@cons.org> http://www.cons.org/cracauer
   Tel.: (private) +4940 5221829 Fax.: (private) +4940 5228536
   Paper: (private) Waldstrasse 200, 22846 Norderstedt, Germany

From: Garance A Drosihn <drosih@rpi.edu>
To: Martin Cracauer <cracauer@cons.org>, freebsd-gnats-submit@freebsd.org,
        dima@best.net
Cc: Martin Cracauer <cracauer@cons.org>,
        "Greg A. Woods" <woods@zeus.leitch.com>
Subject: Re: bin/6557: /bin/sh is broken
Date: Thu, 10 Sep 1998 14:48:17 -0400

 At 12:15 PM +0200 9/10/98, Martin Cracauer wrote:
 > 2) The construct autoconf is falling over is very simple to
 >    replace with one extra variable assignment
 
 It could also be fixed if autoconf was changed from doing:
    for ckpath in $PATH:dir1:dir1 ; do
 to
    for ckpath in $PATH dir1 dir2 ; do
 
 My question is, has anyone brought this up with the autoconf
 guys?  It is inconceivable that the above change would cause
 any script on any platform to break, and it is no additional
 overhead.  Should I point that out to the autoconf guys, or
 have other people already talked to them only to find out
 that they not interested?
 
 I have no strong feelings on how IFS itself is processed, or
 how it should be processed.  I'd just like autoconf scripts
 to work...
 
 > Said that, Thor Egge commited some syntax fixes to sh a few
 > days ago. He did not change the basic IFS behaviour, but
 > fixed some real bugs related to list handling. It is possible
 > that the code you have problems with is affected by this.
 
 I doubt it, but at least this gives me a chance to point
 people at the bug-reproducing script that I wrote last
 night (*before* checking gnats, sigh...).  My only freebsd-
 current system is in a "production mode" status now, so I
 wouldn't want to be making changes on that.  However, the
 following script will demonstrate the bug:
 
    ftp://eclipse.its.rpi.edu/pub/fbsd/bugs/ifs_bug.sh
 
 the script works on all unix's I have access to, except
 for freebsd-stable and freebsd-current.  It's a minimal
 version of what autoconf likes to do, so if this script
 worked then the autofconf issues would be gone.  All you
 have to do is run it (on any unix platform), and the
 output should be pretty self-explanatory (although you
 would want to read the script first, just on general
 principles :-)
 
 
 ---
 Garance Alistair Drosehn           =   gad@eclipse.its.rpi.edu
 Senior Systems Programmer          or  drosih@rpi.edu
 Rensselaer Polytechnic Institute
Responsible-Changed-From-To: freebsd-bugs->cracauer 
Responsible-Changed-By: nbm 
Responsible-Changed-When: Thu Jul 13 04:04:02 PDT 2000 
Responsible-Changed-Why:  
Martin can decide what to do with this. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=6557 
State-Changed-From-To: open->closed 
State-Changed-By: cracauer 
State-Changed-When: Wed Aug 16 12:50:56 MEST 2000 
State-Changed-Why:  
Same as for bin/5263: 

Field splitting does not happen on simple command lines, only on the 
portions of the fields generated by tilde expansion, parameter 
expansion, command substitution and arithmetic expansion, as explained 
by Tor in the discussion. 

bash and pdksh now work the same way as our sh. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=6557 
>Unformatted:
