From rfg@tristatelogic.com  Fri Nov 12 01:02:12 2010
Return-Path: <rfg@tristatelogic.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 262311065674
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 12 Nov 2010 01:02:12 +0000 (UTC)
	(envelope-from rfg@tristatelogic.com)
Received: from outgoing.tristatelogic.com (segfault.tristatelogic.com [69.62.255.118])
	by mx1.freebsd.org (Postfix) with ESMTP id 02EC38FC13
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 12 Nov 2010 01:02:11 +0000 (UTC)
Received: by segfault.tristatelogic.com (Postfix, from userid 1237)
	id 87641BDC46; Thu, 11 Nov 2010 17:02:10 -0800 (PST)
Message-Id: <20101112010210.87641BDC46@segfault.tristatelogic.com>
Date: Thu, 11 Nov 2010 17:02:10 -0800 (PST)
From: Ronald F.Guilmette <rfg@tristatelogic.com>
Reply-To: Ronald F.Guilmette <rfg@tristatelogic.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc: rfg@tristatelogic.com
Subject: script(1) -k malfunctions with certain shells (e.g. tcsh, bash, zsh)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         152154
>Category:       bin
>Synopsis:       script(1) -k malfunctions with certain shells (e.g. tcsh, bash, zsh)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Nov 12 01:10:09 UTC 2010
>Closed-Date:    
>Last-Modified:  Wed Nov 17 21:43:23 UTC 2010
>Originator:     Ronald F. Guilmette
>Release:        FreeBSD 7.0-RELEASE i386
>Organization:
Infinite Monkeys & Co.
>Environment:

System: FreeBSD segfault.tristatelogic.com 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Tue Aug 5 02:38:40 PDT 2008 root@segfault.monkeys.com:/usr/src/sys/i386/compile/rfg20080805 i386

>Description:

Apparently, /bin/csh (aka /bin/tcsh) is diddling termios flags, in particular
the ECHO flag, for no apparently good reason and without ever even having been
asked to do so.  The result is that the /usr/bin/script program, when invoked
with the -k option, will log every input character the user types in TWICE,
but only if the user has SHELL set in the environment to either /bin/csh or
to /bin/tcsh.  This behavior does not seem to occur when and if SHELL is set
to /bin/sh.

>How-To-Repeat:

setenv SHELL /bin/tcsh
/usr/bin/script -k

Then just type something... anything... and then exit the shell and
then look at the typescript file to see the doubled characters.

I further traced this problem down by inserting the following changes into
/usr/bin/script.  These show what the termios flags look like before and
after the first call to select() within script.c, and thus they will show
how once tcsh is fired up (as a child process) by script, it changes the
termios flags on its pty, unsetting MANY of the bitflags, including ECHO.

Note that if you run /usr/bin/script with these changes installed, and do
so with SHELL set to /bin/sh then you will see that in that case, the
termios bit flags DO NOT get diddled by the child shell process, and that
the ECHO bitflag, in particular, does not get un-set, and thus, "script -k"
works properly in that case.

*** script.c.orig	2004-02-15 09:30:13.000000000 -0800
--- script.c	2010-11-11 16:41:15.000000000 -0800
***************
*** 92,95 ****
--- 92,96 ----
  	fd_set rfd;
  	int flushtime = 30;
+ 	int first_time = 1;
  
  	aflg = kflg = 0;
***************
*** 174,178 ****
--- 175,184 ----
  			tv.tv_usec = 0;
  		}
+ 		if (first_time & tcgetattr(master, &stt) >= 0)
+ 			fprintf (stderr, "Pre-select: 0x%08x\r\n", stt.c_lflag);
  		n = select(master + 1, &rfd, 0, 0, tvp);
+ 		if (first_time & tcgetattr(master, &stt) >= 0)
+ 			fprintf (stderr, "Post-select: 0x%08x\r\n", stt.c_lflag);
+ 		first_time = 0;
  		if (n < 0 && errno != EINTR)
  			break;

>Fix:

I have no idea how to fix this.  I tried to dredge into the tcsh source code,
but it is just too imposing for me right now.  So I'm sort-of hoping that
somebody more familiarity with tcsh than me will dredge into it and find the
problem.

P.S.  Strangely, when you run /usr/bin/script with SHELL set to /bin/tcsh
and when you then run a program from the tcsh command prompt, that program
will see a properly set group of termios bitflags, e.g. when and if it
calls tcgetattr().
>Release-Note:
>Audit-Trail:

From: Bruce Evans <brde@optusnet.com.au>
To: "Ronald F.Guilmette" <rfg@tristatelogic.com>
Cc: FreeBSD-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org
Subject: Re: bin/152154: /bin/csh & /bin/tcsh improperly diddle termios flags
Date: Fri, 12 Nov 2010 20:34:52 +1100 (EST)

 On Thu, 11 Nov 2010, Ronald F.Guilmette wrote:
 
 >> Description:
 >
 > Apparently, /bin/csh (aka /bin/tcsh) is diddling termios flags, in particular
 > the ECHO flag, for no apparently good reason and without ever even having been
 > asked to do so.  The result is that the /usr/bin/script program, when invoked
 > with the -k option, will log every input character the user types in TWICE,
 > but only if the user has SHELL set in the environment to either /bin/csh or
 > to /bin/tcsh.  This behavior does not seem to occur when and if SHELL is set
 > to /bin/sh.
 
 /bin/sh now does this too.  bash does this too (in at least versions 1.14.7
 and 4.0.35).  I think the mode switch in the shell is an unavoidable
 consequence of line editing in the shell, and script is depending on the
 line editing being done in the kernel.  The old versions of /bin/sh that
 don't have the problem differ in not starting up in line editing mode.
 
 > P.S.  Strangely, when you run /usr/bin/script with SHELL set to /bin/tcsh
 > and when you then run a program from the tcsh command prompt, that program
 > will see a properly set group of termios bitflags, e.g. when and if it
 > calls tcgetattr().
 
 This also happens with at least bash, since bash is careful to virtualize
 the terminal state by context switching it to a different value only
 while in its line editor.
 
 I don't understand why clearing ECHO in line editing mode doubles the
 script -k output instead of suppressing it.
 
 Bruce

From: Bruce Evans <brde@optusnet.com.au>
To: Bruce Evans <brde@optusnet.com.au>
Cc: "Ronald F.Guilmette" <rfg@tristatelogic.com>, freebsd-bugs@FreeBSD.org,
        FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: bin/152154: /bin/csh & /bin/tcsh improperly diddle termios flags
Date: Fri, 12 Nov 2010 20:48:08 +1100 (EST)

 On Fri, 12 Nov 2010, Bruce Evans wrote:
 
 > On Thu, 11 Nov 2010, Ronald F.Guilmette wrote:
 >
 >>> Description:
 >> 
 >> Apparently, /bin/csh (aka /bin/tcsh) is diddling termios flags, in 
 >> particular
 >> the ECHO flag, for no apparently good reason and without ever even having
 
 Perhaps I shouldn't have replied to this, since my ISP was blacklisted.
 
 >> ...
 >> P.S.  Strangely, when you run /usr/bin/script with SHELL set to /bin/tcsh
 >> and when you then run a program from the tcsh command prompt, that program
 >> will see a properly set group of termios bitflags, e.g. when and if it
 >> calls tcgetattr().
 >
 > This also happens with at least bash, since bash is careful to virtualize
 > the terminal state by context switching it to a different value only
 > while in its line editor.
 > ...
 
 PS: probably starting the subshell in non-line-editing-mode would work, but
 this is no good in general since you want the subshell to run in normal
 mode.  I don't know how to force non-line-editing-mode, especially using
 $SHELL.  script even starts the shell with -i, which tends to imply line
 editing mode.
 
 I just remembered a related problem.  I often mistype command lines,
 or do complicated editing of them involving many backspaces and other
 control characters to move around or search history.  script makes a
 mess of this by showing all the control characters, even without -k.
 The only way to handle this right might be to have the line editor in
 script itself, or perhaps just between the kernel and script (the ile
 utility might work for this).
 
 Bruce

From: Robert Bonomi <bonomi@mail.r-bonomi.com>
To: brde@optusnet.com.au
Cc: freebsd-bugs@freebsd.org, FreeBSD-gnats-submit@freebsd.org,
        rfg@tristatelogic.com
Subject: Re: bin/152154: /bin/csh & /bin/tcsh improperly diddle termios flags
Date: Sat, 13 Nov 2010 09:36:39 -0600 (CST)

 > From owner-freebsd-bugs@freebsd.org  Fri Nov 12 03:46:22 2010
 > Date: Fri, 12 Nov 2010 20:48:08 +1100 (EST)
 > From: Bruce Evans <brde@optusnet.com.au>
 > Subject: Re: bin/152154: /bin/csh & /bin/tcsh improperly diddle termios flags
 >
 > On Fri, 12 Nov 2010, Bruce Evans wrote:
 >
 >
 > I just remembered a related problem.  I often mistype command lines,
 > or do complicated editing of them involving many backspaces and other
 > control characters to move around or search history.  script makes a
 > mess of this by showing all the control characters, even without -k.
 
 This is  _documented behavior_.   It is, also, essentially inescapable.
 
   " BUGS
       Script places everything in the log file, including linefeeds and
       backspaces.  This is not what the naive user expects. "
 
 
 It is also documented in the body of the description:
 
   " The script utility works best with commands that do not manipulate 
     the screen.  The results are meant to emulate a hardcopy terminal, 
     not an addressable one. "
 
 Aside: some of us _count_  on script behaving that way -- capturing _every_ 
 character sent out.  
 
 Script has -no- way of knowing what is echoed 'editing' characters, and 
 what is 'significant' output.
 
 *ALL* script 'guarantees' -- in the situation involving an 'addressable'
 terminal -- is that if you play back the logfile to the _same_ type of 
 terminal is that 'what you see' on the screen on replay is "what you got'
 originally.   True "WYSIWYG".  <grin>
 
 
 > The only way to handle this right might be to have the line editor in
 > script itself ...
 
 That works _UNTIL_ you need to pass a 'line edit' character to an application.
 Heaven forbid you forget you have script running, and invoke a full-screen 
 editor (or any-other full-scrren utility for that matter) that uses one of
 the line-edit characters as it's "exit" sequence.
 
 And, even doing _that_ does not help with the 'mess' created by rummaging
 through, say, the csh cmommand history.
 
 A script functionality integrated into whichever CLI (shell) one uses
 could eliminate the line-edit / history, etc. "mess" from the logfile,
 but that is 'no help' in dealing with _all_ the other programs that 
 produce output "assuming" an 'addressable' display.  How do you log a
 'vi' session, for example?  <evil grin>
 
 For a 'cleaner' typescript, set the TERM environment variable to 'dumb'.
 

From: "Ronald F. Guilmette" <rfg@tristatelogic.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:  
Subject: Re: bin/152154: /bin/csh & /bin/tcsh improperly diddle termios flags 
Date: Wed, 17 Nov 2010 02:36:20 -0800

 Gentlemen,
 
 Thanks for the discussion of this problem.
 
 As someone else just pointed out to me (via the tcsh discussion list)
 _any_ shell that allows command line editing (not just tcsh but also
 bash and zsh) are going to put the pseudo terminal into character-at-a-time
 non-echo mode, and thus, one will get doubled characters in the typescript
 file when using "script -k" in conjunction with any such shell.
 
 So I guess we should say that the REAL non-feature (aka "bug") here is not
 really in tcsh but rather in the way that script(1) decides whether or
 not to write input characters to the typescript file when operating in
 the -k mode.
 
 Here is the code which involves that decision:
 
                         if (cc > 0) {
                                 (void)write(master, ibuf, cc);
                                 if (kflg && (tcgetattr(master, &stt) >= 0) &&
                                     ((stt.c_lflag & ECHO) == 0)) {
                                         (void)fwrite(ibuf, 1, cc, fscript);
                                 }
                         }
 
 My feeling now is that I should withdraw/cancel this PR (bin/152154) which
 is on tcsh and instead file a new & different one on script(1), because
 it is clear now that -k, as implemented, just won't work in conjunction
 with shells that allow command line editing.
 
 (Can someone plesse instruct me how to cancel a PR?  I've never done it
 before, and I'm ignorant.)
 
 I want to say also that I do believe that there is a solution to this
 problem.  I cannot be 100% sure yet that there will never be any circumstances
 under which it might perhaps malfunction slightly, but my early (and so far
 minimal) testing has not shown any signs of any malfunctions, so I am hopeful
 that this may be a solution that will work OK, even for the shells that
 allow command line editing.
 
 So anyway, here's my idea...
 
 Right now, script(1) is simply writing 100% of all characters which the
 child shell (or its children) write to stdout/stderr into the typescript
 file.  In the -k case however, this is augmented by script also writing
 any characters that are INPUT (to the stdin of the child shell) to the
 typescript file also, but only (as shown by the code above) when the
 the child shell (or any of its children) has set its pseudo-tty to _not_
 ECHO.  Clearly, as we now know, this approach won't work consistantly
 across all shells.
 
 So instead of this approach, I propose that script(1) be modified so that
 when -k mode is in effect, it will write 100% of all INPUT characters to
 the typescript file, but then each time it receives a block of characters
 that are comming _out_ from the shell, instead of writing 100% of those
 OUTPUT characters to the typescript file, script(1) would instead compare
 the buffer full of output bytes it just received (frm the child shell and/or
 its children) to the last buffer full of input bytes that script(1) sent
 down to the child shell.  If the contents of the two buffers are equal,
 then the current block of output characters that are coming up from the
 child shell would simply be suppressed and NOT written to the typescript
 file, thus, in efffect, nixing any echoed characters which simply duplicate
 characters that were already added to the typescript file.
 
 A patch implementing the change I have just described is included below.
 I would appreciate any and all comments on it.  I've just tried it out
 a little and it does seem to work OK, at least with tcsh.
 
 Of course, the whole idea here is, admittedly, based upon a presumption
 which may or may not be true in all circumstances and/or in all contexts,
 i.e. that any input from the _actual_ controlling tty... which will usually
 be, you know, one character at a time, especially for a hunt-and-pecker
 like me... will be echoed back before more input comes in, i.e. if it is
 going to be echoed back at all.  And I am allowing here for even the super
 duper input rate that may come from doing, e.g. a cut-and-paste, where many
 characters may be read from the real terminal device in one glup.  As long
 as those all get echoed back, immediately, in one single glup too, then this
 approach to implementing -k still works.  (I did try some cut-and-pasting
 and that also seemed to work OK with the patche below.)
 
 The only thing that isn't all that clear to me (and that frankly, I haven't
 tested yet) is what would happen if input is coming from something non-
 interactive, like, you know, a file.  I actually don't know what would
 happen in that case, but I suspect that this patch should work OK even for
 that case.
 
 I guess that the possibility that something might cause some characters (e.g
 control characters and/or terminal control sequences) not to "echo" back
 verbatim, as themselves, might also be an issue, but in this case I rather
 doubt that the patch below make things any worse/stranger than they already
 are, when/if such oddities are being logged in the (current) script-k+/bin/sh
 case.
 
 
 Regards,
 rfg
 
 
 P.S.  My apologies to Bruce Evans about the local blacklisting of your ISP.
 It was nothing personal, believe me.  Just got too much spam from them,
 that's all.  Anyway, I have removed that local blacklist entry now so for
 the time being at least, please do feel free to mail me direct, if you are
 so inclined.
 
 
 
 *** script.c.orig	2004-02-15 09:30:13.000000000 -0800
 --- script.c	2010-11-14 16:22:15.000000000 -0800
 ***************
 *** 82,86 ****
   main(int argc, char *argv[])
   {
 ! 	int cc;
   	struct termios rtt, stt;
   	struct winsize win;
 --- 82,86 ----
   main(int argc, char *argv[])
   {
 ! 	int icc = 0, occ;
   	struct termios rtt, stt;
   	struct winsize win;
 ***************
 *** 178,200 ****
   			break;
   		if (n > 0 && FD_ISSET(STDIN_FILENO, &rfd)) {
 ! 			cc = read(STDIN_FILENO, ibuf, BUFSIZ);
 ! 			if (cc < 0)
   				break;
 ! 			if (cc == 0)
   				(void)write(master, ibuf, 0);
 ! 			if (cc > 0) {
 ! 				(void)write(master, ibuf, cc);
 ! 				if (kflg && tcgetattr(master, &stt) >= 0 &&
 ! 				    ((stt.c_lflag & ECHO) == 0)) {
 ! 					(void)fwrite(ibuf, 1, cc, fscript);
 ! 				}
   			}
   		}
   		if (n > 0 && FD_ISSET(master, &rfd)) {
 ! 			cc = read(master, obuf, sizeof (obuf));
 ! 			if (cc <= 0)
   				break;
 ! 			(void)write(STDOUT_FILENO, obuf, cc);
 ! 			(void)fwrite(obuf, 1, cc, fscript);
   		}
   		tvec = time(0);
 --- 178,201 ----
   			break;
   		if (n > 0 && FD_ISSET(STDIN_FILENO, &rfd)) {
 ! 			icc = read(STDIN_FILENO, ibuf, BUFSIZ);
 ! 			if (icc < 0)
   				break;
 ! 			if (icc == 0)
   				(void)write(master, ibuf, 0);
 ! 			if (icc > 0) {
 ! 				(void)write(master, ibuf, icc);
 ! 				if (kflg)
 ! 					(void)fwrite(ibuf, 1, icc, fscript);
   			}
   		}
   		if (n > 0 && FD_ISSET(master, &rfd)) {
 ! 			occ = read(master, obuf, sizeof (obuf));
 ! 			if (occ <= 0)
   				break;
 ! 			(void)write(STDOUT_FILENO, obuf, occ);
 ! 			if (!kflg || (occ != icc) || memcmp (obuf, ibuf, occ)) {
 ! 				(void)fwrite(obuf, 1, occ, fscript);
 ! 				icc = 0;
 ! 			}
   		}
   		tvec = time(0);
 
 
>Unformatted:
