From jhs@berklix.com  Mon Jan 31 03:53:33 2011
Return-Path: <jhs@berklix.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 2D76B106566B
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Jan 2011 03:53:33 +0000 (UTC)
	(envelope-from jhs@berklix.com)
Received: from tower.berklix.org (tower.berklix.org [83.236.223.114])
	by mx1.freebsd.org (Postfix) with ESMTP id 8C1D58FC12
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Jan 2011 03:53:32 +0000 (UTC)
Received: from park.js.berklix.net (p5B22D186.dip.t-dialin.net [91.34.209.134])
	(authenticated bits=0)
	by tower.berklix.org (8.14.2/8.14.2) with ESMTP id p0V3rR4B075213;
	Mon, 31 Jan 2011 03:53:28 GMT
	(envelope-from jhs@berklix.com)
Received: from blak.js.berklix.net (blak.js.berklix.net [192.168.91.66])
	by park.js.berklix.net (8.13.8/8.13.8) with ESMTP id p0V3rLUS041456;
	Mon, 31 Jan 2011 04:53:22 +0100 (CET)
	(envelope-from jhs@blak.js.berklix.net)
Received: from blak.js.berklix.net (localhost.js.berklix.net [127.0.0.1])
	by blak.js.berklix.net (8.14.4/8.14.4) with ESMTP id p0V3r2TC048461;
	Mon, 31 Jan 2011 04:53:07 +0100 (CET)
	(envelope-from jhs@blak.js.berklix.net)
Received: (from jhs@localhost)
	by blak.js.berklix.net (8.14.4/8.14.4/Submit) id p0V3qTeZ048460;
	Mon, 31 Jan 2011 04:52:29 +0100 (CET)
	(envelope-from jhs)
Message-Id: <201101310352.p0V3qTeZ048460@blak.js.berklix.net>
Date: Mon, 31 Jan 2011 04:52:29 +0100 (CET)
From: "Julian H. Stacey" <jhs@berklix.com>
Reply-To: "Julian H. Stacey" <jhs@berklix.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc: "Julian H. Stacey" <jhs@berklix.com>,
        "Tim Kientzle" <kientzle@freebsd.org>,
        "Marc Recht (for NetBSD)" <m.recht@devnet.de>,
        "Gary Jennejohn (Home)" <gljennjohn@googlemail.com>
Subject: usr.bin/tar/ ignores error codes from read() just silently pads nulls
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         154407
>Category:       bin
>Synopsis:       [patch] tar(1) ignores error codes from read() just silently pads nulls
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kientzle
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jan 31 04:00:21 UTC 2011
>Closed-Date:    Thu Jun 13 17:11:50 UTC 2013
>Last-Modified:  Thu Jun 13 17:11:50 UTC 2013
>Originator:     "Julian H. Stacey" <jhs@berklix.com>
>Release:        FreeBSD 8.1-RELEASE amd64
>Organization:
http://berklix.com BSD Linux Unix Consultancy, Munich/Muenchen.
>Environment:
System: FreeBSD blak.js.berklix.net 8.1-RELEASE FreeBSD 8.1-RELEASE #2: Fri Aug 20 15:11:19 CEST 2010 jhs@blak.js.berklix.net:/usr/src/sys/amd64/compile/BLAK.small amd64


	
>Description:
	usr.bin/tar/ ignores error codes from read() silently pads with nulls,
	corrupting copied data without warning !
	Log of how I discovered problem + diffs + copy of send-pr:
	  http://berklix.com/~jhs/src/bsd/fixes/FreeBSD/src/gen/usr.bin/tar/
	(various files have .ignore extension to avoid my customise script).

	Comparison with gtar:
		gtar also delivers false nulls unfortunately,
		gtar warns users it is getting device errors.

	Comparison with cp:
		cp warns of errors,
		cp delivers no false nulls, truncates at read fail.

>How-To-Repeat:
- Get a normal data CD or DVD you have created )
  (not commercial recorded hollywood deliberately sector crippled media),
- Damage the media so it will have some read errors,
  mount /dev/acd0 /cdrom
  cd /tmp
  mkdir -p cp tar/cdrom
  cp -R /cdrom cp
  (cd /cdrom ; tar cf - . ) | ( cd tar/cdrom ; tar xf - )
  cd tar/cdrom
  # http://berklix.com/~jhs/src/bsd/jhs/bin/public/cmpd/
  find . -type f -exex cmpd -d {} ../../cp/cdrom \;
  # http://berklix.com/~jhs/src/bsd/jhs/bin/public/8f/
  foreach i ( `find . -type f | xargs 8f -n 2048 -b 0 -l` )
	echo $i
	xargs od -c $i | yail
	end

>Fix:
My patches fix code to avoid unwarned damaged data.  As it's Tim
K.'s recently written code, I presume he will want to fix it himself,
so my patch lines as well as fixing, are also appended with edit mark
// JHS for easy searching.

write.c & util.c were bad in 8.1-RELEASE 
	(& the critical bit in 
	./-current/src/usr/bin/write.c
	^write_file_data() 
	the final 
	bytes_read = read(fd, bsdtar->buff, FILEDATABUFLEN);
	is also bad)
Various return values silently ignored.  
All invocations of [f]read() & [f]write() & all invocations of
*_read_* & *_write_* etc should be checked whether return value is
properly used.

Later fixes if any will be in
http://berklix.com/~jhs/src/bsd/fixes/FreeBSD/src/gen/usr.bin/tar/
A copy appended.

*** 8.1-RELEASE-generic/src/usr.bin/tar/write.c	Mon Jun 14 04:09:06 2010
--- 8.1-RELEASE+jhs-error-detect/src/usr.bin/tar/write.c	Sat Jan 29 18:34:55 2011
***************
*** 217,222 ****
--- 217,224 ----
  	if (ARCHIVE_OK != archive_write_open_file(a, bsdtar->filename))
  		bsdtar_errc(bsdtar, 1, 0, archive_error_string(a));
  	write_archive(a, bsdtar);
+ 		// JHS write_archive is declared void,
+ 		// JHS is there a static extern ret val to check ?
  }
  
  /*
***************
*** 301,312 ****
  		archive_write_set_format(a, format);
  	}
  	lseek(bsdtar->fd, end_offset, SEEK_SET); /* XXX check return val XXX */
  	if (ARCHIVE_OK != archive_write_set_options(a, bsdtar->option_options))
  		bsdtar_errc(bsdtar, 1, 0, archive_error_string(a));
  	if (ARCHIVE_OK != archive_write_open_fd(a, bsdtar->fd))
  		bsdtar_errc(bsdtar, 1, 0, archive_error_string(a));
  
! 	write_archive(a, bsdtar); /* XXX check return val XXX */
  
  	close(bsdtar->fd);
  	bsdtar->fd = -1;
--- 303,316 ----
  		archive_write_set_format(a, format);
  	}
  	lseek(bsdtar->fd, end_offset, SEEK_SET); /* XXX check return val XXX */
+ 								// JHS
  	if (ARCHIVE_OK != archive_write_set_options(a, bsdtar->option_options))
  		bsdtar_errc(bsdtar, 1, 0, archive_error_string(a));
  	if (ARCHIVE_OK != archive_write_open_fd(a, bsdtar->fd))
  		bsdtar_errc(bsdtar, 1, 0, archive_error_string(a));
  
! 	write_archive(a, bsdtar); /* XXX check return val XXX */	
! 		// JHS what return val ? write_archive is declared void
  
  	close(bsdtar->fd);
  	bsdtar->fd = -1;
***************
*** 390,395 ****
--- 394,401 ----
  		bsdtar_errc(bsdtar, 1, 0, archive_error_string(a));
  
  	write_archive(a, bsdtar);
+ 		// JHS write_archive is declared void,
+ 		// JHS is there a static extern ret val to check ?
  
  	close(bsdtar->fd);
  	bsdtar->fd = -1;
***************
*** 926,931 ****
--- 932,945 ----
  	off_t	progress = 0;
  
  	bytes_read = read(fd, bsdtar->buff, FILEDATABUFLEN);
+ 	if (bytes_read < 0) {					// JHS
+ 		perror(NULL) ;					// JHS
+ 		fprintf(stderr,"File: %s, Line %d, Ret %d\n",	// JHS
+ 			__FILE__, __LINE__, (int)bytes_read );	// JHS
+ 		return(-1) ;					// JHS
+ 		// I've not checked to see if caller 		// JHS
+ 		// appropriately detects & deals with -1	// JHS
+ 		}						// JHS
  	while (bytes_read > 0) {
  		siginfo_printinfo(bsdtar, progress);
  
***************
*** 945,951 ****
  		}
  		progress += bytes_written;
  		bytes_read = read(fd, bsdtar->buff, FILEDATABUFLEN);
! 	}
  	return 0;
  }
  
--- 959,995 ----
  		}
  		progress += bytes_written;
  		bytes_read = read(fd, bsdtar->buff, FILEDATABUFLEN);
! 		if (bytes_read < 0)				// JHS
! 			{					// JHS
! 			perror(NULL) ;				// JHS
! 			fprintf(stderr,"Read error.\n");	// JHS
! 			// fprintf(stderr,"Read error on %s.\n",	// JHS
! 			// Please pass in name as parameter to print. // JHS
! 			//	"" ); 				// JHS
! 			fprintf(stderr,				// JHS
! 			 "Output will contain false trailing nulls.\n"); // JHS
! 			// This prints when 
! 			fprintf(stderr,				// JHS
! 				"File: %s, Line %d, Ret %d\n",	// JHS
! 			 __FILE__, __LINE__, (int)bytes_read );	// JHS
! 			return(-1) ;				// JHS
! 			// I've not read code to see if caller 	// JHS
! 			// appropriately detects & deals with -1 // JHS
! 			// But Ive tested it, with		// JHS
! 			// tar cvf junk.tar aa bb,		// JHS
! 			// & if aa has dev errors,		// JHS
! 			// bb does not get archived.		// JHS
! 			}					// JHS
! 	}
! 	if (bytes_read < 0) 					// JHS
! 		{						// JHS
! 		perror(NULL) ;					// JHS
! 		fprintf(stderr,"File: %s, Line %d\n", 		// JHS
! 			__FILE__, __LINE__ ); 			// JHS
! 		return(-1) ;					// JHS
! 		// I've not read code to see if caller 		// JHS
! 		// appropriately detects & deals with -1 	// JHS
! 		}						// JHS
  	return 0;
  }
  
*** 8.1-RELEASE-generic/src/usr.bin/tar/util.c	Mon Jun 14 04:09:06 2010
--- 8.1-RELEASE+jhs-error-detect/src/usr.bin/tar/util.c	Sat Jan 29 16:36:45 2011
***************
*** 234,239 ****
--- 234,241 ----
  	exit(eval);
  }
  
+ /* Get confirmation from user to do something, eg add/ copy 		// JHS
+ 	Return: 1=yes, 0=no */						// JHS
  int
  yes(const char *fmt, ...)
  {
***************
*** 249,254 ****
--- 251,263 ----
  	fflush(stderr);
  
  	l = read(2, buff, sizeof(buff) - 1);
+ 	if (l < 0) 							// JHS
+ 		{							// JHS
+ 		perror(NULL) ;						// JHS
+ 		fprintf(stderr,"Failed to read yes/no\n" ); 		// JHS
+ 		fprintf(stderr,"File: %s, Line %d, Ret %d\n", 		// JHS
+ 			__FILE__, __LINE__, (int)l );			// JHS	
+ 		}							// JHS
  	if (l <= 0)
  		return (0);
  	buff[l] = 0;
***************
*** 307,312 ****
--- 316,333 ----
  		/* Get some more data into the buffer. */
  		bytes_wanted = buff + buff_length - buff_end;
  		bytes_read = fread(buff_end, 1, bytes_wanted, f);
+ 		if (							// JHS
+ 			( bytes_read < bytes_wanted )			// JHS
+ 			|| 						// JHS
+ 			(bytes_read = 0)				// JHS
+ 		   )							// JHS
+ 		if ((bytes_read = 0) || ( bytes_read < bytes_wanted ))	// JHS
+ 			{						// JHS
+ 			perror(NULL) ;					// JHS
+ 			fprintf(stderr,					// JHS
+ 				"File: %s, Line %d, Ret %d\n",		// JHS
+ 				__FILE__, __LINE__, (int)bytes_read ); 	// JHS
+ 			}						// JHS
  		buff_end += bytes_read;
  		/* Process all complete lines in the buffer. */
  		while (line_end < buff_end) {
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->kientzle 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Jan 31 08:13:53 UTC 2011 
Responsible-Changed-Why:  
Over to maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=154407 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/154407: commit references a PR
Date: Sat, 25 Jun 2011 16:28:03 +0000 (UTC)

 Author: kientzle
 Date: Sat Jun 25 16:27:49 2011
 New Revision: 223541
 URL: http://svn.freebsd.org/changeset/base/223541
 
 Log:
   If there is a read error reading Y/N confirmation from the keyboard,
   exit immediately with an error.
   
   If there is an error opening or reading a file to put into the archive,
   set the return value for a deferred error exit.
   
   PR:		bin/154407
 
 Modified:
   head/usr.bin/tar/util.c
   head/usr.bin/tar/write.c
 
 Modified: head/usr.bin/tar/util.c
 ==============================================================================
 --- head/usr.bin/tar/util.c	Sat Jun 25 16:13:56 2011	(r223540)
 +++ head/usr.bin/tar/util.c	Sat Jun 25 16:27:49 2011	(r223541)
 @@ -226,7 +226,11 @@ yes(const char *fmt, ...)
  	fflush(stderr);
  
  	l = read(2, buff, sizeof(buff) - 1);
 -	if (l <= 0)
 +	if (l < 0) {
 +	  fprintf(stderr, "Keyboard read failed\n");
 +	  exit(1);
 +	}
 +	if (l == 0)
  		return (0);
  	buff[l] = 0;
  
 
 Modified: head/usr.bin/tar/write.c
 ==============================================================================
 --- head/usr.bin/tar/write.c	Sat Jun 25 16:13:56 2011	(r223540)
 +++ head/usr.bin/tar/write.c	Sat Jun 25 16:27:49 2011	(r223541)
 @@ -919,6 +919,7 @@ write_entry_backend(struct bsdtar *bsdta
  		const char *pathname = archive_entry_sourcepath(entry);
  		fd = open(pathname, O_RDONLY | O_BINARY);
  		if (fd == -1) {
 +			bsdtar->return_value = 1;
  			if (!bsdtar->verbose)
  				bsdtar_warnc(errno,
  				    "%s: could not open file", pathname);
 @@ -1020,6 +1021,12 @@ write_file_data(struct bsdtar *bsdtar, s
  		progress += bytes_written;
  		bytes_read = read(fd, bsdtar->buff, FILEDATABUFLEN);
  	}
 +	if (bytes_read < 0) {
 +		bsdtar_warnc(errno,
 +			     "%s: Read error",
 +			     archive_entry_pathname(entry));
 +		bsdtar->return_value = 1;
 +	}
  	return 0;
  }
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: kientzle 
State-Changed-When: Sat Jun 25 16:36:28 UTC 2011 
State-Changed-Why:  
r223541 fixes this in trunk 

http://www.freebsd.org/cgi/query-pr.cgi?pr=154407 

From: Tim Kientzle <kientzle@freebsd.org>
To: bug-followup@FreeBSD.org,
 jhs@berklix.com
Cc:  
Subject: Re: bin/154407: [patch] tar(1) ignores error codes from read() just silently pads nulls
Date: Sat, 25 Jun 2011 09:30:59 -0700

 Please take a look at r223541, which I believe fixes this problem in =
 -CURRENT.
 
 My solution is a little different from yours; read errors will not abort =
 archiving immediately
 but will instead report the error and set a deferred exit value for tar.
 
 Tim
 
State-Changed-From-To: patched->closed 
State-Changed-By: gavin 
State-Changed-When: Thu Jun 13 17:10:07 UTC 2013 
State-Changed-Why:  
This appears to have been fixed in all supported release some 
time ago. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=154407 
>Unformatted:
