From nobody@FreeBSD.org  Mon Oct 11 06:09:07 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 00D05106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 11 Oct 2010 06:09:07 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id E46738FC12
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 11 Oct 2010 06:09:06 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o9B696Xk037284
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 11 Oct 2010 06:09:06 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o9B696bo037279;
	Mon, 11 Oct 2010 06:09:06 GMT
	(envelope-from nobody)
Message-Id: <201010110609.o9B696bo037279@www.freebsd.org>
Date: Mon, 11 Oct 2010 06:09:06 GMT
From: Dominic Fandrey <kamikaze@bsdforen.de>
To: freebsd-gnats-submit@FreeBSD.org
Subject: rs(1) damages data
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         151384
>Category:       bin
>Synopsis:       rs(1) truncates lines longer than 1024 bytes (BUFSIZ)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    jh
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Oct 11 06:10:01 UTC 2010
>Closed-Date:    Fri Oct 07 14:35:53 UTC 2011
>Last-Modified:  Fri Oct 07 14:35:53 UTC 2011
>Originator:     Dominic Fandrey
>Release:        RELENG_8
>Organization:
private
>Environment:
FreeBSD mobileKamikaze.norad 8.1-STABLE FreeBSD 8.1-STABLE #0: Mon Sep  6 17:08:51 CEST 2010     root@mobileKamikaze.norad:/usr/obj/HP6510b-8/amd64/usr/src/sys/HP6510b-8  amd64
>Description:
Recently rs has adopted a habit of damaging data in long lines
of input. I've been relying on it to reformat dependency data,
I had to switch to sed, which has a slight performance impact
for my use case.
>How-To-Repeat:
# jot -s\  -b 01234567 1000 | rs 0 1 | grep -vxF 01234567
012345
67
012
34567
012345
67
012
34567
012345
67
012
34567

The jot command prints the string 01234567 a thousand times in
a single row.

The rs command is supposed to generate an automatic(0) number of
rows with 1 column per row. I.e. every word stands in its own line.

The grep filters all intact words, so everything that is printed,
was damaged by rs.

This has the look of a repetitive pattern to me, probably this
happens at a fixed buffer boundary.

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->uqs 
Responsible-Changed-By: uqs 
Responsible-Changed-When: Tue Oct 12 11:53:22 UTC 2010 
Responsible-Changed-Why:  
I'll have a look at this. 

Can you please specify when this broke, ie. for which release did this work? 

Also, you may want to use xargs(1) to deal with overly long lines. Your example 
works fine like this: 

jot -s  -b 01234567 1000 | xargs -n1 | grep -vxF 01234567 


http://www.freebsd.org/cgi/query-pr.cgi?pr=151384 

From: Bryan Hahn <hahnbc@gmail.com>
To: bug-followup@FreeBSD.org, kamikaze@bsdforen.de
Cc:  
Subject: Re: bin/151384: rs(1) damages data
Date: Wed, 13 Oct 2010 15:18:58 -0500

 Question was:
 "Can you please specify when this broke, ie. for which release did this work?"
 
 Out of curiousity, I checked a handful of systems and couldn't find a
 working version.
 
 It was broken on the following:
 
 FreeBSD 4.11-RELEASE-p14
 FreeBSD 5.4-RELEASE-p1
 FreeBSD 6.3-RELEASE-p14
 FreeBSD 7.2-RELEASE-p6
 FreeBSD 8.1-RELEASE
 
 And just for fun, I tried it on a REALLY old system where it was also broken:
 
 BSDI BSD/OS 3.1 (from 1998)
 
 (The Description comment:  "Recently rs has adopted...." may be misleading.)
 
 ==
 
    - B

From: Dominic Fandrey <kamikaze@bsdforen.de>
To: Bryan Hahn <hahnbc@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: bin/151384: rs(1) damages data
Date: Wed, 13 Oct 2010 23:00:22 +0200

 On 13/10/2010 22:18, Bryan Hahn wrote:
 > Question was:
 > "Can you please specify when this broke, ie. for which release did this work?"
 
 I've been using rs for a long time in pkg_upgrade and I only recognized
 this problem recently, figuring out how some dependencies disappeared.
 
 > Out of curiousity, I checked a handful of systems and couldn't find a
 > working version.
 > 
 > It was broken on the following:
 > 
 > FreeBSD 4.11-RELEASE-p14
 > FreeBSD 5.4-RELEASE-p1
 > FreeBSD 6.3-RELEASE-p14
 > FreeBSD 7.2-RELEASE-p6
 > FreeBSD 8.1-RELEASE
 > 
 > And just for fun, I tried it on a REALLY old system where it was also broken:
 > 
 > BSDI BSD/OS 3.1 (from 1998)
 > 
 > (The Description comment:  "Recently rs has adopted...." may be misleading.)
 
 Considering the /massive/ amount of testing I perform I figured
 I'd have recognized this if it had happened before. I guess I was
 wrong.
 
 Thank you for your exhaustive effort to verify this problem.
 
 As far as I'm concerned rs is salted ground, now. I will not use
 it in my scripts any more until support for all branches where
 it's broken has run out.
 
 Considering your findings I'd expect it to be broken on Net-, Open-
 and DragonFlyBSD, too.

From: Dominic Fandrey <kamikaze@bsdforen.de>
To: bug-followup@FreeBSD.org, kamikaze@bsdforen.de
Cc:  
Subject: Re: bin/151384: rs(1) damages data
Date: Wed, 13 Oct 2010 23:29:10 +0200

 In response to uqs, about "when this broke" Bryan is probably a
 better source.
 
 The runtime of your xargs example is really terrible:
 
 # jot -b 12345678 -s\  1000000 | time -h rs 0 1 > /dev/null
 	0.27s real		0.10s user		0.01s sys
 
 # jot -b 12345678 -s\  1000000 | time -h awk -F\  '{gsub(" ", "\n"); print}' > /dev/null
 	0.39s real		0.17s user		0.04s sys
 
 # jot -b 12345678 -s\  1000000 | time -h awk -F\  'BEGIN{RS=" "} {print $0}' > /dev/null
 	0.57s real		0.53s user		0.00s sys
 
 # jot -b 12345678 -s\  1000000 | time -h sed 's/ /\
 /g' > /dev/null
 	1.06s real		0.80s user		0.06s sys
 
 # jot -b 12345678 -s\  1000000 | time -h xargs -n1 > /dev/null
 	10m3.00s real		2m26.65s user		7m32.41s sys
 
 Note the runtime factor between rs and xargs is somewhere
 around 2000.
 
 At least now I know that I should use AWK for this task.
 
 Regards

From: Dominic Fandrey <kamikaze@bsdforen.de>
To: Bryan Hahn <hahnbc@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: bin/151384: rs(1) damages data
Date: Fri, 29 Oct 2010 20:37:39 +0200

 On 13/10/2010 22:18, Bryan Hahn wrote:
 > Question was:
 > "Can you please specify when this broke, ie. for which release did this work?"
 > 
 > Out of curiousity, I checked a handful of systems and couldn't find a
 > working version.
 > ...
 
 Line 333 of of src/usr.bin/rs/rs.c states the nature of the problem:
 /* two screenfuls should do */
 
 There's a 2048 byte buffer and if your lines are longer than that,
 you're screwed.
 
 -- 
 A: Because it fouls the order in which people normally read text.
 Q: Why is top-posting such a bad thing?
 A: Top-posting.
 Q: What is the most annoying thing on usenet and in e-mail? 
Responsible-Changed-From-To: uqs->jh 
Responsible-Changed-By: jh 
Responsible-Changed-When: Sat Jan 22 15:20:31 UTC 2011 
Responsible-Changed-Why:  
Take. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=151384 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/151384: commit references a PR
Date: Mon,  7 Feb 2011 18:10:24 +0000 (UTC)

 Author: jh
 Date: Mon Feb  7 18:10:18 2011
 New Revision: 218411
 URL: http://svn.freebsd.org/changeset/base/218411
 
 Log:
   - Use LINE_MAX from limits.h as the maximum line length instead of
     BUFSIZ. Use LINE_MAX * 2 as the buffer size (BSIZE).
   - Error out if we encounter a line longer than LINE_MAX. The previous
     behavior was to silently split long lines and produce corrupted
     output.
   
   PR:		bin/151384
 
 Modified:
   head/usr.bin/rs/rs.c
 
 Modified: head/usr.bin/rs/rs.c
 ==============================================================================
 --- head/usr.bin/rs/rs.c	Mon Feb  7 18:05:56 2011	(r218410)
 +++ head/usr.bin/rs/rs.c	Mon Feb  7 18:10:18 2011	(r218411)
 @@ -48,6 +48,7 @@ __FBSDID("$FreeBSD$");
  
  #include <err.h>
  #include <ctype.h>
 +#include <limits.h>
  #include <stdio.h>
  #include <stdlib.h>
  #include <string.h>
 @@ -328,8 +329,8 @@ prepfile(void)
  		warnx("%d is colwidths, nelem %d", colwidths[i], nelem);*/
  }
  
 -#define	BSIZE	2048
 -char	ibuf[BSIZE];		/* two screenfuls should do */
 +#define	BSIZE	(LINE_MAX * 2)
 +char	ibuf[BSIZE];
  
  int
  getline(void)	/* get line; maintain curline, curlen; manage storage */
 @@ -350,7 +351,7 @@ getline(void)	/* get line; maintain curl
  			curline = ibuf;
  		}
  	}
 -	if (!putlength && endblock - curline < BUFSIZ) {   /* need storage */
 +	if (!putlength && endblock - curline < LINE_MAX + 1) { /* need storage */
  		/*ww = endblock-curline; tt += ww;*/
  		/*printf("#wasted %d total %d\n",ww,tt);*/
  		if (!(curline = (char *) malloc(BSIZE)))
 @@ -358,11 +359,16 @@ getline(void)	/* get line; maintain curl
  		endblock = curline + BSIZE;
  		/*printf("#endb %d curline %d\n",endblock,curline);*/
  	}
 -	for (p = curline, i = 1; i < BUFSIZ; *p++ = c, i++)
 -		if ((c = getchar()) == EOF || c == '\n')
 +	for (p = curline, i = 0;; *p++ = c, i++) {
 +		if ((c = getchar()) == EOF)
  			break;
 +		if (i >= LINE_MAX)
 +			errx(1, "maximum line length (%d) exceeded", LINE_MAX);
 +		if (c == '\n')
 +			break;
 +	}
  	*p = '\0';
 -	curlen = i - 1;
 +	curlen = i;
  	return(c);
  }
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: jh 
State-Changed-When: Mon Feb 7 18:59:59 UTC 2011 
State-Changed-Why:  
Patched in head (r218411). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=151384 

From: Dominic Fandrey <kamikaze@bsdforen.de>
To: bug-followup@FreeBSD.org, kamikaze@bsdforen.de
Cc:  
Subject: Re: bin/151384: rs(1) truncates lines longer than 1024 bytes (BUFSIZ)
Date: Thu, 10 Feb 2011 14:46:07 +0100

 I propose documenting the change:
 
 diff -u usr.bin/rs/rs.1.orig usr.bin/rs/rs.1
 --- usr.bin/rs/rs.1.orig	2011-02-10 14:10:45.000000000 +0100
 +++ usr.bin/rs/rs.1	2011-02-10 14:42:09.000000000 +0100
 @@ -168,6 +168,18 @@
  unless the first non-ignored line is longer than the display width.
  Option letters which take numerical arguments interpret a missing
  number as zero unless otherwise indicated.
 +.Sh LIMITATIONS
 +Lines exceeding
 +.Dv LINE_MAX
 +defined in
 +.In limits.h
 +are not processed by
 +.Nm
 +and result in immediate termination. This behaviour is new as of
 +.Fx 9.0 ,
 +previous versions of
 +.Nm
 +silently split lines exceeding 2048 bytes and produce corrupted output.
  .Sh EXAMPLES
  The
  .Nm

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/151384: commit references a PR
Date: Wed, 17 Aug 2011 15:23:12 +0000 (UTC)

 Author: jh
 Date: Wed Aug 17 15:22:58 2011
 New Revision: 224945
 URL: http://svn.freebsd.org/changeset/base/224945
 
 Log:
   MFC r218411:
   
   - Use LINE_MAX from limits.h as the maximum line length instead of
     BUFSIZ. Use LINE_MAX * 2 as the buffer size (BSIZE).
   - Error out if we encounter a line longer than LINE_MAX. The previous
     behavior was to silently split long lines and produce corrupted
     output.
   
   PR:		bin/151384
 
 Modified:
   stable/8/usr.bin/rs/rs.c
 Directory Properties:
   stable/8/usr.bin/rs/   (props changed)
 
 Modified: stable/8/usr.bin/rs/rs.c
 ==============================================================================
 --- stable/8/usr.bin/rs/rs.c	Wed Aug 17 15:19:25 2011	(r224944)
 +++ stable/8/usr.bin/rs/rs.c	Wed Aug 17 15:22:58 2011	(r224945)
 @@ -52,6 +52,7 @@ __FBSDID("$FreeBSD$");
  
  #include <err.h>
  #include <ctype.h>
 +#include <limits.h>
  #include <stdio.h>
  #include <stdlib.h>
  #include <string.h>
 @@ -332,8 +333,8 @@ prepfile(void)
  		warnx("%d is colwidths, nelem %d", colwidths[i], nelem);*/
  }
  
 -#define	BSIZE	2048
 -char	ibuf[BSIZE];		/* two screenfuls should do */
 +#define	BSIZE	(LINE_MAX * 2)
 +char	ibuf[BSIZE];
  
  int
  getline(void)	/* get line; maintain curline, curlen; manage storage */
 @@ -354,7 +355,7 @@ getline(void)	/* get line; maintain curl
  			curline = ibuf;
  		}
  	}
 -	if (!putlength && endblock - curline < BUFSIZ) {   /* need storage */
 +	if (!putlength && endblock - curline < LINE_MAX + 1) { /* need storage */
  		/*ww = endblock-curline; tt += ww;*/
  		/*printf("#wasted %d total %d\n",ww,tt);*/
  		if (!(curline = (char *) malloc(BSIZE)))
 @@ -362,11 +363,16 @@ getline(void)	/* get line; maintain curl
  		endblock = curline + BSIZE;
  		/*printf("#endb %d curline %d\n",endblock,curline);*/
  	}
 -	for (p = curline, i = 1; i < BUFSIZ; *p++ = c, i++)
 -		if ((c = getchar()) == EOF || c == '\n')
 +	for (p = curline, i = 0;; *p++ = c, i++) {
 +		if ((c = getchar()) == EOF)
  			break;
 +		if (i >= LINE_MAX)
 +			errx(1, "maximum line length (%d) exceeded", LINE_MAX);
 +		if (c == '\n')
 +			break;
 +	}
  	*p = '\0';
 -	curlen = i - 1;
 +	curlen = i;
  	return(c);
  }
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/151384: commit references a PR
Date: Fri,  7 Oct 2011 14:29:29 +0000 (UTC)

 Author: jh
 Date: Fri Oct  7 14:29:15 2011
 New Revision: 226107
 URL: http://svn.freebsd.org/changeset/base/226107
 
 Log:
   MFC r218411:
   
   - Use LINE_MAX from limits.h as the maximum line length instead of
     BUFSIZ. Use LINE_MAX * 2 as the buffer size (BSIZE).
   - Error out if we encounter a line longer than LINE_MAX. The previous
     behavior was to silently split long lines and produce corrupted
     output.
   
   PR:		bin/151384
 
 Modified:
   stable/7/usr.bin/rs/rs.c
 Directory Properties:
   stable/7/usr.bin/rs/   (props changed)
 
 Modified: stable/7/usr.bin/rs/rs.c
 ==============================================================================
 --- stable/7/usr.bin/rs/rs.c	Fri Oct  7 14:27:20 2011	(r226106)
 +++ stable/7/usr.bin/rs/rs.c	Fri Oct  7 14:29:15 2011	(r226107)
 @@ -52,6 +52,7 @@ __FBSDID("$FreeBSD$");
  
  #include <err.h>
  #include <ctype.h>
 +#include <limits.h>
  #include <stdio.h>
  #include <stdlib.h>
  #include <string.h>
 @@ -333,8 +334,8 @@ prepfile(void)
  		warnx("%d is colwidths, nelem %d", colwidths[i], nelem);*/
  }
  
 -#define	BSIZE	2048
 -char	ibuf[BSIZE];		/* two screenfuls should do */
 +#define	BSIZE	(LINE_MAX * 2)
 +char	ibuf[BSIZE];
  
  int
  getline(void)	/* get line; maintain curline, curlen; manage storage */
 @@ -355,7 +356,7 @@ getline(void)	/* get line; maintain curl
  			curline = ibuf;
  		}
  	}
 -	if (!putlength && endblock - curline < BUFSIZ) {   /* need storage */
 +	if (!putlength && endblock - curline < LINE_MAX + 1) { /* need storage */
  		/*ww = endblock-curline; tt += ww;*/
  		/*printf("#wasted %d total %d\n",ww,tt);*/
  		if (!(curline = (char *) malloc(BSIZE)))
 @@ -363,11 +364,16 @@ getline(void)	/* get line; maintain curl
  		endblock = curline + BSIZE;
  		/*printf("#endb %d curline %d\n",endblock,curline);*/
  	}
 -	for (p = curline, i = 1; i < BUFSIZ; *p++ = c, i++)
 -		if ((c = getchar()) == EOF || c == '\n')
 +	for (p = curline, i = 0;; *p++ = c, i++) {
 +		if ((c = getchar()) == EOF)
  			break;
 +		if (i >= LINE_MAX)
 +			errx(1, "maximum line length (%d) exceeded", LINE_MAX);
 +		if (c == '\n')
 +			break;
 +	}
  	*p = '\0';
 -	curlen = i - 1;
 +	curlen = i;
  	return(c);
  }
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: jh 
State-Changed-When: Fri Oct 7 14:35:51 UTC 2011 
State-Changed-Why:  
Fixed in head, stable/8 and stable/7. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=151384 
>Unformatted:
