From nobody@FreeBSD.org  Sun Aug  8 12:50:26 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 9B9F2106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  8 Aug 2010 12:50:26 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 7060C8FC18
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  8 Aug 2010 12:50:26 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o78CoQng092932
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 8 Aug 2010 12:50:26 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o78CoQhL092931;
	Sun, 8 Aug 2010 12:50:26 GMT
	(envelope-from nobody)
Message-Id: <201008081250.o78CoQhL092931@www.freebsd.org>
Date: Sun, 8 Aug 2010 12:50:26 GMT
From: Shuichi KITAGUCHI <ki@hh.iij4u.or.jp>
To: freebsd-gnats-submit@FreeBSD.org
Subject: BSD grep accesses incorrect memory area.
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         149425
>Category:       bin
>Synopsis:       [patch] grep(1): BSD grep accesses incorrect memory area.
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    gabor
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Aug 08 13:00:16 UTC 2010
>Closed-Date:    Sun Aug 15 22:22:17 UTC 2010
>Last-Modified:  Sun Aug 15 22:22:17 UTC 2010
>Originator:     Shuichi KITAGUCHI
>Release:        9.0-CURRENT
>Organization:
>Environment:
FreeBSD rhea.k.ysnb.net 9.0-CURRENT FreeBSD 9.0-CURRENT #0 r210927: Fri Aug  6 22:55:43 JST 2010     root@rhea.k.ysnb.net:/usr/obj/data/rhea/usr/src/sys/RHEA  amd64

>Description:
BSD grep accesses incorrect memory region and may abort.
Additionally, valgrind detectes some errors.
>How-To-Repeat:
1)grep aborts (allocated buffer is short)
% cd /usr/src/usr.bin/grep
% grep -r -B 2 char .
..
grep: (malloc) /data/rhea/usr/src/lib/libc/stdlib/malloc.c:3176: Failed assertion: "bin->runcur->magic == ARENA_RUN_MAGIC"
Abort

2)valgrind detects some errors. (take 1, 0 bytes file)
% cd /usr/src/usr.bin/grep
% make
% touch x
% valgrind -v ./grep a x
..
==3775== Invalid read of size 1
==3775==    at 0x25C8E9: memchr (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
==3775==    by 0x4027C4: grep_fgetln (in /data/home/kit/tmp/g/grep)
==3775==    by 0x404156: procfile (in /data/home/kit/tmp/g/grep)
==3775==    by 0x4037DA: main (in /data/home/kit/tmp/g/grep)
..

3)valgrind detects some errors. (take 2, incorrect free)
% cd /usr/src/usr.bin/grep
% make
% valgrind -v ./grep -r -C 2 char .
..
==3716== Invalid free() / delete / delete[]
==3716==    at 0x25B260: free (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
==3716==    by 0x4044FD: procfile (in /data/home/kit/tmp/grep/grep)
==3716==    by 0x4049CD: grep_tree (in /data/home/kit/tmp/grep/grep)
==3716==    by 0x403857: main (in /data/home/kit/tmp/grep/grep)
..

>Fix:
attached patch is to be fixed for 1) and 2).
I cannot resolve 3).

Patch attached with submission follows:

diff --git a/queue.c b/queue.c
index 5f4d4ef..1fd05c5 100644
--- a/queue.c
+++ b/queue.c
@@ -56,7 +56,7 @@ enqueue(struct str *x)
 	struct qentry *item;
 
 	item = grep_malloc(sizeof(struct qentry));
-	item->data.dat = grep_malloc(sizeof(char) * x->len);
+	item->data.dat = grep_malloc(sizeof(char) * x->len + 1);
 	item->data.len = x->len;
 	item->data.line_no = x->line_no;
 	item->data.off = x->off;

diff --git a/file.c b/file.c
index 7d9715f..ab658fc 100644
--- a/file.c
+++ b/file.c
@@ -132,6 +132,10 @@ grep_fgetln(struct file *f, size_t *len)
 			else if (stat(fname, &st) != 0)
 				err(2, NULL);
 
+			/* no need to allocate buffer. */
+			if (st.st_size == 0)
+				return (NULL);
+
 			bufsiz = (MAXBUFSIZ > (st.st_size * PREREAD_M)) ?
 			    (st.st_size / 2) : MAXBUFSIZ;
 


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->gabor 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sun Aug 8 19:10:56 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=149425 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/149425: commit references a PR
Date: Sun, 15 Aug 2010 22:15:21 +0000 (UTC)

 Author: gabor
 Date: Sun Aug 15 22:15:04 2010
 New Revision: 211364
 URL: http://svn.freebsd.org/changeset/base/211364
 
 Log:
   - Revert strlcpy() changes to memcpy() because it's more efficient and
     former may be safer but in this case it doesn't add extra
     safety [1]
   - Fix -w option [2]
   - Fix handling of GREP_OPTIONS [3]
   - Fix --line-buffered
   - Make stdin input imply --line-buffered so that tail -f can be piped
     to grep [4]
   - Imply -h if single file is grepped, this is the GNU behaviour
   - Reduce locking overhead to gain some more performance [5]
   - Inline some functions to help the compiler better optimize the code
   - Use shortcut for empty files [6]
   
   PR:		bin/149425 [6]
   Prodded by:	jilles [1]
   Reported by:	Alex Kozlov <spam@rm-rf.kiev.ua> [2] [3],
   		swell.k@gmail.com [2],
   		poyopoyo@puripuri.plala.or.jp [4]
   Submitted by:	scf [5],
   		Shuichi KITAGUCHI <ki@hh.iij4u.or.jp> [6]
   Approved by:	delphij (mentor)
 
 Modified:
   head/usr.bin/grep/fastgrep.c
   head/usr.bin/grep/file.c
   head/usr.bin/grep/grep.c
   head/usr.bin/grep/grep.h
   head/usr.bin/grep/queue.c
   head/usr.bin/grep/util.c
 
 Modified: head/usr.bin/grep/fastgrep.c
 ==============================================================================
 --- head/usr.bin/grep/fastgrep.c	Sun Aug 15 22:09:43 2010	(r211363)
 +++ head/usr.bin/grep/fastgrep.c	Sun Aug 15 22:15:04 2010	(r211364)
 @@ -46,8 +46,8 @@ __FBSDID("$FreeBSD$");
  
  #include "grep.h"
  
 -static int	grep_cmp(const unsigned char *, const unsigned char *, size_t);
 -static void	grep_revstr(unsigned char *, int);
 +static inline int	grep_cmp(const unsigned char *, const unsigned char *, size_t);
 +static inline void	grep_revstr(unsigned char *, int);
  
  void
  fgrepcomp(fastgrep_t *fg, const char *pat)
 @@ -273,7 +273,7 @@ grep_search(fastgrep_t *fg, unsigned cha
   * Returns:	i >= 0 on failure (position that it failed)
   *		-1 on success
   */
 -static int
 +static inline int
  grep_cmp(const unsigned char *pat, const unsigned char *data, size_t len)
  {
  	size_t size;
 @@ -318,7 +318,7 @@ grep_cmp(const unsigned char *pat, const
  	return (-1);
  }
  
 -static void
 +static inline void
  grep_revstr(unsigned char *str, int len)
  {
  	int i;
 
 Modified: head/usr.bin/grep/file.c
 ==============================================================================
 --- head/usr.bin/grep/file.c	Sun Aug 15 22:09:43 2010	(r211363)
 +++ head/usr.bin/grep/file.c	Sun Aug 15 22:15:04 2010	(r211364)
 @@ -67,14 +67,14 @@ static int	 bzerr;
   * Returns a single character according to the file type.
   * Returns -1 on failure.
   */
 -int
 +static inline int
  grep_fgetc(struct file *f)
  {
  	unsigned char c;
  
  	switch (filebehave) {
  	case FILE_STDIO:
 -		return (fgetc(f->f));
 +		return (getc_unlocked(f->f));
  	case FILE_GZIP:
  		return (gzgetc(f->gzf));
  	case FILE_BZIP:
 @@ -92,13 +92,13 @@ grep_fgetc(struct file *f)
   * Returns true if the file position is a EOF, returns false
   * otherwise.
   */
 -int
 +static inline int
  grep_feof(struct file *f)
  {
  
  	switch (filebehave) {
  	case FILE_STDIO:
 -		return (feof(f->f));
 +		return (feof_unlocked(f->f));
  	case FILE_GZIP:
  		return (gzeof(f->gzf));
  	case FILE_BZIP:
 @@ -131,6 +131,9 @@ grep_fgetln(struct file *f, size_t *len)
  				st.st_size = MAXBUFSIZ;
  			else if (stat(fname, &st) != 0)
  				err(2, NULL);
 +			/* no need to allocate buffer. */
 +			if (st.st_size == 0)
 +				return (NULL);
  
  			bufsiz = (MAXBUFSIZ > (st.st_size * PREREAD_M)) ?
  			    (st.st_size / 2) : MAXBUFSIZ;
 @@ -142,6 +145,8 @@ grep_fgetln(struct file *f, size_t *len)
  				if (ch == EOF)
  					break;
  				binbuf[i++] = ch;
 +				if ((ch == '\n') && lbflag)
 +					break;
  			}
  
  			f->binary = memchr(binbuf, (filebehave != FILE_GZIP) ?
 @@ -184,11 +189,16 @@ grep_stdin_open(void)
  {
  	struct file *f;
  
 +	/* Processing stdin implies --line-buffered for tail -f to work. */
 +	lbflag = true;
 +
  	snprintf(fname, sizeof fname, "%s", getstr(1));
  
  	f = grep_malloc(sizeof *f);
  
 +	binbuf = NULL;
  	if ((f->f = fdopen(STDIN_FILENO, "r")) != NULL) {
 +		flockfile(f->f);
  		f->stdin = true;
  		return (f);
  	}
 @@ -209,11 +219,14 @@ grep_open(const char *path)
  
  	f = grep_malloc(sizeof *f);
  
 +	binbuf = NULL;
  	f->stdin = false;
  	switch (filebehave) {
  	case FILE_STDIO:
 -		if ((f->f = fopen(path, "r")) != NULL)
 +		if ((f->f = fopen(path, "r")) != NULL) {
 +			flockfile(f->f);
  			return (f);
 +		}
  		break;
  	case FILE_GZIP:
  		if ((f->gzf = gzopen(fname, "r")) != NULL)
 @@ -238,6 +251,7 @@ grep_close(struct file *f)
  
  	switch (filebehave) {
  	case FILE_STDIO:
 +		funlockfile(f->f);
  		fclose(f->f);
  		break;
  	case FILE_GZIP:
 @@ -251,5 +265,4 @@ grep_close(struct file *f)
  	/* Reset read buffer for the file we are closing */
  	binbufptr = NULL;
  	free(binbuf);
 -
  }
 
 Modified: head/usr.bin/grep/grep.c
 ==============================================================================
 --- head/usr.bin/grep/grep.c	Sun Aug 15 22:09:43 2010	(r211363)
 +++ head/usr.bin/grep/grep.c	Sun Aug 15 22:15:04 2010	(r211364)
 @@ -121,8 +121,8 @@ int	 devbehave = DEV_READ;		/* -D: handl
  int	 dirbehave = DIR_READ;		/* -dRr: handling of directories */
  int	 linkbehave = LINK_READ;	/* -OpS: handling of symlinks */
  
 -bool	 dexclude, dinclude;	/* --exclude amd --include */
 -bool	 fexclude, finclude;	/* --exclude-dir and --include-dir */
 +bool	 dexclude, dinclude;	/* --exclude-dir and --include-dir */
 +bool	 fexclude, finclude;	/* --exclude and --include */
  
  enum {
  	BIN_OPT = CHAR_MAX + 1,
 @@ -236,7 +236,8 @@ add_pattern(char *pat, size_t len)
  		--len;
  	/* pat may not be NUL-terminated */
  	pattern[patterns] = grep_malloc(len + 1);
 -	strlcpy(pattern[patterns], pat, len + 1);
 +	memcpy(pattern[patterns], pat, len);
 +	pattern[patterns][len] = '\0';
  	++patterns;
  }
  
 @@ -355,38 +356,33 @@ main(int argc, char *argv[])
  
  	eopts = getenv("GREP_OPTIONS");
  
 -	eargc = 1;
 +	/* support for extra arguments in GREP_OPTIONS */
 +	eargc = 0;
  	if (eopts != NULL) {
  		char *str;
  
 -		for(i = 0; i < strlen(eopts); i++)
 -			if (eopts[i] == ' ')
 +		/* make an estimation of how many extra arguments we have */
 +		for (unsigned int j = 0; j < strlen(eopts); j++)
 +			if (eopts[j] == ' ')
  				eargc++;
  
  		eargv = (char **)grep_malloc(sizeof(char *) * (eargc + 1));
  
 -		str = strtok(eopts, " ");
  		eargc = 0;
 -
 -		while(str != NULL) {
 -			eargv[++eargc] = (char *)grep_malloc(sizeof(char) *
 -			    (strlen(str) + 1));
 -			strlcpy(eargv[eargc], str, strlen(str) + 1);
 -			str = strtok(NULL, " ");
 -		}
 -		eargv[++eargc] = NULL;
 +		/* parse extra arguments */
 +		while ((str = strsep(&eopts, " ")) != NULL)
 +			eargv[eargc++] = grep_strdup(str);
  
  		aargv = (char **)grep_calloc(eargc + argc + 1,
  		    sizeof(char *));
 -		aargv[0] = argv[0];
  
 -		for(i = 1; i < eargc; i++)
 -			aargv[i] = eargv[i];
 -		for(int j = 1; j < argc; j++)
 -			aargv[i++] = argv[j];
 -
 -		aargc = eargc + argc - 1;
 +		aargv[0] = argv[0];
 +		for (i = 0; i < eargc; i++)
 +			aargv[i + 1] = eargv[i];
 +		for (int j = 1; j < argc; j++, i++)
 +			aargv[i + 1] = argv[j];
  
 +		aargc = eargc + argc;
  	} else {
  		aargv = argv;
  		aargc = argc;
 @@ -609,11 +605,11 @@ main(int argc, char *argv[])
  			add_fpattern(optarg, EXCL_PAT);
  			break;
  		case R_DINCLUDE_OPT:
 -			dexclude = true;
 +			dinclude = true;
  			add_dpattern(optarg, INCL_PAT);
  			break;
  		case R_DEXCLUDE_OPT:
 -			dinclude = true;
 +			dexclude = true;
  			add_dpattern(optarg, EXCL_PAT);
  			break;
  		case HELP_OPT:
 @@ -685,12 +681,15 @@ main(int argc, char *argv[])
  
  	if (dirbehave == DIR_RECURSE)
  		c = grep_tree(aargv);
 -	else 
 +	else {
 +		if (aargc == 1)
 +			hflag = true;
  		for (c = 0; aargc--; ++aargv) {
  			if ((finclude || fexclude) && !file_matching(*aargv))
  				continue;
  			c+= procfile(*aargv);
  		}
 +	}
  
  #ifndef WITHOUT_NLS
  	catclose(catalog);
 
 Modified: head/usr.bin/grep/grep.h
 ==============================================================================
 --- head/usr.bin/grep/grep.h	Sun Aug 15 22:09:43 2010	(r211363)
 +++ head/usr.bin/grep/grep.h	Sun Aug 15 22:15:04 2010	(r211364)
 @@ -115,7 +115,7 @@ extern int	 cflags, eflags;
  extern bool	 Eflag, Fflag, Gflag, Hflag, Lflag,
  		 bflag, cflag, hflag, iflag, lflag, mflag, nflag, oflag,
  		 qflag, sflag, vflag, wflag, xflag;
 -extern bool	 dexclude, dinclude, fexclude, finclude, nullflag;
 +extern bool	 dexclude, dinclude, fexclude, finclude, lbflag, nullflag;
  extern unsigned long long Aflag, Bflag, mcount;
  extern char	*label;
  extern const char *color;
 @@ -134,7 +134,6 @@ extern fastgrep_t *fg_pattern;
  extern char	 re_error[RE_ERROR_BUF + 1];	/* Seems big enough */
  
  /* util.c */
 -bool	 dir_matching(const char *dname);
  bool	 file_matching(const char *fname);
  int	 procfile(const char *fn);
  int	 grep_tree(char **argv);
 @@ -153,8 +152,6 @@ void	 clearqueue(void);
  void		 grep_close(struct file *f);
  struct file	*grep_stdin_open(void);
  struct file	*grep_open(const char *path);
 -int		 grep_feof(struct file *f);
 -int		 grep_fgetc(struct file *f);
  char		*grep_fgetln(struct file *f, size_t *len);
  
  /* fastgrep.c */
 
 Modified: head/usr.bin/grep/queue.c
 ==============================================================================
 --- head/usr.bin/grep/queue.c	Sun Aug 15 22:09:43 2010	(r211363)
 +++ head/usr.bin/grep/queue.c	Sun Aug 15 22:15:04 2010	(r211364)
 @@ -60,7 +60,7 @@ enqueue(struct str *x)
  	item->data.len = x->len;
  	item->data.line_no = x->line_no;
  	item->data.off = x->off;
 -	strcpy(item->data.dat, x->dat);
 +	memcpy(item->data.dat, x->dat, x->len);
  	item->data.file = x->file;
  
  	STAILQ_INSERT_TAIL(&queue, item, list);
 
 Modified: head/usr.bin/grep/util.c
 ==============================================================================
 --- head/usr.bin/grep/util.c	Sun Aug 15 22:09:43 2010	(r211363)
 +++ head/usr.bin/grep/util.c	Sun Aug 15 22:15:04 2010	(r211364)
 @@ -72,7 +72,7 @@ file_matching(const char *fname)
  	return (ret);
  }
  
 -bool
 +static inline bool
  dir_matching(const char *dname)
  {
  	bool ret;
 @@ -144,9 +144,10 @@ grep_tree(char **argv)
  			if (dexclude || dinclude) {
  				if ((d = strrchr(p->fts_path, '/')) != NULL) {
  					dir = grep_malloc(sizeof(char) *
 -					    (d - p->fts_path + 2));
 -					strlcpy(dir, p->fts_path,
  					    (d - p->fts_path + 1));
 +					memcpy(dir, p->fts_path,
 +					    d - p->fts_path);
 +					dir[d - p->fts_path] = '\0';
  				}
  				ok = dir_matching(dir);
  				free(dir);
 @@ -276,7 +277,7 @@ procfile(const char *fn)
   * matches.  The matching lines are passed to printline() to display the
   * appropriate output.
   */
 -static int
 +static inline int
  procline(struct str *l, int nottext)
  {
  	regmatch_t matches[MAX_LINE_MATCHES];
 @@ -317,30 +318,20 @@ procline(struct str *l, int nottext)
  					    (size_t)pmatch.rm_eo != l->len)
  						r = REG_NOMATCH;
  				/* Check for whole word match */
 -				if (r == 0 && wflag && pmatch.rm_so != 0 &&
 -				    (size_t)pmatch.rm_eo != l->len) {
 -					wchar_t *wbegin;
 -					wint_t wend;
 -					size_t size;
 +				if (r == 0 && wflag && pmatch.rm_so != 0) {
 +					wint_t wbegin, wend;
  
 -					size = mbstowcs(NULL, l->dat,
 -					    pmatch.rm_so);
 -
 -					if (size == ((size_t) - 1))
 +					wbegin = wend = L' ';
 +					if (pmatch.rm_so != 0 &&
 +					    sscanf(&l->dat[pmatch.rm_so - 1],
 +					    "%lc", &wbegin) != 1)
 +						r = REG_NOMATCH;
 +					else if ((size_t)pmatch.rm_eo != l->len &&
 +					    sscanf(&l->dat[pmatch.rm_eo],
 +					    "%lc", &wend) != 1)
 +						r = REG_NOMATCH;
 +					else if (iswword(wbegin) || iswword(wend))
  						r = REG_NOMATCH;
 -					else {
 -						wbegin = grep_malloc(size);
 -						if (mbstowcs(wbegin, l->dat,
 -						    pmatch.rm_so) == ((size_t) - 1))
 -							r = REG_NOMATCH;
 -						else if (sscanf(&l->dat[pmatch.rm_eo],
 -						    "%lc", &wend) != 1)
 -							r = REG_NOMATCH;
 -						else if (iswword(wbegin[wcslen(wbegin)]) ||
 -						    iswword(wend))
 -							r = REG_NOMATCH;
 -						free(wbegin);
 -					}
  				}
  				if (r == 0) {
  					if (m == 0)
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->closed 
State-Changed-By: gabor 
State-Changed-When: Sun Aug 15 22:21:21 UTC 2010 
State-Changed-Why:  
Committed, thanks. I used another approach for the first problem and checking 
the third unfixed problem is still on my TODO list. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=149425 
>Unformatted:
