From nobody@FreeBSD.org  Mon Aug  6 12:29:36 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id E47C1106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  6 Aug 2012 12:29:36 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id B64DF8FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  6 Aug 2012 12:29:36 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q76CTahf025291
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 6 Aug 2012 12:29:36 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q76CTaTM025290;
	Mon, 6 Aug 2012 12:29:36 GMT
	(envelope-from nobody)
Message-Id: <201208061229.q76CTaTM025290@red.freebsd.org>
Date: Mon, 6 Aug 2012 12:29:36 GMT
From: Stephen McKay <mckay@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: file(1) fails to detect troff (and other file types)
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         170415
>Category:       bin
>Synopsis:       [patch] file(1) fails to detect troff (and other file types)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    mckay
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 06 12:30:01 UTC 2012
>Closed-Date:    
>Last-Modified:  Fri Sep  7 00:30:01 UTC 2012
>Originator:     Stephen McKay
>Release:        9.1-beta1
>Organization:
>Environment:
>Description:
A bug was introduced into file(1) 5.11 when it was imported into the FreeBSD base.  Detection of troff files (and others) has been broken.  This is because "search" and "regex" patterns are no longer correctly processed by "mkmagic", the program that compiles /usr/share/misc/magic.mgc from the individual pattern files.
>How-To-Repeat:
$ echo '.tr' > foo
$ file foo
foo: ASCII text

Expected behaviour is:

$ file foo
foo: troff or preprocessor input, ASCII text

>Fix:
Apply the (hopefully) attached patch against 10-current.

Patch attached with submission follows:

Index: contrib/file/apprentice.c
===================================================================
RCS file: /cvs/src/contrib/file/apprentice.c,v
retrieving revision 1.7
diff -u -r1.7 apprentice.c
--- contrib/file/apprentice.c	19 Apr 2012 03:20:13 -0000	1.7
+++ contrib/file/apprentice.c	6 Aug 2012 12:01:01 -0000
@@ -648,7 +648,6 @@
 		break;
 	case FILE_REGEX:
 	case FILE_SEARCH:
-#ifndef COMPILE_ONLY
 		/* Check for override */
 		if (mstart->str_flags & STRING_BINTEST)
 			mstart->flag |= BINTEST;
@@ -664,7 +663,6 @@
 			mstart->flag |= BINTEST;
 		else
 			mstart->flag |= TEXTTEST;
-#endif
 		break;
 	case FILE_DEFAULT:
 		/* can't deduce anything; we shouldn't see this at the
Index: lib/libmagic/Makefile
===================================================================
RCS file: /cvs/src/lib/libmagic/Makefile,v
retrieving revision 1.18
diff -u -r1.18 Makefile
--- lib/libmagic/Makefile	19 May 2012 02:30:10 -0000	1.18
+++ lib/libmagic/Makefile	6 Aug 2012 12:02:03 -0000
@@ -39,7 +39,7 @@
 
 CLEANFILES+=	mkmagic
 build-tools: mkmagic
-mkmagic: apprentice.c funcs.c getline.c magic.c print.c
+mkmagic: apprentice.c encoding.c funcs.c getline.c magic.c print.c
 	${CC} ${CFLAGS} -DCOMPILE_ONLY -DHOSTPROG ${LDFLAGS} \
 	    -o ${.TARGET} ${.ALLSRC}
 


>Release-Note:
>Audit-Trail:

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/170415: commit references a PR
Date: Fri, 17 Aug 2012 02:27:31 +0000 (UTC)

 Author: mckay
 Date: Fri Aug 17 02:27:17 2012
 New Revision: 239348
 URL: http://svn.freebsd.org/changeset/base/239348
 
 Log:
   Correct a regression introduced during the import of file(1) 5.11.
   
   Magic tests containing "search" or "regex" directives were incorrectly
   compiled by "mkmagic" and were effectively ignored.  This caused troff
   files (for example) to be detected as simply "ASCII text" instead of
   as "troff or preprocessor input, ASCII text".
   
   PR:		bin/170415
   Approved by:	consensus on developers@
   MFC after:	3 days
 
 Modified:
   head/contrib/file/apprentice.c
   head/lib/libmagic/Makefile
 
 Modified: head/contrib/file/apprentice.c
 ==============================================================================
 --- head/contrib/file/apprentice.c	Fri Aug 17 02:26:31 2012	(r239347)
 +++ head/contrib/file/apprentice.c	Fri Aug 17 02:27:17 2012	(r239348)
 @@ -648,7 +648,6 @@ set_test_type(struct magic *mstart, stru
  		break;
  	case FILE_REGEX:
  	case FILE_SEARCH:
 -#ifndef COMPILE_ONLY
  		/* Check for override */
  		if (mstart->str_flags & STRING_BINTEST)
  			mstart->flag |= BINTEST;
 @@ -664,7 +663,6 @@ set_test_type(struct magic *mstart, stru
  			mstart->flag |= BINTEST;
  		else
  			mstart->flag |= TEXTTEST;
 -#endif
  		break;
  	case FILE_DEFAULT:
  		/* can't deduce anything; we shouldn't see this at the
 
 Modified: head/lib/libmagic/Makefile
 ==============================================================================
 --- head/lib/libmagic/Makefile	Fri Aug 17 02:26:31 2012	(r239347)
 +++ head/lib/libmagic/Makefile	Fri Aug 17 02:27:17 2012	(r239348)
 @@ -39,7 +39,7 @@ magic.mgc: mkmagic magic
  
  CLEANFILES+=	mkmagic
  build-tools: mkmagic
 -mkmagic: apprentice.c funcs.c getline.c magic.c print.c
 +mkmagic: apprentice.c encoding.c funcs.c getline.c magic.c print.c
  	${CC} ${CFLAGS} -DCOMPILE_ONLY -DHOSTPROG ${LDFLAGS} \
  	    -o ${.TARGET} ${.ALLSRC}
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->feedback 
State-Changed-By: obrien 
State-Changed-When: Fri Aug 17 15:22:06 UTC 2012 
State-Changed-Why:  
Does this happen with stock file(1) 5.11? 


Responsible-Changed-From-To: freebsd-bugs->mckay 
Responsible-Changed-By: obrien 
Responsible-Changed-When: Fri Aug 17 15:22:06 UTC 2012 
Responsible-Changed-Why:  
Does this happen with stock file(1) 5.11? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=170415 

From: mckay@FreeBSD.org
To: obrien@FreeBSD.org
Cc: mckay@FreeBSD.org, bug-followup@FreeBSD.org
Subject: Re: bin/170415: [patch] file(1) fails to detect troff (and other file types)
Date: Sat, 18 Aug 2012 12:19:14 +1000

 On Friday, 17th August 2012, obrien@FreeBSD.org wrote:
 
 >Does this happen with stock file(1) 5.11?
 
 No.
 
 Before patching, "/usr/bin/file -C" and ports "file -C" would generate
 the same compiled output given the same magic tests as input.  However,
 mkmagic would produce different output.  "file -l" on that output
 would show missing patterns when compared to "file -l" on a working
 compiled magic file.
 
 The faulty mkmagic was generating tests with an incorrect "flag"
 value of 0 (which once meant "text") when the correct value is 0x40
 (for text) or 0x20 (for binary).  Note that the value of TEXTTEST
 changed from 0 to 0x40 in the 5.11 upgrade commit.
 
 There are binary tests too, so we can't simply assign TEXTTEST to them
 all like previous versions (this may have been a bug in itself but
 I've not verified it).  It looks as if the call to file_looks_utf8()
 is essential.
 
 In short, the patch makes mkmagic and file itself match the behaviour
 of file 5.11 from ports.  Hence I believe that it is correct and also
 safe for 9.1-R.
 
 Stephen.

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/170415: commit references a PR
Date: Thu,  6 Sep 2012 06:18:00 +0000 (UTC)

 Author: mckay
 Date: Thu Sep  6 06:17:39 2012
 New Revision: 240157
 URL: http://svn.freebsd.org/changeset/base/240157
 
 Log:
   MFC r235638,r239348:
   - Work around failure to compile on FreeBSD 7.x machines.
   - Correct a regression introduced during the import of file(1) 5.11.
   
   PR:             bin/170415
   Reviewed by:	obrien@
 
 Modified:
   stable/9/contrib/file/apprentice.c
   stable/9/lib/libmagic/Makefile
   stable/9/lib/libmagic/config.h
 Directory Properties:
   stable/9/   (props changed)
   stable/9/contrib/file/   (props changed)
   stable/9/lib/   (props changed)
 
 Modified: stable/9/contrib/file/apprentice.c
 ==============================================================================
 --- stable/9/contrib/file/apprentice.c	Thu Sep  6 03:19:48 2012	(r240156)
 +++ stable/9/contrib/file/apprentice.c	Thu Sep  6 06:17:39 2012	(r240157)
 @@ -648,7 +648,6 @@ set_test_type(struct magic *mstart, stru
  		break;
  	case FILE_REGEX:
  	case FILE_SEARCH:
 -#ifndef COMPILE_ONLY
  		/* Check for override */
  		if (mstart->str_flags & STRING_BINTEST)
  			mstart->flag |= BINTEST;
 @@ -664,7 +663,6 @@ set_test_type(struct magic *mstart, stru
  			mstart->flag |= BINTEST;
  		else
  			mstart->flag |= TEXTTEST;
 -#endif
  		break;
  	case FILE_DEFAULT:
  		/* can't deduce anything; we shouldn't see this at the
 
 Modified: stable/9/lib/libmagic/Makefile
 ==============================================================================
 --- stable/9/lib/libmagic/Makefile	Thu Sep  6 03:19:48 2012	(r240156)
 +++ stable/9/lib/libmagic/Makefile	Thu Sep  6 06:17:39 2012	(r240157)
 @@ -40,8 +40,9 @@ magic.mgc: mkmagic magic
  
  CLEANFILES+=	mkmagic
  build-tools: mkmagic
 -mkmagic: apprentice.c funcs.c magic.c print.c
 -	${CC} ${CFLAGS} -DCOMPILE_ONLY ${LDFLAGS} -o ${.TARGET} ${.ALLSRC}
 +mkmagic: apprentice.c encoding.c funcs.c getline.c magic.c print.c
 +	${CC} ${CFLAGS} -DCOMPILE_ONLY -DHOSTPROG ${LDFLAGS} \
 +	    -o ${.TARGET} ${.ALLSRC}
  
  FILEVER!= awk '$$1 == "\#define" && $$2 == "VERSION" { print $$3; exit }' \
  			${.CURDIR}/config.h
 
 Modified: stable/9/lib/libmagic/config.h
 ==============================================================================
 --- stable/9/lib/libmagic/config.h	Thu Sep  6 03:19:48 2012	(r240156)
 +++ stable/9/lib/libmagic/config.h	Thu Sep  6 06:17:39 2012	(r240157)
 @@ -39,7 +39,9 @@
  #define HAVE_FSEEKO 1
  
  /* Define to 1 if you have the `getline' function. */
 +#ifndef HOSTPROG
  #define HAVE_GETLINE 1
 +#endif
  
  /* Define to 1 if you have the <getopt.h> header file. */
  #define HAVE_GETOPT_H 1
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/170415: commit references a PR
Date: Fri,  7 Sep 2012 00:20:56 +0000 (UTC)

 Author: mckay
 Date: Fri Sep  7 00:20:46 2012
 New Revision: 240179
 URL: http://svn.freebsd.org/changeset/base/240179
 
 Log:
   MFS r240157 (MFC r235638,r239348):
   - Work around failure to compile on FreeBSD 7.x machines.
   - Correct a regression introduced during the import of file(1) 5.11.
   
   PR:		170415
   Reviewed by:	obrien
   Approved by:	re (kib)
 
 Modified:
   releng/9.1/contrib/file/apprentice.c
   releng/9.1/lib/libmagic/Makefile
   releng/9.1/lib/libmagic/config.h
 Directory Properties:
   releng/9.1/   (props changed)
   releng/9.1/contrib/file/   (props changed)
   releng/9.1/lib/   (props changed)
 
 Modified: releng/9.1/contrib/file/apprentice.c
 ==============================================================================
 --- releng/9.1/contrib/file/apprentice.c	Thu Sep  6 20:59:49 2012	(r240178)
 +++ releng/9.1/contrib/file/apprentice.c	Fri Sep  7 00:20:46 2012	(r240179)
 @@ -648,7 +648,6 @@ set_test_type(struct magic *mstart, stru
  		break;
  	case FILE_REGEX:
  	case FILE_SEARCH:
 -#ifndef COMPILE_ONLY
  		/* Check for override */
  		if (mstart->str_flags & STRING_BINTEST)
  			mstart->flag |= BINTEST;
 @@ -664,7 +663,6 @@ set_test_type(struct magic *mstart, stru
  			mstart->flag |= BINTEST;
  		else
  			mstart->flag |= TEXTTEST;
 -#endif
  		break;
  	case FILE_DEFAULT:
  		/* can't deduce anything; we shouldn't see this at the
 
 Modified: releng/9.1/lib/libmagic/Makefile
 ==============================================================================
 --- releng/9.1/lib/libmagic/Makefile	Thu Sep  6 20:59:49 2012	(r240178)
 +++ releng/9.1/lib/libmagic/Makefile	Fri Sep  7 00:20:46 2012	(r240179)
 @@ -40,8 +40,9 @@ magic.mgc: mkmagic magic
  
  CLEANFILES+=	mkmagic
  build-tools: mkmagic
 -mkmagic: apprentice.c funcs.c magic.c print.c
 -	${CC} ${CFLAGS} -DCOMPILE_ONLY ${LDFLAGS} -o ${.TARGET} ${.ALLSRC}
 +mkmagic: apprentice.c encoding.c funcs.c getline.c magic.c print.c
 +	${CC} ${CFLAGS} -DCOMPILE_ONLY -DHOSTPROG ${LDFLAGS} \
 +	    -o ${.TARGET} ${.ALLSRC}
  
  FILEVER!= awk '$$1 == "\#define" && $$2 == "VERSION" { print $$3; exit }' \
  			${.CURDIR}/config.h
 
 Modified: releng/9.1/lib/libmagic/config.h
 ==============================================================================
 --- releng/9.1/lib/libmagic/config.h	Thu Sep  6 20:59:49 2012	(r240178)
 +++ releng/9.1/lib/libmagic/config.h	Fri Sep  7 00:20:46 2012	(r240179)
 @@ -39,7 +39,9 @@
  #define HAVE_FSEEKO 1
  
  /* Define to 1 if you have the `getline' function. */
 +#ifndef HOSTPROG
  #define HAVE_GETLINE 1
 +#endif
  
  /* Define to 1 if you have the <getopt.h> header file. */
  #define HAVE_GETOPT_H 1
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
>Unformatted:
