From rea-fbsd@codelabs.ru  Fri Nov 27 14:00:48 2009
Return-Path: <rea-fbsd@codelabs.ru>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id CD5AF1065670;
	Fri, 27 Nov 2009 14:00:48 +0000 (UTC)
	(envelope-from rea-fbsd@codelabs.ru)
Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45])
	by mx1.freebsd.org (Postfix) with ESMTP id 5C8CE8FC12;
	Fri, 27 Nov 2009 14:00:48 +0000 (UTC)
Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25])
	by 0.mx.codelabs.ru with esmtps (TLSv1:CAMELLIA256-SHA:256)
	id 1NE1NP-000L3b-F9; Fri, 27 Nov 2009 17:00:47 +0300
Message-Id: <20091127140047.6A690DA81A@void.codelabs.ru>
Date: Fri, 27 Nov 2009 17:00:47 +0300 (MSK)
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Reply-To: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc: nectar@freebsd.org, miwi@freebsd.org
Subject: [patch] security/vuxml: fix and extend files/newentry.sh
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         140939
>Category:       ports
>Synopsis:       [patch] security/vuxml: fix and extend files/newentry.sh
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-ports-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Nov 27 14:10:01 UTC 2009
>Closed-Date:    Thu May 03 10:10:31 UTC 2012
>Last-Modified:  Thu May 03 10:10:31 UTC 2012
>Originator:     Eygene Ryabinkin
>Release:        FreeBSD 8.0-RC1 amd64
>Organization:
Code Labs
>Environment:

System: FreeBSD 8.0-RC1 amd64

>Description:

There are two problems in the current implementation of newentry.sh:
1. cleanup handler is installed after the temporary file is created,
   so there is a small window during which the script can receive the
   signal and temporary file won't be deleted;

2. cleanup handler isn't called when external editor is executed, so
   temporary file is left in its place.

Moreover, since patches to the vuln.xml tend to expire quickly: once the
new entry is added to the vuln.xml, the patches that are created for the
previous versions won't apply anymore.  Since vuln.xml changes rather
often, it is desirable to find a way to apply these patches
automatically and newentry.sh can be extended to handle it.

miwi@ sometimes asks me to redo my diffs for the VuXML entries since
they went out-of-date, so this functionality should help in such cases.

>How-To-Repeat:

Point 1: look at the code:
-----
tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1
doclean="yes"
cleanup() {
  if [ "${doclean}" = "yes" ]; then
    rm -f "${tmp}"
  fi
}
trap cleanup EXIT 1 2 13 15
-----
Compare with
-----
doclean="yes"
cleanup() {
  if [ "${doclean}" = "yes" ]; then
    [ -n "${tmp}" ] && rm -f "${tmp}"
  fi
}
trap cleanup EXIT 1 2 13 15
tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1
-----

Point 2: try to invoke 'make newentry' and watch for file(s)
vuxml.<anything> in the /tmp or $TMPDIR.

>Fix:

The following patch fixes the point 1:
--- 0001-install-cleanup-handler-earlier.diff begins here ---
From 4375203e1602acacfc0ef230fd2cabd42d142463 Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Fri, 27 Nov 2009 12:35:17 +0300

There is a gap between 'mktemp' and installation of the trap handler.

We can initialize 'doclean' before trapping signals and additionally
check if ${tmp} is empty inside the cleanup handler.  This will
practically eliminate the gap where temporary file won't be removed
if the script will receive some signal.

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 security/vuxml/files/newentry.sh |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/security/vuxml/files/newentry.sh b/security/vuxml/files/newentry.sh
index b6337da..d08bbc8 100644
--- a/security/vuxml/files/newentry.sh
+++ b/security/vuxml/files/newentry.sh
@@ -6,14 +6,14 @@ if [ -z "${vuxml_file}" ]; then
   exit 1
 fi
 
-tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1
-doclean="yes"
 cleanup() {
   if [ "${doclean}" = "yes" ]; then
-    rm -f "${tmp}"
+    [ -n "${tmp}" ] && rm -f "${tmp}"
   fi
 }
+doclean="yes"
 trap cleanup EXIT 1 2 13 15
+tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1
 
 vid="`uuidgen | tr '[:upper:]' '[:lower:]'`"
 [ -z "$vid" ] && exit 1
-- 
1.6.5.3
--- 0001-install-cleanup-handler-earlier.diff ends here ---

The following patch fixes the point 2:
--- 0002-cleanup-before-spawning-editor.diff begins here ---
From b5ebab7ccd3ac8d484bbe6bc037b94601dc0c7be Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Fri, 27 Nov 2009 12:52:07 +0300
Subject: [PATCH 2/4] security/vuxml: newentry.sh: cleanup before executing editor

When we do 'exec EDITOR', cleanup handler isn't called, so the temporary
file is left in place (/tmp or $TMPDIR).  We fix it by explicitely
calling cleanup() before exec'ing editor.

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 security/vuxml/files/newentry.sh |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/security/vuxml/files/newentry.sh b/security/vuxml/files/newentry.sh
index d08bbc8..f84d0ea 100644
--- a/security/vuxml/files/newentry.sh
+++ b/security/vuxml/files/newentry.sh
@@ -51,6 +51,7 @@ awk '/^[[:space:]]+<vuln /,/^NONE$/ { print }' \
   "${vuxml_file}" >> "${tmp}" || exit 1
 
 if cp "${tmp}" "${vuxml_file}"; then
+  cleanup
   exec ${EDITOR:-vi} "${vuxml_file}"
 else
   doclean="no"
-- 
1.6.5.3
--- 0002-cleanup-before-spawning-editor.diff ends here ---

Two next patches implement the functionality to add existing VuXML
entries or single-entry patches for vuln.xml automatically.

First patch refactors newentry.sh to place currently spreaded code for
creation of a new entry to a single function.

Second patch implements new functionality.

--- 0003-refactor-newentry.sh.diff begins here ---
From 277a3c98c2449fc9c965ba3d8a46a6e78d4db5f0 Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Fri, 27 Nov 2009 13:11:58 +0300
Subject: [PATCH 3/4] security/vuxml: refactor newentry.sh

Make new procedure, newentry(), that creates new VuXML entry,
thus incapsulating the needed functionality into the single
function.  Main code just calls it when needed.

Also make usage() to be stand-alone function.

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 security/vuxml/files/newentry.sh |   54 ++++++++++++++++++++++++--------------
 1 files changed, 34 insertions(+), 20 deletions(-)

diff --git a/security/vuxml/files/newentry.sh b/security/vuxml/files/newentry.sh
index f84d0ea..f2b9e61 100644
--- a/security/vuxml/files/newentry.sh
+++ b/security/vuxml/files/newentry.sh
@@ -1,27 +1,17 @@
 #! /bin/sh
-vuxml_file="$1"
-if [ -z "${vuxml_file}" ]; then
-  exec >&2
-  echo "Usage: newentry.sh /path/to/vuxml/document"
-  exit 1
-fi
 
-cleanup() {
-  if [ "${doclean}" = "yes" ]; then
-    [ -n "${tmp}" ] && rm -f "${tmp}"
-  fi
-}
-doclean="yes"
-trap cleanup EXIT 1 2 13 15
-tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1
+new_entry () {
+  local vid discovery entry
 
-vid="`uuidgen | tr '[:upper:]' '[:lower:]'`"
-[ -z "$vid" ] && exit 1
-discovery="`date -u '+%Y-%m'`-FIXME" || exit 1
-entry="`date -u '+%Y-%m-%d'`" || exit 1
+  vid="`uuidgen | tr '[:upper:]' '[:lower:]'`"
+  discovery="`date -u '+%Y-%m'`-FIXME" || exit 1
+  entry="`date -u '+%Y-%m-%d'`" || exit 1
+  if [ -z "$vid" -o -z "$discovery" -o -z "$entry" ]; then
+    false
+    return
+  fi
 
-awk '/^<\?/,/^<vuxml/ { print }' "${vuxml_file}" >> "${tmp}" || exit 1
-cat << EOF >> "${tmp}" || exit 1
+  cat << EOF
   <vuln vid="${vid}">
     <topic> -- </topic>
     <affects>
@@ -47,6 +37,30 @@ cat << EOF >> "${tmp}" || exit 1
   </vuln>
 
 EOF
+}
+
+usage () {
+	echo "Usage: newentry.sh /path/to/vuxml/document" >&2
+}
+
+cleanup() {
+  if [ "${doclean}" = "yes" ]; then
+    [ -n "${tmp}" ] && rm -f "${tmp}"
+  fi
+}
+
+vuxml_file="$1"
+if [ -z "${vuxml_file}" ]; then
+  usage
+  exit 1
+fi
+
+doclean="yes"
+trap cleanup EXIT 1 2 13 15
+tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1
+
+awk '/^<\?/,/^<vuxml/ { print }' "${vuxml_file}" >> "${tmp}" || exit 1
+new_entry >> "${tmp}" || exit 1
 awk '/^[[:space:]]+<vuln /,/^NONE$/ { print }' \
   "${vuxml_file}" >> "${tmp}" || exit 1
 
-- 
1.6.5.3
--- 0003-refactor-newentry.sh.diff ends here ---

--- 0004-teach-to-accept-pre-made-entries.diff begins here ---
From 94dd35b3f635ef665426c4b339fe69982a8fb392 Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Fri, 27 Nov 2009 15:06:09 +0300
Subject: [PATCH 4/4] security/vuxml: teach newentry.sh to accept existing entries

Now newentry.sh can add not only generic template for the new entry,
but existing entries as well.  Both plain entries and patches are
accepted.

This mode was created mainly due to the fact that vuln.xml changes
frequently and if one sends patch for the obsolete version of vuln.xml,
the patch won't be applied cleanly and manual work is needed.  Script
automates such insertions.

Moreover, plain VuXML entries that are not patches, but are just pieces
of vuln.xml can be sent via PRs -- they are inserted automatically too.

'make addentry E=/path/to/entry/file' will do the work -- Makefile was
extended to handle the new target.

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 security/vuxml/Makefile          |    6 ++
 security/vuxml/files/newentry.sh |  115 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 119 insertions(+), 2 deletions(-)

diff --git a/security/vuxml/Makefile b/security/vuxml/Makefile
index d347c39..8aa177e 100644
--- a/security/vuxml/Makefile
+++ b/security/vuxml/Makefile
@@ -83,4 +83,10 @@ tidy:
 newentry:
 	@${SH} ${FILESDIR}/newentry.sh "${VUXML_FILE}"
 
+addentry:
+.if empty(E)
+.error "Set make variable E to the location of the VuXML entry to be added."
+.endif
+	@${SH} ${FILESDIR}/newentry.sh -i "${E}" "${VUXML_FILE}"
+
 .include <bsd.port.mk>
diff --git a/security/vuxml/files/newentry.sh b/security/vuxml/files/newentry.sh
index f2b9e61..d546a79 100644
--- a/security/vuxml/files/newentry.sh
+++ b/security/vuxml/files/newentry.sh
@@ -1,5 +1,87 @@
 #! /bin/sh
 
+# Patch detection strategy: we are looking for the three successive
+# lines starting with "--- ", "+++ " and "@@ [+-]".  The next line
+# should start with space.  This catches unified diffs.
+#
+# The number of the line where the patch contents start (the line with
+# first '@@ [+-]') is printed to the standard output if the patch was
+# really found.
+is_patch () {
+  awk '
+BEGIN { seen_minus = 0; seen_plus = 0; seen_at = 0; diff = 0 }
+/^--- / { seen_minus = 1; seen_plus = 0; seen_at = 0; diff = 0; next; }
+seen_minus == 1 && /^\+\+\+ / { seen_plus = 1; next; }
+seen_plus == 1 && /^@@ [+-]/ { seen_at = 1; next; }
+seen_at == 1 && /^ / { print FNR; diff = 1; exit(0); }
+{ seen_minus = 0; seen_plus = 0; seen_at = 0; }
+END { exit (diff != 1); }
+' "$1"
+}
+
+# Checks if the patch contains one continuous hunk that contains
+# only additions.
+# Arguments:
+# $1 -- patch filename;
+# $2 -- line where patch contents start.
+is_continuous_patch () {
+
+  if [ -z "$1" -o -z "$2" ]; then
+    false
+    return
+  fi
+  awk -v start="$2" '
+BEGIN { blks = 0; in_blk = 0; }
+FNR < start { next; }
+/^-/ { exit(1); }
+in_blk == 0 && /^+/ { in_blk = 1; blks++; if (blks > 1) { exit(1); } next; }
+in_blk == 1 && !/^+/ { in_blk = 0; next; }
+END { exit(blks != 1); };
+' "$1"
+}
+
+# $1 is the file with the existing VuXML entry; it is guaranteed
+# to be a real file.
+#
+# We should check if the supplied file contains a patch that has
+# one continuous entry that is meant to be added -- only such files
+# are treated as patches that add VuXML entries.  If patch has more
+# than one chunk or it deletes anything, we won't accept it.
+#
+# For the non-patch files, we will add them "as is".
+
+existing_entry () {
+  local patch_start
+
+  if [ -z "$1" ]; then
+    false
+    return
+  fi
+
+  # Plain file?  Just cat it!
+  patch_start=`is_patch "$1"`
+  if [ "$?" != 0 ]; then
+    cat "$1"
+    echo
+    return
+  fi
+
+  if is_continuous_patch "$1" "$patch_start"; then
+    awk -v start="$patch_start" '
+FNR < start { next; }
+{ print; }
+' "$1" | grep '^\+' | sed -e's/^\+//'
+  else
+    cat >&2 << EOF
+Input VuXML entry looks like a patch, but it doesn't contain a single
+hunk that I can use as the entry body.  Human insight is needed, may
+be it is better to merge the entry by hand.
+EOF
+    false
+    return
+  fi
+}
+
 new_entry () {
   local vid discovery entry
 
@@ -40,7 +122,7 @@ EOF
 }
 
 usage () {
-	echo "Usage: newentry.sh /path/to/vuxml/document" >&2
+	echo "Usage: newentry.sh [-i /existing/entry] /path/to/vuxml/document" >&2
 }
 
 cleanup() {
@@ -49,6 +131,31 @@ cleanup() {
   fi
 }
 
+while getopts "i:h" opt; do
+  case "$opt" in
+  h)
+    usage
+    exit 0
+    ;;
+  i)
+    if [ ! -e "$OPTARG" ]; then
+      echo "File '$OPTARG' does not exist." >&2
+      exit 1
+    fi
+    entry="$OPTARG"
+    ;;
+  ?)
+    usage
+    exit 1
+    ;;
+  -)
+    break
+    ;;
+  esac
+done
+
+shift $(($OPTIND - 1))
+
 vuxml_file="$1"
 if [ -z "${vuxml_file}" ]; then
   usage
@@ -60,7 +167,11 @@ trap cleanup EXIT 1 2 13 15
 tmp="`mktemp ${TMPDIR:-/tmp}/vuxml.XXXXXXXXXX`" || exit 1
 
 awk '/^<\?/,/^<vuxml/ { print }' "${vuxml_file}" >> "${tmp}" || exit 1
-new_entry >> "${tmp}" || exit 1
+if [ -n "$entry" ]; then
+	existing_entry "$entry" >> "${tmp}" || exit 1
+else
+	new_entry >> "${tmp}" || exit 1
+fi
 awk '/^[[:space:]]+<vuln /,/^NONE$/ { print }' \
   "${vuxml_file}" >> "${tmp}" || exit 1
 
-- 
1.6.5.3
--- 0004-teach-to-accept-pre-made-entries.diff ends here ---

To ease the things, patches are mirrored at the following locations:
  http://codelabs.ru/fbsd/patches/vuxml/newentry.sh/0001-install-cleanup-handler-earlier.diff
  http://codelabs.ru/fbsd/patches/vuxml/newentry.sh/0002-cleanup-before-spawning-editor.diff
  http://codelabs.ru/fbsd/patches/vuxml/newentry.sh/0003-refactor-newentry.sh.diff
  http://codelabs.ru/fbsd/patches/vuxml/newentry.sh/0004-teach-to-accept-pre-made-entries.diff
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-ports-bugs->secteam 
Responsible-Changed-By: edwin 
Responsible-Changed-When: Fri Nov 27 14:10:14 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer (via the GNATS Auto Assign Tool) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140939 
Responsible-Changed-From-To: secteam->simon 
Responsible-Changed-By: remko 
Responsible-Changed-When: Tue Dec 28 08:37:13 UTC 2010 
Responsible-Changed-Why:  
Simon, 

Can you have a look at this please, and allow me to commit 
this, or commit it yourself if this looks reasonable... 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140939 
Responsible-Changed-From-To: simon->miwi 
Responsible-Changed-By: miwi 
Responsible-Changed-When: Sat Feb 5 09:02:08 UTC 2011 
Responsible-Changed-Why:  
I'll take it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140939 
State-Changed-From-To: open->feedback 
State-Changed-By: miwi 
State-Changed-When: Sat Feb 5 09:02:33 UTC 2011 
State-Changed-Why:  
0004 break the port build, also it breaks the "make newentry", it should 
work in the way wish it currently do. please check it again. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140939 

From: Eygene Ryabinkin <rea@freebsd.org>
To: FreeBSD GNATS followup <bug-followup@freebsd.org>
Cc:  
Subject: Re: ports/140939: [patch] security/vuxml: fix and extend
 files/newentry.sh
Date: Sat, 5 Feb 2011 15:11:58 +0300

 Will take a look, thanks!
 -- 
 Eygene Ryabinkin                                        ,,,^..^,,,
 [ Life's unfair - but root password helps!           | codelabs.ru ]
 [ 82FE 06BC D497 C0DE 49EC  4FF0 16AF 9EAE 8152 ECFB | freebsd.org ]
Responsible-Changed-From-To: miwi->freebsd-ports-bugs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Jul 12 05:21:00 UTC 2011 
Responsible-Changed-Why:  
Back to pool at assignee request for the moment. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140939 
Responsible-Changed-From-To: freebsd-ports-bugs->rea 
Responsible-Changed-By: arved 
Responsible-Changed-When: Mon Aug 1 18:27:04 UTC 2011 
Responsible-Changed-Why:  
Submitter promised to look at this 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140939 
State-Changed-From-To: feedback->open 
State-Changed-By: pgollucci 
State-Changed-When: Thu Feb 9 02:36:58 UTC 2012 
State-Changed-Why:  
maintainer timeout (rea ; 192 days) / last commit 68 days ago 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140939 
Responsible-Changed-From-To: rea->freebsd-ports-bugs 
Responsible-Changed-By: eadler 
Responsible-Changed-When: Sat Mar 10 20:37:38 UTC 2012 
Responsible-Changed-Why:  
timeout 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140939 
State-Changed-From-To: open->closed 
State-Changed-By: scheidell 
State-Changed-When: Thu May 3 10:10:29 UTC 2012 
State-Changed-Why:  
old pr, no feedback, no fix, breaks existing port. Fix and create a new 
pr if this is still a problem. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140939 
>Unformatted:
