From skv@FreeBSD.org  Mon Aug  1 17:16:38 2005
Return-Path: <skv@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id F26A116A41F
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  1 Aug 2005 17:16:37 +0000 (GMT)
	(envelope-from skv@FreeBSD.org)
Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21])
	by mx1.FreeBSD.org (Postfix) with ESMTP id AF13443D48
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  1 Aug 2005 17:16:37 +0000 (GMT)
	(envelope-from skv@FreeBSD.org)
Received: from freefall.freebsd.org (skv@localhost [127.0.0.1])
	by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id j71HGbes069288
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 1 Aug 2005 17:16:37 GMT
	(envelope-from skv@freefall.freebsd.org)
Received: (from skv@localhost)
	by freefall.freebsd.org (8.13.3/8.13.1/Submit) id j71HGbq3069287;
	Mon, 1 Aug 2005 17:16:37 GMT
	(envelope-from skv)
Message-Id: <200508011716.j71HGbq3069287@freefall.freebsd.org>
Date: Mon, 1 Aug 2005 17:16:37 GMT
From: skv@freebsd.org
Reply-To: skv@freebsd.org
To: FreeBSD-gnats-submit@freebsd.org
Subject: Add "make test" to ports
X-Send-Pr-Version: 3.2

>Number:         84448
>Category:       ports
>Synopsis:       [patch] bsd.port.mk: Add "make test" to ports
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    portmgr
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 01 17:20:12 GMT 2005
>Closed-Date:    Fri May 25 03:17:52 GMT 2007
>Last-Modified:  Fri May 25 03:17:52 GMT 2007
>Originator:     Sergey Skvortsov
>Release:        FreeBSD 4.11-STABLE i386
>Organization:
<Organization of PR author (multiple lines)>
>Environment:

	<Relevant environment information (multiple lines)>

>Description:

Add to ports ability to run tests.

To enable "test" target just define HAS_TEST variable in Makefile.
All perl ports have automatically activated tests.

New variables are introduced: TEST_DEPENDS, TEST_TARGET, TEST_WRKSRC, TEST_ENV, TEST_ARGS.

>How-To-Repeat:

	<Code/input/activities to reproduce the problem (multiple lines)>

>Fix:

Index: bsd.port.mk
===================================================================
RCS file: /home/ncvs/ports/Mk/bsd.port.mk,v
retrieving revision 1.515
diff -u -r1.515 bsd.port.mk
--- bsd.port.mk	24 Jun 2005 09:18:54 -0000	1.515
+++ bsd.port.mk	1 Aug 2005 17:11:22 -0000
@@ -477,6 +477,12 @@
 #				  package depends on.  "lib" is the name of a shared library.
 #				  make will use "ldconfig -r" to search for the library.
 #				  lib can contain extended regular expressions.
+# TEST_DEPENDS	- A list of "path:dir[:target]" tuples of other ports this
+#				  package depends to test.  The test done to determine
+#				  the existence of the dependency is the same as
+#				  FETCH_DEPENDS.  This will be checked during the
+#				  "test" stage.  If the third field ("target") exists,
+#				  it will be used instead of ${DEPENDS_TARGET}.
 # DEPENDS		- A list of "dir[:target]" tuples of other ports this
 #				  package depends on being made first.  Use this only for
 #				  things that don't fall into the above four categories.
@@ -628,6 +634,9 @@
 # build-depends-list
 #				- Show all directories which are build-dependencies
 #				  for this port.
+# test-depends-list
+#				- Show all directories which are test-dependencies
+#				  for this port.
 # package-depends-list
 #				- Show all directories which are
 #				  package-dependencies for this port.
@@ -808,6 +817,19 @@
 # MAKE_ARGS		- Any extra arguments to sub-make in build and install stages.
 #				  Default: none
 #
+# For test:
+#
+# HAS_TEST	- If set, this port has its own test suite.  The
+#				  test stage will not do anything if this is not set.
+# TEST_TARGET
+#				- Default target for sub-make in test stage.
+#				  Default: test
+# TEST_WRKSRC	- Directory to do test in (default: ${WRKSRC}).
+# TEST_ENV		- Additional environment vars passed to sub-make in test stage.
+#				  Default: see below
+# TEST_ARGS		- Any extra arguments to sub-make in test stage.
+#				  Default: none
+#
 # For install:
 #
 # INSTALL_TARGET
@@ -1399,6 +1421,7 @@
 PATCH_WRKSRC?=	${WRKSRC}
 CONFIGURE_WRKSRC?=	${WRKSRC}
 BUILD_WRKSRC?=	${WRKSRC}
+TEST_WRKSRC?=	${WRKSRC}
 INSTALL_WRKSRC?=${WRKSRC}
 
 PLIST_SUB+=	OSREL=${OSREL} PREFIX=%D LOCALBASE=${LOCALBASE} X11BASE=${X11BASE}
@@ -1425,6 +1448,7 @@
 CONFIGURE_ENV+=	SHELL=${SH} CONFIG_SHELL=${SH} PORTOBJFORMAT=${PORTOBJFORMAT}
 SCRIPTS_ENV+=	PORTOBJFORMAT=${PORTOBJFORMAT}
 MAKE_ENV+=		SHELL=${SH} PORTOBJFORMAT=${PORTOBJFORMAT}
+TEST_ENV+=		SHELL=${SH}
 PLIST_SUB+=		PORTOBJFORMAT=${PORTOBJFORMAT}
 
 .if defined(MANCOMPRESSED)
@@ -1692,6 +1716,7 @@
 .if defined(PERL_MODBUILD)
 PERL_CONFIGURE=		yes
 CONFIGURE_SCRIPT?=	Build.PL
+HAS_TEST?=		yes
 .if ${PORTNAME} != Module-Build
 BUILD_DEPENDS+=		${SITE_PERL}/Module/Build.pm:${PORTSDIR}/devel/p5-Module-Build
 .endif
@@ -1707,6 +1732,7 @@
 	install_path=bindoc="${MAN1PREFIX}/man/man1"
 .elif defined(PERL_CONFIGURE)
 CONFIGURE_ARGS+=	INSTALLDIRS="site"
+HAS_TEST?=		yes
 .endif
 
 .if defined(PERL_CONFIGURE)
@@ -2156,6 +2182,7 @@
 MOTIFLIB?=	-L${X11BASE}/lib -lXm -lXp
 
 ALL_TARGET?=		all
+TEST_TARGET?=		test
 INSTALL_TARGET?=	install
 
 # This is a mid-term solution patch while pkg-comment files are
@@ -3378,6 +3405,35 @@
 .endif
 .endif
 
+# Test
+
+.if !target(do-test)
+do-test:
+.if defined(HAS_TEST)
+.if defined(USE_GMAKE)
+	@(cd ${BUILD_WRKSRC}; ${SETENV} ${TEST_ENV} ${GMAKE} ${MAKE_FLAGS} ${MAKEFILE} ${TEST_ARGS} ${TEST_TARGET})
+.else
+.if defined(PERL_MODBUILD)
+	@(cd ${BUILD_WRKSRC}; ${SETENV} ${TEST_ENV} ${PERL5} ${PL_BUILD} ${TEST_ARGS} ${TEST_TARGET})
+.else
+	@(cd ${BUILD_WRKSRC}; ${SETENV} ${TEST_ENV} ${MAKE} ${MAKE_FLAGS} ${MAKEFILE} ${TEST_ARGS} ${TEST_TARGET})
+.endif
+.endif
+.endif
+.endif
+
+.if !target(test)
+test: build test-depends
+	@${ECHO_MSG} "===>  Testing for ${PKGNAME}"
+.if target(pre-test)
+	@cd ${.CURDIR} && ${MAKE} ${__softMAKEFLAGS} pre-test
+.endif
+	@cd ${.CURDIR} && ${MAKE} ${__softMAKEFLAGS} do-test
+.if target(post-test)
+	@cd ${.CURDIR} && ${MAKE} ${__softMAKEFLAGS} post-test
+.endif
+.endif
+
 # Check conflicts
 
 .if !target(check-conflicts)
@@ -4289,7 +4345,7 @@
 _DEPEND_ALWAYS=	0
 .endif
 
-.for deptype in EXTRACT PATCH FETCH BUILD RUN
+.for deptype in EXTRACT PATCH FETCH BUILD RUN TEST
 ${deptype:L}-depends:
 .if defined(${deptype}_DEPENDS)
 .if !defined(NO_DEPENDS)
@@ -4464,13 +4520,13 @@
 # Dependency lists: both build and runtime, recursive.  Print out directory names.
 
 all-depends-list:
-.if defined(EXTRACT_DEPENDS) || defined(PATCH_DEPENDS) || defined(FETCH_DEPENDS) || defined(BUILD_DEPENDS) || defined(LIB_DEPENDS) || defined(RUN_DEPENDS) || defined(DEPENDS)
+.if defined(EXTRACT_DEPENDS) || defined(PATCH_DEPENDS) || defined(FETCH_DEPENDS) || defined(BUILD_DEPENDS) || defined(LIB_DEPENDS) || defined(RUN_DEPENDS) || defined(TEST_DEPENDS) || defined(DEPENDS)
 	@${ALL-DEPENDS-LIST}
 .endif
 
 ALL-DEPENDS-LIST= \
 	checked="${PARENT_CHECKED}"; \
-	for dir in $$(${ECHO_CMD} "${EXTRACT_DEPENDS} ${PATCH_DEPENDS} ${FETCH_DEPENDS} ${BUILD_DEPENDS} ${LIB_DEPENDS} ${RUN_DEPENDS}" | ${SED} -e 'y/ /\n/' | ${CUT} -f 2 -d ':') $$(${ECHO_CMD} ${DEPENDS} | ${SED} -e 'y/ /\n/' | ${CUT} -f 1 -d ':'); do \
+	for dir in $$(${ECHO_CMD} "${EXTRACT_DEPENDS} ${PATCH_DEPENDS} ${FETCH_DEPENDS} ${BUILD_DEPENDS} ${LIB_DEPENDS} ${RUN_DEPENDS} ${TEST_DEPENDS}" | ${SED} -e 'y/ /\n/' | ${CUT} -f 2 -d ':') $$(${ECHO_CMD} ${DEPENDS} | ${SED} -e 'y/ /\n/' | ${CUT} -f 1 -d ':'); do \
 		if [ -d $$dir ]; then \
 			if (${ECHO_CMD} $$checked | ${GREP} -qwv "$$dir"); then \
 				child=$$(cd $$dir; ${MAKE} PARENT_CHECKED="$$checked" all-depends-list); \
@@ -4607,6 +4663,20 @@
 		fi; \
 	done | ${SORT} -u
 
+test-depends-list:
+.if defined(TEST_DEPENDS) || defined(DEPENDS)
+	@${TEST-DEPENDS-LIST}
+.endif
+
+TEST-DEPENDS-LIST= \
+	for dir in $$(${ECHO_CMD} "${TEST_DEPENDS}" | ${TR} '\040' '\012' | ${SED} -e 's/^[^:]*://' -e 's/:.*//' | ${SORT} -u) $$(${ECHO_CMD} ${DEPENDS} | ${TR} '\040' '\012' | ${SED} -e 's/:.*//' | ${SORT} -u); do \
+		if [ -d $$dir ]; then \
+			${ECHO_CMD} $$dir; \
+		else \
+			${ECHO_MSG} "${PKGNAME}: \"$$dir\" non-existent -- dependency list incomplete" >&2; \
+		fi; \
+	done | ${SORT} -u
+
 # Package (recursive runtime) dependency list.  Print out both directory names
 # and package names.
 
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-ports-bugs->portmgr 
Responsible-Changed-By: skv 
Responsible-Changed-When: Mon Aug 1 17:21:39 GMT 2005 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=84448 
State-Changed-From-To: open->feedback 
State-Changed-By: kris 
State-Changed-When: Sat May 6 01:34:03 UTC 2006 
State-Changed-Why:  
There are a couple of issues with this patch: 

* I am not sure that TEST_DEPENDS is needed - but if it is, it needs 
to be added into the INDEX in an appropriate way (i.e. without 
breaking backwards compatibility with tools that parse it). 

* I am also not sure that the other variables you add are all needed 
(in general we don't like to add new variables except for good 
reason).  Please provide a sample patch that enables self-tests in a 
suitable number of ports demonstrating why they are all needed. 

* Turning on the perl self-tests now is a recipe for disaster; I 
guarantee that many ports will break if you do this (since I already 
tested it myself).  That will only delay the acceptance of your 
patch since it will need possibly many dedicated test builds to get 
to 100% working state. 

* The pointyhat cluster currently runs a 'make regression-test' port 
makefile target if it exists, and I think some ports are already 
using this for self-testing. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=84448 

From: Sergey Skvortsov <skv@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: ports/84448: [patch] bsd.port.mk: Add "make test" to ports
Date: Sat, 06 May 2006 13:46:52 +0400

 Kris Kennaway wrote:
 
 > There are a couple of issues with this patch:
 >
 > * I am not sure that TEST_DEPENDS is needed - but if it is, it needs
 >   to be added into the INDEX in an appropriate way (i.e. without
 >   breaking backwards compatibility with tools that parse it).
 
 I don't think that TEST_DEPENDS should be added into INDEX.
 "test" target is not required option for ports building but obviously
 essential feature.
 
 By the way TEST_DEPENDS is required because testing suite can be
 depended on additional modules which are not required for build/run
 purposes.
 
 > * I am also not sure that the other variables you add are all needed
 >   (in general we don't like to add new variables except for good
 >   reason).  Please provide a sample patch that enables self-tests in a
 >   suitable number of ports demonstrating why they are all needed.
 
 This test suite is intended to cover all needs in ports testing.
 I can not see good reason to afraid of adding new variables if desired
 level of functionality requires them.
 
 If test suite will be added then it must be full-featured and easily
 extensible. IMHO any palliatives are unacceptable.
 
 I does not understand "self-test" term. There is another kinds of testing?
 
 Requested example of patch for lang/perl5.8/Makefile (very trivial):
 
 @@ -121,8 +121,7 @@
  .endif
  MAN3PREFIX=    ${PREFIX}/lib/perl5/${PERL_VER}/perl
 
 -test:
 -       @(cd ${WRKSRC}; make test)
 +HAS_TEST=      yes
 
  BSDPAN_DEST=   ${PREFIX}/lib/perl5/${PERL_VER}/BSDPAN
  BSDPAN_FILES=  BSDPAN.pm BSDPAN/Override.pm Config.pm \
 
 > * Turning on the perl self-tests now is a recipe for disaster; I
 >   guarantee that many ports will break if you do this (since I already
 >   tested it myself).  That will only delay the acceptance of your
 >   patch since it will need possibly many dedicated test builds to get
 >   to 100% working state.
 
 I can not see can how 'test' target will break ports' building.
 
 For perl ports extracting 'test' target can reduce number of "main"
 dependencies. New perl's build/installation suites (like Module::Build,
 Module::Install) splits test dependencies from the rest.
 
 Currently many perl ports require 'Test::More' and even more exotic modules.
 If we extract 'test' target then p5-* port building will be faster and
 simpler.
 
 > * The pointyhat cluster currently runs a 'make regression-test' port
 >   makefile target if it exists, and I think some ports are already
 >   using this for self-testing.
 
 'regression-test' and 'test' have similar goals but different semantics.
 
 'test' target is suited for general functionality testing.
 'test' can be automatically processed while port's upgrade.
 End-user can tune testing parameters to test port in your own
 environment. For example, apache can be checked that it is runned in
 given test host:port (i.e. it is not crashed because of missed libraries
 and so on). Or RDBMS port can contain test suite with concrete end-users
 databases/procedures. Ideally, test suite can be good helper for
 developing seamless upgrade procedures - currently ports tree lacks good
 support for handling of updates.
 
 For example portupgrade can be extended to add option to require
 successful port's test passing before install/upgrade port.
 This feature can be very useful in environment with serious requirements
 in deployment procedures.
 
 'regression-test' is regression (harness) testing.
 
 Generally if 'regression-test' target is missed then 'test' target
 should be executed (as fallback case).
 
 Or, additional variable TEST_HARNESS=yes can be set to indicate user's
 wish to run additional regression tests.
 
 -- 
 Sergey Skvortsov
 mailto: skv@FreeBSD.org

From: Kris Kennaway <kris@obsecurity.org>
To: Sergey Skvortsov <skv@FreeBSD.org>
Cc: portmgr@FreeBSD.org, freebsd-gnats-submit@FreeBSD.org
Subject: Re: ports/84448: [patch] bsd.port.mk: Add "make test" to ports
Date: Sat, 6 May 2006 10:59:47 -0400

 --M9NhX3UHpAaciwkO
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 On Sat, May 06, 2006 at 09:50:23AM +0000, Sergey Skvortsov wrote:
 > The following reply was made to PR ports/84448; it has been noted by GNAT=
 S.
 >=20
 > From: Sergey Skvortsov <skv@FreeBSD.org>
 > To: bug-followup@FreeBSD.org
 > Cc: =20
 > Subject: Re: ports/84448: [patch] bsd.port.mk: Add "make test" to ports
 > Date: Sat, 06 May 2006 13:46:52 +0400
 >=20
 >  Kris Kennaway wrote:
 > =20
 >  > There are a couple of issues with this patch:
 >  >
 >  > * I am not sure that TEST_DEPENDS is needed - but if it is, it needs
 >  >   to be added into the INDEX in an appropriate way (i.e. without
 >  >   breaking backwards compatibility with tools that parse it).
 > =20
 >  I don't think that TEST_DEPENDS should be added into INDEX.
 >  "test" target is not required option for ports building but obviously
 >  essential feature.
 
 If it is going to be used, then it must be in INDEX - there is no
 other way for tinderboxes to know about it.
 
 >  By the way TEST_DEPENDS is required because testing suite can be
 >  depended on additional modules which are not required for build/run
 >  purposes.
 > =20
 >  > * I am also not sure that the other variables you add are all needed
 >  >   (in general we don't like to add new variables except for good
 >  >   reason).  Please provide a sample patch that enables self-tests in a
 >  >   suitable number of ports demonstrating why they are all needed.
 > =20
 >  This test suite is intended to cover all needs in ports testing.
 >  I can not see good reason to afraid of adding new variables if desired
 >  level of functionality requires them.
 
 What I am saying is that you've made a guess about what kinds of
 features might be needed in the most general case, but we need to see
 a patch that gives good evidence that:
 
 a) They are all needed somewhere or another, to avoid adding useless
 complications to b.p.m
 
 b) You are not missing anything because you didn't think hard enough.
 Sometimes people propose changes that sounds good on paper but turn
 out to not be usable in all situations because of something they
 didn't think of.
 
 >  Requested example of patch for lang/perl5.8/Makefile (very trivial):
 > =20
 >  @@ -121,8 +121,7 @@
 >   .endif
 >   MAN3PREFIX=3D    ${PREFIX}/lib/perl5/${PERL_VER}/perl
 > =20
 >  -test:
 >  -       @(cd ${WRKSRC}; make test)
 >  +HAS_TEST=3D      yes
 > =20
 >   BSDPAN_DEST=3D   ${PREFIX}/lib/perl5/${PERL_VER}/BSDPAN
 >   BSDPAN_FILES=3D  BSDPAN.pm BSDPAN/Override.pm Config.pm \
 > =20
 >  > * Turning on the perl self-tests now is a recipe for disaster; I
 >  >   guarantee that many ports will break if you do this (since I already
 >  >   tested it myself).  That will only delay the acceptance of your
 >  >   patch since it will need possibly many dedicated test builds to get
 >  >   to 100% working state.
 > =20
 >  I can not see can how 'test' target will break ports' building.
 
 When pointyhat runs the tests, many ports will fail them because they
 have never been tested before.
 
 >  For perl ports extracting 'test' target can reduce number of "main"
 >  dependencies. New perl's build/installation suites (like Module::Build,
 >  Module::Install) splits test dependencies from the rest.
 > =20
 >  Currently many perl ports require 'Test::More' and even more exotic modu=
 les.
 >  If we extract 'test' target then p5-* port building will be faster and
 >  simpler.
 > =20
 >  > * The pointyhat cluster currently runs a 'make regression-test' port
 >  >   makefile target if it exists, and I think some ports are already
 >  >   using this for self-testing.
 > =20
 >  'regression-test' and 'test' have similar goals but different semantics.
 
 That is not clear to me - and I think it will be unclear to others.
 We need to be clear on what your goal is with this patch.
 
 >  'test' target is suited for general functionality testing.
 >  'test' can be automatically processed while port's upgrade.
 >  End-user can tune testing parameters to test port in your own
 >  environment. For example, apache can be checked that it is runned in
 >  given test host:port (i.e. it is not crashed because of missed libraries
 >  and so on). Or RDBMS port can contain test suite with concrete end-users
 >  databases/procedures. Ideally, test suite can be good helper for
 >  developing seamless upgrade procedures - currently ports tree lacks good
 >  support for handling of updates.
 > =20
 >  For example portupgrade can be extended to add option to require
 >  successful port's test passing before install/upgrade port.
 >  This feature can be very useful in environment with serious requirements
 >  in deployment procedures.
 > =20
 >  'regression-test' is regression (harness) testing.
 > =20
 >  Generally if 'regression-test' target is missed then 'test' target
 >  should be executed (as fallback case).
 > =20
 >  Or, additional variable TEST_HARNESS=3Dyes can be set to indicate user's
 >  wish to run additional regression tests.
 > =20
 >  --=20
 >  Sergey Skvortsov
 >  mailto: skv@FreeBSD.org
 >=20
 
 --M9NhX3UHpAaciwkO
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.3 (FreeBSD)
 
 iD8DBQFEXLniWry0BWjoQKURAsaCAJ9K0Aq6vlVQOYnDlfn4ou2HTdWRAgCgz2F5
 a//wG93eQ1VAZ7czpgZ3C+Y=
 =Arkm
 -----END PGP SIGNATURE-----
 
 --M9NhX3UHpAaciwkO--
State-Changed-From-To: feedback->closed 
State-Changed-By: linimon 
State-Changed-When: Fri May 25 03:17:12 UTC 2007 
State-Changed-Why:  
At the portmgr meeting at BSDCan 2007, we agreed that this is too fragile, 
because too many ports will fail with it on the build cluster.  So we're 
not going to pursue this one; sorry. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=84448 
>Unformatted:
