From netchild@freebsd.org  Fri May 18 14:51:53 2007
Return-Path: <netchild@freebsd.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 6D8AE16A401;
	Fri, 18 May 2007 14:51:53 +0000 (UTC)
	(envelope-from netchild@freebsd.org)
Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217])
	by mx1.freebsd.org (Postfix) with ESMTP id 9EC5D13C44B;
	Fri, 18 May 2007 14:51:52 +0000 (UTC)
	(envelope-from netchild@freebsd.org)
Received: from outgoing.leidinger.net (p54A5CDFD.dip.t-dialin.net [84.165.205.253])
	by redbull.bpaserver.net (Postfix) with ESMTP id 503FA2E137;
	Fri, 18 May 2007 16:30:49 +0200 (CEST)
Received: from deskjail.leidinger.net (deskjail.Leidinger.net [192.168.1.109])
	by outgoing.leidinger.net (Postfix) with ESMTP id 838005B48A3;
	Fri, 18 May 2007 16:30:33 +0200 (CEST)
Message-Id: <1179498648.13221@deskjail.leidinger.net>
Date: Fri, 18 May 2007 16:30:48 +0200
From: "Alexander Leidinger" <netchild@FreeBSD.org>
To: "FreeBSD gnats submit" <FreeBSD-gnats-submit@freebsd.org>
Cc: Stephen Montgomery-Smith <stephen@math.missouri.edu>
Subject: [patch] Speed improvements for package registration (about 30 times faster dependency computing)
X-Send-Pr-Version: gtk-send-pr 0.4.8 
X-GNATS-Notify:

>Number:         112765
>Category:       ports
>Synopsis:       [patch] bsd.port.mk - Speed improvements for package registration (about 30 times faster dependency computing)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    portmgr
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Fri May 18 15:00:14 GMT 2007
>Closed-Date:    Sat Jun 16 19:07:53 GMT 2007
>Last-Modified:  Sat Jun 16 19:10:04 GMT 2007
>Originator:     Alexander Leidinger
>Release:        FreeBSD 7.0-CURRENT i386
>Organization:
FreeBSD 
>Environment:


System: FreeBSD 7.0-CURRENT #249: Tue Apr 24 10:56:32 CEST 2007
    root@Magellan.Leidinger.net:/usr/src/sys/i386/compile/WORK



>Description:


Based upon a patch from Stephen (CCed) and a hint from rdivacky, I produced a patch which 
 - changes the package registration (package-depends) from pkgnames/dependencies in the ports collection to pkgnames/dependencies as installed on the current system (actual-package-depends)
 - speeds up the process from about 2min 30sec to about 5sec for gnome2 on my system

Depending on the config options used to install a system, the output of package-depends is different from actual-package-depends. This is intended and there should be no change for a stock run on the ports build cluster.


>How-To-Repeat:





>Fix:


--- bsd.port.mk.diff begins here ---
--- Mk/bsd.port.mk	Sat Apr  7 14:51:47 2007
+++ Mk/bsd.port.mk	Fri May 18 11:36:58 2007
@@ -692,8 +692,16 @@
 #				- Show all directories which are build-dependencies
 #				  for this port.
 # package-depends-list
-#				- Show all directories which are
-#				  package-dependencies for this port.
+#				- Show all directories which are package-dependencies
+#				  for this port. This is based upon the dependency
+#				  tree as recorded in the Makefiles of the ports
+#				  collection, not as recorded in the currently
+#				  installed ports.
+# actual-package-depends-list
+#				- Like package-depends-list but with the difference
+#				  that the dependencies of the currently installed
+#				  ports are used instead of the dependencies as
+#				  recorded in the ports collection.
 # run-depends-list
 #				- Show all directories which are run-dependencies
 #				  for this port.
@@ -2321,7 +2329,7 @@
 DISABLE_CONFLICTS=	YES
 .endif
 .if !defined(PKG_ARGS)
-PKG_ARGS=		-v -c -${COMMENT:Q} -d ${DESCR} -f ${TMPPLIST} -p ${PREFIX} -P "`cd ${.CURDIR} && ${MAKE} package-depends | ${GREP} -v -E ${PKG_IGNORE_DEPENDS} | ${SORT} -u`" ${EXTRA_PKG_ARGS} $${_LATE_PKG_ARGS}
+PKG_ARGS=		-v -c -${COMMENT:Q} -d ${DESCR} -f ${TMPPLIST} -p ${PREFIX} -P "`cd ${.CURDIR} && ${MAKE} actual-package-depends | ${GREP} -v -E ${PKG_IGNORE_DEPENDS} | ${SORT} -u`" ${EXTRA_PKG_ARGS} $${_LATE_PKG_ARGS}
 .if !defined(NO_MTREE)
 PKG_ARGS+=		-m ${MTREE_FILE}
 .endif
@@ -5145,16 +5153,45 @@
 				shift 3; \
 			done; \
 			checked="$$dir $$childdir $$checked"; \
-		else \\
+		else \
 			${ECHO_MSG} "${PKGNAME}: \"$$dir\" non-existent -- dependency list incomplete" >&2; \
 		fi; \
 	done
 
+ACTUAL-PACKAGE-DEPENDS?= \
+	if [ "${_LIB_RUN_DEPENDS}" != "  " ]; then \
+		origins=$$(for pkgname in ${PKG_DBDIR}/*; do \
+			if [ -e $$pkgname/+CONTENTS ]; then \
+				${ECHO_CMD} $${pkgname\#\#*/}; \
+				${SED} -n -e "s/@comment ORIGIN://p" $$pkgname/+CONTENTS; \
+			fi; \
+		done); \
+		for dir in ${_LIB_RUN_DEPENDS:C,[^:]*:([^:]*):?.*,\1,}; do \
+			tmp=$${dir%/*}; \
+			dir=$${tmp\#\#*/}/$${dir\#\#*/}; \
+			set -- $$origins; \
+			while [ $$\# != 0 ]; do \
+				if [ $$dir = $$2 ]; then \
+					${ECHO_CMD} $$1:$$dir; \
+					if [ -e ${PKG_DBDIR}/$$1/+CONTENTS ]; then \
+						packagelist="$$packagelist ${PKG_DBDIR}/$$1/+CONTENTS"; \
+					fi; \
+					break; \
+				fi; \
+				shift 2; \
+			done; \
+		done; \
+		[ -z "$$packagelist" ] || ${AWK} -F '( |:)' 'BEGIN { pkgname="broken_contents" } /@pkgdep / { pkgname=$$2 } /@comment DEPORIGIN:/ { printf "%s:%s\n", pkgname, $$3; pkgname="broken_contents" }' $$packagelist; \
+	fi
+
 # Print out package names.
 
 package-depends:
 	@${PACKAGE-DEPENDS-LIST} | ${AWK} '{print $$1":"$$3}'
 
+actual-package-depends:
+	@${ACTUAL-PACKAGE-DEPENDS}
+
 # Build packages for port and dependencies
 
 package-recursive: package
--- bsd.port.mk.diff ends here ---



>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-ports-bugs->portmgr 
Responsible-Changed-By: pav 
Responsible-Changed-When: Fri May 18 15:04:21 UTC 2007 
Responsible-Changed-Why:  
Looks interesting 

http://www.freebsd.org/cgi/query-pr.cgi?pr=112765 

From: Stephen Montgomery-Smith <stephen@math.missouri.edu>
To: bug-followup@FreeBSD.org, netchild@FreeBSD.org
Cc:  
Subject: Re: ports/112765: [patch] Speed improvements for package registration
 (about 30 times faster dependency computing)
Date: Fri, 18 May 2007 15:16:22 -0500

 Let me say a couple more things about this PR.
 
 1. The result of "make actual-package-depends" is actually more accurate 
 than "make package-depends."  This is because "make 
 actual-package-depends" reflects the actual state of the packages 
 already installed, whereas "make package-depends" simply tells you what 
 would be installed if you were to install them just before installing 
 the current port.
 
 2.  The super magnificent speed increases reported also require the 
 application of http://www.freebsd.org/cgi/query-pr.cgi?pr=112630.  Both 
 PR's may be applied independently, and solve two entirely different 
 speed bottlenecks in the package registration process, but applying just 
 one will just result in something like only a 50% speed increase, as 
 opposed to the 30-fold increase we are experiencing.
 
 

From: "Peter Johnson" <johnson.peter@gmail.com>
To: bug-followup@FreeBSD.org, netchild@FreeBSD.org
Cc:  
Subject: Re: ports/112765: [patch] bsd.port.mk - Speed improvements for package registration (about 30 times faster dependency computing)
Date: Sat, 19 May 2007 22:29:23 -0700

 I ran into a problem on my system where occasionally I'd end up with a
 blank space (?) at the end of $$origins.  This caused "[: =: argument
 expected" and make garbage in the depends output, or if I put in x's
 into "if [ $$dir = $$2 ]", just an Error 1 due to trying to shift 2
 something only 1 long.  I fixed it by changing the while loop from
 "while [ $$\# != 0 ]" to "while [ $$\# -gt 1 ]".  This works without
 error even when there's an odd number of $$origins for some reason.

From: Alexander Leidinger <netchild@FreeBSD.org>
To: "Peter Johnson" <johnson.peter@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: ports/112765: [patch] bsd.port.mk - Speed improvements for
 package registration (about 30 times faster dependency computing)
Date: Sun, 20 May 2007 09:08:26 +0200

 Quoting "Peter Johnson" <johnson.peter@gmail.com> (Sat, 19 May 2007 22:29:23 -0700):
 
 > I ran into a problem on my system where occasionally I'd end up with a
 > blank space (?) at the end of $$origins.  This caused "[: =: argument
 
 Ugh...
 
 > expected" and make garbage in the depends output, or if I put in x's
 > into "if [ $$dir = $$2 ]", just an Error 1 due to trying to shift 2
 
 In my local tree I quote $$dir and $$2 now instead, this should work
 too.
 
 > something only 1 long.  I fixed it by changing the while loop from
 > "while [ $$\# != 0 ]" to "while [ $$\# -gt 1 ]".  This works without
 > error even when there's an odd number of $$origins for some reason.
 
 Good idea. Incorporated into my local version.
 
 Thanks,
 Alexander.
 
 -- 
       ...and that is how we know the Earth to be banana-shaped.
 http://www.Leidinger.net  Alexander @ Leidinger.net: PGP ID = B0063FE7
 http://www.FreeBSD.org     netchild @ FreeBSD.org  : PGP ID = 72077137

From: Stephen Montgomery-Smith <stephen@math.missouri.edu>
To: bug-followup@FreeBSD.org,  netchild@FreeBSD.org
Cc:  
Subject: Re: ports/112765: [patch] bsd.port.mk - Speed improvements for package
 registration (about 30 times faster dependency computing)
Date: Thu, 24 May 2007 23:38:49 -0500

 This is a multi-part message in MIME format.
 --------------090802010508040501020601
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit
 
 I just saw that you guys have also experienced the "argument expected" 
 error.  I believe that this is caused when ORIGINS is absent from 
 +CONTENTS.  This will happen if you are building several ports at the 
 same time, and make try to look inside +CONTENTS of another package 
 while it is still being written out.  If this happens, and you are 
 *lucky* you will get the error you reported.  If you are unlucky you 
 will get no errors, but nevertheless a bogus collection of depends.
 
 I enclose as an attachment what I believe to be the fix.  Apply this 
 patch on top of the one already provided.
 
 
 --------------090802010508040501020601
 Content-Type: text/plain;
  name="eee"
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline;
  filename="eee"
 
 --- bsd.port.mk2	Thu May 24 23:09:17 2007
 +++ bsd.port.mk	Thu May 24 23:31:11 2007
 @@ -5231,12 +5231,15 @@
  
  ACTUAL-PACKAGE-DEPENDS?= \
  	if [ "${_LIB_RUN_DEPENDS}" != "  " ]; then \
 -		origins=$$(for pkgname in ${PKG_DBDIR}/*; do \
 +		for pkgname in ${PKG_DBDIR}/*; do \
  			if [ -e $$pkgname/+CONTENTS ]; then \
 -				${ECHO_CMD} $${pkgname\#\#*/}; \
 -				${SED} -n -e "s/@comment ORIGIN://p" $$pkgname/+CONTENTS; \
 +				a=$${pkgname\#\#*/}; \
 +				b=`${SED} -n -e "s/@comment ORIGIN://p" $$pkgname/+CONTENTS`; \
 +				if [ ! -z $$b ]; then \
 +					origins="$$origins $$a $$b"; \
 +				fi; \
  			fi; \
 -		done); \
 +		done; \
  		for dir in ${_LIB_RUN_DEPENDS:C,[^:]*:([^:]*):?.*,\1,}; do \
  			tmp=$${dir%/*}; \
  			dir=$${tmp\#\#*/}/$${dir\#\#*/}; \
 
 --------------090802010508040501020601--

From: Alexander Leidinger <netchild@FreeBSD.org>
To: Stephen Montgomery-Smith <stephen@math.missouri.edu>
Cc: bug-followup@FreeBSD.org
Subject: Re: ports/112765: [patch] bsd.port.mk - Speed improvements for
	package registration (about 30 times faster dependency computing)
Date: Fri, 25 May 2007 08:59:46 +0200

 Quoting Stephen Montgomery-Smith <stephen@math.missouri.edu> (from =20
 Thu, 24 May 2007 23:38:49 -0500):
 
 > I just saw that you guys have also experienced the "argument expected"
 > error.  I believe that this is caused when ORIGINS is absent from
 > +CONTENTS.  This will happen if you are building several ports at the
 > same time, and make try to look inside +CONTENTS of another package
 > while it is still being written out.  If this happens, and you are
 > *lucky* you will get the error you reported.  If you are unlucky you
 > will get no errors, but nevertheless a bogus collection of depends.
 >
 > I enclose as an attachment what I believe to be the fix.  Apply this
 > patch on top of the one already provided.
 
 Looks good. As the port with the missing origin is not needed for this =20
 package, not adding it is the right way to go.
 
 Bye,
 Alexander.
 
 --=20
 http://www.Leidinger.net  Alexander @ Leidinger.net: PGP ID =3D B0063FE7
 http://www.FreeBSD.org     netchild @ FreeBSD.org  : PGP ID =3D 72077137
 Chisolm's First Corollary to Murphy's Second Law:
 =09When things just can't possibly get any worse, they will.
 
State-Changed-From-To: open->closed 
State-Changed-By: pav 
State-Changed-When: Sat Jun 16 19:05:17 UTC 2007 
State-Changed-Why:  
Patch committed to CVS, thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=112765 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: ports/112765: commit references a PR
Date: Sat, 16 Jun 2007 19:07:47 +0000 (UTC)

 pav         2007-06-16 19:07:42 UTC
 
   FreeBSD ports repository
 
   Modified files:
     Mk                   bsd.port.mk 
   Log:
   - Provide actual-package-depends target, which prints actual dependencies of
     freshly installed port, according to records in /var/db/pkg
   - Use this new target when registering package, providing massive speedup
   
   PR:             ports/112765
   Submitted by:   Stephen Montgomery-Smith <stephen@math.missouri.edu>
   Tested by:      bento amd64 exprun
   
   - Fix double slash typo
   
   Submitted by:   Alexander Logvinov <ports@logvinov.com>
   Tested by:      bento amd64 exprun
   
   Revision  Changes    Path
   1.569     +45 -5     ports/Mk/bsd.port.mk
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
>Unformatted:
