From stephen@cauchy.math.missouri.edu  Sun May 13 14:16:18 2007
Return-Path: <stephen@cauchy.math.missouri.edu>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 30C2616A416
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 13 May 2007 14:16:18 +0000 (UTC)
	(envelope-from stephen@cauchy.math.missouri.edu)
Received: from cauchy.math.missouri.edu (cauchy.math.missouri.edu [128.206.184.213])
	by mx1.freebsd.org (Postfix) with ESMTP id DE33D13C459
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 13 May 2007 14:16:17 +0000 (UTC)
	(envelope-from stephen@cauchy.math.missouri.edu)
Received: from cauchy.math.missouri.edu (localhost [127.0.0.1])
	by cauchy.math.missouri.edu (8.14.1/8.13.4) with ESMTP id l4DDpkq1063383
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 13 May 2007 08:51:46 -0500 (CDT)
	(envelope-from stephen@cauchy.math.missouri.edu)
Received: (from stephen@localhost)
	by cauchy.math.missouri.edu (8.14.1/8.13.4/Submit) id l4DDpk6r063382;
	Sun, 13 May 2007 08:51:46 -0500 (CDT)
	(envelope-from stephen)
Message-Id: <200705131351.l4DDpk6r063382@cauchy.math.missouri.edu>
Date: Sun, 13 May 2007 08:51:46 -0500 (CDT)
From: Stephen Montgomery-Smith <stephen@cauchy.math.missouri.edu>
Reply-To: Stephen Montgomery-Smith <stephen@math.missouri.edu>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Registering of ports and creation of packages takes a long time
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         112630
>Category:       bin
>Synopsis:       pkg_install [patch] Registering of ports and creation of packages takes a long time
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    portmgr
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Sun May 13 14:20:09 GMT 2007
>Closed-Date:    Sat Jul 21 22:50:45 GMT 2007
>Last-Modified:  Sat Jul 21 22:50:45 GMT 2007
>Originator:     Stephen Montgomery-Smith
>Release:        FreeBSD 6.2-STABLE i386
>Organization:
>Environment:
System: FreeBSD cauchy 6.2-STABLE FreeBSD 6.2-STABLE #0: Sat May 5 23:55:26 CDT 2007 stephen@cauchy:/usr/obj/usr/src/sys/cauchy i386


	
>Description:
Registering of ports when doing "make install", for ports that have many 
dependencies (e.g. x11/gnome2), or making packages using "make package",
takes a very long time.  A significant part of the problem is in the
function sortdeps in deps.c, which uses a rather inefficient bubble sort.

>How-To-Repeat:
	
>Fix:

--- /usr/src/usr.sbin/pkg_install/lib/deps.c	Wed Dec  6 20:14:13 2006
+++ usr.sbin/pkg_install/lib/deps.c	Sat May 12 23:53:36 2007
@@ -26,98 +26,144 @@
 #include <err.h>
 #include <stdio.h>
 
+void list_deps(const char *pkgname, char **pkgs, char *listed, 
+               char *check_loop, char **newpkgs, int *nrnewpkgs,
+               int *err_cnt);
+
 /*
  * Sort given NULL-terminated list of installed packages (pkgs) in
  * such a way that if package A depends on package B then after
  * sorting A will be listed before B no matter how they were
  * originally positioned in the list.
+ *
+ * Works by performing a recursive depth-first search on the 
+ * required-by lists.
  */
+
 int
 sortdeps(char **pkgs)
 {
-    char *tmp;
-    int i, j, loop_cnt;
-    int err_cnt = 0;
+    int i, err_cnt=0;
+    int nrpkgs, nrnewpkgs;
+    char *listed, *check_loop, **newpkgs;
+    char *cp;
 
     if (pkgs[0] == NULL || pkgs[1] == NULL)
 	return (0);
 
-    for (i = 0; pkgs[i + 1]; i++) {
-	/*
-	 * Check to see if any other package in pkgs[i+1:] depends
-	 * on pkgs[i] and swap those two packages if so.
-	 */
-	loop_cnt = 0;
-	for (j = i + 1; pkgs[j]; j++) {
-	    if (chkifdepends(pkgs[j], pkgs[i]) == 1) {
-		/*
-		 * Try to avoid deadlock if package A depends on B which in
-		 * turn depends on C and C due to an error depends on A.
-		 * Use ugly but simple method, becase it Should Never
-		 * Happen[tm] in the real life anyway.
-		 */
-		if (loop_cnt > 4096) {
-		    warnx("dependency loop detected for package %s", pkgs[j]);
-		    err_cnt++;
-		    break;
-		}
-		loop_cnt++;
-		tmp = pkgs[i];
-		pkgs[i] = pkgs[j];
-		pkgs[j] = tmp;
-		/*
-		 * Another iteration requred to check if new pkgs[i]
-		 * itself has any packages that depend on it
-		 */
-		j = i + 1;
-	    }
-	}
+    nrpkgs = 0;
+    while (pkgs[nrpkgs]) nrpkgs++;
+    listed = alloca(nrpkgs);
+    if (listed == NULL) {
+	warnx("%s(): alloca() failed", __func__);
+	return 1;
+    }
+    bzero(listed,nrpkgs);
+    check_loop = alloca(nrpkgs);
+    if (check_loop == NULL) {
+	warnx("%s(): alloca() failed", __func__);
+	return 1;
+    }
+    bzero(check_loop,nrpkgs);
+    newpkgs = alloca(nrpkgs*sizeof(char*));
+    if (newpkgs == NULL) {
+	warnx("%s(): alloca() failed", __func__);
+	return 1;
+    }
+    nrnewpkgs = 0;
+
+    for (i = 0; pkgs[i]; i++) if (!listed[i]) {
+	check_loop[i] = 1;
+	cp = strchr(pkgs[i], ':');
+	if (cp != NULL)
+	    *cp = '\0';
+	list_deps(pkgs[i],pkgs,listed,check_loop,newpkgs,&nrnewpkgs,&err_cnt);
+	if (cp != NULL)
+	    *cp = ':';
+	listed[i] = 1;
+	newpkgs[nrnewpkgs] = pkgs[i];
+	nrnewpkgs++;
+    }
+
+    if (nrnewpkgs != nrpkgs) {
+	fprintf(stderr,"This shouldn't happen, and indicates a huge error in the code.\n");
+	exit(1);
     }
+    for (i = 0; i < nrnewpkgs; i++) pkgs[i] = newpkgs[i];
+
     return err_cnt;
 }
 
 /*
- * Check to see if pkgname1 depends on pkgname2.
- * Returns 1 if depends, 0 if not, and -1 if error occured.
- */ 
-int
-chkifdepends(const char *pkgname1, const char *pkgname2)
-{
-    char *cp1, *cp2;
-    int errcode;
+ * This recursive function lists the dependencies (that is, the 
+ * "required-by"s) for pkgname, putting them into newpkgs.
+ */
+
+void list_deps(const char *pkgname, char **pkgs, char *listed, 
+               char *check_loop, char **newpkgs, int *nrnewpkgs,
+               int *err_cnt) {
+    char **rb, **rbtmp;
+    char *cp;
+    int errcode, i, j;
     struct reqr_by_entry *rb_entry;
     struct reqr_by_head *rb_list;
 
-    cp2 = strchr(pkgname2, ':');
-    if (cp2 != NULL)
-	*cp2 = '\0';
-    cp1 = strchr(pkgname1, ':');
-    if (cp1 != NULL)
-	*cp1 = '\0';
-
-    errcode = 0;
-    /* Check that pkgname2 is actually installed */
-    if (isinstalledpkg(pkgname2) <= 0)
-	goto exit;
+    if (isinstalledpkg(pkgname) <= 0)
+	return;
 
-    errcode = requiredby(pkgname2, &rb_list, FALSE, TRUE);
+    errcode = requiredby(pkgname, &rb_list, FALSE, TRUE);
     if (errcode < 0)
-	goto exit;
-
-    errcode = 0;
+	return;
+    /*
+     * We put rb_list into an argv style NULL terminated list,
+     * because requiredby uses some static storage, and list_deps
+     * is a recursive function.
+     */
+
+    rbtmp = rb = alloca((errcode + 1) * sizeof(*rb));
+    if (rb == NULL) {
+	warnx("%s(): alloca() failed", __func__);
+	(*err_cnt)++;
+	return;
+    }
     STAILQ_FOREACH(rb_entry, rb_list, link) {
-	if (strcmp(rb_entry->pkgname, pkgname1) == 0) {	/* match */
-	    errcode = 1;
-	    break;
+	*rbtmp = alloca(strlen(rb_entry->pkgname) + 1);
+	if (*rbtmp == NULL) {
+	    warnx("%s(): alloca() failed", __func__);
+	    (*err_cnt)++;
+	    return;
 	}
+	strcpy(*rbtmp, rb_entry->pkgname);
+	rbtmp++;
     }
+    *rbtmp = NULL;
 
-exit:
-    if (cp1 != NULL)
-	*cp1 = ':';
-    if (cp2 != NULL)
-	*cp2 = ':';
-    return errcode;
+    for (i = 0; rb[i]; i++)
+	for (j = 0; pkgs[j]; j++) if (!listed[j]) {
+	    cp = strchr(pkgs[j], ':');
+	    if (cp != NULL)
+		*cp = '\0';
+	    if (strcmp(rb[i], pkgs[j]) == 0) { /*match */
+		/*
+		 * Try to avoid deadlock if package A depends on B which in
+		 * turn depends on C and C due to an error depends on A.
+		 * It Should Never Happen[tm] in real life.
+		 */
+		if (check_loop[j]) {
+		    warnx("dependency loop detected for package %s", pkgs[j]);
+		    (*err_cnt)++;
+		}
+		else {
+		    check_loop[j] = 1;
+		    list_deps(pkgs[j],pkgs,listed,check_loop,newpkgs,nrnewpkgs,err_cnt);
+		    listed[j] = 1;
+		    newpkgs[*nrnewpkgs] = pkgs[j];
+		    (*nrnewpkgs)++;
+		}
+	    }
+	    if (cp != NULL)
+		*cp = ':';
+	}
 }
 
 /*

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-ports-bugs->portmgr 
Responsible-Changed-By: pav 
Responsible-Changed-When: Sun May 13 14:31:58 UTC 2007 
Responsible-Changed-Why:  
Move over to bin, assign to portmgr 

http://www.freebsd.org/cgi/query-pr.cgi?pr=112630 

From: Stanislav Sedov <stas@FreeBSD.org>
Subject: Re: pkg_install [patch] Registering of ports and creation of packages takes a long time
Date: unknown

 Hi!
 
 Thanks for the patch! However, it contains a some of style(9) bugs:
 
 +    nrpkgs = 0;
 +    while (pkgs[nrpkgs]) nrpkgs++;
 
 Should looks like:
 +    while (pkgs[nrpkgs])
 +	nrpkgs++;
 
 +    listed = alloca(nrpkgs);
 +    if (listed == NULL) {
 +        warnx("%s(): alloca() failed", __func__);
 +        return 1;
 +    }
 +    bzero(listed,nrpkgs);
 
 missing spaces
 
 +    check_loop = alloca(nrpkgs);
 +    if (check_loop == NULL) {
 +        warnx("%s(): alloca() failed", __func__);
 +        return 1;
 +    }
 +    bzero(check_loop,nrpkgs);
 
 missing spaces
 
 +    newpkgs = alloca(nrpkgs*sizeof(char*));
 
 the same
 
 +    if (newpkgs == NULL) {
 +        warnx("%s(): alloca() failed", __func__);
 +        return 1;
 +    }
 +    nrnewpkgs = 0;
 +
 +    for (i = 0; pkgs[i]; i++) if (!listed[i]) {
 
 missing newline & indentation. Also non style(9) check (should be
 listed[i] != 0).
 
      }
 +    for (i = 0; i < nrnewpkgs; i++) pkgs[i] = newpkgs[i];
 +
 missing newline.
 
 
 +                    list_deps(pkgs
 [j],pkgs,listed,check_loop,newpkgs,nrnewpkgs,err_cnt);
 +                    listed[j] = 1;
 +                    newpkgs[*nrnewpkgs] = pkgs[j];
 +                    (*nrnewpkgs)++;
 +                }
 
 spaces.
 
 Again, thanks for your work!
 

From: Toomas Pelberg <toomasp@gmx.net>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/112630: pkg_install [patch] Registering of ports and
	creation of packages takes a long time
Date: Fri, 01 Jun 2007 07:47:27 +0300

 Tested the patch, appears to "work" for me.
 

From: "[LoN]Kamikaze" <LoN_Kamikaze@gmx.de>
To: bug-followup@FreeBSD.org,  stephen@math.missouri.edu
Cc:  
Subject: Re: bin/112630: pkg_install [patch] Registering of ports and creation
 of packages takes a long time
Date: Sat, 16 Jun 2007 22:46:43 +0200

 I am using the patch since Stephen first posted it on the mailing list:
 
 http://lists.freebsd.org/pipermail/freebsd-ports/2007-May/040794.html
 
 It did not cause any trouble so far and the speed improvement is extremely
 significant. Together with Stephens other patch that has just been commited:
 
 http://www.freebsd.org/cgi/query-pr.cgi?pr=112765
 
 the registration time of x11/xorg went down from ~15 minutes to 18 seconds on
 one of my systems.
State-Changed-From-To: open->analyzed 
State-Changed-By: pav 
State-Changed-When: Sat Jun 16 20:54:14 UTC 2007 
State-Changed-Why:  
This patch is currently being tested in exprun 

http://www.freebsd.org/cgi/query-pr.cgi?pr=112630 
State-Changed-From-To: analyzed->patched 
State-Changed-By: pav 
State-Changed-When: Mon Jun 18 22:43:59 UTC 2007 
State-Changed-Why:  
Committed to HEAD; pending MFC 

http://www.freebsd.org/cgi/query-pr.cgi?pr=112630 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/112630: commit references a PR
Date: Mon, 18 Jun 2007 22:49:21 +0000 (UTC)

 pav         2007-06-18 22:49:13 UTC
 
   FreeBSD src repository (doc,ports committer)
 
   Modified files:
     usr.sbin/pkg_install/lib deps.c 
   Log:
   - Replace rather inefficient bubble sort with a recursive depth-first search.
     This speeds up registration of packages considerably.
   - style(9) police welcome!
   
   PR:             bin/112630
   Submitted by:   Stephen Montgomery-Smith <stephen@cauchy.math.missouri.edu>
   Tested by:      bento i386 experimental run
   MFC after:      14 days
   
   Revision  Changes    Path
   1.12      +111 -65   src/usr.sbin/pkg_install/lib/deps.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: pav 
State-Changed-When: Sat Jul 21 22:50:24 UTC 2007 
State-Changed-Why:  
This was MFC'ed to 6-STABLE some time ago. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=112630 
>Unformatted:
