From wollman@hergotha.csail.mit.edu  Wed Sep 19 02:43:12 2007
Return-Path: <wollman@hergotha.csail.mit.edu>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 68AAB16A417
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 19 Sep 2007 02:43:12 +0000 (UTC)
	(envelope-from wollman@hergotha.csail.mit.edu)
Received: from hergotha.csail.mit.edu (hergotha.csail.mit.edu [66.92.79.170])
	by mx1.freebsd.org (Postfix) with ESMTP id DE40D13C45E
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 19 Sep 2007 02:43:11 +0000 (UTC)
	(envelope-from wollman@hergotha.csail.mit.edu)
Received: from hergotha.csail.mit.edu (localhost [127.0.0.1])
	by hergotha.csail.mit.edu (8.13.8/8.13.8) with ESMTP id l8J2hAru041539
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 18 Sep 2007 22:43:10 -0400 (EDT)
	(envelope-from wollman@hergotha.csail.mit.edu)
Received: (from wollman@localhost)
	by hergotha.csail.mit.edu (8.13.8/8.13.8/Submit) id l8J2hAl0041538;
	Tue, 18 Sep 2007 22:43:10 -0400 (EDT)
	(envelope-from wollman)
Message-Id: <200709190243.l8J2hAl0041538@hergotha.csail.mit.edu>
Date: Tue, 18 Sep 2007 22:43:10 -0400 (EDT)
From: Garrett Wollman <wollman@hergotha.csail.mit.edu>
Reply-To: Garrett Wollman <wollman@hergotha.csail.mit.edu>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: pkg_create uses bogosort?
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         116452
>Category:       bin
>Synopsis:       pkg_create uses bogosort?
>Confidential:   no
>Severity:       non-critical
>Priority:       high
>Responsible:    krion
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Sep 19 02:50:01 GMT 2007
>Closed-Date:    Sat Nov 10 22:04:58 UTC 2007
>Last-Modified:  Sat Nov 10 22:10:02 UTC 2007
>Originator:     Garrett Wollman
>Release:        FreeBSD 6.2-RELEASE-p3 i386
>Organization:
>Environment:
System: FreeBSD hergotha.csail.mit.edu 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #5: Sun Mar 18 00:59:50 EDT 2007 wollman@hergotha.csail.mit.edu:/usr/obj/usr/src/sys/HERGOTHA i386


>Description:

	pkg_create makes an insane number of pointless filesystem
	operations (quadratic? exponential?) when being run as part of
	the "Registering installation" step of a port install.
	It is definitely a function of the number of ports installed,
	as it has gotten much worse since X was fragmented into 200
	separate packages.  ktrace shows:

[root@hergotha /home/wollman]# kdump -t n | wc -l
  937814
[root@hergotha /home/wollman]# kdump -t n | sort -u | wc -l
    1023

	(This is actually only a partial ktrace, started about halfway
	through the process.)  What it is actually trying to do, I do
	not know.  A representative sample from the trace:

 97936 pkg_create CALL  lstat(0xbfbfbe20,0xbfbfb990)
 97936 pkg_create NAMI  "/var/db/pkg/glitz-0.5.6_1"
 97936 pkg_create RET   lstat 0
 97936 pkg_create CALL  access(0xbfbfbe20,0x4)
 97936 pkg_create NAMI  "/var/db/pkg/glitz-0.5.6_1"
 97936 pkg_create RET   access 0
 97936 pkg_create CALL  stat(0xbfbfba20,0xbfbfb990)
 97936 pkg_create NAMI  "/var/db/pkg/glitz-0.5.6_1/+CONTENTS"
 97936 pkg_create RET   stat 0
 97936 pkg_create CALL  access(0xbfbfba20,0x4)
 97936 pkg_create NAMI  "/var/db/pkg/glitz-0.5.6_1/+CONTENTS"
 97936 pkg_create RET   access 0

>How-To-Repeat:
	Install an already-compiled port on a machine that has lots of
	packages.  Wait a few minutes.  Wonder why it's not done yet.

>Fix:

	Either use a less-stupid algorithm or at least memoize the
	result of whatever it's trying to do with this sequence of
	calls.

>Release-Note:
>Audit-Trail:

From: Garrett Wollman <wollman@bimajority.org>
To: FreeBSD-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: bin/116452: pkg_create uses bogosort?
Date: Wed, 19 Sep 2007 00:04:29 -0400

 I tracked this issue down as far as isinstalledpkg().  Here is a patch
 that memoizes the result of isinstalledpkg().  I haven't tested this
 code outside of pkg_create so I do not know if there needs to be an
 invalidation mechanism (e.g., for pkg_install when installing
 dependencies recursively).  It's possible that only "yes" responses
 should be memoized.
 
 -GAWollman
 
 Index: lib/match.c
 ===================================================================
 RCS file: /home/ncvs/src/usr.sbin/pkg_install/lib/match.c,v
 retrieving revision 1.19.8.1
 diff -u -r1.19.8.1 match.c
 --- lib/match.c	27 Sep 2005 13:39:06 -0000	1.19.8.1
 +++ lib/match.c	19 Sep 2007 03:11:07 -0000
 @@ -307,6 +307,17 @@
  }
  
  /*
 + * Small linked list to memoize results of isinstalledpkg().  A hash table
 + * would be faster but for n ~= 1000 may be overkill.
 + */
 +struct iip_memo {
 +	LIST_ENTRY(iip_memo) iip_link;
 +	char	*iip_name;
 +	int	 iip_result;
 +};
 +LIST_HEAD(, iip_memo) iip_memo = LIST_HEAD_INITIALIZER(iip_memo);
 +
 +/*
   * 
   * Return 1 if the specified package is installed,
   * 0 if not, and -1 if an error occured.
 @@ -314,18 +325,53 @@
  int
  isinstalledpkg(const char *name)
  {
 -    char buf[FILENAME_MAX];
 -    char buf2[FILENAME_MAX];
 -
 -    snprintf(buf, sizeof(buf), "%s/%s", LOG_DIR, name);
 -    if (!isdir(buf) || access(buf, R_OK) == FAIL)
 -	return 0;
 -
 -    snprintf(buf2, sizeof(buf2), "%s/%s", buf, CONTENTS_FNAME);
 -    if (!isfile(buf2) || access(buf2, R_OK) == FAIL)
 -	return -1;
 +    int result;
 +    char *buf, *buf2;
 +    struct iip_memo *memo;
 +
 +    LIST_FOREACH(memo, &iip_memo, iip_link) {
 +	if (strcmp(memo->iip_name, name) == 0)
 +	    return memo->iip_result;
 +    }
 +    
 +    buf2 = NULL;
 +    asprintf(&buf, "%s/%s", LOG_DIR, name);
 +    if (buf == NULL)
 +	goto errout;
 +    if (!isdir(buf) || access(buf, R_OK) == FAIL) {
 +	result = 0;
 +    } else {
 +	asprintf(&buf2, "%s/%s", buf, CONTENTS_FNAME);
 +	if (buf2 == NULL)
 +	    goto errout;
 +
 +	if (!isfile(buf2) || access(buf2, R_OK) == FAIL)
 +	    result = -1;
 +	else
 +	    result = 1;
 +    }
  
 -    return 1;
 +    free(buf);
 +    buf = strdup(name);
 +    if (buf == NULL)
 +	goto errout;
 +    free(buf2);
 +    buf2 = NULL;
 +
 +    memo = malloc(sizeof *memo);
 +    if (memo == NULL)
 +	goto errout;
 +    memo->iip_name = buf;
 +    memo->iip_result = result;
 +    LIST_INSERT_HEAD(&iip_memo, memo, iip_link);
 +    return result;
 +
 +errout:
 +    if (buf != NULL)
 +	free(buf);
 +    if (buf2 != NULL)
 +	free(buf2);
 +    return -1;
  }
  
  /*
Responsible-Changed-From-To: freebsd-bugs->krion 
Responsible-Changed-By: rodrigc 
Responsible-Changed-When: Wed Sep 19 04:20:00 UTC 2007 
Responsible-Changed-Why:  
Over to maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=116452 

From: Kirill Ponomarew <krion@voodoo.bawue.com>
To: rodrigc@FreeBSD.org
Cc: Garrett Wollman <wollman@bimajority.org>, bug-followup@FreeBSD.org
Subject: Re: bin/116452: pkg_create uses bogosort?
Date: Wed, 19 Sep 2007 10:28:51 +0200

 On Wed, Sep 19, 2007 at 04:20:38AM +0000, rodrigc@FreeBSD.org wrote:
 > Synopsis: pkg_create uses bogosort?
 > 
 > Responsible-Changed-From-To: freebsd-bugs->krion
 > Responsible-Changed-By: rodrigc
 > Responsible-Changed-When: Wed Sep 19 04:20:00 UTC 2007
 > Responsible-Changed-Why: 
 > Over to maintainer.
 
 I would suggest we run Garrett's patch on pointyhat for a while to
 catch possible issues.
 
 -Kirill

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/116452: commit references a PR
Date: Fri,  2 Nov 2007 20:18:55 +0000 (UTC)

 krion       2007-11-02 20:18:47 UTC
 
   FreeBSD src repository
 
   Modified files:
     usr.sbin/pkg_install/lib match.c 
   Log:
   Momoize the results of isinstalledpkg()
   
   PR:             bin/116452
   Submitted by:   wollmann
   MFC after:      7 days
   
   Revision  Changes    Path
   1.21      +57 -11    src/usr.sbin/pkg_install/lib/match.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: krion 
State-Changed-When: Fri Nov 2 22:54:05 UTC 2007 
State-Changed-Why:  
MFC follows in 7 days 

http://www.freebsd.org/cgi/query-pr.cgi?pr=116452 
State-Changed-From-To: patched->closed 
State-Changed-By: krion 
State-Changed-When: Sat Nov 10 22:04:57 UTC 2007 
State-Changed-Why:  
Changes committed, thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=116452 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/116452: commit references a PR
Date: Sat, 10 Nov 2007 22:04:39 +0000 (UTC)

 krion       2007-11-10 22:04:31 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_6)
     usr.sbin/pkg_install/lib match.c 
   Log:
   MFC: match.c: 1.21
   
     date: 2007/11/02 20:18:47;  author: krion;  state: Exp;  lines: +57 -11
     Memoize the results of isinstalledpkg()
   
     PR:             bin/116452
     Submitted by:   wollmann
   
   Revision  Changes    Path
   1.19.8.2  +57 -11    src/usr.sbin/pkg_install/lib/match.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
>Unformatted:
