From nick@iol.ie  Tue Dec  2 11:30:47 1997
Received: from iol.ie (auth1.auth.iol.ie [194.125.2.227])
          by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id LAA25888
          for <FreeBSD-gnats-submit@freebsd.org>; Tue, 2 Dec 1997 11:30:41 -0800 (PST)
          (envelope-from nick@iol.ie)
Received: (from nick@localhost)
	by iol.ie (8.8.7/8.8.7) id TAA02446;
	Tue, 2 Dec 1997 19:30:31 GMT
	(envelope-from nick)
Message-Id: <199712021930.TAA02446@iol.ie>
Date: Tue, 2 Dec 1997 19:30:31 GMT
From: nick@foobar.org
Reply-To: nick@foobar.org
To: FreeBSD-gnats-submit@freebsd.org
Cc: nick@foobar.org
Subject: Long rebuild times for pwd_mkdb with large master.passwd files
X-Send-Pr-Version: 3.2

>Number:         5193
>Category:       bin
>Synopsis:       It takes ages to rebuild master.passwd if it's large.
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Tue Dec  2 11:40:01 PST 1997
>Closed-Date:    Sun Apr 19 00:16:01 PDT 1998
>Last-Modified:  Sun Apr 19 00:16:18 PDT 1998
>Originator:     Nick Hilliard
>Release:        FreeBSD 2.2.5-RELEASE i386
>Organization:
Ireland On-Line
>Environment:

	FreeBSD 2.2.5-RELEASE with large master.passwd file

>Description:

	By default, FreeBSD doesn't cope well with rebuilding large
	passwd files.  On an unloaded P5/200 with 30000 passwd
	entries with the default DB cache size of 2Mb, it takes
	almost 30 minutes to do a complete passwd rebuild.  Increasing
	the cache size dramatically reduces the rebuild time.
	I've included a patch below for implementing a "-s" option which
	will specify the cache size in megabytes.  This leads to
	the following results:

# pwd_mkdb -p -s 2 /etc/master.passwd
14.772u 41.534s 26:11.28 3.5% 16+5850k 57+97145io 0pf+0w
# pwd_mkdb -p -s 4 /etc/master.passwd
13.720u 27.744s 17:45.82 3.8% 16+11227k 8+65377io 0pf+0w
# pwd_mkdb -p -s 6 /etc/master.passwd
11.786u 15.848s 9:49.47 4.6% 16+16253k 8+36842io 0pf+0w
# pwd_mkdb -p -s 8 /etc/master.passwd
10.224u 7.435s 4:30.88 6.5% 16+20428k 7+17582io 0pf+0w
# pwd_mkdb -p -s 10 /etc/master.passwd
9.927u 4.358s 2:34.70 9.2% 17+24852k 9+10910io 0pf+0w
# pwd_mkdb -p -s 12 /etc/master.passwd
10.253u 3.449s 1:45.71 12.9% 17+28356k 7+8043io 0pf+0w
# pwd_mkdb -p -s 14 /etc/master.passwd
10.814u 2.894s 1:24.57 16.1% 16+30982k 7+6657io 0pf+0w
# pwd_mkdb -p -s 16 /etc/master.passwd
12.406u 3.128s 1:13.88 21.0% 16+35270k 7+6153io 0pf+0w
# pwd_mkdb -p -s 18 /etc/master.passwd
12.515u 3.148s 1:12.56 21.5% 16+37672k 18+6303io 0pf+0w
# pwd_mkdb -p -s 20 /etc/master.passwd
13.733u 3.255s 1:17.34 21.9% 17+42232k 51+6223io 0pf+0w

	As you can see, the rebuild time decreases assymptotically.

	Is there any interest in putting this into the default distribution?

>How-To-Repeat:

	Create large master.passwd file, type pwd_mkdb -p /etc/master.passwd,
	and go on vacation for several weeks while waiting for the code to
	execute.

>Fix:
	

diff -u -r1.1 pwd_mkdb.8
--- pwd_mkdb.8	1997/12/02 18:59:13	1.1
+++ pwd_mkdb.8	1997/12/02 19:14:03
@@ -76,6 +76,14 @@
 Only update the record for the specified user.  Utilities that
 operate on a single user can use this option to avoid the
 overhead of rebuilding the entire database.
+.It Fl s Ar cachesize
+Specify in megabytes the size of the memory cache used by the
+hashing library.  On systems with a large user base, a small cache
+size can lead to prohibitively long database file rebuild times.
+As a rough guide, the memory usage of
+.Nm pwd_mkdb
+in megabytes will be a little bit more than twice the figure
+specified here.  The default is 2 megabytes.
 .El
 .Pp
 The two databases differ in that the secure version contains the user's 
diff -u -r1.1 pwd_mkdb.c
--- pwd_mkdb.c	1997/12/02 18:59:13	1.1
+++ pwd_mkdb.c	1997/12/02 18:59:18
@@ -110,7 +110,7 @@
 	strcpy(prefix, _PATH_PWD);
 	makeold = 0;
 	username = NULL;
-	while ((ch = getopt(argc, argv, "cd:pu:v")) != -1)
+	while ((ch = getopt(argc, argv, "cd:ps:u:v")) != -1)
 		switch(ch) {
 		case 'c':                       /* verify only */
 			cflag = 1;
@@ -123,6 +123,9 @@
 			break;
 		case 'u':			/* only update this record */
 			username = optarg;
+			break;
+		case 's':			/* change default cachesize */
+			openinfo.cachesize = atoi(optarg) * 1024 * 1024;
 			break;
 		case 'v':                       /* backward compatible */
 			break;
>Release-Note:
>Audit-Trail:

From: Bill Fenner <fenner@parc.xerox.com>
To: nick@foobar.org
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/5193: Long rebuild times for pwd_mkdb with large master.passwd files 
Date: Tue, 2 Dec 1997 13:54:48 PST

 Would it make sense to try to guess a good default based on the size of
 master.passwd, so the user doesn't have to decide what value to pass?
 
   Bill

From: Wolfram Schneider <wosch@cs.tu-berlin.de>
To: nick@foobar.org
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: bin/5193: Long rebuild times for pwd_mkdb with large master.passwd files
Date: 03 Dec 1997 02:01:40 +0100

 nick@foobar.org writes:
 > 	As you can see, the rebuild time decreases assymptotically.
 > 	Is there any interest in putting this into the default distribution?
 
 BSDI has a similar fix for the problem. FreeBSD should use the same
 option name and semantic as BSDI. See the pwd_mkdb manpage at
 www.bsdi.com .
 
 -- 
 Wolfram Schneider   <wosch@apfel.de>   http://www.apfel.de/~wosch/

From: Nick Hilliard <nick@iol.ie>
To: fenner@parc.xerox.com (Bill Fenner)
Cc: nick@foobar.org, FreeBSD-gnats-submit@freebsd.org, wosch@cs.tu-berlin.de
Subject: Re: bin/5193: Long rebuild times for pwd_mkdb with large master.passwd files
Date: Wed, 3 Dec 1997 10:00:50 +0000 (GMT)

 Bill:
 > Would it make sense to try to guess a good default based on the size of
 > master.passwd, so the user doesn't have to decide what value to pass?
 
 Yeah, I thought about this, but it would have taken more experimentation to
 find out some direct relationship between the two.
 
 The other problem with this is that if you make pwd_mkdb set the cache size
 automatically based on the size of /etc/master.passwd, it's also going to
 have to do other things like checking the total amount of RAM, and compare
 that against the amount of available RAM.  Otherwise, if someone had, say, a
 50,000 line passwd file, it might trash the system if there were other
 memory intensive processes lying around.
 
 Apart from anything else, if you have large passwd files, you're probably a
 sys-admin in a large establishment, and you're probably going to be clueful
 enough to figure out the "right" amount of cache to use.
 
 Wolfram:
 > BSDI has a similar fix for the problem. FreeBSD should use the same
 > option name and semantic as BSDI. See the pwd_mkdb manpage at
 > www.bsdi.com .
 
 BSDI uses '-c' for specifying the cache size, which conflicts with the
 FreeBSD '-c', which is used for doing a syntactic check of master.passwd. 
 So the only option here would be to change the current FreeBSD '-c' option
 to something else.
 
 Nick
State-Changed-From-To: open->closed 
State-Changed-By: phk 
State-Changed-When: Sun Apr 19 00:16:01 PDT 1998 
State-Changed-Why:  
committed,thanks! 
>Unformatted:
