From nobody  Thu Nov 19 10:48:05 1998
Received: (from nobody@localhost)
          by hub.freebsd.org (8.8.8/8.8.8) id KAA04559;
          Thu, 19 Nov 1998 10:48:05 -0800 (PST)
          (envelope-from nobody)
Message-Id: <199811191848.KAA04559@hub.freebsd.org>
Date: Thu, 19 Nov 1998 10:48:05 -0800 (PST)
From: lem@cantv.net
To: freebsd-gnats-submit@freebsd.org
Subject: pwd_mkdb is slow on many users
X-Send-Pr-Version: www-1.0

>Number:         8764
>Category:       misc
>Synopsis:       pwd_mkdb is slow on many users
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Thu Nov 19 10:50:01 PST 1998
>Closed-Date:    Fri Nov 16 15:09:52 PST 2001
>Last-Modified:  Fri Nov 16 15:10:20 PST 2001
>Originator:     Luis Munoz
>Release:        2.2.6-RELEASE
>Organization:
CANTV Servicios
>Environment:
bash-2.01# uname -a
FreeBSD rs8s1.datacenter.cha.cantv.net 2.2.6-RELEASE FreeBSD 2.2.6-RELEASE #0: F
ri Aug 21 11:05:02 GMT 1998     root@myname.my.domain:/usr/src/sys/compile/CCSER
V  i386

>Description:
On systems with large user counts (>= 50k), the time to rebuild
tha password database becomes quite large. One of our test systems
clocked 60+ minutes with 150k users. Increasing the buffer space
as suggested on one of the freebsd-* lists reduced this time to
44+ minutes, which is roughly a 25% improvement.

Probably further increasing cache could reduce the time by 50%.

>How-To-Repeat:

>Fix:
This is a patch for pwd_mkdb on i386 2.2.6-RELEASE. The patch allows
two new command-line options that reduce the time required to
update the password database. Those are documented in the man page
(after patches are applied :) In our text box, PII w256M RAM
and RAID disks, with 150k users, time to build a password database
went from 60+ minutes to 44+ minutes. Perhaps it could be improved
even more...

[If the patches do not work properly, please contact me directly
to have them sent via email -lem]

*** pwd_mkdb.8-orig	Thu Nov 19 18:40:36 1998
--- pwd_mkdb.8	Wed Nov 18 22:18:11 1998
***************
*** 39,46 ****
--- 39,48 ----
  .Nd "generate the password databases"
  .Sh SYNOPSIS
  .Nm pwd_mkdb
+ .Op Fl b Ar base size
  .Op Fl c
  .Op Fl p
+ .Op Fl s Ar cache size
  .Op Fl d Ar directory
  .Op Fl u Ar username
  .Ar file
***************
*** 63,74 ****
--- 65,89 ----
  .Pp
  The options are as follows:
  .Bl -tag -width flag
+ .It Fl b Ar base size
+ Use a factor to scale up certain parameters in the definition of
+ the database (see
+ .Xr hash 3 ) .
+ For large ammounts of users, this should be set to a rather large
+ factor, 64 or so, to speed up database creation. If left undefined
+ it defaults to reasonable values for around 500 users.
  .It Fl c
  Check if the password file is in the correct format. Do not
  change, add, or remove any files.
  .It Fl p
  Create a Version 7 style password file and install it into
  .Pa /etc/passwd .
+ .It Fl s Ar cache size
+ The database functions use a cache to speedup the process. This
+ flag allows the specification of a cache size. It's defined in
+ kilobytes. A larger cache tipically produces quicker processing
+ times. 8192 kilobytes provides plenty of space for larger user
+ databases.
  .It Fl d Ar directory
  Store databases into specified destination directory instead of
  .Pa /etc .

*** pwd_mkdb.c-orig	Wed Nov 18 21:34:59 1998
--- pwd_mkdb.c	Wed Nov 18 22:08:49 1998
***************
*** 100,106 ****
  	DBT data, sdata, key;
  	FILE *fp, *oldfp;
  	sigset_t set;
! 	int ch, cnt, ypcnt, len, makeold, tfd, yp_enabled = 0;
  	char *p, *t;
  	char buf[MAX(MAXPATHLEN, LINE_MAX * 2)], tbuf[1024];
  	char sbuf[MAX(MAXPATHLEN, LINE_MAX * 2)];
--- 100,107 ----
  	DBT data, sdata, key;
  	FILE *fp, *oldfp;
  	sigset_t set;
! 	int cachesize, basesize, ch, cnt, ypcnt, len, 
! 	  makeold, tfd, yp_enabled = 0;
  	char *p, *t;
  	char buf[MAX(MAXPATHLEN, LINE_MAX * 2)], tbuf[1024];
  	char sbuf[MAX(MAXPATHLEN, LINE_MAX * 2)];
***************
*** 114,121 ****
  	strcpy(prefix, _PATH_PWD);
  	makeold = 0;
  	username = NULL;
! 	while ((ch = getopt(argc, argv, "cd:pu:v")) != -1)
  		switch(ch) {
  		case 'c':                       /* verify only */
  			cflag = 1;
  			break;
--- 115,127 ----
  	strcpy(prefix, _PATH_PWD);
  	makeold = 0;
  	username = NULL;
! 	basesize = 1;		/* Default base size for DB */
! 	cachesize = 2048;	/* Default cache size for DB */
! 	while ((ch = getopt(argc, argv, "s:b:cd:pu:v")) != -1)
  		switch(ch) {
+ 		case 'b':	/* Set bucket size */
+ 		  basesize = atoi(optarg);
+ 		  break;
  		case 'c':                       /* verify only */
  			cflag = 1;
  			break;
***************
*** 125,130 ****
--- 131,139 ----
  		case 'p':			/* create V7 "file.orig" */
  			makeold = 1;
  			break;
+ 		case 's':	/* Set buffer/cache size */
+ 		  cachesize = atoi(optarg);
+ 		  break;
  		case 'u':			/* only update this record */
  			username = optarg;
  			break;
***************
*** 139,144 ****
--- 148,173 ----
  	if (argc != 1 || (username && (*username == '+' || *username == '-')))
  		usage();
  
+ 	if (basesize <= 0) {
+ 	  error("Base size must be an integer >= 0");
+ 	  usage();
+ 	}
+ 
+ 	if (cachesize <= 0) {
+ 	  error("Cache size must be an integer >= 0");
+ 	  usage();
+ 	}
+ 
+ 				/* update the openinfo structure */
+ 
+ 	openinfo.bsize = (u_int) 4096 * basesize;
+ 	if (openinfo.bsize > 16384) {
+ 	  openinfo.bsize = 16384;
+ 	}
+ 	openinfo.ffactor = (u_int) 32 * basesize;
+ 	openinfo.nelem = (u_int) 256 * basesize * basesize;
+ 	openinfo.cachesize = cachesize * 1024;
+ 
  	/*
  	* This could be changed to allow the user to interrupt.
  	* Probably not worth the effort.
***************
*** 577,582 ****
  {
  
  	(void)fprintf(stderr,
! "usage: pwd_mkdb [-c] [-p] [-d <dest dir>] [-u <local username>] file\n");
  	exit(1);
  }
--- 606,611 ----
  {
  
  	(void)fprintf(stderr,
! "usage: pwd_mkdb [-b <size>] [-c] [-p] [-d <dest dir>] [-s <cache size>] [-u <local username>] file\n");
  	exit(1);
  }

>Release-Note:
>Audit-Trail:

From: Ruslan Ermilov <ru@ucb.crimea.ua>
To: lem@cantv.net
Cc: freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: misc/8764: pwd_mkdb is slow on many users
Date: Fri, 20 Nov 1998 09:51:17 +0200

 On Thu, Nov 19, 1998 at 10:48:05AM -0800, lem@cantv.net wrote:
 
 > On systems with large user counts (>= 50k), the time to rebuild
 > tha password database becomes quite large. One of our test systems
 > clocked 60+ minutes with 150k users. Increasing the buffer space
 > as suggested on one of the freebsd-* lists reduced this time to
 > 44+ minutes, which is roughly a 25% improvement.
 > 
 > Probably further increasing cache could reduce the time by 50%.
 > 
 
 Did you try `-u' switch of pwd_mkdb?
 
 * -u username
 *       Only update the record for the specified user.  Utilities that op-
 *       erate on a single user can use this option to avoid the overhead of
 *       rebuilding the entire database.
 
 
 -- 
 Ruslan Ermilov		Sysadmin and DBA of the
 ru@ucb.crimea.ua	United Commercial Bank
 +380.652.247.647	Simferopol, Ukraine
 
 http://www.FreeBSD.org	The Power To Serve
 http://www.oracle.com	Enabling The Information Age

From: Luis Munoz <lem@cantv.net>
To: Ruslan Ermilov <ru@ucb.crimea.ua>
Cc: lem@cantv.net, freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: misc/8764: pwd_mkdb is slow on many users
Date: Fri, 20 Nov 1998 09:40:43 -0400

 At 03:51 AM 11/20/98 , Ruslan Ermilov wrote:
 >On Thu, Nov 19, 1998 at 10:48:05AM -0800, lem@cantv.net wrote:
 >
 >> On systems with large user counts (>= 50k), the time to rebuild
 >> tha password database becomes quite large. One of our test systems
 >> clocked 60+ minutes with 150k users. Increasing the buffer space
 >> as suggested on one of the freebsd-* lists reduced this time to
 >> 44+ minutes, which is roughly a 25% improvement.
 >> 
 >> Probably further increasing cache could reduce the time by 50%.
 >> 
 >
 >Did you try `-u' switch of pwd_mkdb?
 
 Yep. Takes about the same time (~35 - ~40 min per user). In our
 case, we have to make some 800 mods a day to the user database, 
 so this gets a little impractical :)
 
 The included patch allows pwd_mkdb to resize the buffers used 
 by the DB routines, actually reducing the time it takes for
 a full rebuild. I'm not sure wether this patch might have an
 effect in the -u case.
 
 Anyway, me filling a pr was a suggestion from Doug White. My
 intent is to offer you guys the patch so that they can be
 included in future releases of the code. There are significant
 improvements in speed using the patch with 30K or so users, so
 I think many people might find it useful.
 
 Thanks a lot for your answer.
 
 -lem
 
 >
 >* -u username
 >*       Only update the record for the specified user.  Utilities that op-
 >*       erate on a single user can use this option to avoid the overhead of
 >*       rebuilding the entire database.
 >
 >
 >-- 
 >Ruslan Ermilov		Sysadmin and DBA of the
 >ru@ucb.crimea.ua	United Commercial Bank
 >+380.652.247.647	Simferopol, Ukraine
 >
 >http://www.FreeBSD.org	The Power To Serve
 >http://www.oracle.com	Enabling The Information Age
 
  ------------
 Luis E. Muoz
 Centro de Operaciones de la Red
 CANTV Servicios
State-Changed-From-To: open->closed 
State-Changed-By: jedgar 
State-Changed-When: Fri Nov 16 15:09:52 PST 2001 
State-Changed-Why:  
Support for this was added to 2.2.8-STABLE. 

http://www.FreeBSD.org/cgi/query-pr.cgi?pr=8764 
>Unformatted:
