From nobody@FreeBSD.org  Tue Jul 17 12:31:05 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id C75DE16A400
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 17 Jul 2007 12:31:05 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (groups.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id B85B913C4AA
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 17 Jul 2007 12:31:05 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.1/8.14.1) with ESMTP id l6HCV5Lb090878
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 17 Jul 2007 12:31:05 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.1/8.14.1/Submit) id l6HCV5kn090877;
	Tue, 17 Jul 2007 12:31:05 GMT
	(envelope-from nobody)
Message-Id: <200707171231.l6HCV5kn090877@www.freebsd.org>
Date: Tue, 17 Jul 2007 12:31:05 GMT
From: Michael Hanselmann <freebsd@hansmi.ch>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Implement getgroupmembership in nss_ldap
X-Send-Pr-Version: www-3.0
X-GNATS-Notify: kazakov@gmail.com

>Number:         114655
>Category:       kern
>Synopsis:       Implement getgroupmembership in net/nss_ldap
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    bushman
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          update
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jul 17 12:40:03 GMT 2007
>Closed-Date:    Thu Apr 29 21:41:25 UTC 2010
>Last-Modified:  Thu Apr 29 21:41:25 UTC 2010
>Originator:     Michael Hanselmann
>Release:        6.x, 7
>Organization:
>Environment:
>Description:
(Copied from freebsd-current@)
I was working with a company which plans to migrate its FreeBSD servers
from using /etc/{passwd,group} to LDAP. They will have about 45'000
users and as much groups in the directory.

Tests showed that any function retrieving the groups a user is member
of, for example getgrouplist(3) or initgroups(3), is very slow. In our
case, it was about 7 seconds per invocation. Further investigation
showed how inefficient these functions are implemented through
getgrouplist(3). FreeBSD's implementation loops through all groups and
their members to check whether a user is member of it, in which case it
adds the group to a list. In our case, this means retrieving 45'000
search results from the LDAP server.

Directory services like LDAP or Winbind allow queries to have filters,
enabling us to write a much more efficient implementation. The attached
patches (nss-getgroupmembership-try9.diff for FreeBSD 6,
nss-getgroupmembership-fbsd7-try3.diff for FreeBSD 7) use an nss
module's getgroupmembership(3) function if available. Otherwise it uses
a fallback which then uses the old algorithm with some modifications.
After applying it, getgrouplist(3) takes only a few milliseconds to
retrieve all groups of a user.

Another patch, attached as bsdnss.diff, is needed for nss_ldap. It
applies to the ports/net/nss_ldap/files/bsdnss.c file and exports the
required getgroupmembership function. Most of the code there is from
NetBSD.
>How-To-Repeat:

>Fix:
Apply the patch.

Patch attached with submission follows:

--- bsdnss.c.orig	Mon May 14 14:25:05 2007
+++ bsdnss.c	Mon May 14 15:01:06 2007
@@ -1,9 +1,11 @@
+#include <stdlib.h>
 #include <errno.h>
 #include <sys/param.h>
 #include <netinet/in.h>
 #include <pwd.h>
 #include <grp.h>
 #include <nss.h>
+#include <nsswitch.h>
 #include <netdb.h>
 
 extern enum nss_status _nss_ldap_getgrent_r(struct group *, char *, size_t,
@@ -34,12 +36,15 @@ extern enum nss_status _nss_ldap_gethost
 extern enum nss_status _nss_ldap_gethostbyaddr_r (struct in_addr * addr, int len, int type,
 			   struct hostent * result, char *buffer,
 			   size_t buflen, int *errnop, int *h_errnop);
+extern enum nss_status _nss_ldap_initgroups_dyn(const char *, gid_t, long int *,
+			   long int *, gid_t **, long int, int *);
 
 NSS_METHOD_PROTOTYPE(__nss_compat_getgrnam_r);
 NSS_METHOD_PROTOTYPE(__nss_compat_getgrgid_r);
 NSS_METHOD_PROTOTYPE(__nss_compat_getgrent_r);
 NSS_METHOD_PROTOTYPE(__nss_compat_setgrent);
 NSS_METHOD_PROTOTYPE(__nss_compat_endgrent);
+static NSS_METHOD_PROTOTYPE(__freebsd_getgroupmembership);
 
 NSS_METHOD_PROTOTYPE(__nss_compat_getpwnam_r);
 NSS_METHOD_PROTOTYPE(__nss_compat_getpwuid_r);
@@ -57,6 +62,7 @@ static ns_mtab methods[] = {
 { NSDB_GROUP, "getgrent_r", __nss_compat_getgrent_r, _nss_ldap_getgrent_r },
 { NSDB_GROUP, "setgrent",   __nss_compat_setgrent,   _nss_ldap_setgrent },
 { NSDB_GROUP, "endgrent",   __nss_compat_endgrent,   _nss_ldap_endgrent },
+{ NSDB_GROUP, "getgroupmembership", __freebsd_getgroupmembership, NULL },
 
 { NSDB_PASSWD, "getpwnam_r", __nss_compat_getpwnam_r, _nss_ldap_getpwnam_r },
 { NSDB_PASSWD, "getpwuid_r", __nss_compat_getpwuid_r, _nss_ldap_getpwuid_r },
@@ -155,4 +161,62 @@ int __nss_compat_gethostbyaddr(void *ret
 	status = __nss_compat_result(status,errnop);
 	h_errno = h_errnop;
 	return (status);
+}
+
+static int
+__gr_addgid(gid_t gid, gid_t *groups, int maxgrp, int *groupc)
+{
+	int	ret, dupc;
+
+						/* skip duplicates */
+	for (dupc = 0; dupc < MIN(maxgrp, *groupc); dupc++) {
+		if (groups[dupc] == gid)
+			return 1;
+	}
+
+	ret = 1;
+	if (*groupc < maxgrp)			/* add this gid */
+		groups[*groupc] = gid;
+	else
+		ret = 0;
+	(*groupc)++;
+	return ret;
+}
+
+static int
+__freebsd_getgroupmembership(void *retval, void *mdata, va_list ap)
+{
+	int err;
+	enum nss_status s;
+	int *result = va_arg(ap, int *);
+	const char *user = va_arg(ap, const char *);
+	gid_t group = va_arg(ap, gid_t);
+	gid_t *groups = va_arg(ap, gid_t *);
+	int limit = va_arg(ap, int);
+	int *size = va_arg(ap, int*);
+	gid_t *tmpgroups;
+	long int lstart, lsize;
+	int i;
+
+	tmpgroups = malloc(limit * sizeof(gid_t));
+	if (tmpgroups == NULL)
+		return NS_TRYAGAIN;
+
+	/* insert primary membership */
+	__gr_addgid(group, groups, limit, size);
+
+	lstart = 0;
+	lsize = limit;
+	s = _nss_ldap_initgroups_dyn(user, group, &lstart, &lsize,
+		&tmpgroups, 0, &err);
+	if (s == NSS_STATUS_SUCCESS) {
+		for (i = 0; i < lstart; i++)
+			if (! __gr_addgid(tmpgroups[i], groups, limit, size))
+				*result = -1;
+		s = NSS_STATUS_NOTFOUND;
+	}
+
+	free(tmpgroups);
+
+	return __nss_compat_result(s, 0);
 }


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: edwin 
State-Changed-When: Tue Jul 17 22:33:02 UTC 2007 
State-Changed-Why:  
Awaiting maintainers feedback 

http://www.freebsd.org/cgi/query-pr.cgi?pr=114655 
Responsible-Changed-From-To: freebsd-ports-bugs->tmclaugh 
Responsible-Changed-By: tmclaugh 
Responsible-Changed-When: Sat Jul 21 05:25:14 UTC 2007 
Responsible-Changed-Why:  
Dear maintainer, can you provide feedback on this patch?  By the way, 
1.256 is out.  I've been usuing it for a week with no issues. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=114655 

From: Michael Hanselmann <freebsd@hansmi.ch>
To: bug-followup@FreeBSD.org
Cc: kazakov@gmail.com
Subject: Re: ports/114655: Implement getgroupmembership in net/nss_ldap
Date: Mon, 30 Jul 2007 20:43:33 +0200

 Hi kazakov
 
 Can you please review my patch attached to PR 114655?
 
 Thanks,
 Michael

From: Craig Rodrigues <rodrigc@crodrigues.org>
To: bug-followup@FreeBSD.org, freebsd@hansmi.ch
Cc:  
Subject: Re: ports/114655: Implement getgroupmembership in net/nss_ldap
Date: Sat, 4 Aug 2007 11:07:52 -0400

 Hi,
 
 See related patch against libc at:
 http://www.freebsd.org/cgi/query-pr.cgi?pr=115196
 -- 
 Craig Rodrigues        
 rodrigc@crodrigues.org

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: ports/114655: commit references a PR
Date: Wed, 12 Dec 2007 10:08:09 +0000 (UTC)

 bushman     2007-12-12 10:08:03 UTC
 
   FreeBSD src repository
 
   Modified files:
     include              nsswitch.h 
     lib/libc/gen         getgrent.c getgrouplist.c 
     lib/libc/net         nsdispatch.c 
   Log:
   Implementing 'fallback' nsswitch source. 'fallback' source is used
   when particular function can't be found in nsswitch-module. For
   example, getgrouplist(3) will use module-supplied 'getgroupmembership'
   function (which can work in an optimal way for such source as LDAP) and
   will fall back to the stanard iterate-through-all-groups implementation
   otherwise.
   
   PR:             ports/114655
   Submitted by:   Michael Hanselmann <freebsd AT hansmi DOT ch>
   Reviewed by:    brooks (mentor)
   
   Revision  Changes    Path
   1.5       +3 -1      src/include/nsswitch.h
   1.37      +190 -74   src/lib/libc/gen/getgrent.c
   1.16      +4 -39     src/lib/libc/gen/getgrouplist.c
   1.15      +26 -3     src/lib/libc/net/nsdispatch.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: feedback->closed 
State-Changed-By: linimon 
State-Changed-When: Fri Feb 29 03:13:23 UTC 2008 
State-Changed-Why:  
Feedback timeout (several months).  tmclaugh had noted that he was not 
seeing any issues with the latest version; a fix has already been 
committed to libc. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=114655 
State-Changed-From-To: closed->patched 
State-Changed-By: linimon 
State-Changed-When: Fri Feb 29 03:14:56 UTC 2008 
State-Changed-Why:  
Reopen.  The libc patches that were made as a consequence of this PR 
have not been MFCed to RELENG_6 yet. 


Responsible-Changed-From-To: tmclaugh->bushman 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Feb 29 03:14:56 UTC 2008 
Responsible-Changed-Why:  
Assign to committer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=114655 
State-Changed-From-To: patched->closed 
State-Changed-By: linimon 
State-Changed-When: Thu Apr 29 21:40:13 UTC 2010 
State-Changed-Why:  
Committer has turned in his commit bit for safekeeping.  Since this PR is about 
an MFC to 6, and 6 is nearing EOL, it is unlikely that it will be committed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=114655 
>Unformatted:
