From nobody@FreeBSD.org  Tue Mar 15 16:32:27 2005
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id DD5D216A4CF
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 15 Mar 2005 16:32:27 +0000 (GMT)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 7E3E943D2D
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 15 Mar 2005 16:32:27 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id j2FGWRXQ004945
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 15 Mar 2005 16:32:27 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id j2FGWQGA004944;
	Tue, 15 Mar 2005 16:32:26 GMT
	(envelope-from nobody)
Message-Id: <200503151632.j2FGWQGA004944@www.freebsd.org>
Date: Tue, 15 Mar 2005 16:32:26 GMT
From: Cyril Vechera <cv-c@fluid.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: nfs client cache negative lookups
X-Send-Pr-Version: www-2.3

>Number:         78884
>Category:       kern
>Synopsis:       [nfs] [patch] nfs client cache negative lookups
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          suspended
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Tue Mar 15 16:40:02 GMT 2005
>Closed-Date:    
>Last-Modified:  Sun Mar 02 02:16:31 UTC 2008
>Originator:     Cyril Vechera
>Release:        RELENG_5
>Organization:
Net Ltd.
>Environment:
FreeBSD cv.ws.local 5.3-RELEASE-p5 FreeBSD 5.3-RELEASE-p5 #1: Tue Mar 15 20:54:40 MSK 2005     root@5.devel.incru.net:/usr/src/sys/i386/compile/OFFICIER  i386
>Description:
Current NFS client does not cache negative lookups that results significant
network overload with identical lookup requests when searching in the PATH,
libraries, configs etc.

When the caching of negative lookups is active it reduces meta-data traffic
close to 80% and increases applictions startup time.


>How-To-Repeat:

>Fix:
Implementation below does not fit strong caching policy requirements due
to lack of cached entries time-to-live support.  So negative caching is off
by default and can be turned on via sysctl or mount_nfs option -C
(negcache).  But it is reasonable enough to turn it on everywhere with
read-only NFS root and mounts with single-client. For more effective
negative caching I'd recommend to set sysctl debug.ncnegfactor less than
default 16, maybe to 4 or 2.

Here are two patches - with sysctl control and with mount flag control

variant 1, sysctl:
====cut here
*** /usr/src/sys/nfs_vnops.c.orig       Tue Mar 15 00:17:26 2005
--- /usr/src/sys/nfs_vnops.c    Tue Mar 15 00:20:08 2005
***************
*** 238,243 ****
--- 238,247 ----
  static int    nfsv3_commit_on_close = 0;
  SYSCTL_INT(_vfs_nfs, OID_AUTO, nfsv3_commit_on_close, CTLFLAG_RW,
           &nfsv3_commit_on_close, 0, "write+commit on close, else only write");
+
+ static int    nfs_cache_negative_lookup = 0;
+ SYSCTL_INT(_vfs_nfs, OID_AUTO, cache_negative_lookup, CTLFLAG_RW,
+          &nfs_cache_negative_lookup, 0, "cache negative lookup");
  #if 0
  SYSCTL_INT(_vfs_nfs, OID_AUTO, access_cache_hits, CTLFLAG_RD,
           &nfsstats.accesscache_hits, 0, "NFS ACCESS cache hit count");
***************
*** 767,776 ****
        wantparent = flags & (LOCKPARENT|WANTPARENT);
        nmp = VFSTONFS(dvp->v_mount);
        np = VTONFS(dvp);
!       if ((error = cache_lookup(dvp, vpp, cnp)) && error != ENOENT) {
                struct vattr vattr;
                int vpid;

                if ((error = VOP_ACCESS(dvp, VEXEC, cnp->cn_cred, td)) != 0) {
                        *vpp = NULLVP;
                        return (error);
--- 767,782 ----
        wantparent = flags & (LOCKPARENT|WANTPARENT);
        nmp = VFSTONFS(dvp->v_mount);
        np = VTONFS(dvp);
!       error = cache_lookup(dvp, vpp, cnp);
!       if (error) {
                struct vattr vattr;
                int vpid;

+               if (error == ENOENT) {
+                       nfsstats.lookupcache_hits++;
+                       return (error);
+               }
+
                if ((error = VOP_ACCESS(dvp, VEXEC, cnp->cn_cred, td)) != 0) {
                        *vpp = NULLVP;
                        return (error);
***************
*** 941,946 ****
--- 946,953 ----
                }
                if (cnp->cn_nameiop != LOOKUP && (flags & ISLASTCN))
                        cnp->cn_flags |= SAVENAME;
+               if (error == ENOENT && nfs_cache_negative_lookup && (cnp->cn_flags & MAKEENTRY))
+                       cache_enter(dvp, NULL, cnp);
        }
        return (error);
  }
====cut here

variant 2, mount_nfs option:
====cut here
*** sys/nfs_vnops.c.orig        Tue Mar 15 00:17:26 2005
--- sys/nfs_vnops.c     Tue Mar 15 00:22:02 2005
***************
*** 767,776 ****
        wantparent = flags & (LOCKPARENT|WANTPARENT);
        nmp = VFSTONFS(dvp->v_mount);
        np = VTONFS(dvp);
!       if ((error = cache_lookup(dvp, vpp, cnp)) && error != ENOENT) {
                struct vattr vattr;
                int vpid;

                if ((error = VOP_ACCESS(dvp, VEXEC, cnp->cn_cred, td)) != 0) {
                        *vpp = NULLVP;
                        return (error);
--- 767,782 ----
        wantparent = flags & (LOCKPARENT|WANTPARENT);
        nmp = VFSTONFS(dvp->v_mount);
        np = VTONFS(dvp);
!       error = cache_lookup(dvp, vpp, cnp);
!       if (error) {
                struct vattr vattr;
                int vpid;

+               if (error == ENOENT) {
+                       nfsstats.lookupcache_hits++;
+                       return (error);
+               }
+
                if ((error = VOP_ACCESS(dvp, VEXEC, cnp->cn_cred, td)) != 0) {
                        *vpp = NULLVP;
                        return (error);
***************
*** 941,946 ****
--- 942,949 ----
                }
                if (cnp->cn_nameiop != LOOKUP && (flags & ISLASTCN))
                        cnp->cn_flags |= SAVENAME;
+               if (error == ENOENT && (nmp->nm_flag & NFSMNT_NEGCACHE) && (cnp->cn_flags & MAKEENTRY))
+                       cache_enter(dvp, NULL, cnp);
        }
        return (error);
  }
*** sys/nfsargs.h.orig  Tue Mar 15 00:25:40 2005
--- sys/nfsargs.h       Tue Mar 15 00:24:36 2005
***************
*** 78,83 ****
--- 78,84 ----
  #define       NFSMNT_NOCONN           0x00000080  /* Don't Connect the socket */
  /* 0x100 free, was NFSMNT_NQNFS */
  #define       NFSMNT_NFSV3            0x00000200  /* Use NFS Version 3 protocol */
+ #define       NFSMNT_NEGCACHE         0x00000400  /* Cache negative lookups */
  /* 0x400 free, was NFSMNT_KERB */
  #define       NFSMNT_DUMBTIMR         0x00000800  /* Don't estimate rtt dynamically */
  /* 0x1000 free, was NFSMNT_LEASETERM */
====cut here

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->cel 
Responsible-Changed-By: cel 
Responsible-Changed-When: Wed May 24 19:40:16 UTC 2006 
Responsible-Changed-Why:  
An interesting performance enhancement.  Lots of testing will be needed 
here because negative caching can be dangerous for some workloads. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=78884 
Responsible-Changed-From-To: cel->freebsd-bugs 
Responsible-Changed-By: cel 
Responsible-Changed-When: Mon Mar 12 15:23:55 UTC 2007 
Responsible-Changed-Why:  
Back to the public pool. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=78884 
State-Changed-From-To: open->feedback 
State-Changed-By: kmacy 
State-Changed-When: Sun Nov 18 08:33:13 UTC 2007 
State-Changed-Why:  

Would you be OK with this only being enabled for R/O mounts? 
If so, could you update the patch for HEAD? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=78884 
State-Changed-From-To: feedback->suspended 
State-Changed-By: linimon 
State-Changed-When: Sun Mar 2 02:16:01 UTC 2008 
State-Changed-Why:  
An updated patch was never received, but it sounds as though this could 
still be a problem. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=78884 
>Unformatted:
