From sysmaint@filbert.contek.com  Thu Nov 11 10:06:32 2004
Return-Path: <sysmaint@filbert.contek.com>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id C1B8916A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 11 Nov 2004 10:06:32 +0000 (GMT)
Received: from filbert.contek.com (contek.ultranet.com [146.115.254.63])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 4233243D31
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 11 Nov 2004 10:06:31 +0000 (GMT)
	(envelope-from sysmaint@filbert.contek.com)
Received: by filbert.contek.com (Postfix, from userid 10112)
	id E4CA710730; Wed, 10 Nov 2004 21:54:02 -0500 (EST)
Message-Id: <20041111025402.E4CA710730@filbert.contek.com>
Date: Wed, 10 Nov 2004 21:54:02 -0500 (EST)
From: Douglas Wells <sysmaint@filbert.contek.com>
Reply-To: Douglas Wells <sysmaint@filbert.contek.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc: Douglas Wells <sysmaint@filbert.contek.com>
Subject: panic: mutex nfsd_mtx not owned in NFS server
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         73807
>Category:       kern
>Synopsis:       panic: mutex nfsd_mtx not owned in NFS server
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    rwatson
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Nov 11 10:10:21 GMT 2004
>Closed-Date:    Sat Apr 09 22:19:23 GMT 2005
>Last-Modified:  Sat Apr 09 22:19:23 GMT 2005
>Originator:     Douglas Wells
>Release:        FreeBSD 5.3
>Organization:
Connection Technologies
>Environment:
System: FreeBSD filbert.contek.com 5.2.1-RELEASE-p11 FreeBSD 5.2.1-RELEASE-p11 #2: Wed Nov 10 16:59:00 EST 2004 root@filbert.contek.com:/var/obj/usr/src.5.2.1/sys/FILBERT i386

	The problem environment is the same as the system above
	except that the OS is FreeBSD 5.3 - cvsup updated (RELENG_5_3)
	November 9 evening. Updated from 5.2.1 via recommended
	buildworld / buildkernel sequence.

>Description:

	The FreeBSD system panics when I attempt to boot a diskless
	Sun 3 that mounts all filesystems from the FreeBSD system.

	The problem began as a NULL pointer dereference when I
	used a custom SMP kernel. The same NULL pointer deference
	occurred when I repeated the problem with a stock GENERIC
	kernel compiled from the same sources. I then ran a variant
	of GENERIC with the following options:
		options	INVARIANT_SUPPORT
		options	INVARIANTS
		options	KDB_TRACE
		options	KDB
		options	DDB

	Repeating the bootload of the Sun 3 led to the following
	panic, which I presume to be due to the same cause as the
	NULL pointer deference: (This information is retyped.)

	    panic: mutex nfsd_mtx not owned at ... nfs_serv.c:4493

	    mtx_assert+5c (c08d97c0, 1, c08160f7, 118d)
	    nfsrv_access+28 (c7217420, 80, c2142180, 0, c1a79190)
	    nfsrv_create+84c (c22142100, c1d62d80, c1a79190, d5469c98, c08d97c0)
	    nfssvc_nfsd+3d5 (c1a79190, c08067a3, 12b, c1c360bc, c1c36000)
	    nfssvc+18c (c1a79190, d5469d14, 2, 0, 296)
	    syscall+213 (2f, 2f, 2f, bf??eec4, 4)

	    thread 100051

	I repeated the experiment several times and generally the
	addresses and thread id vary from run to run. The exceptions
	are the last arg (c08d97c0) to nfsrv_create and the second
	arg (c08067a3) to nfssvc_nfsd.

	Analysis:

	I believe that the problem is due to the system mishandling
	RPC requests for mounts of regular files (mountd(8) -r):
	  - If I remove the -r option to mountd and attempt to
	    boot the Sun 3, the FreeBSD system does not panic (but
	    of course, the Sun 3 does not boot properly).
	  - If I leave the Sun 3 up from having booted from a 5.2.1
	    system, the FreeBSD system panics very soon after it
	    starts up system daemons. (Note that the mountd daemon
	    would not be invoked in this case. The NFS request
	    would use information retained from the previous (5.2.1)
	    boot.)

	Additional notes:
	  - This bootload operationg has previously worked regularly
	    on both 5.1 and 5.2+.
	  - I have modified mountd to support v2 mounts, but I
	    disabled my extensions during at least some of the
	    tests above. These extensions are simple.
	  - Note that the hardware system has 2 AMD Athlon CPUs,
	    but I understand that by not including the SMP kernel
	    option, the OS behaves as though the hardware were
	    uniprocessor.

>How-To-Repeat:

	This happens repeatably whenever I attempt to bootload the
	Sun 3. I presume that sending the NFS server an I/O request
	for any regular file that is being treated as a mount
	point would also trigger this problem.
>Fix:

	None known.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->rwatson 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Thu Nov 11 18:17:08 GMT 2004 
Responsible-Changed-Why:  
I'll grab this one, since I've had my hands in NFS server locking. 
Sounds like we fail to re-grab a required mutex following a VFS operation. 
I will investigate.  Thanks! 


http://www.freebsd.org/cgi/query-pr.cgi?pr=73807 

From: Robert Watson <rwatson@freebsd.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:  
Subject: Re: kern/73807: panic: mutex nfsd_mtx not owned in NFS server
Date: Thu, 11 Nov 2004 20:09:34 +0000 (GMT)

 On Thu, 11 Nov 2004, Robert Watson wrote:
 
 > I'll grab this one, since I've had my hands in NFS server locking. 
 > Sounds like we fail to re-grab a required mutex following a VFS
 > operation.  I will investigate.  Thanks! 
 
 Coult you try the following patch, which cleans up a locking bug
 associated with calls to nfsrv_access()?  Prior to this patch, the NFS
 server lock must be held entering nfsrv_access(), and it drops that lock
 and acquires Giant to enter VFS.  This patch splits the function into two:
 one to be called w/o the NFS server lock but with Giant, and the other to
 be called with the lock and w/o Giant.  It also cleans up some
 inefficiencies in locking.  I ran a basic set of NFS server tests here,
 but this is not yet extensively tested.
 
 Thanks,
 
 Index: nfs_serv.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/nfsserver/nfs_serv.c,v
 retrieving revision 1.150
 diff -u -r1.150 nfs_serv.c
 --- nfs_serv.c	22 Oct 2004 08:47:20 -0000	1.150
 +++ nfs_serv.c	11 Nov 2004 18:38:36 -0000
 @@ -137,6 +137,9 @@
  SYSCTL_STRUCT(_vfs_nfsrv, NFS_NFSRVSTATS, nfsrvstats, CTLFLAG_RD,
  	&nfsrvstats, nfsrvstats, "S,nfsrvstats");
  
 +static int	nfsrv_access_withgiant(struct vnode *vp, int flags,
 +		    struct ucred *cred, int rdonly, struct thread *td,
 +		    int override);
  static int	nfsrv_access(struct vnode *, int, struct ucred *, int,
  		    struct thread *, int);
  static void	nfsrvw_coalesce(struct nfsrv_descript *,
 @@ -195,8 +198,10 @@
  		goto nfsmout;
  	}
  	nfsmode = fxdr_unsigned(u_int32_t, *tl);
 +	NFSD_UNLOCK();
 +	mtx_lock(&Giant);	/* VFS */
  	if ((nfsmode & NFSV3ACCESS_READ) &&
 -		nfsrv_access(vp, VREAD, cred, rdonly, td, 0))
 +		nfsrv_access_withgiant(vp, VREAD, cred, rdonly, td, 0))
  		nfsmode &= ~NFSV3ACCESS_READ;
  	if (vp->v_type == VDIR)
  		testmode = (NFSV3ACCESS_MODIFY | NFSV3ACCESS_EXTEND |
 @@ -204,17 +209,15 @@
  	else
  		testmode = (NFSV3ACCESS_MODIFY | NFSV3ACCESS_EXTEND);
  	if ((nfsmode & testmode) &&
 -		nfsrv_access(vp, VWRITE, cred, rdonly, td, 0))
 +		nfsrv_access_withgiant(vp, VWRITE, cred, rdonly, td, 0))
  		nfsmode &= ~testmode;
  	if (vp->v_type == VDIR)
  		testmode = NFSV3ACCESS_LOOKUP;
  	else
  		testmode = NFSV3ACCESS_EXECUTE;
  	if ((nfsmode & testmode) &&
 -		nfsrv_access(vp, VEXEC, cred, rdonly, td, 0))
 +		nfsrv_access_withgiant(vp, VEXEC, cred, rdonly, td, 0))
  		nfsmode &= ~testmode;
 -	NFSD_UNLOCK();
 -	mtx_lock(&Giant);	/* VFS */
  	getret = VOP_GETATTR(vp, vap, cred, td);
  	vput(vp);
  	mtx_unlock(&Giant);	/* VFS */
 @@ -857,12 +860,14 @@
  		else
  			error = (vp->v_type == VDIR) ? EISDIR : EACCES;
  	}
 -	if (!error) {
 -		if ((error = nfsrv_access(vp, VREAD, cred, rdonly, td, 1)) != 0)
 -			error = nfsrv_access(vp, VEXEC, cred, rdonly, td, 1);
 -	}
  	NFSD_UNLOCK();
  	mtx_lock(&Giant);	/* VFS */
 +	if (!error) {
 +		if ((error = nfsrv_access_withgiant(vp, VREAD, cred, rdonly,
 +		    td, 1)) != 0)
 +			error = nfsrv_access_withgiant(vp, VEXEC, cred,
 +			    rdonly, td, 1);
 +	}
  	getret = VOP_GETATTR(vp, vap, cred, td);
  	if (!error)
  		error = getret;
 @@ -1497,9 +1502,11 @@
  		} else {
  		    vp = NULL;
  		}
 -		if (!error)
 -		    error = nfsrv_access(vp, VWRITE, cred, rdonly, td, 1);
  		NFSD_UNLOCK();
 +		mtx_lock(&Giant);	/* VFS */
 +		if (!error)
 +		    error = nfsrv_access_withgiant(vp, VWRITE, cred, rdonly,
 +			td, 1);
  		if (nfsd->nd_stable == NFSV3WRITE_UNSTABLE)
  		    ioflags = IO_NODELOCKED;
  		else if (nfsd->nd_stable == NFSV3WRITE_DATASYNC)
 @@ -1511,7 +1518,6 @@
  		uiop->uio_td = NULL;
  		uiop->uio_offset = nfsd->nd_off;
  		uiop->uio_resid = nfsd->nd_eoff - nfsd->nd_off;
 -		mtx_lock(&Giant);	/* VFS */
  		if (uiop->uio_resid > 0) {
  		    mp = mrep;
  		    i = 0;
 @@ -1910,8 +1916,8 @@
  		}
  	} else {
  		if (vap->va_size != -1) {
 -			error = nfsrv_access(nd.ni_vp, VWRITE, cred,
 -			    (nd.ni_cnd.cn_flags & RDONLY), td, 0);
 +			error = nfsrv_access_withgiant(nd.ni_vp, VWRITE,
 +			    cred, (nd.ni_cnd.cn_flags & RDONLY), td, 0);
  			if (!error) {
  				tempsize = vap->va_size;
  				VATTR_NULL(vap);
 @@ -3370,13 +3376,8 @@
  			error = NFSERR_BAD_COOKIE;
  #endif
  	}
 -	if (!error) {
 -		mtx_unlock(&Giant);	/* VFS */
 -		NFSD_LOCK();
 -		error = nfsrv_access(vp, VEXEC, cred, rdonly, td, 0);
 -		NFSD_UNLOCK();
 -		mtx_lock(&Giant);	/* VFS */
 -	}
 +	if (!error)
 +		error = nfsrv_access_withgiant(vp, VEXEC, cred, rdonly, td, 0);
  	if (error) {
  		vput(vp);
  		mtx_unlock(&Giant);	/* VFS */
 @@ -3685,13 +3686,8 @@
  	if (!error && toff && verf && verf != at.va_filerev)
  		error = NFSERR_BAD_COOKIE;
  #endif
 -	if (!error) {
 -		mtx_unlock(&Giant);	/* VFS */
 -		NFSD_LOCK();
 -		error = nfsrv_access(vp, VEXEC, cred, rdonly, td, 0);
 -		NFSD_UNLOCK();
 -		mtx_lock(&Giant);	/* VFS */
 -	}
 +	if (!error)
 +		error = nfsrv_access_withgiant(vp, VEXEC, cred, rdonly, td, 0);
  	if (error) {
  		vput(vp);
  		mtx_unlock(&Giant);	/* VFS */
 @@ -4473,7 +4469,8 @@
   * Perform access checking for vnodes obtained from file handles that would
   * refer to files already opened by a Unix client. You cannot just use
   * vn_writechk() and VOP_ACCESS() for two reasons.
 - * 1 - You must check for exported rdonly as well as MNT_RDONLY for the write case
 + * 1 - You must check for exported rdonly as well as MNT_RDONLY for the write
 + *     case.
   * 2 - The owner is to be given access irrespective of mode bits for some
   *     operations, so that processes that chmod after opening a file don't
   *     break. I don't like this because it opens a security hole, but since
 @@ -4482,17 +4479,23 @@
   *
   * The exception to rule 2 is EPERM. If a file is IMMUTABLE, VOP_ACCESS()
   * will return EPERM instead of EACCESS. EPERM is always an error.
 + *
 + * There are two versions: one to be called while holding Giant (which is
 + * needed due to use of VFS), and the other called with the NFS server lock
 + * (which will be dropped and reacquired).  This is necessary because
 + * nfsrv_access checks are required from both classes of contexts.
   */
  static int
 -nfsrv_access(struct vnode *vp, int flags, struct ucred *cred, int rdonly,
 -    struct thread *td, int override)
 +nfsrv_access_withgiant(struct vnode *vp, int flags, struct ucred *cred,
 +    int rdonly, struct thread *td, int override)
  {
  	struct vattr vattr;
  	int error;
  
 -	NFSD_LOCK_ASSERT();
 +	GIANT_REQUIRED;
  
  	nfsdbprintf(("%s %d\n", __FILE__, __LINE__));
 +
  	if (flags & VWRITE) {
  		/* Just vn_writechk() changed to check rdonly */
  		/*
 @@ -4517,11 +4520,10 @@
  		if (vp->v_vflag & VV_TEXT)
  			return (ETXTBSY);
  	}
 -	NFSD_UNLOCK();
 -	mtx_lock(&Giant);	/* VFS */
 +
  	error = VOP_GETATTR(vp, &vattr, cred, td);
  	if (error)
 -		goto out;
 +		return (error);
  	error = VOP_ACCESS(vp, flags, cred, td);
  	/*
  	 * Allow certain operations for the owner (reads and writes
 @@ -4529,9 +4531,21 @@
  	 */
  	if (override && error == EACCES && cred->cr_uid == vattr.va_uid)
  		error = 0;
 -out:
 -	NFSD_UNLOCK_ASSERT();
 +	return (error);
 +}
 +
 +static int
 +nfsrv_access(struct vnode *vp, int flags, struct ucred *cred, int rdonly,
 +    struct thread *td, int override)
 +{
 +	int error;
 +
 +	NFSD_LOCK_ASSERT();
 +
 +	NFSD_UNLOCK();
 +	mtx_lock(&Giant);	/* VFS */
 +	error = nfsrv_access_withgiant(vp, flags, cred, rdonly, td, override);
  	mtx_unlock(&Giant);	/* VFS */
  	NFSD_LOCK();
 -	return error;
 +	return (error);
  }
 
 
State-Changed-From-To: open->patched 
State-Changed-By: rwatson 
State-Changed-When: Fri Nov 19 12:05:28 GMT 2004 
State-Changed-Why:  
This patch has been committed to HEAD as nfs_serv.c:1.151, and now merged 
to RELENG_5 as 1.147.2.3.  I will merge to RELENG_5_3 as an errata branch 
fix after additional testing. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=73807 

From: Matteo Riondato <rionda@gufi.org>
To: bug-followup@freebsd.org
Cc: rwatson@freebsd.org
Subject: Re: kern/73807 panic: mutex nfsd_mtx not owned in NFS server
Date: Fri, 8 Apr 2005 22:28:28 +0200

 --4WCFFtl4AQpQKunj
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 This was fixed and merged to RELENG_5 and RELENG_5_3, so this PR can
 be closed.
 Thanks.
 Best Regards
 
 --=20
 Rionda aka Matteo Riondato
 Disinformato per default
 G.U.F.I. Staff Member (http://www.gufi.org)
 FreeSBIE Developer (http://www.freesbie.org)
 
 --4WCFFtl4AQpQKunj
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.0 (FreeBSD)
 
 iD8DBQFCVuls2Mp4pR7Fa+wRAlhvAJ9iVTQd4y6BPz/4yTZD9EDBkq5JUwCfWLm0
 o7KkJmezWImz3QETxPQW0jM=
 =g10l
 -----END PGP SIGNATURE-----
 
 --4WCFFtl4AQpQKunj--
State-Changed-From-To: patched->closed 
State-Changed-By: rwatson 
State-Changed-When: Sat Apr 9 22:18:55 GMT 2005 
State-Changed-Why:  
Closed.   The fix has been committed and merged.  Thanks for the report 
(and reminder!). 


http://www.freebsd.org/cgi/query-pr.cgi?pr=73807 
>Unformatted:
