From nobody@FreeBSD.org  Sun Mar 24 11:27:59 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	by hub.freebsd.org (Postfix) with ESMTP id 468581BCF
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 24 Mar 2013 11:27:59 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 25D6C3E8
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 24 Mar 2013 11:27:59 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r2OBRwPg092465
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 24 Mar 2013 11:27:58 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r2OBRw70092464;
	Sun, 24 Mar 2013 11:27:58 GMT
	(envelope-from nobody)
Message-Id: <201303241127.r2OBRw70092464@red.freebsd.org>
Date: Sun, 24 Mar 2013 11:27:58 GMT
From: Svetlin Ivanov <izrodix@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Sleeping on "vmopar" with the following non-sleepable locks held: exclusive sleep mutex NFSnode lock (NFSnode lock) 
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         177335
>Category:       kern
>Synopsis:       [nfs] [panic] Sleeping on "vmopar" with the following non-sleepable locks held: exclusive sleep mutex NFSnode lock (NFSnode lock)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-fs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Mar 24 11:30:00 UTC 2013
>Closed-Date:    Fri Jun 14 00:21:15 UTC 2013
>Last-Modified:  Fri Jun 14 00:21:15 UTC 2013
>Originator:     Svetlin Ivanov
>Release:        FreeBSD 8.3-STABLE
>Organization:
Global Electronic Solutions LTD.
>Environment:
FreeBSD www-02.gesbg.com 8.3-STABLE FreeBSD 8.3-STABLE #0: Tue Mar 12 12:40:17 EET 2013     root@GainesTown.gesbg.com:/usr/obj/usr/src/sys/GESBG80  amd64

>Description:
Hi list,

 We have 5 freebsd 8.3-STABLE (cvsuped Mar 12) that keeps crashing in a bad way.

 The are doing heavy apache/ nginx/ php5 web serving from a nfs mount and panic at least once a day with the following message:

1st 0xffffff0174f10ca8 vm object (standard object) @ /usr/src/sys/vm/vnode_pager.c:375
 2nd 0xffffffff80796020 Giant (Giant) @ /usr/src/sys/modules/usb/ukbd/../../../dev/usb/input/ukbd.c:2017
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x80a
_mtx_lock_flags() at _mtx_lock_flags+0x85
ukbd_poll() at ukbd_poll+0x44
kbdmux_poll() at kbdmux_poll+0x3f
sc_cngetc() at sc_cngetc+0xed
cncheckc() at cncheckc+0x65
cngetc() at cngetc+0x1e
db_readline() at db_readline+0x77
db_read_line() at db_read_line+0x15
db_command_loop() at db_command_loop+0x38
db_trap() at db_trap+0x89
kdb_trap() at kdb_trap+0xe1
trap() at trap+0x188
calltrap() at calltrap+0x8
--- trap 0x3, rip = 0xffffffff802efa2b, rsp = 0xffffff8362f99040, rbp = 0xffffff8362f99060 ---
kdb_enter() at kdb_enter+0x3b
witness_warn() at witness_warn+0x2c4
_sleep() at _sleep+0x64
vm_object_page_remove() at vm_object_page_remove+0x1df
vnode_pager_setsize() at vnode_pager_setsize+0x11f
nfs_loadattrcache() at nfs_loadattrcache+0x4e6
nfsm_loadattr_xx() at nfsm_loadattr_xx+0x2b
nfs_getattr() at nfs_getattr+0x2e9
VOP_GETATTR_APV() at VOP_GETATTR_APV+0xaf
nfs_lookup() at nfs_lookup+0x43a
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb3
lookup() at lookup+0x3d5
namei() at namei+0x481
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
amd64_syscall() at amd64_syscall+0x2b7
Xfast_syscall() at Xfast_syscall+0xfc
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x801488f5c, rsp = 0x7fffffffa608, rbp = 0x7fffffffa700 ---

db> gdb
(ctrl-c will return control to ddb)
Switching to gdb back-end


These servers are diskless and we are using online remote console debug.

[root@GainesTown /usr/obj/usr/src/sys/GESBG80]# kgdb -r /dev/cuau0 ./kernel.debug 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...Switching to remote protocol
kdb_enter (why=0xffffffff805acb07 "witness", msg=0xa <Address 0xa out of bounds>) at /usr/src/sys/kern/subr_kdb.c:454
454                     kdb_why = KDB_WHY_UNSET;

(kgdb) bt
#0  kdb_enter (why=0xffffffff805acb07 "witness", msg=0xa <Address 0xa out of bounds>) at /usr/src/sys/kern/subr_kdb.c:454
#1  0xffffffff80304d14 in witness_warn (flags=<value optimized out>, lock=0xffffff0174f10ca8, fmt=0xffffffff805a6178 "Sleeping on \"%s\"") at /usr/src/sys/kern/subr_witness.c:1691
#2  0xffffffff802c55e4 in _sleep (ident=0xffffff031b8dc990, lock=0xffffff0174f10ca8, priority=68, wmesg=0xffffffff805dba6e "vmopar", timo=0) at /usr/src/sys/kern/kern_synch.c:163
#3  0xffffffff8050038f in vm_object_page_remove (object=0xffffff0174f10ca8, start=0, end=22, clean_only=0) at vm_page.h:359
#4  0xffffffff8050b84f in vnode_pager_setsize (vp=<value optimized out>, nsize=0) at /usr/src/sys/vm/vnode_pager.c:389
#5  0xffffffff804683a6 in nfs_loadattrcache (vpp=<value optimized out>, mdp=<value optimized out>, dposp=<value optimized out>, vaper=0xffffff8362f993b0, dontshrink=0)
    at /usr/src/sys/nfsclient/nfs_subs.c:609
#6  0xffffffff804684bb in nfsm_loadattr_xx (v=0xffffff8362f99498, va=<value optimized out>, md=<value optimized out>, dpos=<value optimized out>)
    at /usr/src/sys/nfsclient/nfs_subs.c:973
#7  0xffffffff80470959 in nfs_getattr (ap=<value optimized out>) at /usr/src/sys/nfsclient/nfs_vnops.c:726
#8  0xffffffff8056d9bf in VOP_GETATTR_APV (vop=0xffffffff8074a720, a=0xffffff8362f996b0) at vnode_if.c:699
#9  0xffffffff8047442a in nfs_lookup (ap=<value optimized out>) at vnode_if.h:309
#10 0xffffffff8056e073 in VOP_LOOKUP_APV (vop=0xffffffff8074a720, a=0xffffff8362f997e0) at vnode_if.c:122
#11 0xffffffff8034ac95 in lookup (ndp=0xffffff8362f99900) at vnode_if.h:54
#12 0xffffffff8034bc71 in namei (ndp=0xffffff8362f99900) at /usr/src/sys/kern/vfs_lookup.c:269
#13 0xffffffff8035b18f in kern_statat_vnhook (td=0xffffff000adbd470, flag=<value optimized out>, fd=<value optimized out>, path=<value optimized out>, pathseg=<value optimized out>, 
    sbp=0xffffff8362f99a80, hook=0) at /usr/src/sys/kern/vfs_syscalls.c:2339
#14 0xffffffff8035b335 in kern_statat (td=<value optimized out>, flag=<value optimized out>, fd=<value optimized out>, path=<value optimized out>, pathseg=<value optimized out>, 
    sbp=<value optimized out>) at /usr/src/sys/kern/vfs_syscalls.c:2320
#15 0xffffffff8035b3fa in lstat (td=<value optimized out>, uap=0xffffff8362f99bb0) at /usr/src/sys/kern/vfs_syscalls.c:2383
#16 0xffffffff805300e7 in amd64_syscall (td=0xffffff000adbd470, traced=0) at subr_syscall.c:114
#17 0xffffffff8051750c in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:387
#18 0x0000000801488f5c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 


root@www-02:~ # mount
/dev/md0 on / (ufs, local)
devfs on /dev (devfs, local, multilabel)
/dev/md1 on /etc (ufs, local)
tmpfs on /tmp (tmpfs, local, noexec)
/dev/md2 on /usr (ufs, local, soft-updates)
/dev/md3 on /var (ufs, local)
10.10.10.1:/storage/jails/store-01/usr/lib on /usr/lib (nfs)
10.10.10.1:/storage/jails/store-01/usr/local on /usr/local (nfs)
10.10.10.1:/storage/jails/store-01/storage on /storage (nfs, noatime)


>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon May 20 03:32:02 UTC 2013 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=177335 
State-Changed-From-To: open->feedback 
State-Changed-By: rmacklem 
State-Changed-When: Mon May 20 21:04:48 UTC 2013 
State-Changed-Why:  

I have emailed the reporter a variant of r248567, r248581 
for the 8.3/old NFS client and am waiting to hear from him 
w.r.t. this. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=177335 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/177335: commit references a PR
Date: Wed, 29 May 2013 00:32:57 +0000 (UTC)

 Author: rmacklem
 Date: Wed May 29 00:32:49 2013
 New Revision: 251089
 URL: http://svnweb.freebsd.org/changeset/base/251089
 
 Log:
   Add a patch analygous to r248567, r248581, r251079 to the
   old NFS client to avoid the panic reported in the PR by
   doing the vnode_pager_setsize() call after unlocking the mutex.
   
   PR:		177335
   MFC after:	2 weeks
 
 Modified:
   head/sys/nfsclient/nfs_subs.c
 
 Modified: head/sys/nfsclient/nfs_subs.c
 ==============================================================================
 --- head/sys/nfsclient/nfs_subs.c	Wed May 29 00:19:58 2013	(r251088)
 +++ head/sys/nfsclient/nfs_subs.c	Wed May 29 00:32:49 2013	(r251089)
 @@ -463,6 +463,8 @@ nfs_loadattrcache(struct vnode **vpp, st
  	struct timespec mtime, mtime_save;
  	int v3 = NFS_ISV3(vp);
  	int error = 0;
 +	u_quad_t nsize;
 +	int setnsize;
  
  	md = *mdp;
  	t1 = (mtod(md, caddr_t) + md->m_len) - *dposp;
 @@ -565,6 +567,8 @@ nfs_loadattrcache(struct vnode **vpp, st
  		vap->va_filerev = 0;
  	}
  	np->n_attrstamp = time_second;
 +	setnsize = 0;
 +	nsize = 0;
  	if (vap->va_size != np->n_size) {
  		if (vap->va_type == VREG) {
  			if (dontshrink && vap->va_size < np->n_size) {
 @@ -591,7 +595,8 @@ nfs_loadattrcache(struct vnode **vpp, st
  				np->n_size = vap->va_size;
  				np->n_flag |= NSIZECHANGED;
  			}
 -			vnode_pager_setsize(vp, np->n_size);
 +			setnsize = 1;
 +			nsize = vap->va_size;
  		} else {
  			np->n_size = vap->va_size;
  		}
 @@ -628,6 +633,8 @@ nfs_loadattrcache(struct vnode **vpp, st
  		KDTRACE_NFS_ATTRCACHE_LOAD_DONE(vp, &np->n_vattr, 0);
  #endif
  	mtx_unlock(&np->n_mtx);
 +	if (setnsize)
 +		vnode_pager_setsize(vp, nsize);
  out:
  #ifdef KDTRACE_HOOKS
  	if (error)
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: feedback->closed 
State-Changed-By: rmacklem 
State-Changed-When: Fri Jun 14 00:19:54 UTC 2013 
State-Changed-Why:  

The patch that stops this crash (int head as r251089) has 
been MFC'd to stable/8 as r251719. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=177335 
>Unformatted:
