From hsu@mail.clinet.fi  Sat Sep 20 08:26:51 1997
Received: from hauki.clinet.fi (root@hauki.clinet.fi [194.100.0.1])
          by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id IAA10110
          for <FreeBSD-gnats-submit@freebsd.org>; Sat, 20 Sep 1997 08:26:16 -0700 (PDT)
Received: from zetor.clinet.fi (root@zetor.clinet.fi [194.100.0.11])
	by hauki.clinet.fi (8.8.6/8.8.6) with ESMTP id SAA04431
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 20 Sep 1997 18:25:43 +0300 (EET DST)
Received: (hsu@localhost) by zetor.clinet.fi (8.8.7/8.6.4) id SAA23098; Sat, 20 Sep 1997 18:25:42 +0300 (EEST)
Message-Id: <199709201525.SAA23098@zetor.clinet.fi>
Date: Sat, 20 Sep 1997 18:25:42 +0300 (EEST)
From: Heikki Suonsivu <hsu@mail.clinet.fi>
Reply-To: hsu@mail.clinet.fi
To: FreeBSD-gnats-submit@freebsd.org
Subject: NFS access locks up
X-Send-Pr-Version: 3.2

>Number:         4588
>Category:       kern
>Synopsis:       NFS access locks up
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    peter
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Sep 20 08:30:01 PDT 1997
>Closed-Date:    Sat Dec 11 23:58:21 PST 1999
>Last-Modified:  Sat Dec 11 23:59:31 PST 1999
>Originator:     Heikki Suonsivu
>Release:        FreeBSD 2.2-STABLE i386
>Organization:
Clinet, Espoo, Finland
>Environment:

FreeBSD-2.2, mostly late STABLE kernels but this seems to have been around
for some time.  These are usually SCSI, either NCR or 2940 configurations,
with 64M to 128M of memory, Pentiums from 133MHz and up.

>Description:

NFS clients deadlock on something, directories or files.  Nothing kills the
processes, not even kill -9.  For some reason the files are almost always
files which are WWW server related, but not necessarily on WWW server's
disks.  So it is access logs and users WWW directories which lock up.  It
could be either the directory or one of the files in the directory which go
into this state.  I think the reason for WWW stuff to lock up is related to
usage pattern, as we have seen the problem with other files, just it has
been rare.

Usually rebooting the client fixes it temporarily, but some cases it won't.
In particular if it failed when user was using microsoft frontpage to
update the contents of the directory, it usually locks the directory up
again as soon as he tries to update the pages after the reboot.

It seems to be the client which is the problem, as while some clients lock
up reading the file/directory I can still see it from other nfs clients.

ps axuwwwl:

USER       PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED       TIME COMMAND            UID   PID  PPID CPU PRI NI   VSZ  RSS WCHAN  STAT  TT       TIME COMMAND
hsu       9170  0.0  0.3   228  208  p0  D+    5:27PM    0:00.03 less access_log.   105  9170 20972   1  -4  0   228  208 nfsvin D+    p0    0:00.03 less access_log.874368000
hsu      19077  0.0  0.1   196   68  p7  R+    6:08PM    0:00.01 egrep (less|USER   105 19077 18879   3  28  0   196   68 -      R+    p7    0:00.01 egrep (less|USER)

fstat:

hsu      less        9170   wd /m/varasto/usr  21696 drwxr-xr-x     512  r
hsu      less        9170 text /usr       9064 -r-xr-xr-x   73728  r
hsu      less        9170    0 /          4189 crw--w----   ttyp0 rw
hsu      less        9170    1 /          4189 crw--w----   ttyp0 rw
hsu      less        9170    2 /          4189 crw--w----   ttyp0 rw
hsu      less        9170    3 /          3993 crw-rw-rw-     tty  r

I cannot get sensible tcpdump unless I know how to look for a specific
thing, the nfs server is very busy.  Any ideas what to look for ? 

>How-To-Repeat:

Busy clients seem to exhibit this more frequently.

>Fix:
	
-- 
Heikki Suonsivu, T{ysikuu 10 C 83/02210 Espoo/FINLAND, hsu@clinet.fi
mobile +358-40-5519679 work +358-9-43542270 fax -4555276
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->peter 
Responsible-Changed-By: peter 
Responsible-Changed-When: Sun Apr 26 00:43:41 PDT 1998 
Responsible-Changed-Why:  
I'll check into this, this is similar to a couple of other reports. 

From: peter@sirius.com
To: freebsd-gnats-submit@freebsd.org
Cc: peter@staff.sirius.com (Peter Preuss), hsu@clinet.fi
Subject: re:kern/4588:nfsaccesslocksup
Date: Fri, 22 May 1998 18:01:45 -0700 (PDT)

 Hello,
 
 I'm certain we've been bitten by the same bug many times as our web-servers
 used to lock up with the same problem (unkillable processes) sometimes
 several times per week.
 
 Analysis of the code in question seems to indicate a fundamental 
 problem with the NFS kernel code (and other parts?) in all 
 4.4 BSD derived OSes (I could get my hands on. FreeBSD 2.X, 3.X,
 NetBSD, OpenBSD, Rhapsody -- just kidding [but who knows] ;)
 
 The problem is an overly optimistic use of 
 tsleep(void *ident, int priority, char *wmesg, int timo) 
 with priority not including the PCATCH flag *and* timeout set to eternity.
 
 This causes problems in situations where either the wakeup has happened
 in the past (as I believe is the case here), does not happen, or takes 
 longer than "short term" (ie., due to underlying hardware problems).
 
 The workaround (in production here on 5 heavy-used web-servers for three
 weeks now) is to simply ensure a reasonable time-out in the nfs_vinvalbuf()
 function (module nfs_bio.c):
 
 *** nfs_bio.c   Fri May 22 16:35:49 1998
 --- nfs_bio.c.bak       Sat Mar 28 08:05:05 1998
 ***************
 *** 898,904 ****
         while (np->n_flag & NFLUSHINPROG) {
                 np->n_flag |= NFLUSHWANT;
                 error = tsleep((caddr_t)&np->n_flag, PRIBIO + 2, "nfsvinval",
 !                       2 * hz);
                 if (error && intrflg && nfs_sigintr(nmp, (struct nfsreq *)0, p))
                         return (EINTR);
         }
 --- 898,904 ----
         while (np->n_flag & NFLUSHINPROG) {
                 np->n_flag |= NFLUSHWANT;
                 error = tsleep((caddr_t)&np->n_flag, PRIBIO + 2, "nfsvinval",
 !                       slptimeo);
                 if (error && intrflg && nfs_sigintr(nmp, (struct nfsreq *)0, p))
                         return (EINTR);
         }
 
 A better fix would be to inspect the whole kernel for similar loopholes
 (no PCATCH *and* no time-out in tsleep()).
 
 Background:
 
 We experienced more and more locked-up httpd processes on our busy 
 web-servers. At first only 5..10 out of 200 (or so) processes enter
 the ``D'' status (as per "ps lax"), they never seem to recover and
 eventually spread to all remaining httpds. The web-server slows down
 and eventually stops responding to http requests. Only reboot works
 around the problem.
 
 The "ps lax" output hints to "nfsvin" as the culprit:
 
   UID   PID  PPID CPU PRI NI   VSZ  RSS WCHAN  STAT  TT       TIME COMMAND
 ...
 65534  8622 24378   0  -4  0  2876  828 nfsvin D     ??    0:00.31 /usr/local/sb
 in/httpd
 65534  8758 24378   0  -4  0  2876  784 nfsvin D     ??    0:00.53 /usr/local/sb
 in/httpd
 65534  8759 24378   0  -4  0  2872  876 nfsvin D     ??    0:00.46 /usr/local/sb
 in/httpd
 65534  8766 24378   1  -4  0  2864  740 nfsvin D     ??    0:00.15 /usr/local/sb
 in/httpd
 65534  8838 24378   1  -4  0  2864  652 nfsvin D     ??    0:00.28 /usr/local/sb
 in/httpd
 65534  8844 24378   0  -4  0  2876  728 nfsvin D     ??    0:00.16 /usr/local/sb
 in/httpd
 65534  8849 24378   0  -4  0  2876  892 nfsvin D     ??    0:00.46 /usr/local/sb
 in/httpd
 ...
 
 which occurs only in nfs_bio.c:
                 error = tsleep((caddr_t)&np->n_flag, PRIBIO + 2, "nfsvinval",
 
 The function nfs_vinvalbuf() seems to be in charge of flushing dirty buffers
 (so claims the comment ;), all httpds have the same set of log-files open
 and seem to trip over each other's flush requests (onto a NFS partition).
 The first process picks up the task and sets NFLUSHINPROG; followers
 discover that NFLUSHINPROG, set NFLUSHWANT, and go to sleep waiting for
 a wakeup from the leader (which checks NFLUSHWANT to see whether a wakeup()
 is neeed).
 
 But if the wakeup() happens after NFLUSHINPROG is discovered by a "follower"
 yet *before* tsleep() has been performed, that "follower" is doomed: no
 wakeup() kiss nor "kill -whatever" can bring it back to life. In the
 Apache case, the other lemmings follow suit whithin minutes.
 
 Agreed, the chances are slim, yet
                 np->n_flag |= NFLUSHWANT;
 
 might take a couple of cycles leaving a chance for a premature or
 missed wakeup().
 
 But then again, it happened only every 42 million web-hits...
 
 
 Peter Preuss <peter@sirius.com>
 Sirius Connections, San Francisco
State-Changed-From-To: open->closed 
State-Changed-By: dillon 
State-Changed-When: Sat Dec 11 23:58:21 PST 1999 
State-Changed-Why:  
Several NFS bugs relating to lockups, especially with TCP mounts (but could 
also occur with UDP mounts) have been fixed in 3.x and 4.x.  I believe this 
bug has been fixed in the later kernels. 
>Unformatted:
