From nobody@FreeBSD.org  Mon Aug 15 14:38:06 2005
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 2963316A41F
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 15 Aug 2005 14:38:06 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 4CEEC43D5F
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 15 Aug 2005 14:38:04 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id j7FEc3HM074053
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 15 Aug 2005 14:38:03 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id j7FEc3HN074052;
	Mon, 15 Aug 2005 14:38:03 GMT
	(envelope-from nobody)
Message-Id: <200508151438.j7FEc3HN074052@www.freebsd.org>
Date: Mon, 15 Aug 2005 14:38:03 GMT
From: Vladimir Sharun <sharun@ukr.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: NFS locking issue in RELENG_6/i386/SMP
X-Send-Pr-Version: www-2.3

>Number:         84953
>Category:       kern
>Synopsis:       [nfs] NFS locking issue in RELENG_6/i386/SMP
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kuriyama
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 15 14:40:13 GMT 2005
>Closed-Date:    Wed Feb 24 15:50:47 UTC 2010
>Last-Modified:  Wed Feb 24 15:50:47 UTC 2010
>Originator:     Vladimir Sharun
>Release:        FreeBSD6
>Organization:
UkrNet ISP
>Environment:
FreeBSD krokodil.ukr.net 6.0-BETA2 FreeBSD 6.0-BETA2 #5: Sun Aug  7 19:15:28 EEST 2005     root@krokodil.ukr.net:/usr/obj/usr/src/sys/FISH.MP  i386
    
>Description:
While setting a lock via fcntl on NFSv3 mount point, we stuck in
uninterruptible wait (ps axc show D state for process.

Mounting NFS share without rpcbind/rpc.statd/rpc.lockd gaves errno 45
EOPNOTSUPP Operation not supported
>How-To-Repeat:
#include <stdio.h>
#include <errno.h>
#include <fcntl.h>

int main() {
  int lockfd;
  char* tempfile="/mnt/nfs/testfile";

  lockfd=open(tempfile,O_CREAT);
  printf("Open errno:   %d\n",errno);
  if (flock(lockfd, LOCK_SH|LOCK_NB)==-1) {
    printf("ERROR shared lock:  %d\n",errno);
  }
  if (flock(lockfd, LOCK_EX|LOCK_NB)==-1) {
    printf("ERROR exclusive lock:  %d\n",errno);
  }
   close(lockfd);
}

>Fix:
Rollback to RELENG_5
>Release-Note:
>Audit-Trail:

From: Jun Kuriyama <kuriyama@imgsrc.co.jp>
To: bug-followup@FreeBSD.org, sharun@ukr.net
Cc:  
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP 
Date: Wed, 15 Feb 2006 23:11:24 +0900

 Can you take a tcpdump at this stucked situation?
 
 I tried to reproduce this on my 5.5-BETA1 (server) and 6.1-BETA1
 (client) with you sample program.  But file was locked without
 problem.
 
 
 -- 
 Jun Kuriyama <kuriyama@imgsrc.co.jp> // IMG SRC, Inc.
              <kuriyama@FreeBSD.org> // FreeBSD Project
Responsible-Changed-From-To: freebsd-bugs->kuriyama 
Responsible-Changed-By: kuriyama 
Responsible-Changed-When: Wed Feb 15 15:17:16 UTC 2006 
Responsible-Changed-Why:  
Seems to be my PR. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=84953 

From: Vladimir Sharun <sharun@ukr.net>
To: Jun Kuriyama <kuriyama@imgsrc.co.jp>
Cc: bug-followup@FreeBSD.org, sharun@ukr.net
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Wed, 15 Feb 2006 22:05:20 +0200

 Jun Kuriyama wrote:
  JK> Can you take a tcpdump at this stucked situation?
 
 I'm running non-stop production and can't test. Second:
 we set an workaround and stops using NFS locking at all.
 
  JK> I tried to reproduce this on my 5.5-BETA1 (server) and 6.1-BETA1
  JK> (client) with you sample program.  But file was locked without
  JK> problem.
 
 Run test in a loop for few billion times (few hours on Gigabit).
 
 -- 
 UKR.NET Postmaster

From: Palle Girgensohn <girgen@pingpong.net>
To: bug-followup@FreeBSD.org, sharun@ukr.net
Cc:  
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Fri, 24 Feb 2006 17:27:31 +0100

 Hi,
 
 I found this PR.
 
 I can get you a tcpdump later tonight if it helps?
 
 /Palle
 

From: Kris Kennaway <kris@obsecurity.org>
To: Jun Kuriyama <kuriyama@imgsrc.co.jp>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Fri, 10 Mar 2006 17:44:10 -0500

 On Wed, Feb 15, 2006 at 02:20:06PM +0000, Jun Kuriyama wrote:
 > The following reply was made to PR kern/84953; it has been noted by GNATS.
 > 
 > From: Jun Kuriyama <kuriyama@imgsrc.co.jp>
 > To: bug-followup@FreeBSD.org, sharun@ukr.net
 > Cc:  
 > Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP 
 > Date: Wed, 15 Feb 2006 23:11:24 +0900
 > 
 >  Can you take a tcpdump at this stucked situation?
 >  
 >  I tried to reproduce this on my 5.5-BETA1 (server) and 6.1-BETA1
 >  (client) with you sample program.  But file was locked without
 >  problem.
 
 I'm also not able to reproduce this with 6.1 client and server (after
 leaving it running for about 8 hours).  The only thing I can think of
 is that your rpc.lockd may have crashed, in which case even
 non-blocking lock requests will block on the client (see bin/94258)
 until/if the server comes back up and they recover state.
 
 Kris
 
 P.S. rpc.lockd *is* broken with nfsv2, but you say you're using nfsv3
 only.

From: Palle Girgensohn <girgen@pingpong.net>
To: bug-followup@FreeBSD.org, sharun@ukr.net,
 Jun Kuriyama <kuriyama@imgsrc.co.jp>, Kris Kennaway <kris@obsecurity.org>
Cc:  
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Mon, 03 Apr 2006 04:06:58 +0200

 Hi!
 
 I gathered up two tcpdumps.
 
 They're on
 
  http://people.freebsd.org/~girgen/eclipse.tcpdump.bz2
 
 and
 
  http://people.freebsd.org/~girgen/openoffice_open_file.tcpdump.bz2
 
 
 Both relate to a fresh 6.1-prerelease client and a 4.11-release server.
 
 The problematic applications use nfs file locking. I've tested two apps, 
 namely eclipse (java based) and openoffice.org-2.0.1. I ran tcpdump first 
 and then started the app. App hung. After a while I kill the app and 
 tcpdump. tcpdump was run on the client side.
 
 rpc.lockd and rpc.statd has to be switched on on the server, or else the 
 apps will fail.
 
 nfs_server_flags="-t -u -n 12"
 
 Also see the discussion on "nfs locking broken" around Feb 24 2006. 
 <http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2006-02/msg00227.html>
 
 At first I thought that running a 6.x server would fix the problem for me, 
 but it doesn't, although it seems somewhat more stable. I'll run some dumps 
 with the example application tomorrow.
 
 Regards,
 Palle
 
 
 > I've tried the following combinations:
 >
 > client server
 > 4.11 4.11 OK
 > 5.4 4.11 OK
 > 6.0 4.11 Hangs
 > 6.1pre 4.11 Hangs
 >
 > 4.11 5.4 OK
 > 5.4 5.4 OK
 > 6.0 5.4 Hangs
 > 6.1pre 5.4 Hangs
 >
 > 6.0 6.0 OK
 > 6.1pre 6.0 OK
 
 This is not quite correct anymore, alla clients will fail with a 
 6.1prerelease server. If you want, I can test with the 

From: Kris Kennaway <kris@obsecurity.org>
To: Palle Girgensohn <girgen@pingpong.net>
Cc: bug-followup@FreeBSD.org, sharun@ukr.net,
	Jun Kuriyama <kuriyama@imgsrc.co.jp>,
	Kris Kennaway <kris@obsecurity.org>
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Sun, 2 Apr 2006 22:25:43 -0400

 --eAbsdosE1cNLO4uF
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 On Mon, Apr 03, 2006 at 04:06:58AM +0200, Palle Girgensohn wrote:
 > Hi!
 >=20
 > I gathered up two tcpdumps.
 >=20
 > They're on
 >=20
 > http://people.freebsd.org/~girgen/eclipse.tcpdump.bz2
 >=20
 > and
 >=20
 > http://people.freebsd.org/~girgen/openoffice_open_file.tcpdump.bz2
 >=20
 >=20
 > Both relate to a fresh 6.1-prerelease client and a 4.11-release server.
 >=20
 > The problematic applications use nfs file locking. I've tested two apps,=
 =20
 > namely eclipse (java based) and openoffice.org-2.0.1. I ran tcpdump first=
 =20
 > and then started the app. App hung. After a while I kill the app and=20
 > tcpdump. tcpdump was run on the client side.
 >=20
 > rpc.lockd and rpc.statd has to be switched on on the server, or else the=
 =20
 > apps will fail.
 
 Yes, this much is expected.
 
 Anyway it turns out based on my subsequent testing that rpc.lockd is
 basically chock full o' bugs in all versions of FreeBSD, so even if
 your issue is one I haven't seen, nothing much is going to change
 until the utility gets a maintainer (and they do a lot of work).
 
 Not using rpc.lockd (e.g. mount_nfs -L) may be an option if you don't
 really need locking between multiple clients.
 
 Kris
 --eAbsdosE1cNLO4uF
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.2.2 (FreeBSD)
 
 iD8DBQFEMIemWry0BWjoQKURAqwoAKDou9GMjeCkhiMUiTGp64BHUQ8XwwCg6lye
 lpCqgkVhArQtPA98WR4fIjQ=
 =Cpub
 -----END PGP SIGNATURE-----
 
 --eAbsdosE1cNLO4uF--

From: Jun Kuriyama <kuriyama@imgsrc.co.jp>
To: Kris Kennaway <kris@obsecurity.org>
Cc: Palle Girgensohn <girgen@pingpong.net>,
	bug-followup@FreeBSD.org,
	sharun@ukr.net,
	Jun Kuriyama <kuriyama@imgsrc.co.jp>
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Mon, 03 Apr 2006 11:32:06 +0900

 At Sun, 2 Apr 2006 22:25:43 -0400,
 Kris Kennaway wrote:
 > > Both relate to a fresh 6.1-prerelease client and a 4.11-release server.
 > > 
 > > The problematic applications use nfs file locking. I've tested two apps, 
 > > namely eclipse (java based) and openoffice.org-2.0.1. I ran tcpdump first 
 > > and then started the app. App hung. After a while I kill the app and 
 > > tcpdump. tcpdump was run on the client side.
 > > 
 > > rpc.lockd and rpc.statd has to be switched on on the server, or else the 
 > > apps will fail.
 > 
 > Yes, this much is expected.
 > 
 > Anyway it turns out based on my subsequent testing that rpc.lockd is
 > basically chock full o' bugs in all versions of FreeBSD, so even if
 > your issue is one I haven't seen, nothing much is going to change
 > until the utility gets a maintainer (and they do a lot of work).
 > 
 > Not using rpc.lockd (e.g. mount_nfs -L) may be an option if you don't
 > really need locking between multiple clients.
 
 I'm for all to revert my modifications if that will solve some of the
 problems.  But there are people who said reverting my modification
 solve the problem, and does not solve the problem.  What do you think
 I should do for 6.1-RELEASE?
 
 
 -- 
 Jun Kuriyama <kuriyama@imgsrc.co.jp> // IMG SRC, Inc.
              <kuriyama@FreeBSD.org> // FreeBSD Project

From: Kris Kennaway <kris@obsecurity.org>
To: Jun Kuriyama <kuriyama@imgsrc.co.jp>
Cc: Kris Kennaway <kris@obsecurity.org>,
	Palle Girgensohn <girgen@pingpong.net>, bug-followup@FreeBSD.org,
	sharun@ukr.net
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Sun, 2 Apr 2006 23:04:04 -0400

 --sm4nu43k4a2Rpi4c
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 On Mon, Apr 03, 2006 at 11:32:06AM +0900, Jun Kuriyama wrote:
 > At Sun, 2 Apr 2006 22:25:43 -0400,
 > Kris Kennaway wrote:
 > > > Both relate to a fresh 6.1-prerelease client and a 4.11-release serve=
 r.
 > > >=20
 > > > The problematic applications use nfs file locking. I've tested two ap=
 ps,=20
 > > > namely eclipse (java based) and openoffice.org-2.0.1. I ran tcpdump f=
 irst=20
 > > > and then started the app. App hung. After a while I kill the app and=
 =20
 > > > tcpdump. tcpdump was run on the client side.
 > > >=20
 > > > rpc.lockd and rpc.statd has to be switched on on the server, or else =
 the=20
 > > > apps will fail.
 > >=20
 > > Yes, this much is expected.
 > >=20
 > > Anyway it turns out based on my subsequent testing that rpc.lockd is
 > > basically chock full o' bugs in all versions of FreeBSD, so even if
 > > your issue is one I haven't seen, nothing much is going to change
 > > until the utility gets a maintainer (and they do a lot of work).
 > >=20
 > > Not using rpc.lockd (e.g. mount_nfs -L) may be an option if you don't
 > > really need locking between multiple clients.
 >=20
 > I'm for all to revert my modifications if that will solve some of the
 > problems.  But there are people who said reverting my modification
 > solve the problem, and does not solve the problem.  What do you think
 > I should do for 6.1-RELEASE?
 
 I haven't been able to get anyone to confirm the claim made by anholt
 about your patches and obtain a tcpdump (including anholt).  So I'd
 recommend not doing anything until someone can do so.
 
 Kris
 
 
 --sm4nu43k4a2Rpi4c
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.2.2 (FreeBSD)
 
 iD8DBQFEMJCkWry0BWjoQKURAm39AKCCjrgD+x0A5uxy1WjbKPAGcu1V8ACfUXYy
 jf/dXUb1TqduuqZm0HbTEjk=
 =nCg5
 -----END PGP SIGNATURE-----
 
 --sm4nu43k4a2Rpi4c--

From: Palle Girgensohn <girgen@FreeBSD.org>
To: Kris Kennaway <kris@obsecurity.org>
Cc: bug-followup@FreeBSD.org, sharun@ukr.net,
        Jun Kuriyama <kuriyama@imgsrc.co.jp>,
        Kris Kennaway <kris@obsecurity.org>
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Tue, 30 May 2006 16:35:52 +0200

 --On s=F6ndag, april 02, 2006 22.25.43 -0400 Kris Kennaway=20
 <kris@obsecurity.org> wrote:
 
 > Anyway it turns out based on my subsequent testing that rpc.lockd is
 > basically chock full o' bugs in all versions of FreeBSD, so even if
 > your issue is one I haven't seen, nothing much is going to change
 > until the utility gets a maintainer (and they do a lot of work).
 >
 > Not using rpc.lockd (e.g. mount_nfs -L) may be an option if you don't
 > really need locking between multiple clients.
 
 >From my point of view, this seems like a very urgent matter. We have our=20
 office set up with a bunch of FreeBSD machines mounting /home and some=20
 volumes for documents (also accessed by windows machines via samba and macs =
 
 via netatalk).
 
 Since installing FreeBSD 6.x, "nothing works"... Many applications, like=20
 OpenOffice, Eclipse, Amarok, KDE apps in general, simply lock forever,=20
 often requiring a reboot (!) to get back on track.
 
 Mounting -L helps, but most applications also need to be tuned to not using =
 
 file locking, or else they fail or hang anyway. Settings are of course=20
 different for every app. This becomes an administrative pain, and the users =
 
 complain very vividly, shouting things like "In Linux this works out of the =
 
 box" :-(
 
 Backing to FreeBSD 5.x might help, but it does not seem like a realistic=20
 alternative; the app we develop runs on FreeBSD 6.1 servers.
 
 In my users' view, as long as rpc.lockd is broken, FreeBSD is severely=20
 broken... I don't have the time, probably not knowledge enough, to dig into =
 
 the code myself. How can we help fix this? Can we find someone willing to=20
 fix and maintain rpc.lockd?
 
 /Palle
 

From: Kris Kennaway <kris@obsecurity.org>
To: Palle Girgensohn <girgen@FreeBSD.org>
Cc: Kris Kennaway <kris@obsecurity.org>, bug-followup@FreeBSD.org,
	sharun@ukr.net, Jun Kuriyama <kuriyama@imgsrc.co.jp>
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Tue, 30 May 2006 14:17:47 -0400

 --mxv5cy4qt+RJ9ypb
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 On Tue, May 30, 2006 at 04:35:52PM +0200, Palle Girgensohn wrote:
 >=20
 >=20
 > --On s?ndag, april 02, 2006 22.25.43 -0400 Kris Kennaway=20
 > <kris@obsecurity.org> wrote:
 >=20
 > >Anyway it turns out based on my subsequent testing that rpc.lockd is
 > >basically chock full o' bugs in all versions of FreeBSD, so even if
 > >your issue is one I haven't seen, nothing much is going to change
 > >until the utility gets a maintainer (and they do a lot of work).
 > >
 > >Not using rpc.lockd (e.g. mount_nfs -L) may be an option if you don't
 > >really need locking between multiple clients.
 >=20
 > >From my point of view, this seems like a very urgent matter. We have our=
 =20
 > office set up with a bunch of FreeBSD machines mounting /home and some=20
 > volumes for documents (also accessed by windows machines via samba and ma=
 cs=20
 > via netatalk).
 >=20
 > Since installing FreeBSD 6.x, "nothing works"... Many applications, like=
 =20
 > OpenOffice, Eclipse, Amarok, KDE apps in general, simply lock forever,=20
 > often requiring a reboot (!) to get back on track.
 >=20
 > Mounting -L helps, but most applications also need to be tuned to not usi=
 ng=20
 > file locking, or else they fail or hang anyway. Settings are of course=20
 > different for every app. This becomes an administrative pain, and the use=
 rs=20
 > complain very vividly, shouting things like "In Linux this works out of t=
 he=20
 > box" :-(
 >=20
 > Backing to FreeBSD 5.x might help, but it does not seem like a realistic=
 =20
 > alternative; the app we develop runs on FreeBSD 6.1 servers.
 >=20
 > In my users' view, as long as rpc.lockd is broken, FreeBSD is severely=20
 > broken... I don't have the time, probably not knowledge enough, to dig in=
 to=20
 > the code myself. How can we help fix this? Can we find someone willing to=
 =20
 > fix and maintain rpc.lockd?
 
 It will take serious attention from someone qualified, possibly a
 rewrite.  If you need this urgently then you may need to contract
 someone to do it.
 
 Kris
 
 --mxv5cy4qt+RJ9ypb
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.3 (FreeBSD)
 
 iD8DBQFEfIxKWry0BWjoQKURAvCOAJ9JAy8OOpkfakf9g7QBr2It8CFCGgCfTAkS
 BTxxP79JfkpeC5bBOYvbeZA=
 =ew5W
 -----END PGP SIGNATURE-----
 
 --mxv5cy4qt+RJ9ypb--

From: Palle Girgensohn <girgen@FreeBSD.org>
To: bug-followup@FreeBSD.org, sharun@ukr.net
Cc:  
Subject: Re: kern/84953: [nfs] NFS locking issue in RELENG_6/i386/SMP
Date: Mon, 07 Aug 2006 17:38:22 +0200

 Hello,
 
 A status inquiry; I'm curious, is there any work being done on rpc.lockd?
 
 Cheers,
 Palle
 
State-Changed-From-To: open->closed 
State-Changed-By: jh 
State-Changed-When: Wed Feb 24 15:39:56 UTC 2010 
State-Changed-Why:  
This has been probably fixed by the new NFS locking implementation. 
If not, this PR can be reopened. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=84953 
>Unformatted:
