From nobody@FreeBSD.org  Sun Apr  1 18:46:44 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id D73A516A403
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  1 Apr 2007 18:46:44 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id C8CC013C44C
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  1 Apr 2007 18:46:44 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l31IkihZ069509
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 1 Apr 2007 18:46:44 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id l31IfhZ4068924;
	Sun, 1 Apr 2007 18:41:43 GMT
	(envelope-from nobody)
Message-Id: <200704011841.l31IfhZ4068924@www.freebsd.org>
Date: Sun, 1 Apr 2007 18:41:43 GMT
From: Arne Woerner<arne_woerner@yahoo.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: /usr/bin/lockf: when lockf blocks due to another lockf and no -k is specified and the other lockf ends, the file is away
X-Send-Pr-Version: www-3.0

>Number:         111101
>Category:       bin
>Synopsis:       lockf(1): when lockf blocks due to another lockf and no -k is specified and the other lockf ends, the file is away
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Apr 01 18:50:04 GMT 2007
>Closed-Date:    Sun Apr 01 20:19:01 GMT 2007
>Last-Modified:  Fri Oct 12 15:00:08 UTC 2007
>Originator:     Arne Woerner
>Release:        R6.2
>Organization:
>Environment:
FreeBSD neo.riddick.homeunix.org 6.2-RELEASE FreeBSD 6.2-RELEASE #8: Fri Mar 30 10:00:56 UTC 2007     aw@neo.riddick.homeunix.org:/usr/src/sys/i386/compile/NEO  i386

>Description:
I use lockf in order to coordinate the access to my TV frame grabbers
(/dev/cxm?).

When another recording starts (process B) it can happen that a previous
recording (process A) still needs some seconds, I end up with an unlink-ed
lock file.

I would expect the following order:

1. process A: holds file LCK
2. process B: blocks on file LCK
3. process A: unlinks file LCK
4. process A: releases lock on file LCK
5. process B: locks the already opened but unlinked file LCK
6. process B: checks, if file LCK still exists
7. process B: creates file LCK
8. process B: locks file LCK

As a workaround I use lockf with option "-k", which should solve that problem...

Furthermore I would like to recommend, to make the "-k" behaviour the
default behaviour, so that we will not remove data files, that were
co-used as lock files...

-Arne
>How-To-Repeat:
Let LCK be a file, that does not exist in the work directory
% lockf LCK sleep 10 & sleep 1 ; lockf LCK ls -l LCK
[1] 38788
ls: LCK: No such file or directory
[1]  + Done                          lockf LCK sleep 10

I was able to repeat that on a i586 (Pentium) and a i686 (Athlon XP),
which both used R6.2 with multimedia/pvr250 related kernel patches.
>Fix:
1.
Use "-k" as a work around.

2.
Check if file exists after holding the lock. And if it doesnt exist
anymore, re-create it and close the old (in the meantime unlinked) lock
file and start again... <-- Maybe that contains another error, that I
dont see yet...
>Release-Note:
>Audit-Trail:

From: Kris Kennaway <kris@obsecurity.org>
To: Arne Woerner <arne_woerner@yahoo.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: bin/111101: /usr/bin/lockf: when lockf blocks due to another lockf and no -k is specified and the other lockf ends, the file is away
Date: Sun, 1 Apr 2007 15:53:08 -0400

 On Sun, Apr 01, 2007 at 06:41:43PM +0000, Arne Woerner wrote:
 
 > >Description:
 > I use lockf in order to coordinate the access to my TV frame grabbers (/dev/cxm?).
 > 
 > When another recording starts (process B) it can happen that a previous recording (process A) still needs some seconds, I end up with an unlink-ed lock file.
 > 
 > I would expect the following order:
 > 
 > 1. process A: holds file LCK
 > 2. process B: blocks on file LCK
 > 3. process A: unlinks file LCK
 > 4. process A: releases lock on file LCK
 > 5. process B: locks the already opened but unlinked file LCK
 > 6. process B: checks, if file LCK still exists
 > 7. process B: creates file LCK
 > 8. process B: locks file LCK
 > 
 > As a workaround I use lockf with option "-k", which should solve that problem...
 > 
 > Furthermore I would like to recommend, to make the "-k" behaviour the default behaviour, so that we will not remove data files, that were co-used as lock files...
 
 We cannot do that, not using -k is required in some situations, so
 those scripts will break.
 
 It appears that lockf is working as designed, and also as explicitly
 documented.  It looks like no action can be taken here.
 
 Kris

From: Kris Kennaway <kris@obsecurity.org>
To: "R. B. Riddick" <arne_woerner@yahoo.com>
Cc: Kris Kennaway <kris@obsecurity.org>, freebsd-gnats-submit@FreeBSD.org
Subject: Re: bin/111101: /usr/bin/lockf: when lockf blocks due to another lockf and no -k is specified and the other lockf ends, the file is away
Date: Sun, 1 Apr 2007 16:04:28 -0400

 On Sun, Apr 01, 2007 at 12:59:53PM -0700, R. B. Riddick wrote:
 > --- Kris Kennaway <kris@obsecurity.org> wrote:
 > > On Sun, Apr 01, 2007 at 06:41:43PM +0000, Arne Woerner wrote:
 > > > Furthermore I would like to recommend, to make the "-k" behaviour the
 > > > default behaviour, so that we will not remove data files, that were co-used
 > > > as lock files...
 > > 
 > > We cannot do that, not using -k is required in some situations, so
 > > those scripts will break.
 > >
 > OK - recommendation withdrawn...
 > 
 > > It appears that lockf is working as designed, and also as explicitly
 > > documented.  It looks like no action can be taken here.
 > > 
 > Explicitly? Documented? Since I have not found that documentation, can u help
 > me a little bit by showing it?
 > 
 > In the mean time I show a piece of my documentation (man page lockf(1)):
 > "DESCRIPTION
 >      The lockf utility acquires an exclusive lock on a file, creating it if
 >      necessary, and removing the file on exit unless explicitly told not to.
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This is even underlined.
 
 >      While holding the lock, it executes a command with optional arguments.
 >      After the command completes, lockf releases the lock, and removes the
 >      file unless the -k option is specified.  BSD-style locking is used, as
 >      described in flock(2); the mere existence of the file is not considered
 >      to constitute a lock."
 > I repeat: "CREATING IT IF NECESSARY"
 > 
 > If think, the explicit documentation says, that the file is created (although
 > this "if necessary" is somewhat weaselish, so that the man page should be
 > changed, too; it should say: "creating it if it does not exist").
 
 But that is what "if necessary" means.
 
 > Currently without the "-k" option given only the first conflict is solved
 > properly, while the next is not, which is surely not what we want, and which is
 > surely nowhere documented.
 
 If you would like to submit a manpage change to expand upon the
 problems that can occur when not using -k, please go ahead.
 
 Kris
State-Changed-From-To: open->closed 
State-Changed-By: linimon 
State-Changed-When: Sun Apr 1 20:18:39 UTC 2007 
State-Changed-Why:  
Followup notes that this is working as designed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=111101 

From: "R. B. Riddick" <arne_woerner@yahoo.com>
To: Kris Kennaway <kris@obsecurity.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: bin/111101: /usr/bin/lockf: when lockf blocks due to another lockf and no -k is specified and the other lockf ends, the file is away
Date: Sun, 1 Apr 2007 13:11:04 -0700 (PDT)

 --- Kris Kennaway <kris@obsecurity.org> wrote:
 > > Explicitly? Documented? Since I have not found that documentation, can u
 > > help me a little bit by showing it?
 > > 
 > > In the mean time I show a piece of my documentation (man page lockf(1)):
 > > "DESCRIPTION
 > >      The lockf utility acquires an exclusive lock on a file, creating it if
 > >      necessary, and removing the file on exit unless explicitly told not
 > >      to.
 >                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 > 
 > This is even underlined.
 >
 Hmm... I was talking of the second lockf process, that does not create the
 file, but that holds an exclusive lock on an unlinked file, which is completely
 useless.
 
 > > If think, the explicit documentation says, that the file is created
 > > (although
 > > this "if necessary" is somewhat weaselish, so that the man page should be
 > > changed, too; it should say: "creating it if it does not exist").
 > 
 > But that is what "if necessary" means.
 >
 Water on my mills... There u have the contradiction...
 
 > If you would like to submit a manpage change to expand upon the
 > problems that can occur when not using -k, please go ahead.
 > 
 Hmm... As long as u say, that I dont understand the use of lockf(1), I will not
 say/request so much about it...
 
 :-)
 
 -Arne
 
 
  
 ____________________________________________________________________________________
 Don't pick lemons.
 See all the new 2007 cars at Yahoo! Autos.
 http://autos.yahoo.com/new_cars.html 

From: Kris Kennaway <kris@obsecurity.org>
To: "R. B. Riddick" <arne_woerner@yahoo.com>
Cc: Kris Kennaway <kris@obsecurity.org>, freebsd-gnats-submit@FreeBSD.org
Subject: Re: bin/111101: /usr/bin/lockf: when lockf blocks due to another lockf and no -k is specified and the other lockf ends, the file is away
Date: Sun, 1 Apr 2007 16:15:16 -0400

 On Sun, Apr 01, 2007 at 01:11:04PM -0700, R. B. Riddick wrote:
 
 > Hmm... As long as u say, that I dont understand the use of lockf(1), I will not
 > say/request so much about it...
 
 OK, if you dont want to pursue a documentation improvement then I'll
 close the PR.
 
 Thanks,
 Kris

From: "R. B. Riddick" <arne_woerner@yahoo.com>
To: Kris Kennaway <kris@obsecurity.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: bin/111101: /usr/bin/lockf: when lockf blocks due to another lockf and no -k is specified and the other lockf ends, the file is away
Date: Sun, 1 Apr 2007 13:18:44 -0700 (PDT)

 --- Kris Kennaway <kris@obsecurity.org> wrote:
 > On Sun, Apr 01, 2007 at 01:11:04PM -0700, R. B. Riddick wrote:
 > OK, if you dont want to pursue a documentation improvement then I'll
 > close the PR.
 > 
 Then I will continue to think, that lockf without -k is useless in many cases
 and very fnny/surprising/undcumented.
 
 -Arne
 
 
  
 ____________________________________________________________________________________
 Expecting? Get great news right away with email Auto-Check. 
 Try the Yahoo! Mail Beta.
 http://advision.webevents.yahoo.com/mailbeta/newmail_tools.html 

From: Kris Kennaway <kris@obsecurity.org>
To: "R. B. Riddick" <arne_woerner@yahoo.com>
Cc: Kris Kennaway <kris@obsecurity.org>, freebsd-gnats-submit@FreeBSD.org
Subject: Re: bin/111101: /usr/bin/lockf: when lockf blocks due to another lockf and no -k is specified and the other lockf ends, the file is away
Date: Sun, 1 Apr 2007 16:21:12 -0400

 On Sun, Apr 01, 2007 at 01:18:44PM -0700, R. B. Riddick wrote:
 > --- Kris Kennaway <kris@obsecurity.org> wrote:
 > > On Sun, Apr 01, 2007 at 01:11:04PM -0700, R. B. Riddick wrote:
 > > OK, if you dont want to pursue a documentation improvement then I'll
 > > close the PR.
 > > 
 > Then I will continue to think, that lockf without -k is useless in many cases
 > and very f?nny/surprising/und?cumented.
 
 That's fine, the question is whether you are going to do something
 about improving the documentation that you feel is insufficient.
 
 Please advise :)
 
 Kris

From: "R. B. Riddick" <arne_woerner@yahoo.com>
To: Kris Kennaway <kris@obsecurity.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: bin/111101: /usr/bin/lockf: when lockf blocks due to another lockf and no -k is specified and the other lockf ends, the file is away
Date: Sun, 1 Apr 2007 12:59:53 -0700 (PDT)

 --- Kris Kennaway <kris@obsecurity.org> wrote:
 > On Sun, Apr 01, 2007 at 06:41:43PM +0000, Arne Woerner wrote:
 > > Furthermore I would like to recommend, to make the "-k" behaviour the
 > > default behaviour, so that we will not remove data files, that were co-used
 > > as lock files...
 > 
 > We cannot do that, not using -k is required in some situations, so
 > those scripts will break.
 >
 OK - recommendation withdrawn...
 
 > It appears that lockf is working as designed, and also as explicitly
 > documented.  It looks like no action can be taken here.
 > 
 Explicitly? Documented? Since I have not found that documentation, can u help
 me a little bit by showing it?
 
 In the mean time I show a piece of my documentation (man page lockf(1)):
 "DESCRIPTION
      The lockf utility acquires an exclusive lock on a file, creating it if
      necessary, and removing the file on exit unless explicitly told not to.
      While holding the lock, it executes a command with optional arguments.
      After the command completes, lockf releases the lock, and removes the
      file unless the -k option is specified.  BSD-style locking is used, as
      described in flock(2); the mere existence of the file is not considered
      to constitute a lock."
 I repeat: "CREATING IT IF NECESSARY"
 
 If think, the explicit documentation says, that the file is created (although
 this "if necessary" is somewhat weaselish, so that the man page should be
 changed, too; it should say: "creating it if it does not exist").
 
 Currently without the "-k" option given only the first conflict is solved
 properly, while the next is not, which is surely not what we want, and which is
 surely nowhere documented.
 
 -Arne
 
 
 
  
 ____________________________________________________________________________________
 8:00? 8:25? 8:40? Find a flick in no time 
 with the Yahoo! Search movie showtime shortcut.
 http://tools.search.yahoo.com/shortcuts/#news

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/111101: commit references a PR
Date: Fri, 12 Oct 2007 14:57:04 +0000 (UTC)

 csjp        2007-10-12 14:56:52 UTC
 
   FreeBSD src repository
 
   Modified files:
     usr.bin/lockf        lockf.1 lockf.c 
   Log:
   Revision 1.12 of lockf.c fixed a "thundering herd" scenario when the
   lock experienced contention a number of processes would race to acquire
   lock when it was released.  This problem resulted in a lot of CPU
   load as well as locks being picked up out of order.
   
   Unfortunately, a regression snuck in which allowed multiple threads
   to pickup the same lock when -k was not used.  This could occur when
   multiple processes open a file descriptor to inode X (one process
   will be blocked) and the file is unlinked on unlock (thereby removing
   the directory entry allow another process to create a new directory
   entry for the same file name and lock it).
   
   This changes restores the old algorithm of: wait for the lock, then
   acquire lock when we want to unlink the file on exit (specifically
   when -k is not used) and keeps the new algorithm for when -k is used,
   which yields fairness and improved performance.
   
   Also, update the man page to inform users that if lockf(1) is being
   used to facilitate concurrency between a number of processes, it
   is recommended that -k be used to reduce CPU load and yeld
   fairness with regard to lock ordering.
   
   Collaborated with:      jdp
   PR:             bin/114341
   PR:             bin/116543
   PR:             bin/111101
   MFC after:      1 week
   
   Revision  Changes    Path
   1.19      +13 -1     src/usr.bin/lockf/lockf.1
   1.17      +59 -10    src/usr.bin/lockf/lockf.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
>Unformatted:
