From nobody@FreeBSD.org  Sun Aug 19 23:52:10 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 4BD07106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 19 Aug 2012 23:52:10 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 2B8E08FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 19 Aug 2012 23:52:10 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q7JNq9QR085121
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 19 Aug 2012 23:52:09 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q7JNq9dU085120;
	Sun, 19 Aug 2012 23:52:09 GMT
	(envelope-from nobody)
Message-Id: <201208192352.q7JNq9dU085120@red.freebsd.org>
Date: Sun, 19 Aug 2012 23:52:09 GMT
From: Matthew Story <matthewstory@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: lockf(1) should support a flag to not create a lock file if one does not exist
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         170775
>Category:       bin
>Synopsis:       [patch] lockf(1) should support a flag to not create a lock file if one does not exist
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    eadler
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          update
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 20 00:00:23 UTC 2012
>Closed-Date:    Thu Jun 13 21:02:21 UTC 2013
>Last-Modified:  Thu Jun 13 21:02:21 UTC 2013
>Originator:     Matthew Story
>Release:        10.0-CURRENT
>Organization:
>Environment:
>Description:
I sometimes want to scan a directory and dispatch a job per file in the directory.  In such cases I typically run a bunch of scanning/executing jobs on the directory and use lock(1) to control concurrent and potentially competing  attempts to scan, think something like:

jot - 1 10 | while read line; do find work/ -depth 1 -print0 | xargs -0 -n1 -I {} lockf -k -t0 {} sh -c '
    # do something
    mv "$1" done/' worker {} &
done

In cases where I want to lock on a file that I know exists, do some work and move on, there is a small race-condition were a secondary job to attempt to lock on a file that had been moved.

In other cases, it would be nice to use mv to short-circuit a coordinated lockf (with -k) effort:

lockf -k foo/bar sh -c 'something || rm foo/bar' &
lockf -k foo/bar sh -c 'second thing || rm foo/bar' & 

To this end, I propose the attached `-n' flag, which augments the behavior of lockf(1) by removing the O_CREAT flag at open(2) time: making the above

jot - 1 10 | while read line; do find work/ -depth 1 -print0 | xargs -0 -n1 -I {} lockf -kn -t0 {} sh -c '
    # do something
    mv "$1" done/' worker {} &
done

and

lockf -k foo/bar sh -c 'something || rm foo/bar' &
lockf -kn foo/bar sh -c 'second thing || rm foo/bar'

The patch causes exit due to nonexistence to exit with EX_UNAVAILABLE if -n is specified and the lock file does not exist.  

The patch does not distinguish between failure due to missing intermediate directory and missing lock-file, although lockf could probably be cleaned up to use open + fopenat to cut down on stats for retried locks, which would also allow -n to distinguish between the two failure cases trivially.

The patch does not cause failure due to missing lock file to have it's diagnostic information suppressed by the -s (silent) flag, although perhaps it should (I was unsure about which way to go on this one).
>How-To-Repeat:

>Fix:
patch attached, also available via HTTP: 

http://axe0.blackskyresearch.net/patches/matt/freebsd.nocreate_lockf.patch.txt

Patch attached with submission follows:

Index: usr.bin/lockf/lockf.c
===================================================================
--- usr.bin/lockf/lockf.c	(revision 239407)
+++ usr.bin/lockf/lockf.c	(working copy)
@@ -56,12 +56,13 @@
 int
 main(int argc, char **argv)
 {
-	int ch, silent, status, waitsec;
+	int ch, flags, silent, status, waitsec;
 	pid_t child;
 
 	silent = keep = 0;
+	flags = O_CREAT;
 	waitsec = -1;	/* Infinite. */
-	while ((ch = getopt(argc, argv, "skt:")) != -1) {
+	while ((ch = getopt(argc, argv, "sknt:")) != -1) {
 		switch (ch) {
 		case 'k':
 			keep = 1;
@@ -69,6 +70,9 @@
 		case 's':
 			silent = 1;
 			break;
+		case 'n':
+			flags &= ~O_CREAT;
+			break;
 		case 't':
 		{
 			char *endptr;
@@ -118,13 +122,13 @@
 	 * avoiding the separate step of waiting for the lock.  This
 	 * yields fairness and improved performance.
 	 */
-	lockfd = acquire_lock(lockname, O_NONBLOCK);
+	lockfd = acquire_lock(lockname, flags|O_NONBLOCK);
 	while (lockfd == -1 && !timed_out && waitsec != 0) {
-		if (keep)
-			lockfd = acquire_lock(lockname, 0);
+		if (keep) 
+			lockfd = acquire_lock(lockname, flags);
 		else {
 			wait_for_lock(lockname);
-			lockfd = acquire_lock(lockname, O_NONBLOCK);
+			lockfd = acquire_lock(lockname, flags|O_NONBLOCK);
 		}
 	}
 	if (waitsec > 0)
@@ -165,9 +169,12 @@
 {
 	int fd;
 
-	if ((fd = open(name, O_RDONLY|O_CREAT|O_EXLOCK|flags, 0666)) == -1) {
+	if ((fd = open(name, O_RDONLY|O_EXLOCK|flags, 0666)) == -1) {
 		if (errno == EAGAIN || errno == EINTR)
 			return (-1);
+		else if (errno == ENOENT && (flags&O_CREAT) == 0)
+			err(EX_UNAVAILABLE, "%s", name);
+
 		err(EX_CANTCREAT, "cannot open %s", name);
 	}
 	return (fd);
@@ -215,7 +222,7 @@
 {
 
 	fprintf(stderr,
-	    "usage: lockf [-ks] [-t seconds] file command [arguments]\n");
+	    "usage: lockf [-ksn] [-t seconds] file command [arguments]\n");
 	exit(EX_USAGE);
 }
 
Index: usr.bin/lockf/lockf.1
===================================================================
--- usr.bin/lockf/lockf.1	(revision 239407)
+++ usr.bin/lockf/lockf.1	(working copy)
@@ -90,6 +90,18 @@
 .Nm
 to operate silently.
 Failure to acquire the lock is indicated only in the exit status.
+.It Fl n
+Causes
+.Nm
+to fail if the specified lock
+.Ar file
+does not exist. If
+.Fl n
+is not specified,
+.Nm
+will create
+.Ar file
+if necessary.
 .It Fl t Ar seconds
 Specifies a timeout for waiting for the lock.
 By default,
@@ -130,6 +142,10 @@
 utility
 was unable to create the lock file, e.g., because of insufficient access
 privileges.
+.It Dv EX_UNAVAILABLE
+The
+.Fl n
+option is specified and the specified lock file does not exist.
 .It Dv EX_USAGE
 There was an error on the
 .Nm


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->eadler 
Responsible-Changed-By: eadler 
Responsible-Changed-When: Fri May 10 17:19:22 UTC 2013 
Responsible-Changed-Why:  
I'll take it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=170775 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/170775: commit references a PR
Date: Fri, 10 May 2013 17:30:42 +0000 (UTC)

 Author: eadler
 Date: Fri May 10 17:30:29 2013
 New Revision: 250462
 URL: http://svnweb.freebsd.org/changeset/base/250462
 
 Log:
   Add option to lockf to avoid creating a file if it does not exist.
   
   PR:	bin/170775
   Submitted by:	Matthew Story <matthewstory@gmail.com>
   Reviewed by:	scottl
   MFC after:	1 week
 
 Modified:
   head/usr.bin/lockf/lockf.1
   head/usr.bin/lockf/lockf.c
 
 Modified: head/usr.bin/lockf/lockf.1
 ==============================================================================
 --- head/usr.bin/lockf/lockf.1	Fri May 10 16:48:21 2013	(r250461)
 +++ head/usr.bin/lockf/lockf.1	Fri May 10 17:30:29 2013	(r250462)
 @@ -32,7 +32,7 @@
  .Nd execute a command while holding a file lock
  .Sh SYNOPSIS
  .Nm
 -.Op Fl ks
 +.Op Fl kns
  .Op Fl t Ar seconds
  .Ar file
  .Ar command
 @@ -90,6 +90,18 @@ Causes
  .Nm
  to operate silently.
  Failure to acquire the lock is indicated only in the exit status.
 +.It Fl n
 +Causes
 +.Nm
 +to fail if the specified lock
 +.Ar file
 +does not exist. If
 +.Fl n
 +is not specified,
 +.Nm
 +will create
 +.Ar file
 +if necessary.
  .It Fl t Ar seconds
  Specifies a timeout for waiting for the lock.
  By default,
 @@ -130,6 +142,10 @@ The
  utility
  was unable to create the lock file, e.g., because of insufficient access
  privileges.
 +.It Dv EX_UNAVAILABLE
 +The
 +.Fl n
 +option is specified and the specified lock file does not exist.
  .It Dv EX_USAGE
  There was an error on the
  .Nm
 
 Modified: head/usr.bin/lockf/lockf.c
 ==============================================================================
 --- head/usr.bin/lockf/lockf.c	Fri May 10 16:48:21 2013	(r250461)
 +++ head/usr.bin/lockf/lockf.c	Fri May 10 17:30:29 2013	(r250462)
 @@ -56,16 +56,20 @@ static volatile sig_atomic_t timed_out;
  int
  main(int argc, char **argv)
  {
 -	int ch, silent, status, waitsec;
 +	int ch, flags, silent, status, waitsec;
  	pid_t child;
  
  	silent = keep = 0;
 +	flags = O_CREAT;
  	waitsec = -1;	/* Infinite. */
 -	while ((ch = getopt(argc, argv, "skt:")) != -1) {
 +	while ((ch = getopt(argc, argv, "sknt:")) != -1) {
  		switch (ch) {
  		case 'k':
  			keep = 1;
  			break;
 +		case 'n':
 +			flags &= ~O_CREAT;
 +			break;
  		case 's':
  			silent = 1;
  			break;
 @@ -118,13 +122,13 @@ main(int argc, char **argv)
  	 * avoiding the separate step of waiting for the lock.  This
  	 * yields fairness and improved performance.
  	 */
 -	lockfd = acquire_lock(lockname, O_NONBLOCK);
 +	lockfd = acquire_lock(lockname, flags | O_NONBLOCK);
  	while (lockfd == -1 && !timed_out && waitsec != 0) {
  		if (keep)
 -			lockfd = acquire_lock(lockname, 0);
 +			lockfd = acquire_lock(lockname, flags);
  		else {
  			wait_for_lock(lockname);
 -			lockfd = acquire_lock(lockname, O_NONBLOCK);
 +			lockfd = acquire_lock(lockname, flags | O_NONBLOCK);
  		}
  	}
  	if (waitsec > 0)
 @@ -165,7 +169,7 @@ acquire_lock(const char *name, int flags
  {
  	int fd;
  
 -	if ((fd = open(name, O_RDONLY|O_CREAT|O_EXLOCK|flags, 0666)) == -1) {
 +	if ((fd = open(name, flags|O_RDONLY|O_EXLOCK|flags, 0666)) == -1) {
  		if (errno == EAGAIN || errno == EINTR)
  			return (-1);
  		err(EX_CANTCREAT, "cannot open %s", name);
 @@ -215,7 +219,7 @@ usage(void)
  {
  
  	fprintf(stderr,
 -	    "usage: lockf [-ks] [-t seconds] file command [arguments]\n");
 +	    "usage: lockf [-kns] [-t seconds] file command [arguments]\n");
  	exit(EX_USAGE);
  }
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: eadler 
State-Changed-When: Sat May 11 13:54:37 UTC 2013 
State-Changed-Why:  
committed 


http://www.freebsd.org/cgi/query-pr.cgi?pr=170775 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/170775: commit references a PR
Date: Thu, 13 Jun 2013 20:46:58 +0000 (UTC)

 Author: eadler
 Date: Thu Jun 13 20:46:30 2013
 New Revision: 251705
 URL: http://svnweb.freebsd.org/changeset/base/251705
 
 Log:
   MFC r250462:
   	Add option to lockf to avoid creating a file if it does not exist.
   
   PR:	bin/170775
 
 Modified:
   stable/8/usr.bin/lockf/lockf.1
   stable/8/usr.bin/lockf/lockf.c
 Directory Properties:
   stable/8/usr.bin/lockf/   (props changed)
 
 Modified: stable/8/usr.bin/lockf/lockf.1
 ==============================================================================
 --- stable/8/usr.bin/lockf/lockf.1	Thu Jun 13 20:46:29 2013	(r251704)
 +++ stable/8/usr.bin/lockf/lockf.1	Thu Jun 13 20:46:30 2013	(r251705)
 @@ -32,7 +32,7 @@
  .Nd execute a command while holding a file lock
  .Sh SYNOPSIS
  .Nm
 -.Op Fl ks
 +.Op Fl kns
  .Op Fl t Ar seconds
  .Ar file
  .Ar command
 @@ -90,6 +90,18 @@ Causes
  .Nm
  to operate silently.
  Failure to acquire the lock is indicated only in the exit status.
 +.It Fl n
 +Causes
 +.Nm
 +to fail if the specified lock
 +.Ar file
 +does not exist. If
 +.Fl n
 +is not specified,
 +.Nm
 +will create
 +.Ar file
 +if necessary.
  .It Fl t Ar seconds
  Specifies a timeout for waiting for the lock.
  By default,
 @@ -130,6 +142,10 @@ The
  utility
  was unable to create the lock file, e.g., because of insufficient access
  privileges.
 +.It Dv EX_UNAVAILABLE
 +The
 +.Fl n
 +option is specified and the specified lock file does not exist.
  .It Dv EX_USAGE
  There was an error on the
  .Nm
 
 Modified: stable/8/usr.bin/lockf/lockf.c
 ==============================================================================
 --- stable/8/usr.bin/lockf/lockf.c	Thu Jun 13 20:46:29 2013	(r251704)
 +++ stable/8/usr.bin/lockf/lockf.c	Thu Jun 13 20:46:30 2013	(r251705)
 @@ -56,16 +56,20 @@ static volatile sig_atomic_t timed_out;
  int
  main(int argc, char **argv)
  {
 -	int ch, silent, status, waitsec;
 +	int ch, flags, silent, status, waitsec;
  	pid_t child;
  
  	silent = keep = 0;
 +	flags = O_CREAT;
  	waitsec = -1;	/* Infinite. */
 -	while ((ch = getopt(argc, argv, "skt:")) != -1) {
 +	while ((ch = getopt(argc, argv, "sknt:")) != -1) {
  		switch (ch) {
  		case 'k':
  			keep = 1;
  			break;
 +		case 'n':
 +			flags &= ~O_CREAT;
 +			break;
  		case 's':
  			silent = 1;
  			break;
 @@ -118,13 +122,13 @@ main(int argc, char **argv)
  	 * avoiding the separate step of waiting for the lock.  This
  	 * yields fairness and improved performance.
  	 */
 -	lockfd = acquire_lock(lockname, O_NONBLOCK);
 +	lockfd = acquire_lock(lockname, flags | O_NONBLOCK);
  	while (lockfd == -1 && !timed_out && waitsec != 0) {
  		if (keep)
 -			lockfd = acquire_lock(lockname, 0);
 +			lockfd = acquire_lock(lockname, flags);
  		else {
  			wait_for_lock(lockname);
 -			lockfd = acquire_lock(lockname, O_NONBLOCK);
 +			lockfd = acquire_lock(lockname, flags | O_NONBLOCK);
  		}
  	}
  	if (waitsec > 0)
 @@ -165,7 +169,7 @@ acquire_lock(const char *name, int flags
  {
  	int fd;
  
 -	if ((fd = open(name, O_RDONLY|O_CREAT|O_EXLOCK|flags, 0666)) == -1) {
 +	if ((fd = open(name, flags|O_RDONLY|O_EXLOCK|flags, 0666)) == -1) {
  		if (errno == EAGAIN || errno == EINTR)
  			return (-1);
  		err(EX_CANTCREAT, "cannot open %s", name);
 @@ -215,7 +219,7 @@ usage(void)
  {
  
  	fprintf(stderr,
 -	    "usage: lockf [-ks] [-t seconds] file command [arguments]\n");
 +	    "usage: lockf [-kns] [-t seconds] file command [arguments]\n");
  	exit(EX_USAGE);
  }
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/170775: commit references a PR
Date: Thu, 13 Jun 2013 20:46:42 +0000 (UTC)

 Author: eadler
 Date: Thu Jun 13 20:46:29 2013
 New Revision: 251704
 URL: http://svnweb.freebsd.org/changeset/base/251704
 
 Log:
   MFC r250462:
   	Add option to lockf to avoid creating a file if it does not exist.
   
   PR:	bin/170775
 
 Modified:
   stable/9/usr.bin/lockf/lockf.1
   stable/9/usr.bin/lockf/lockf.c
 Directory Properties:
   stable/9/usr.bin/lockf/   (props changed)
 
 Modified: stable/9/usr.bin/lockf/lockf.1
 ==============================================================================
 --- stable/9/usr.bin/lockf/lockf.1	Thu Jun 13 20:46:03 2013	(r251703)
 +++ stable/9/usr.bin/lockf/lockf.1	Thu Jun 13 20:46:29 2013	(r251704)
 @@ -32,7 +32,7 @@
  .Nd execute a command while holding a file lock
  .Sh SYNOPSIS
  .Nm
 -.Op Fl ks
 +.Op Fl kns
  .Op Fl t Ar seconds
  .Ar file
  .Ar command
 @@ -90,6 +90,18 @@ Causes
  .Nm
  to operate silently.
  Failure to acquire the lock is indicated only in the exit status.
 +.It Fl n
 +Causes
 +.Nm
 +to fail if the specified lock
 +.Ar file
 +does not exist. If
 +.Fl n
 +is not specified,
 +.Nm
 +will create
 +.Ar file
 +if necessary.
  .It Fl t Ar seconds
  Specifies a timeout for waiting for the lock.
  By default,
 @@ -130,6 +142,10 @@ The
  utility
  was unable to create the lock file, e.g., because of insufficient access
  privileges.
 +.It Dv EX_UNAVAILABLE
 +The
 +.Fl n
 +option is specified and the specified lock file does not exist.
  .It Dv EX_USAGE
  There was an error on the
  .Nm
 
 Modified: stable/9/usr.bin/lockf/lockf.c
 ==============================================================================
 --- stable/9/usr.bin/lockf/lockf.c	Thu Jun 13 20:46:03 2013	(r251703)
 +++ stable/9/usr.bin/lockf/lockf.c	Thu Jun 13 20:46:29 2013	(r251704)
 @@ -56,16 +56,20 @@ static volatile sig_atomic_t timed_out;
  int
  main(int argc, char **argv)
  {
 -	int ch, silent, status, waitsec;
 +	int ch, flags, silent, status, waitsec;
  	pid_t child;
  
  	silent = keep = 0;
 +	flags = O_CREAT;
  	waitsec = -1;	/* Infinite. */
 -	while ((ch = getopt(argc, argv, "skt:")) != -1) {
 +	while ((ch = getopt(argc, argv, "sknt:")) != -1) {
  		switch (ch) {
  		case 'k':
  			keep = 1;
  			break;
 +		case 'n':
 +			flags &= ~O_CREAT;
 +			break;
  		case 's':
  			silent = 1;
  			break;
 @@ -118,13 +122,13 @@ main(int argc, char **argv)
  	 * avoiding the separate step of waiting for the lock.  This
  	 * yields fairness and improved performance.
  	 */
 -	lockfd = acquire_lock(lockname, O_NONBLOCK);
 +	lockfd = acquire_lock(lockname, flags | O_NONBLOCK);
  	while (lockfd == -1 && !timed_out && waitsec != 0) {
  		if (keep)
 -			lockfd = acquire_lock(lockname, 0);
 +			lockfd = acquire_lock(lockname, flags);
  		else {
  			wait_for_lock(lockname);
 -			lockfd = acquire_lock(lockname, O_NONBLOCK);
 +			lockfd = acquire_lock(lockname, flags | O_NONBLOCK);
  		}
  	}
  	if (waitsec > 0)
 @@ -165,7 +169,7 @@ acquire_lock(const char *name, int flags
  {
  	int fd;
  
 -	if ((fd = open(name, O_RDONLY|O_CREAT|O_EXLOCK|flags, 0666)) == -1) {
 +	if ((fd = open(name, flags|O_RDONLY|O_EXLOCK|flags, 0666)) == -1) {
  		if (errno == EAGAIN || errno == EINTR)
  			return (-1);
  		err(EX_CANTCREAT, "cannot open %s", name);
 @@ -215,7 +219,7 @@ usage(void)
  {
  
  	fprintf(stderr,
 -	    "usage: lockf [-ks] [-t seconds] file command [arguments]\n");
 +	    "usage: lockf [-kns] [-t seconds] file command [arguments]\n");
  	exit(EX_USAGE);
  }
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: eadler 
State-Changed-When: Thu Jun 13 21:02:21 UTC 2013 
State-Changed-Why:  
Committed. Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=170775 
>Unformatted:
