From danny@cs.huji.ac.il  Tue Jun  9 10:15:35 2009
Return-Path: <danny@cs.huji.ac.il>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 6627210656C4
	for <FreeBSD-gnats-submit@freebsd.org>; Tue,  9 Jun 2009 10:15:35 +0000 (UTC)
	(envelope-from danny@cs.huji.ac.il)
Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84])
	by mx1.freebsd.org (Postfix) with ESMTP id 1F16D8FC08
	for <FreeBSD-gnats-submit@freebsd.org>; Tue,  9 Jun 2009 10:15:35 +0000 (UTC)
	(envelope-from danny@cs.huji.ac.il)
Received: from store-01.cs.huji.ac.il ([132.65.64.104])
	by kabab.cs.huji.ac.il with esmtp
	id 1MDxmM-000Dvh-12
	for FreeBSD-gnats-submit@freebsd.org; Tue, 09 Jun 2009 12:38:02 +0300
Received: from danny by store-01.cs.huji.ac.il with local (Exim 4.69 (FreeBSD))
	(envelope-from <danny@cs.huji.ac.il>)
	id 1MDxmM-0001HJ-0g
	for FreeBSD-gnats-submit@freebsd.org; Tue, 09 Jun 2009 12:38:02 +0300
Message-Id: <E1MDxmM-0001HJ-0g@store-01.cs.huji.ac.il>
Date: Tue, 09 Jun 2009 12:38:02 +0300
From: Danny Braniss <danny@cs.huji.ac.il>
Reply-To: Danny Braniss <danny@cs.huji.ac.il>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: zfs(v13)+nfs and open(..., O_WRONLY|O_CREAT|O_EXCL, ...) returns io error
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         135412
>Category:       kern
>Synopsis:       [zfs] [nfs] zfs(v13)+nfs and open(..., O_WRONLY|O_CREAT|O_EXCL, ...) returns io error
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-fs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jun 09 10:20:03 UTC 2009
>Closed-Date:    Sat Jul 04 21:26:51 UTC 2009
>Last-Modified:  Fri Dec  4 16:40:02 UTC 2009
>Originator:     Danny Braniss
>Release:        FreeBSD 7.2-STABLE amd64
>Organization:
>Environment:
System: FreeBSD store-01 7.2-STABLE FreeBSD 7.2-STABLE #24: Mon Jun 1 12:17:54 IDT 2009 danny@sunfire:/r+d/obj/sunfire/r+d/7/sys/HUJI amd64


	
>Description:
	a file system(zfs-v13) NFS mounted now fails
	       fd = open(..., O_WRONLY|O_CREAT|O_EXCL, 0666);
	file is created with mode 0, and errno=Input/output errno(5).

>How-To-Repeat:
#include <stdio.h>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>

main(int cc, char **vv)
{
     int	fd;
     char	*fn;

     if(cc > 1)
	  fn = vv[1];
     else
	  fn = "lock";
     fd = open(fn, O_WRONLY|O_CREAT|O_EXCL, 0666);
     if(fd < 0) {
	  int err = errno;
	  fprintf(stderr, "%d - %s\n", err, strerror(err));
     }
     exit(0);
}

>Fix:

	


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Jun 10 05:15:55 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135412 

From: Gavin Atkinson <gavin@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc: Danny Braniss <danny@cs.huji.ac.il>
Subject: Re: kern/135412: ZFS issue
Date: Mon, 22 Jun 2009 14:30:27 +0100

 Other useful information (from PR kern/135039):
 
 This appears to only affect 7-STABLE since the ZFS merge, but doesn't
 affect -HEAD.
 
 After the recent import of ZFS v13 into 7-STABLE, an mkstemp() call from
 an NFS client to a ZFS-backed NFS server will fail: the syscall returns
 EIO and the server will have created a 0-byte file with 000 permissions.
 This breaks not just mktemp but also mv, tar, rsync...
 
 Kip Macy said there's a flags check that is too strict, in email message
 <3c1674c90905280025i17039257l573838d33d8493fd@mail.gmail.com>
 Otherwise, use cp and rm instead of mv, or use scp instead of NFS, or
 use UFS2 on the server
 
 To submitter: did you upgrade your on-disk pools to v13, or is running
 the new code and v6 pools enough to show the problem?  What is the
 output of "zpool upgrade"?
 

From: Danny Braniss <danny@cs.huji.ac.il>
To: Gavin Atkinson <gavin@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/135412: ZFS issue 
Date: Mon, 22 Jun 2009 17:53:44 +0300

 > Other useful information (from PR kern/135039):
 > 
 > This appears to only affect 7-STABLE since the ZFS merge, but doesn't
 > affect -HEAD.
 > 
 > After the recent import of ZFS v13 into 7-STABLE, an mkstemp() call from
 > an NFS client to a ZFS-backed NFS server will fail: the syscall returns
 > EIO and the server will have created a 0-byte file with 000 permissions.
 > This breaks not just mktemp but also mv, tar, rsync...
 > 
 > Kip Macy said there's a flags check that is too strict, in email message
 > <3c1674c90905280025i17039257l573838d33d8493fd@mail.gmail.com>
 > Otherwise, use cp and rm instead of mv, or use scp instead of NFS, or
 > use UFS2 on the server
 > 
 > To submitter: did you upgrade your on-disk pools to v13, or is running
 > the new code and v6 pools enough to show the problem?  What is the
 > output of "zpool upgrade"?
 
 it happens ONLY because zfs is v13, irrelevant if the pools have been upgraded,
 created, or not upgraded.
 so yes, just running a newer -stable is enough to show the problem.
 
 Since I am using ZFS + NFS, and providing service to several hundred users,
 telling them not to use tar, svn, rsync, etc is not an option :-)
 also, NFS V2 is broken/un-maintained.
 
 danny
 
 
 
 > 
 
 

From: Andriy Gapon <avg@icyb.net.ua>
To: bug-followup@FreeBSD.org, danny@cs.huji.ac.il
Cc:  
Subject: Re: kern/135412: [zfs] [nfs] zfs(v13)+nfs and open(..., O_WRONLY|O_CREAT|O_EXCL,
 ...) returns io error
Date: Tue, 30 Jun 2009 17:22:47 +0300

 Danny,
 
 maybe you misunderstood Gavin's question or maybe I misunderstood your reply.
 
 I have stable/7 amd64 from Jun 26, I have upgraded zpool to version 13 (as
 reported by 'zpool upgrade' command) and I have upgraded zfs on-disk to version 3
 (as reported by 'zfs upgrade').
 And I can _not_ reproduce your problem using the program you provided - it
 successfully creates a file on the first run and it fails with '17 - File exists'
 on the subsequent ones.
 
 So, I'd like to re-iterate the question - what on-disk versions of zpool and zfs
 you have?
 Please provide output of 'zpool upgrade' and 'zfs upgrade' commands to avoid
 further uncertainties.
 
 -- 
 Andriy Gapon

From: Danny Braniss <danny@cs.huji.ac.il>
To: Andriy Gapon <avg@icyb.net.ua>
Cc: bug-followup@FreeBSD.org, danny@cs.huji.ac.il, danny@cs.huji.ac.il
Subject: Re: kern/135412: [zfs] [nfs] zfs(v13)+nfs and open(..., 
 O_WRONLY|O_CREAT|O_EXCL, ...) returns io error
Date: Tue, 30 Jun 2009 18:05:28 +0300

 > 
 > Danny,
 > 
 > maybe you misunderstood Gavin's question or maybe I misunderstood your reply.
 > 
 > I have stable/7 amd64 from Jun 26, I have upgraded zpool to version 13 (as
 > reported by 'zpool upgrade' command) and I have upgraded zfs on-disk to version 3
 > (as reported by 'zfs upgrade').
 > And I can _not_ reproduce your problem using the program you provided - it
 > successfully creates a file on the first run and it fails with '17 - File exists'
 > on the subsequent ones.
 > 
 > So, I'd like to re-iterate the question - what on-disk versions of zpool and zfs
 > you have?
 > Please provide output of 'zpool upgrade' and 'zfs upgrade' commands to avoid
 > further uncertainties.
 > 
 > -- 
 > Andriy Gapon
 
 you have to run the program on a client that mounted the zfs volume via nfs.
 it fails no matter what pool version, either 6 or 13
 btw, it works fine if the server is solaris/v13
 
 but just to answer your questions, dev is the server host
 
 dev> zfs upgrade
 This system is currently running ZFS filesystem version 3.
 
 All filesystems are formatted with the current version.
 dev> zpool upgrade
 This system is currently running ZFS pool version 13.
 
 All pools are formatted using this version.
 dev> 
 
 danny
 
 

From: Mike Andrews <mandrews@bit0.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/135412: [zfs] [nfs] zfs(v13)+nfs and open(..., O_WRONLY|O_CREAT|O_EXCL,
 ...) returns io error
Date: Tue, 30 Jun 2009 14:40:43 -0400

 I can also confirm that this happens with an un-upgraded v6 pool using the 
 v13 code.

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/135412: commit references a PR
Date: Wed,  1 Jul 2009 12:44:37 +0000 (UTC)

 Author: avg
 Date: Wed Jul  1 12:44:23 2009
 New Revision: 195236
 URL: http://svn.freebsd.org/changeset/base/195236
 
 Log:
   MFC 185586 (kan): Change nfsserver slightly so that it does not trip
   over the timestamp validation code on ZFS.
   
   This should fix O_CREAT|O_EXCL open on NFS where a server is 64-bit
   with v13 ZFS code.
   
   PR:		kern/135412
   Pointed out by:	Jaakko Heinonen <jh@saunalahti.fi>
   Tested by:	Jaakko Heinonen, Danny Braniss <danny@cs.huji.ac.il>
 
 Modified:
   stable/7/sys/   (props changed)
   stable/7/sys/contrib/pf/   (props changed)
   stable/7/sys/nfsserver/nfs_serv.c
 
 Modified: stable/7/sys/nfsserver/nfs_serv.c
 ==============================================================================
 --- stable/7/sys/nfsserver/nfs_serv.c	Wed Jul  1 12:36:10 2009	(r195235)
 +++ stable/7/sys/nfsserver/nfs_serv.c	Wed Jul  1 12:44:23 2009	(r195236)
 @@ -1656,13 +1656,12 @@ nfsrv_create(struct nfsrv_descript *nfsd
  	caddr_t bpos;
  	int error = 0, rdev, len, tsize, dirfor_ret = 1, diraft_ret = 1;
  	int v3 = (nfsd->nd_flag & ND_NFSV3), how, exclusive_flag = 0;
 -	caddr_t cp;
  	struct mbuf *mb, *mreq;
  	struct vnode *dirp = NULL;
  	nfsfh_t nfh;
  	fhandle_t *fhp;
  	u_quad_t tempsize;
 -	u_char cverf[NFSX_V3CREATEVERF];
 +	struct timespec cverf;
  	struct mount *mp = NULL;
  	int tvfslocked;
  	int vfslocked;
 @@ -1741,8 +1740,11 @@ nfsrv_create(struct nfsrv_descript *nfsd
  			nfsm_srvsattr(vap);
  			break;
  		case NFSV3CREATE_EXCLUSIVE:
 -			cp = nfsm_dissect_nonblock(caddr_t, NFSX_V3CREATEVERF);
 -			bcopy(cp, cverf, NFSX_V3CREATEVERF);
 +			tl = nfsm_dissect_nonblock(u_int32_t *,
 +			    NFSX_V3CREATEVERF);
 +			/* Unique bytes, endianness is not important. */
 +			cverf.tv_sec  = tl[0];
 +			cverf.tv_nsec = tl[1];
  			exclusive_flag = 1;
  			break;
  		};
 @@ -1788,8 +1790,7 @@ nfsrv_create(struct nfsrv_descript *nfsd
  				if (exclusive_flag) {
  					exclusive_flag = 0;
  					VATTR_NULL(vap);
 -					bcopy(cverf, (caddr_t)&vap->va_atime,
 -						NFSX_V3CREATEVERF);
 +					vap->va_atime = cverf;
  					error = VOP_SETATTR(nd.ni_vp, vap, cred,
  						td);
  				}
 @@ -1873,7 +1874,7 @@ nfsrv_create(struct nfsrv_descript *nfsd
  	}
  	if (v3) {
  		if (exclusive_flag && !error &&
 -			bcmp(cverf, (caddr_t)&vap->va_atime, NFSX_V3CREATEVERF))
 +		    bcmp(&cverf, &vap->va_atime, sizeof (cverf)))
  			error = EEXIST;
  		if (dirp == nd.ni_dvp)
  			diraft_ret = VOP_GETATTR(dirp, &diraft, cred, td);
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->closed 
State-Changed-By: jilles 
State-Changed-When: Sat Jul 4 21:26:50 UTC 2009 
State-Changed-Why:  
Fix committed, no other branches affected. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135412 

From: John Hein <jhein@symmetricom.com>
To: bug-followup@FreeBSD.org, danny@cs.huji.ac.il
Cc: jilles@FreeBSD.org
Subject: Re: kern/135412: [zfs] [nfs] zfs(v13)+nfs and open(..., O_WRONLY|O_CREAT|O_EXCL, ...) returns io error
Date: Thu, 3 Dec 2009 19:18:39 -0700

 I still get this.
 
 But it happens when the nfs client is a FreeBSD 4.x machine or a linux
 machine (tested with Fedora 10 and 11).  And it does not seem to
 happen with nfs v2, just nfs v3.
 
 This breaks xauth and various programs but the common thread
 is "open(..., O_WRONLY|O_CREAT|O_EXCL, ...)".
 

From: Jaakko Heinonen <jh@FreeBSD.org>
To: John Hein <jhein@symmetricom.com>
Cc: bug-followup@FreeBSD.org, danny@cs.huji.ac.il, jilles@FreeBSD.org
Subject: Re: kern/135412: [zfs] [nfs] zfs(v13)+nfs and open(...,
 O_WRONLY|O_CREAT|O_EXCL, ...) returns io error
Date: Fri, 4 Dec 2009 08:26:17 +0200

 On 2009-12-04, John Hein wrote:
 >  But it happens when the nfs client is a FreeBSD 4.x machine or a linux
 >  machine (tested with Fedora 10 and 11).  And it does not seem to
 >  happen with nfs v2, just nfs v3.
 
 Which FreeBSD version your server is running? There was an additional
 fix (r197525) but it hasn't been MFCd to stable/7.
 
 Here's the patch against stable/7.
 
 %%%
 Index: sys/nfsserver/nfs_serv.c
 ===================================================================
 --- sys/nfsserver/nfs_serv.c	(revision 200062)
 +++ sys/nfsserver/nfs_serv.c	(working copy)
 @@ -1743,7 +1743,7 @@ nfsrv_create(struct nfsrv_descript *nfsd
  			tl = nfsm_dissect_nonblock(u_int32_t *,
  			    NFSX_V3CREATEVERF);
  			/* Unique bytes, endianness is not important. */
 -			cverf.tv_sec  = tl[0];
 +			cverf.tv_sec  = (int32_t)tl[0];
  			cverf.tv_nsec = tl[1];
  			exclusive_flag = 1;
  			break;
 %%%
 
 -- 
 Jaakko

From: John Hein <jhein@symmetricom.com>
To: Jaakko Heinonen <jh@FreeBSD.org>
Cc: bug-followup@FreeBSD.org, danny@cs.huji.ac.il, jilles@FreeBSD.org
Subject: Re: kern/135412: [zfs] [nfs] zfs(v13)+nfs and open(...,
 O_WRONLY|O_CREAT|O_EXCL, ...) returns io error
Date: Fri, 4 Dec 2009 09:37:10 -0700

 Jaakko Heinonen wrote at 08:26 +0200 on Dec  4, 2009:
  > On 2009-12-04, John Hein wrote:
  > >  But it happens when the nfs client is a FreeBSD 4.x machine or a linux
  > >  machine (tested with Fedora 10 and 11).  And it does not seem to
  > >  happen with nfs v2, just nfs v3.
  > 
  > Which FreeBSD version your server is running? There was an additional
  > fix (r197525) but it hasn't been MFCd to stable/7.
  > 
  > Here's the patch against stable/7.
  > 
  > %%%
  > Index: sys/nfsserver/nfs_serv.c
  > ===================================================================
  > --- sys/nfsserver/nfs_serv.c	(revision 200062)
  > +++ sys/nfsserver/nfs_serv.c	(working copy)
  > @@ -1743,7 +1743,7 @@ nfsrv_create(struct nfsrv_descript *nfsd
  >  			tl = nfsm_dissect_nonblock(u_int32_t *,
  >  			    NFSX_V3CREATEVERF);
  >  			/* Unique bytes, endianness is not important. */
  > -			cverf.tv_sec  = tl[0];
  > +			cverf.tv_sec  = (int32_t)tl[0];
  >  			cverf.tv_nsec = tl[1];
  >  			exclusive_flag = 1;
  >  			break;
  > %%%
 
 
 Yes, I saw the same thing, built a new kernel with that change last
 night.  I tested this morning, and it fixes the problem (and causes no
 new ones that I have seen).
 
 +1 for MFC to 7.
 
 Incidentally, is this going to have to be reworked because of
 the 32-bit time_t rollover (admittedly, still quite a ways
 away)?
>Unformatted:
