From mika@dogmatix.cs.caltech.edu  Wed Aug 12 16:38:30 1998
Received: from vlsi.cs.caltech.edu (vlsi.cs.caltech.edu [131.215.131.129])
          by hub.freebsd.org (8.8.8/8.8.8) with SMTP id QAA05485
          for <FreeBSD-gnats-submit@freebsd.org>; Wed, 12 Aug 1998 16:38:21 -0700 (PDT)
          (envelope-from mika@dogmatix.cs.caltech.edu)
Received: from dogmatix.cs.caltech.edu by vlsi.cs.caltech.edu (4.1/1.34.1)
	id AA14077; Wed, 12 Aug 98 16:37:55 PDT
Received: (from mika@localhost)
	by dogmatix.cs.caltech.edu (8.8.8/8.8.7) id QAA20983;
	Wed, 12 Aug 1998 16:37:54 -0700 (PDT)
Message-Id: <199808122337.QAA20983@dogmatix.cs.caltech.edu>
Date: Wed, 12 Aug 1998 16:37:54 -0700 (PDT)
From: Mika Nystrom <mika@cs.caltech.edu>
Reply-To: mika@cs.caltech.edu
To: FreeBSD-gnats-submit@freebsd.org
Subject: serious data integrity problem when reading WHILE writing NFSv3 client-end
X-Send-Pr-Version: 3.2

>Number:         7596
>Category:       kern
>Synopsis:       serious data integrity problem when reading WHILE writing NFSv3 client-end
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Aug 12 16:40:00 PDT 1998
>Closed-Date:    Sat Dec 12 16:58:53 PST 1998
>Last-Modified:  Sat Dec 12 17:01:21 PST 1998
>Originator:     Mika Nystrom
>Release:        FreeBSD 3.0-CURRENT i386
>Organization:
California Institute of Technology
>Environment:

FreeBSD dogmatix.cs.caltech.edu 3.0-CURRENT FreeBSD 3.0-CURRENT #7: Mon Apr  6 15:05:17 PDT 1998     mika@obelix.cs.caltech.edu:/usr/src/sys/compile/P6CLIENT_2CPU_128MB  i386

this system is -current as of the build date (April, 1998)

SMP on dual PPro 200 systems from 128MB to 1GB core.

>Description:

Reading a file over NFS at the same time as it is being written on the
same host causes it to be extended(?) with nulls.  It is possible
that the application that is reading actually seeks past EOF.  The
writer is open in (fopen) "wb" mode, the writer in "rb".  The reader
need *not* have permissions to modify the file (I even did it as a
separate user).

This fails with NFSv3 mounts only (as far as I can tell).  Tested against
server implementations:

NetBSD-1.2/i386
SunOS 5.5/sparc
FreeBSD-3.0/CURRENT-SMP/i386

Seems not to fail against NFSv2 servers
Tested against:
 
SunOS 4.1.4/sparc

>How-To-Repeat:

read a file when someone else is writing it, perhaps try seeking past
EOF when you do.  The file will be ruined (and maybe your day, like 
mine has been innumerable times because of this bug).

Oh, I have a setup that is completely reproducible, if anyone wants it.

>Fix:
Workaround: use NFSv2
Other workaround: use local files

Real Fix:  Unknown	

>Release-Note:
>Audit-Trail:

From: Peter Hawkins <thepish@FreeBSD.ORG>
To: Mika Nystrom <mika@cs.caltech.edu>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/7596: serious data integrity problem when reading WHILE writing NFSv3 client-end
Date: Thu, 13 Aug 1998 20:58:48 +1000 (EST)

 See also PR 7418 - the plot thickens...
 

From: Mika Nystrom <mika@cs.caltech.edu>
To: freebsd-gnats-submit@freebsd.org, mika@cs.caltech.edu
Cc: freebsd-bugs@freebsd.org, thepish@freebsd.org, dillon@best.net,
        freebsd-hackers@freebsd.org, freebsd-current@freebsd.org,
        rajit@dogmatix.cs.caltech.edu
Subject: Re: kern/7596: serious data integrity problem when reading WHILE writing NFSv3 client-end
Date: Thu, 13 Aug 1998 17:45:31 -0700

 All right,
    I have tracked down the cause of the nulls.  I wrote a little program
 to find the sizes of the nulled sections of the files I was writing:
 
 #include <stdio.h>
 main()
 {
   FILE *fp;
   int o,state,when;
   char c;
   state=1;
   o=0;
   when=0;
   fp=fopen("x","rb");
 
   /* read in bytes */
   
   printf("Reading non-zero...");
   while(fread(&c,sizeof(char),1,fp)) {
     if(state==1 && c==0) {
        state^=1; 
        printf("%#x bytes.\nGot zero at %#x, reading...",o-when,o);
        when=o;
     }
     if(state==0 && c!=0) {
        state^=1; 
        printf("%#x bytes.\nGot non-zero at %#x, reading...",o-when,o);
        when=o;
     }
     ++o;
   }
   printf("%#x bytes.\n",o-when);
 }
 
 and ran this on a file written with the following program:
 
 #include <math.h>
 #include <string.h>
 #include <stdlib.h>
 #include <stdio.h>
 main()
 {
    FILE *fp;
    int i=0;
    float d=3.01111;
    fp=fopen("x","w");
    fwrite(&d,sizeof(float),1,fp); 
    while (1) {
       int j;
       int stop;
       int howmany;
       howmany=random()%80+1;
       while(i++%howmany) { fwrite(&d,sizeof(float),1,fp); }
       
       /* delay a bit */
 #if 0
       stop=random()%20000;
 #endif
       stop=50000;
       if (random()%33) fflush(fp);
       for (j=0; j<stop; j++) { volatile double e; e=10.1; e=sin(e); }
    }
 }
 
 while reading repeatedly with the following:
 
 #include <string.h>
 #include <stdio.h>
 
 main()
 {
 
 #define LEN sizeof(float)
    FILE *fp;
    char data[LEN];
 
    int len;
    fp=fopen("x","r");
 
    while(
      fread(data,LEN,1,fp)
    );
 }
 
 As you can see from write.c, the file should contain NO NULLS.
 If you do this on a FreeBSD-current (or probably 2.2) system
 that is set up as an NFSv3 client (the server doesn't matter),
 you get nulls (or at least I do).
 
 I tracked it down to the following:
 
 sys/nfs/nfs_bio.c: line 1114 and following
 
                     if (uiop->uio_resid) {
                         /*
                          * If len > 0, there is a hole in the file and
                          * no writes after the hole have been pushed to
                          * the server yet.
                          * Just zero fill the rest of the valid area.
                          */      
 
                         diff = bp->b_bcount - uiop->uio_resid;
                         len = np->n_size - (((u_quad_t)bp->b_blkno) * DEV_BSIZE 
                                 + diff);
                         if (len > 0) {
                             len = min(len, uiop->uio_resid);
                             bzero((char *)bp->b_data + diff, len);
                             bp->b_validend = diff + len;
                         } else      
                             bp->b_validend = diff;
                     } else
                         bp->b_validend = bp->b_bcount;
 
 Note the bzero!  I inserted a statement in the kernel to print whenever
 it was called and with what "len" argument:
 
 Aug 13 16:46:09 dogmatix /kernel.nfshack: bzeroing 0xe0 bytes in nfs_bio.c
 Aug 13 16:46:09 dogmatix last message repeated 2 times 
 Aug 13 16:46:09 dogmatix /kernel.nfshack: bzeroing 0x8 bytes in nfs_bio.c 
 Aug 13 16:46:09 dogmatix last message repeated 2 times
 Aug 13 16:46:09 dogmatix /kernel.nfshack: bzeroing 0x4 bytes in nfs_bio.c 
 Aug 13 16:46:09 dogmatix last message repeated 2 times
 Aug 13 16:46:09 dogmatix /kernel.nfshack: bzeroing 0x24 bytes in nfs_bio.c
 Aug 13 16:46:09 dogmatix last message repeated 3 times
 Aug 13 16:46:09 dogmatix /kernel.nfshack: bzeroing 0x14 bytes in nfs_bio.c
 Aug 13 16:46:09 dogmatix last message repeated 3 times
 Aug 13 16:46:09 dogmatix /kernel.nfshack: bzeroing 0x2c bytes in nfs_bio.c
 Aug 13 16:46:10 dogmatix last message repeated 8 times
 Aug 13 16:46:11 dogmatix /kernel.nfshack: bzeroing 0x64 bytes in nfs_bio.c
 Aug 13 16:46:11 dogmatix last message repeated 12 times
 Aug 13 16:46:11 dogmatix /kernel.nfshack: bzeroing 0x7c bytes in nfs_bio.c
 Aug 13 16:46:11 dogmatix last message repeated 12 times
 Aug 13 16:46:11 dogmatix /kernel.nfshack: bzeroing 0x44 bytes in nfs_bio.c
 Aug 13 16:46:11 dogmatix last message repeated 13 times
 Aug 13 16:46:12 dogmatix /kernel.nfshack: bzeroing 0x60 bytes in nfs_bio.c
 
 When I ran the null-counter on the file produced, it told me I had a 
 section of valid data, followed by 0xe0 nulls, followed by valid data,
 followed by 0x8 bytes of nulls, etc.  Clearly, the bzero is zeroing
 valid, uncommitted data.
 
 Hmm it appears this bug might occur with NFSv2 also, and not just v3 as
 I thought.  Comments?
 
 My fix consists of either:
 
 just getting rid of the bzero.  I am concerned that this might let 
 reads return garbage at the end of the buffer, though, so what I am
 doing on our systems is:
 
 getting rid of the whole if (len > 0) section
 
      36  *      @(#)nfs_bio.c   8.9 (Berkeley) 3/30/95
      37  * $Id: nfs_bio.c,v 1.54 1998/03/28 16:05:05 steve Exp $
 [...]
    1123 #if 0
    1124                         if (len > 0) {
    1125                             static char x[]="with nfsholes_hack";
    1126                             printf("bzeroing %#x bytes in nfs_bio.c\n",l
 en);
    1127                             len = min(len, uiop->uio_resid);
    1128                             bzero((char *)bp->b_data + diff, len);
    1129                             bp->b_validend = diff + len;
    1130                         } else
    1131 #endif              
    1132                             bp->b_validend = diff;
 
 In other words, if I understand the code correctly, the read is short.
 However, that really shouldn't matter since the file will be brought
 up-to-date eventually, and this is NFS, so that is good enough...
 
 Could someone who understands the purpose of the bzero and/or how this
 NFS stuff works comment?  (And commit a fix for this ASAP because this
 is a real serious data-threatening bug?)
 
 Thanks to David Holland (dholland@eecs.harvard.edu) who helped me
 debug this.
 
     Mika
     <mika@cs.caltech.edu>
 
 
 

From: Mika Nystroem <mika@cs.caltech.edu>
To: freebsd-gnats-submit@freebsd.org, mika@cs.caltech.edu
Cc:  Subject: Re: kern/7596: serious data integrity problem when reading WHILE writing NFSv3 client-end
Date: Mon, 17 Aug 1998 03:15:12 -0700

 Update:
   Also strikes FreeBSD 3.0-CURRENT/i386 (as of April 1998) on
 a single-processor system.
 
    Mika

From: Mika Nystroem <mika@cs.caltech.edu>
To: freebsd-gnats-submit@freebsd.org, mika@cs.caltech.edu
Cc:  Subject: Re: kern/7596: serious data integrity problem when reading WHILE writing NFSv3 client-end
Date: Sun, 30 Aug 1998 21:12:52 -0700

 Another update:
   The problem as described appears to persist even when the programs
 are run on the latest -CURRENT (as of 8/30/98).
 
 
    Mika

From: "David O'Brien" <motminh@NUXI.com>
To: freebsd-gnats-submit@freebsd.org, mika@cs.caltech.edu
Cc:  Subject: Re: kern/7596: serious data integrity problem when reading WHILE writing NFSv3 client-end
Date: Thu, 19 Nov 1998 22:05:06 -0800

 Hi Mika,
 
 Can you please check to see if this bug has been fixed?
 PR 7418 was closely related.  The bug for that PR was fixed
 by dillon@freebsd.org in rev 1.178 of "src/sys/kern/vfs_bio.c".
 
 It may apply to your bug too.
 
 -- David	(obrien@FreeBSD.org)
State-Changed-From-To: open->closed 
State-Changed-By: dillon 
State-Changed-When: Sat Dec 12 16:58:53 PST 1998 
State-Changed-Why:  
I believe this bug is the same as kern/7418 and should now be fixed.  The 
bug was related to DELWRI being set in a bp with RELBUF.  We can reopen 
the PR if you are still seeing the problem.  There were actually two 
serious bugs fixed in this general area. 
>Unformatted:
