From nobody@FreeBSD.org  Wed Mar 21 14:05:12 2001
Return-Path: <nobody@FreeBSD.org>
Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21])
	by hub.freebsd.org (Postfix) with ESMTP id 8119B37B719
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 21 Mar 2001 14:05:11 -0800 (PST)
	(envelope-from nobody@FreeBSD.org)
Received: (from nobody@localhost)
	by freefall.freebsd.org (8.11.1/8.11.1) id f2LM5BJ27956;
	Wed, 21 Mar 2001 14:05:11 -0800 (PST)
	(envelope-from nobody)
Message-Id: <200103212205.f2LM5BJ27956@freefall.freebsd.org>
Date: Wed, 21 Mar 2001 14:05:11 -0800 (PST)
From: jimz@panasas.com
To: freebsd-gnats-submit@FreeBSD.org
Subject: Holes in files do not zero-fill
X-Send-Pr-Version: www-1.0

>Number:         25974
>Category:       kern
>Synopsis:       Holes in files do not zero-fill
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Mar 21 14:10:00 PST 2001
>Closed-Date:    Sat Jun 2 02:45:36 PDT 2001
>Last-Modified:  Mon Jun  4 14:40:04 PDT 2001
>Originator:     Jim Zelenka
>Release:        4.1.1
>Organization:
Panasas, Inc.
>Environment:
FreeBSD natasha.panasas.com 4.1.1-RELEASE FreeBSD 4.1.1-RELEASE #0: Thu Mar  8 14:41:32 EST 2001     jimz@natasha.panasas.com:/usr/src/sys/compile/NATASHA-HZ-DBG  i386

>Description:
If a process opens a file, writes some bytes, seeks past the end of
the file, writes some bytes, then reads the intermediate (hole) area,
it does not see zeroes in the hole. The attached program does just this.
It initializes the buffer that the read goes into to contain all 0x7f,
and then reads into it. Sometimes the buffer (correctly) contains
zeroes in the hole, sometimes some bytes contain 0x7f, suggesting that
it did not write into the buffer, and sometimes the buffer contains
"other" bytes- it is not clear to me if these are uninitialized bytes
from an in-core cache block or uninitialized bytes on the disk.


>How-To-Repeat:
Here is a simple program one of our engineers wrote to reproduce this case:

/*
 * freebsd-zerofill.c 
 * 
 * FreeBSD 4.1.1 seems to have a bug where seeking past the end of a file and then writing
 * the data when read back in, may or may not be zero-filled.
 *
 * @author Edward Hogan
 * @version 1.0
 *
 */
/*
 * @PANASAS_COPYRIGHT@
 */


#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/param.h>
#include <fcntl.h>

#define TOTAL_BUF_LEN (72*1024)
#define BUF1_LEN      (63*1024)
#define HOLE_OFFSET   BUF1_LEN
#define HOLE_LEN      (1025)
#define BUF2_OFFSET   (BUF1_LEN+HOLE_LEN)
#define BUF2_LEN      (1023)
#define READ1_LEN     (BUF2_OFFSET+BUF2_LEN)
#define BUF3_OFFSET   (BUF2_OFFSET+BUF2_LEN)
#define BUF3_LEN      (TOTAL_BUF_LEN-BUF3_OFFSET)

#define WRITE1_LEN    (32*1024)

static char *fname = "freebsd-zerofill.xyz";

int main(){
  char *orig_buf, *target_buf;
  int test_num = 0;
  int fd = -1, ret;
  int i, err_count;
  int bytes_written, bytes_read, new_off;
  
  orig_buf = malloc(TOTAL_BUF_LEN);
  if (orig_buf == NULL) {
    printf("alloc buffer failed\n");
    goto test_failed;
  }
  target_buf = malloc(2*TOTAL_BUF_LEN);
  if (target_buf == NULL) {
    printf("alloc buffer failed\n");
    goto test_failed;
  }
  /* fill orig_buf with the alphabet */
  for(i = 0; i < TOTAL_BUF_LEN; i++) {
    orig_buf[i]   = (char) ('a' + (char)(i % 26));
    target_buf[i] = (char) (0x7f);
  }
  /* fill target_buf completely with 0x7f characters */
  for(i = TOTAL_BUF_LEN; i < (2 * TOTAL_BUF_LEN); i++) {
    target_buf[i] = (char) (0x7f);
  }
  /* */
  fd = open(fname,( O_RDWR | O_CREAT | O_TRUNC ), 0666);
  if (fd < 0) {
    printf("failed opening and creating file\n");
    goto test_failed;
  }
  else {
    printf("file opened               [SUCCESS]\n");
  }
  /* write to the test file */
  bytes_written = write(fd, orig_buf, WRITE1_LEN);
  if (bytes_written != WRITE1_LEN) {
    printf("error writing 1\n");
    goto test_failed;
  }
  else {
    printf("data written 1            [SUCCESS]\n");
  }
  /* write again to the test file */
  bytes_written = write(fd, orig_buf+WRITE1_LEN, BUF1_LEN-WRITE1_LEN);
  if (bytes_written != BUF1_LEN-WRITE1_LEN) {
    printf("write 2 failed\n");
    goto test_failed;
  }
  else {
    printf("data written 2            [SUCCESS]\n");
  }
  /* seek forward in file */
  new_off = lseek(fd, HOLE_LEN, SEEK_CUR);
  if (new_off != BUF2_OFFSET) { 
    printf("seek failed\n");
    goto test_failed;
  }
  else {
    printf("seek past end of file     [SUCCESS]\n");
  }
  /* write at seek point */
  bytes_written = write(fd, orig_buf + BUF2_OFFSET, BUF2_LEN);
  if (bytes_written != BUF2_LEN) {
    printf("write 3 failed\n");
    goto test_failed;
  }
  else {
    printf("data written 3            [SUCCESS]\n");
  }
  /* seek to beginning of file */
  new_off = lseek(fd, 0, SEEK_SET);
  if (new_off != 0) {
    printf("seek to start failed\n");
    goto test_failed;
  }
  else {
    printf("seek to start             [SUCCESS]\n");
  }
  /* read from the beginning */
  bytes_read = read(fd, target_buf, READ1_LEN);
  if (bytes_read != READ1_LEN) {
    printf("read failed\n");
  }
  else {
    printf("read entire file          [SUCCESS]\n");
  }
  /* **************************************************************** */
  /* examine result of read */
  if (memcmp(orig_buf, target_buf, BUF1_LEN) != 0) {
    printf("data before seek is bad   [FAILED]\n");
  }
  else {
    printf("data before seek is good  [SUCCESS]\n");
  }
  err_count = 0;
  for(i = BUF1_LEN; i < BUF2_OFFSET; i++) {
    if (target_buf[i]) { 
      printf("target_buf is dirty (char[%d] = (0x%x))\n", i, target_buf[i]);
      err_count++;
      if (err_count > 10){
        printf("target_buf is dirty       [FAILED]\n");
        break;
      }
    }
  }
  if (err_count == 0){
    printf("target_buf is clean       [SUCCESS]\n");
  }
  if (memcmp(orig_buf + BUF2_OFFSET, target_buf + BUF2_OFFSET, BUF2_LEN)) {
    printf("data after seek is bad   [FAILED]\n");
  }
  else {
    printf("data after seek is good   [SUCCESS]\n");
  }
  close(fd);
  return 0;
 test_failed:
  printf("!!! THE TEST DID NOT RUN !!!\n");
  if (fd != -1) {
    close(fd);
  }
  return 1;
}

>Fix:

>Release-Note:
>Audit-Trail:

From: Peter Pentchev <roam@orbitel.bg>
To: jimz@panasas.com
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/25974: Holes in files do not zero-fill
Date: Thu, 22 Mar 2001 15:06:54 +0200

 On Wed, Mar 21, 2001 at 02:05:11PM -0800, jimz@panasas.com wrote:
 > 
 > >Number:         25974
 > >Category:       kern
 > >Synopsis:       Holes in files do not zero-fill
 > >Originator:     Jim Zelenka
 > >Release:        4.1.1
 > >Organization:
 > Panasas, Inc.
 > >Environment:
 > FreeBSD natasha.panasas.com 4.1.1-RELEASE FreeBSD 4.1.1-RELEASE #0: Thu Mar  8 14:41:32 EST 2001     jimz@natasha.panasas.com:/usr/src/sys/compile/NATASHA-HZ-DBG  i386
 > 
 > >Description:
 > If a process opens a file, writes some bytes, seeks past the end of
 > the file, writes some bytes, then reads the intermediate (hole) area,
 > it does not see zeroes in the hole. The attached program does just this.
 > It initializes the buffer that the read goes into to contain all 0x7f,
 > and then reads into it. Sometimes the buffer (correctly) contains
 > zeroes in the hole, sometimes some bytes contain 0x7f, suggesting that
 > it did not write into the buffer, and sometimes the buffer contains
 > "other" bytes- it is not clear to me if these are uninitialized bytes
 > from an in-core cache block or uninitialized bytes on the disk.
 
 Can you try updating your system to -STABLE (which is currently 4.3-RC),
 and see if the problem persists?  On my 4.3-RC machine, your 'zerofill'
 program outputs nothing but SUCCESS lines.
 
 Information on updating your system can be found at:
 http://www.FreeBSD.org/handbook/synching.html
 
 G'luck,
 Peter
 
 -- 
 because I didn't think of a good beginning of it.
State-Changed-From-To: open->feedback 
State-Changed-By: roam 
State-Changed-When: Thu Mar 22 05:14:47 PST 2001 
State-Changed-Why:  
Awaiting response from submitter on whether upgrading fixes the problem. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=25974 

From: Jim Zelenka <jimz@panasas.com>
To: Peter Pentchev <roam@orbitel.bg>
Cc: jimz@panasas.com, freebsd-gnats-submit@FreeBSD.org,
	nlanza@panasas.com
Subject: Re: kern/25974: Holes in files do not zero-fill 
Date: Fri, 23 Mar 2001 16:45:24 -0500

 Oops! I've been informed that the machine on which I did not see the zero-fill
 problem is not a SCSI machine; it too has IDE disks. In fact, the machines
 are identical.
 
 -jim
 
 Jim Zelenka
 Senior Software Engineer, Panasas, Inc.
 Pioneering Object-Based Network Storage (ONS)
 
 www.panasas.com
 jimz@panasas.com
 412-323-3500
State-Changed-From-To: feedback->closed 
State-Changed-By: kris 
State-Changed-When: Sat Jun 2 02:45:36 PDT 2001 
State-Changed-Why:  
No response from submitter, and this problem appears to be 
no longer present. 

http://www.FreeBSD.org/cgi/query-pr.cgi?pr=25974 

From: Kris Kennaway <kris@obsecurity.org>
To: Jim Zelenka <jimz@panasas.com>
Cc: Kris Kennaway <kris@obsecurity.org>,
	freebsd-gnats-submit@freebsd.org
Subject: Re: kern/25974: Holes in files do not zero-fill
Date: Mon, 4 Jun 2001 14:36:27 -0700

 On Sat, Jun 02, 2001 at 04:45:08PM -0400, Jim Zelenka wrote:
 > >For some reason this response did not appear in GNATS.  I asked Dima
 > >to re-forward it.
 > 
 > That's probably my fault; I sent two responses. In the first, I said
 > that I'd checked briefly, and I didn't see the problem. In the
 > second one, I said that more extensive testing made the problem
 > appear. The problem is particularly likely to occur on busy systems
 > or systems where lots of files were created and deleted between
 > runs of the test program. I suspect that the problem is that something
 > is not zeroing out a block that is getting reallocated, and so
 > old bits are coming back, but that's only based on empirical observation,
 > and not a reading of the code.
 
 We did fix a bug with this kind of symptoms a while back (and released
 an advisory; perhaps that is related?)
 
 Kris
>Unformatted:
