From spe@neo.gen.oleane.fr Wed Sep  1 03:06:44 1999
Return-Path: <spe@neo.gen.oleane.fr>
Received: from neo.gen.oleane.fr (neo.gen.oleane.fr [194.2.203.96])
	by hub.freebsd.org (Postfix) with ESMTP id 199F814D1B
	for <FreeBSD-gnats-submit@freebsd.org>; Wed,  1 Sep 1999 03:06:38 -0700 (PDT)
	(envelope-from spe@neo.gen.oleane.fr)
Received: (from root@localhost)
	by neo.gen.oleane.fr (8.9.3/8.9.3) id MAA00963;
	Wed, 1 Sep 1999 12:02:08 +0200 (CEST)
	(envelope-from spe)
Message-Id: <199909011002.MAA00963@neo.gen.oleane.fr>
Date: Wed, 1 Sep 1999 12:02:08 +0200 (CEST)
From: Sebastien Petit <spe@neo.gen.oleane.fr>
Reply-To: spe@neo.gen.oleane.fr
To: FreeBSD-gnats-submit@freebsd.org
Subject: <panic: getnewbuf: cannot get buffer, infinite recursion failure>
X-Send-Pr-Version: 3.2

>Number:         13517
>Category:       kern
>Synopsis:       <panic: getnewbuf: cannot get buffer, infinite recursion failure>
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Sep  1 03:10:01 PDT 1999
>Closed-Date:    Wed Jun 6 18:05:55 PDT 2001
>Last-Modified:  Wed Jun 06 18:07:01 PDT 2001
>Originator:     Sebastien Petit
>Release:        FreeBSD 3.2-STABLE i386
>Organization:
<Oleane>
>Environment:

FreeBSD ldap.gen.oleane.fr 3.2-19990830-STABLE FreeBSD 3.2-19990830-STABLE #1: Wed Sep  1 11:00:32 CEST 1999     spe@ldap.gen.oleane.fr:/usr/src/sys-3.2-19990830-STABLE/compile/RADLDAP  i386

>Description:



      The system panic when a program do multiples access on nfs mounted directory.
      ie: a program that fork 120 times and do "open" "seek" "write" "close" on this "NFS directory"
      this program write file of 20 Mo
      but after 2 seconds FreeBSD panic with panic: getnewbuf: cannot get buffer, infinite recursion failure with simple user access
      This program work fine on netbsd & linux.

      trace:
      GNU gdb 4.18
      Copyright 1998 Free Software Foundation, Inc.
      GDB is free software, covered by the GNU General Public License, and you
      are
      welcome to change it and/or distribute copies of it under certain
      conditions.
      Type "show copying" to see the conditions.
      There is absolutely no warranty for GDB.  Type "show warranty" for
      details.
      This GDB was configured as "i386-unknown-freebsd".
      (kgdb) symbol-file /usr/src/sys/compile/RADLDAP/kernel.debug
      Reading symbols from /usr/src/sys/compile/RADLDAP/kernel.debug...done.
      (kgdb) exec-file /var/crash/kernel.5
      (kgdb) core-file /var/crash/vmcore.5
      IdlePTD 2723840
      initial pcb at 22eb74
      panicstr: getnewbuf: cannot get buffer, infinite recursion failure
      panic messages:
      ---
      panic: getnewbuf: cannot get buffer, infinite recursion failure

      syncing disks... panic: getnewbuf: cannot get buffer, infinite recursion
      failure

      dumping to dev 20001, offset 272816
      dump 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111
      110 109
       108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88
      87 86 8
      5 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62
      61 60 59
      58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35
      34 33 32
       31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7
      6 5 4 3
       2 1
      ---
      #0  boot (howto=260) at ../../kern/kern_shutdown.c:285
      285                     dumppcb.pcb_cr3 = rcr3();
      (kgdb) where
      #0  boot (howto=260) at ../../kern/kern_shutdown.c:285
      #1  0xc012f87c in at_shutdown (
          function=0xc02075f1
      <__set_sysctl__vfs_sym_sysctl___vfs_kvafreespace+361>,
          arg=0x200, queue=1174437888) at ../../kern/kern_shutdown.c:446
      #2  0xc014dc9f in getnewbuf (vp=0xcc1edb40, blkno=2293824, slpflag=0,
          slptimeo=0, size=8192, maxsize=8192) at ../../kern/vfs_bio.c:1074
      #3  0xc014e58c in getblk (vp=0xcc1edb40, blkno=2293824, size=8192,
      slpflag=0,
          slptimeo=0) at ../../kern/vfs_bio.c:1511
      #4  0xc014cd85 in bread (vp=0xcc1edb40, blkno=2293824, size=8192,
      cred=0x0,
          bpp=0xcc3acbec) at ../../kern/vfs_bio.c:282
      #5  0xc01b60f8 in ffs_update (vp=0xcc21ea40, waitfor=0)
          at ../../ufs/ffs/ffs_inode.c:98
      #6  0xc01ba92f in ffs_fsync (ap=0xcc3acc74) at
      ../../ufs/ffs/ffs_vnops.c:258
      #7  0xc01b8cb7 in ffs_sync (mp=0xc1d01c00, waitfor=2, cred=0xc0756300,
          p=0xc0246624) at vnode_if.h:499
      #8  0xc0155f37 in sync (p=0xc0246624, uap=0x0) at
      ../../kern/vfs_syscalls.c:549
      #9  0xc012f43d in boot (howto=256) at ../../kern/kern_shutdown.c:203
      #10 0xc012f87c in at_shutdown (
          function=0xc02075f1
      <__set_sysctl__vfs_sym_sysctl___vfs_kvafreespace+361>,
          arg=0x2000, queue=12443648) at ../../kern/kern_shutdown.c:446
      #11 0xc014dc9f in getnewbuf (vp=0xcc243280, blkno=1519, slpflag=0,
      slptimeo=0,
          size=8192, maxsize=8192) at ../../kern/vfs_bio.c:1074
      #12 0xc014e58c in getblk (vp=0xcc243280, blkno=1519, size=8192,
      slpflag=0,
          slptimeo=0) at ../../kern/vfs_bio.c:1511
      #13 0xc017b9fa in nfs_getcacheblk (vp=0xcc243280, bn=1519, size=8192,
          p=0xcc36f700) at ../../nfs/nfs_bio.c:904
      #14 0xc017b5a5 in nfs_write (ap=0xcc3acec8) at ../../nfs/nfs_bio.c:765
      #15 0xc0159dea in vn_write (fp=0xc1d40d40, uio=0xcc3acf10,
      cred=0xc1d36300,
          flags=0) at vnode_if.h:331
      #16 0xc013a73a in dofilewrite (p=0xcc36f700, fp=0xc1d40d40, fd=3,
          buf=0x804b000, nbyte=102400, offset=-1, flags=0)
          at ../../kern/sys_generic.c:363
      #17 0xc013a643 in write (p=0xcc36f700, uap=0xcc3acf94)
          at ../../kern/sys_generic.c:298
      #18 0xc01e6edb in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi = 0,
            tf_esi = 12384951, tf_ebp = -1077945584, tf_isp = -868560924,
            tf_ebx = 12384951, tf_edx = 0, tf_ecx = 12384951, tf_eax = 4,
            tf_trapno = 7, tf_err = 2, tf_eip = 671700396, tf_cs = 31,
            tf_eflags = 582, tf_esp = -1077946192, tf_ss = 39})
          at ../../i386/i386/trap.c:1100
      #19 0xc01dda5c in Xint0x80_syscall ()
      #20 0x8048799 in ?? ()

      Another probleme is that FreeBSD is very slow when this program is launch.
      NetBSD and linux doesn't slow down and complete the bench.

>How-To-Repeat:

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <stdio.h>
#include <sys/time.h>
#include <time.h>

void usr1() {
}

int main(int argc, char ** argv) {
  int nbfils;
  int nbopen;
  int tbloc;
  int tfichier;
  char filename[512];
  int i, j, k, f;
  int pid;
  struct timeval start;
  struct timeval end;
  float delay;
  void * bloc;
  
  if (argc<6) {
    fprintf(stderr, "usage: %s prefix nb_fils nb_open tfichier(Kb) tbloc(kb).\n"
, argv[0]);
    exit(EXIT_FAILURE);
  }

  nbfils = atoi(argv[2]);
  nbopen = atoi(argv[3]);
  tfichier = atoi(argv[4]);
  tbloc = atoi(argv[5]);

  bloc = malloc(tbloc * 1024);
  memset(bloc, 0, tbloc * 1024);
  if (!bloc) {
    fprintf(stderr, "%s: ", argv[0]);
    perror("malloc");
    exit(-1);
  }
  
  fprintf(stderr, "forking %d times...\n", nbfils);

  signal(SIGUSR1, &usr1);

  j = 0;
  for(i=0;i<nbfils;i++) {
    pid = fork();
    if (pid<0) {
      perror("fork");
      break;
    } else
      j++;
    if (!pid) break;
  }
  

  if (!pid) {
    pause();
    pid = getpid();
    srand(pid*10);
    fprintf(stderr, "[%d] child %d: Here I go!\n", pid, i);
    sprintf(filename, "%s%d", argv[1], pid);
    for(i=0;i<nbopen;i++) {
      f = open(filename, O_CREAT|O_RDWR, 0666);
      if (f<0) {
        fprintf(stderr, "[%d] file %s ", pid, filename);
        perror("open");
        break;
      }
      k = (rand() % (tfichier * 1024));
      j = lseek(f, k, SEEK_SET);
      if (j!=k) {
        fprintf(stderr, "[%d] ", pid);
        perror("lseek");
        break;
      }
      // read(f, bloc, tbloc*1024);
      if (write(f, bloc, tbloc*1024)!=tbloc*1024) {
        fprintf(stderr, "[%d] ", pid);
        perror("write");
        break;
      }
      if (close(f)) {
        fprintf(stderr, "[%d] ", pid);
        perror("close");
        break;
      }
    }
    exit(0);
  }

  fprintf(stderr, "%d children process created.\nstarting in about 2 sec...\n", 
j);
  sleep(2);
  gettimeofday(&start, NULL);
  kill(0, SIGUSR1);
  
  i = 0;
  while (i<nbfils)
    if (waitpid(-1, NULL, 0)>0)
      i++;
  
  fprintf(stderr, "they're all dead now, exiting.\n");
  gettimeofday(&end, NULL);
  delay = end.tv_sec - start.tv_sec + ((float) (end.tv_usec - start.tv_usec))
    / (float) 1000000;
  
  i = nbopen * tbloc * nbfils;
  fprintf(stderr,
          "\n"
          "total runtime = %g sec,\n"
          "%d io-syscalls => %g io-syscalls/sec,\n"
          "%d Kb written => %g Kb/sec.\n",
          delay, (5*nbopen*nbfils), (float) 5*nbopen*nbfils / delay,
          i, (float) i / delay);
          
  exit(0);
}

Example:
root@ldap:~> mount 194.2.203.100:/nfs /TEST

root@ldap:~> mount
/dev/wd0s1a on / (local, writes: sync 14 async 3095)
/dev/wd0s1f on /usr (local, writes: sync 10010 async 13997)
/dev/wd0s1e on /var (local, writes: sync 177 async 1319)
procfs on /proc (local)
194.2.203.100:/nfs on /TEST

root@ldap:~> ./nfsbench /TEST-nfs 120 100 20000 100

panic: getnewbuf: cannot get buffer, infinite recursion failure

>Fix:
	
don't know how to fix the problem !



>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: gnats-admin->freebsd-bugs 
Responsible-Changed-By: steve 
Responsible-Changed-When: Sun Sep 12 14:29:16 PDT 1999 
Responsible-Changed-Why:  
Misfiled PR. 
State-Changed-From-To: open->feedback 
State-Changed-By: iedowse 
State-Changed-When: Wed Jun 6 17:38:17 PDT 2001 
State-Changed-Why:  

Can you reproduce this on more recent versions of FreeBSD? 

http://www.FreeBSD.org/cgi/query-pr.cgi?pr=13517 
State-Changed-From-To: feedback->closed 
State-Changed-By: iedowse 
State-Changed-When: Wed Jun 6 18:05:55 PDT 2001 
State-Changed-Why:  

Mail to submitter bounces. 

http://www.FreeBSD.org/cgi/query-pr.cgi?pr=13517 
>Unformatted:
