From root@foo.felvi.hu  Thu Dec  4 03:46:59 2003
Return-Path: <root@foo.felvi.hu>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id B70A816A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Thu,  4 Dec 2003 03:46:59 -0800 (PST)
Received: from foo.felvi.hu (foo.felvi.hu [193.6.241.7])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 62F0143FDD
	for <FreeBSD-gnats-submit@freebsd.org>; Thu,  4 Dec 2003 03:46:57 -0800 (PST)
	(envelope-from root@foo.felvi.hu)
Received: from foo.felvi.hu (localhost [127.0.0.1])
	by foo.felvi.hu (8.12.9p2/8.12.9) with ESMTP id hB4BkPlZ065925
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 4 Dec 2003 12:46:25 +0100 (CET)
	(envelope-from root@foo.felvi.hu)
Received: (from root@localhost)
	by foo.felvi.hu (8.12.9p2/8.12.9/Submit) id hB4BkKIY065922;
	Thu, 4 Dec 2003 12:46:20 +0100 (CET)
	(envelope-from root)
Message-Id: <200312041146.hB4BkKIY065922@foo.felvi.hu>
Date: Thu, 4 Dec 2003 12:46:20 +0100 (CET)
From: Admin <root@foo.felvi.hu>
Reply-To: Kovacs Janos <kovacs.janos@ofi.hu>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: nullfs bug: reboot after panic: null_checkvp
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         59945
>Category:       kern
>Synopsis:       [nullfs] [patch] nullfs bug: reboot after panic: null_checkvp [4.9]
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Dec 04 03:50:16 PST 2003
>Closed-Date:    Fri Jan 25 23:04:47 UTC 2008
>Last-Modified:  Fri Jan 25 23:04:47 UTC 2008
>Originator:     Admin
>Release:        FreeBSD 4.9-RELEASE i386
>Organization:
EDUCATIO
>Environment:
System: FreeBSD foo.felvi.hu 4.9-RELEASE FreeBSD 4.9-RELEASE #2: Fri Nov 14 10:52:35 CET 2003 root@foo.felvi.hu:/usr/obj/usr/src/sys/KAAMOS i386

Dmesg:
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.9-RELEASE #2: Fri Nov 14 10:52:35 CET 2003
    root@foo.felvi.hu:/usr/obj/usr/src/sys/KAAMOS
Timecounter "i8254"  frequency 1193182 Hz
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2793.90-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Hyperthreading: 2 logical CPUs
real memory  = 2147397632 (2097068K bytes)
avail memory = 2086674432 (2037768K bytes)
APIC_IO: MP table broken: 8259->APIC entry missing!
Programming 16 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
Programming 16 pins in IOAPIC #1
Programming 16 pins in IOAPIC #2
FreeBSD/SMP: Multiprocessor motherboard: 4 CPUs
 cpu0 (BSP): apic id:  0, version: 0x00050014, at 0xfee00000
 cpu1 (AP):  apic id:  1, version: 0x00050014, at 0xfee00000
 cpu2 (AP):  apic id:  6, version: 0x00050014, at 0xfee00000
 cpu3 (AP):  apic id:  7, version: 0x00050014, at 0xfee00000
 io0 (APIC): apic id: 14, version: 0x000f0011, at 0xfec00000
 io1 (APIC): apic id: 13, version: 0x000f0011, at 0xfec01000
 io2 (APIC): apic id: 12, version: 0x000f0011, at 0xfec02000
Preloaded elf kernel "kernel" at 0xc04a5000.
Warning: Pentium 4 CPU: PSE disabled
Pentium Pro MTRR support enabled
md0: Malloc disk
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
IOAPIC #1 intpin 10 -> irq 2
pci0: <PCI bus> on pcib0
pci0: <ATI Mach64-GR graphics accelerator> at 6.0 irq 2
atapci0: <ServerWorks CSB5 ATA100 controller> port 0x700-0x70f,0x374-0x377,0x170
-0x177,0x3f4-0x3f7,0x1f0-0x1f7 at device 15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pci0: <OHCI USB controller> at 15.2 irq 3
isab0: <PCI to ISA bridge (vendor=1166 device=0225)> at device 15.3 on pci0
isa0: <ISA bus> on isab0
pcib1: <Host to PCI bridge> on motherboard
pci1: <PCI bus> on pcib1
pcib2: <Host to PCI bridge> on motherboard
IOAPIC #1 intpin 4 -> irq 4
pci2: <PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.16> mem 0xfbfd0000-0xf
bfdffff,0xfbfe0000-0xfbffffff irq 4 at device 3.0 on pci2
em0:  Speed:1000 Mbps  Duplex:Full
pcib3: <Host to PCI bridge> on motherboard
pci3: <PCI bus> on pcib3
pcib4: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
IOAPIC #1 intpin 6 -> irq 5
pci4: <PCI bus> on pcib4
aac0: <Adaptec SCSI RAID 2120S> mem 0xf0000000-0xf7ffffff irq 5 at device 4.0 on
 pci4
aac0: i960RX 100MHz, 112MB cache memory, optional battery present
aac0: Kernel 4.0-0, Build 6008, S/N b8b31a
aac0: Supported Options=1f7e<CLUSTERS,WCACHE,DATA64,HOSTTIME,RAID50,WINDOW4GB,SO
FTERR,NORECOND,SGMAP64,ALARM,NONDASD>
pcib5: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
pci5: <PCI bus> on pcib5
pcib6: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
IOAPIC #1 intpin 13 -> irq 7
IOAPIC #1 intpin 14 -> irq 9
pci6: <PCI bus> on pcib6
em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.16> port 0x2500-0x253f
 mem 0xeefe0000-0xeeffffff irq 7 at device 8.0 on pci6
em1:  Speed:N/A  Duplex:N/A
em2: <Intel(R) PRO/1000 Network Connection, Version - 1.7.16> port 0x2540-0x257f
 mem 0xeefc0000-0xeefdffff irq 9 at device 8.1 on pci6
em2:  Speed:N/A  Duplex:N/A
pcib8: <ServerWorks host to PCI bridge(unknown chipset)> on motherboard
IOAPIC #1 intpin 11 -> irq 10
IOAPIC #1 intpin 12 -> irq 11
pci8: <PCI bus> on pcib8
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2600-0x26ff mem 0xecfe0000-0xecfefff
f,0xecff0000-0xecffffff irq 10 at device 7.0 on pci8
mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2700-0x27ff mem 0xecfc0000-0xecfcfff
f,0xecfd0000-0xecfdffff irq 11 at device 7.1 on pci8
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xc9fff,0xc
a000-0xcdfff on isa0
pmtimer0 on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250
sio1: configured irq 3 not in bitmap of probed irqs 0
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: routing 8254 via IOAPIC #0 intpin 2
IP packet filtering initialized, divert enabled, rule-based forwarding enabled,
default to accept, logging limited to 100 packets/entry by default
IPsec: Initialized Security Association Processing.
IP Filter: v3.4.31 initialized.  Default = block all, Logging = enabled
SMP: AP CPU #1 Launched!
acd0: CDROM <CD-224E> at ata0-master PIO4
Waiting 15 seconds for SCSI devices to settle
aacd0: <RAID 1 (Mirror)> on aac0
aacd0: 34712MB (71091456 sectors)
aacd1: <Volume> on aac0
aacd1: 34712MB (71091456 sectors)
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!


>Description:
	4.9 machine running various apps in null-mounted jails crash after various time:  reboot after panic: null_checkvp

	null-mounted jail: 
		jail root: /jail/app (/dev, /var, /tmp, /mnt; +symlinks {/bin,/etc,/usr}-> /mnt/$1)
		read-only content: /jail/ro (/bin, /etc, /usr)
		at startup: mount_null -o ro,noatime /jail/ro /jail/app/mnt

	After starting apps (espec. dnscache) systat shows, that allocated numvnodes started to increment (sometimes 100/sec) and also freevnodes the same way (but freevnodes < numvnodes).
	When numvnodes reach 33695 the system hang up: 2-3 sec while numvnodes does not change, after that kernel panic msg appear.


The gdb output:

GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called a
t /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2
627 in elfstab_build_psymtabs
Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../cont
rib/gdb/gdb/dbxread.c line 933 in fill_symbuf

SMP 4 cpus
IdlePTD at phsyical address 0x004c4000
initial pcb at physical address 0x003fc5c0
panicstr: rslock: cpu: 0, addr: 0xc03fd034, lock: 0x00000001
panic messages:
---
panic: null_checkvp
mp_lock = 00000001; cpuid = 0; lapic.id = 00000000
boot() called on cpu#0
syncing disks... 16 2 1 1 1 1 1 1 1 panic: rslock: cpu: 0, addr: 0xc03fd034, loc
k: 0x00000001
mp_lock = 00000001; cpuid = 0; lapic.id = 00000000
boot() called on cpu#0
Uptime: 30m53s

dumping to dev #aacd/0x20009, offset 4194472
dump 2047 2046 2045 2044 2043 2042 2041 2040 2039 2038 2037 2036 2035 2034 2033
2032 2031 2030 2029 2028 2027 2026 2025 2024 2023 2022 2021 2020 2019 2018 2017
... truncated ...
 6 5 4 3 2 1 0 succeeded
 aac0: shutting down controller...
 ---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487             if (dumping++) {
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc01f930b in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc01f9764 in poweroff_wait (junk=0xc03318ce, howto=0)
    at /usr/src/sys/kern/kern_shutdown.c:595
#3  0xc03318ce in bsl1 ()
#4  0xc02285d8 in vdrop (vp=0xf6fd7c40) at /usr/src/sys/kern/vfs_subr.c:1692
#5  0xc0227b5a in brelvp (bp=0xd61f7b2c) at /usr/src/sys/kern/vfs_subr.c:1074
#6  0xc02204a8 in vfs_vmio_release (bp=0xd61f7b2c)
    at /usr/src/sys/kern/vfs_bio.c:1361
#7  0xc021fead in brelse (bp=0xd61f7b2c) at /usr/src/sys/kern/vfs_bio.c:1140
#8  0xc0227533 in vinvalbuf (vp=0xf6fd7c40, flags=0, cred=0x0, p=0xe84d2860,
    slpflag=0, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:872
#9  0xc02d7a8a in ffs_truncate (vp=0xf6fd7c40, length=0, flags=0, cred=0x0,
    p=0xe84d2860) at /usr/src/sys/ufs/ffs/ffs_inode.c:199
#10 0xc02e2b3c in ufs_inactive (ap=0xf7098b08)
    at /usr/src/sys/ufs/ufs/ufs_inode.c:89
#11 0xc02e8181 in ufs_vnoperate (ap=0xf7098b08)
    at /usr/src/sys/ufs/ufs/ufs_vnops.c:2376
#12 0xc02284ef in vput (vp=0xf6fd7c40) at vnode_if.h:815
#13 0xc02db6e4 in handle_workitem_remove (dirrem=0xc9cb3b40)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852
#14 0xc02d8d55 in process_worklist_item (matchmnt=0x0, flags=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:716
#15 0xc02d8bfe in softdep_process_worklist (matchmnt=0x0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:622
#16 0xc01f9141 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:261
#17 0xc01f9764 in poweroff_wait (junk=0xc0383662, howto=-150369236)
    at /usr/src/sys/kern/kern_shutdown.c:595
#18 0xc0230819 in null_checkvp (vp=0xf70815c0,
    fil=0xc0383780 "/usr/src/sys/miscfs/nullfs/null_vnops.c", lno=824)
    at /usr/src/sys/miscfs/nullfs/null_subr.c:319
#19 0xc0231728 in null_getvobject (ap=0xf7098c2c)
    at /usr/src/sys/miscfs/nullfs/null_vnops.c:824
#20 0xc0227032 in getnewvnode (tag=VT_NULL, mp=0xc9ceb600, vops=0xc9785500,
    vpp=0xf7098c98) at vnode_if.h:1421
#21 0xc0230617 in null_node_alloc (mp=0xc9ceb600, lowervp=0xf7081500,
    vpp=0xf7098c98) at /usr/src/sys/miscfs/nullfs/null_subr.c:172
#22 0xc0230776 in null_node_create (mp=0xc9ceb600, lowervp=0xf7081500,
    newvpp=0xf7098cc8) at /usr/src/sys/miscfs/nullfs/null_subr.c:256
#23 0xc02310f7 in null_lookup (ap=0xf7098d18)
    at /usr/src/sys/miscfs/nullfs/null_vnops.c:396
#24 0xc0226289 in lookup (ndp=0xf7098edc) at vnode_if.h:52
#25 0xc0225d84 in namei (ndp=0xf7098edc) at /usr/src/sys/kern/vfs_lookup.c:153
#26 0xc01f0596 in execve (p=0xe84d2860, uap=0xf7098f80)
    at /usr/src/sys/kern/kern_exec.c:165
#27 0xc033380d in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
      tf_edi = 0, tf_esi = 14, tf_ebp = -1077936992, tf_isp = -150368300,
      tf_ebx = -1077936512, tf_edx = -1077936760, tf_ecx = 134528480,
      tf_eax = 59, tf_trapno = 12, tf_err = 2, tf_eip = 403307776, tf_cs = 31,
      tf_eflags = 663, tf_esp = -1077937036, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1175
#28 0xc031e0db in Xint0x80_syscall ()
#29 0x8048d2b in ?? ()
#30 0x8048a66 in ?? ()
#31 0x80486b6 in ?? ()
(kgdb) q
	

	NULLFS presented in RELENG_4 years ago, but it still in beta: some env works fine for lots of people for years ago, but there lots of bugs - as man page describe it. When talking about a prod release, it must not contain not production stable parts. Instead replying "do not use" and "use nfs" please help to fix it.
(I help to test anything that need to test ;)
		
>How-To-Repeat:
	Just start up nullfs and daemons (tested: bind9, oops, djbdns's dnscache+daemontools).

	I dont know whether an user can do something nasty vnode allocation and halt the system.
	
>Fix:

	dont know. Do not use nullfs, even not for that apps but for any.


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: linimon 
State-Changed-When: Fri Jul 16 04:46:34 GMT 2004 
State-Changed-Why:  
Is this still a problem on 4.10? 


Responsible-Changed-From-To: freebsd-i386->freebsd-bugs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Jul 16 04:46:34 GMT 2004 
Responsible-Changed-Why:  
This problem is probably not i386-specific. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=59945 

From: Brian Buchanan <bwb@holo.org>
To: freebsd-gnats-submit@freebsd.org
Cc:  
Subject: Re: kern/59945: nullfs bug: reboot after panic: null_checkvp [4.9]
Date: Mon, 2 Aug 2004 20:21:22 -0700 (PDT)

 This is almost certainly still an issue in 4.10, however, it will not
 occur unless DIAGNOSTIC is defined.
 
 getnewvnode() attempts a VOP_GETVOBJECT() on a nullfs vnode.
 null_getvobject() uses the NULLVPTOLOWERVP() macro to get the lower vnode
 so that its vm_object can be returned.  However, for a free nullfs vnode,
 there is no lower vnode.  null_checkvp() will be invoked as a result of
 the macro, and it will panic when it finds that lowervp is not set.
 
 #ifdef DIAGNOSTIC
 struct vnode *null_checkvp(struct vnode *vp, char *fil, int lno);
 #define NULLVPTOLOWERVP(vp) null_checkvp((vp), __FILE__, __LINE__)
 #else
 #define NULLVPTOLOWERVP(vp) (VTONULL(vp)->null_lowervp)
 #endif
 
 null_getvobject() needs to test whether the vnode is still alive before it
 tries to descend into its data structures.  Or, alternately, the patch tjr
 was working on will fix this by delaying the cleanup of the null vode and
 its associated data structures until the reclaim operation.
 
 
State-Changed-From-To: feedback->open 
State-Changed-By: linimon 
State-Changed-When: Thu Aug 26 03:57:00 GMT 2004 
State-Changed-Why:  
Feedback received. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=59945 
State-Changed-From-To: open->feedback 
State-Changed-By: rwatson 
State-Changed-When: Fri Jan 25 18:16:01 UTC 2008 
State-Changed-Why:  
nullfs has been substantially rewritten and updated in the FreeBSD 6.x 
release series, fixing many bugs and solving many problems.  Are you able 
to confirm whether these rewrites have corrected the problems you were 
experiencing?  Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=59945 

From: Brian Buchanan <bwb@holo.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/59945: [nullfs] [patch] nullfs bug: reboot after panic: null_checkvp [4.9]
Date: Fri, 25 Jan 2008 11:18:37 -0800

 I have been running multiple nullfs mounts in multiple jails in  
 production under 6.2p6 (and previous 6.x versions) for more than a  
 year without a single crash.  I have not, however, attempted to  
 recreate the conditions needed to trigger this specific bug.
 

From: Brian Buchanan <bwb@holo.org>
To: Robert Watson <rwatson@FreeBSD.org>
Cc:  
Subject: Re: kern/59945: [nullfs] [patch] nullfs bug: reboot after panic:  null_checkvp [4.9] (fwd)
Date: Fri, 25 Jan 2008 11:16:58 -0800

 Wow, I barely even remember this.  I have a production system running  
 6.2p6 that has a number of nullfs mounts in various jails, and it's  
 never experienced a panic.  Unless you know of someone who's still  
 complaining about nullfs crashes, I think this can be pretty safely  
 closed out.
 
 -Brian
 
 On Jan 25, 2008, at 10:20 , Robert Watson wrote:
 
 >
 > You also appear in the PR transcript so I figured I'd forward you a  
 > copy -- not sure if you are interested or not.  If so, could you  
 > follow up on the PR?
 >

From: Robert Watson <rwatson@FreeBSD.org>
To: Brian Buchanan <bwb@holo.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/59945: [nullfs] [patch] nullfs bug: reboot after panic: 
 null_checkvp [4.9] (fwd)
Date: Fri, 25 Jan 2008 20:24:36 +0000 (GMT)

 On Fri, 25 Jan 2008, Brian Buchanan wrote:
 
 > Wow, I barely even remember this.  I have a production system running 6.2p6 
 > that has a number of nullfs mounts in various jails, and it's never 
 > experienced a panic.  Unless you know of someone who's still complaining 
 > about nullfs crashes, I think this can be pretty safely closed out.
 
 Nope -- I don't know of any.  I'll go ahead and close it, thanks!
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge
State-Changed-From-To: feedback->closed 
State-Changed-By: linimon 
State-Changed-When: Fri Jan 25 23:03:54 UTC 2008 
State-Changed-Why:  
Problem is believed to be fixed in recent versions of FreeBSD. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=59945 
>Unformatted:
