From bmeyer@helios.mesoft.com.au  Mon May  2 23:38:49 2011
Return-Path: <bmeyer@helios.mesoft.com.au>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 5EA7D1065672
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  2 May 2011 23:38:49 +0000 (UTC)
	(envelope-from bmeyer@helios.mesoft.com.au)
Received: from helios.mesoft.com.au (helios.mesoft.com.au [203.206.179.92])
	by mx1.freebsd.org (Postfix) with ESMTP id 52C738FC1A
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  2 May 2011 23:38:47 +0000 (UTC)
Received: from helios.mesoft.com.au (localhost [127.0.0.1])
	by helios.mesoft.com.au (8.14.4/8.14.4) with ESMTP id p42NBFvc082976
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Tue, 3 May 2011 09:11:15 +1000 (EST)
	(envelope-from bmeyer@helios.mesoft.com.au)
Received: (from root@localhost)
	by helios.mesoft.com.au (8.14.4/8.14.4/Submit) id p42NBFKT082975;
	Tue, 3 May 2011 09:11:15 +1000 (EST)
	(envelope-from bmeyer)
Message-Id: <201105022311.p42NBFKT082975@helios.mesoft.com.au>
Date: Tue, 3 May 2011 09:11:15 +1000 (EST)
From: bmeyer@mesoft.com.au
Reply-To: bmeyer@mesoft.com.au
To: FreeBSD-gnats-submit@freebsd.org
Cc: bmeyer@mesoft.com.au
Subject: zfs is loosing the snapshot directory,  
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         156781
>Category:       kern
>Synopsis:       [zfs] zfs is losing the snapshot directory,
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon May 02 23:40:07 UTC 2011
>Closed-Date:    
>Last-Modified:  Tue Sep  3 16:30:01 UTC 2013
>Originator:     Charlie &
>Release:        FreeBSD 8.2-STABLE sparc64
>Organization:
MicroElectronic SOFTworks Pty Ltd
>Environment:
System: FreeBSD helios.mesoft.com.au 8.2-STABLE FreeBSD 8.2-STABLE #1: Sun Apr 24 21:28:12 EST 2011 root@helios.mesoft.com.au:/usr/obj/usr/src/sys/GENERIC sparc64

	Sun-Fire V120, FreeBSD 8.2
>Description:
----------------------------------------------------------------------
Command Log Output:

helios.mesoft.com.au : Tue May  3 08:15:30 EST 2011 : /tmp
-> 1013# hostname
helios.mesoft.com.au

helios.mesoft.com.au : Tue May  3 08:15:32 EST 2011 : /tmp
-> 1014# uname -a
FreeBSD helios.mesoft.com.au 8.2-STABLE FreeBSD 8.2-STABLE #1: Sun Apr 24 21:28:12 EST 2011     root@helios.mesoft.com.au:/usr/obj/usr/src/sys/GENERIC  sparc64

helios.mesoft.com.au : Tue May  3 08:15:34 EST 2011 : /tmp
-> 1015# gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  da0
                      da1

helios.mesoft.com.au : Tue May  3 08:15:38 EST 2011 : /tmp
-> 1016# zpool status -v
  pool: pool0
 state: ONLINE
 scrub: scrub completed after 0h2m with 0 errors on Tue May  3 07:15:24 2011
config:

	NAME           STATE     READ WRITE CKSUM
	pool0          ONLINE       0     0     0
	  mirror/gm0h  ONLINE       0     0     0

errors: No known data errors

helios.mesoft.com.au : Tue May  3 08:15:42 EST 2011 : /tmp
-> 1017# zfs list        
NAME              USED  AVAIL  REFER  MOUNTPOINT
pool0            5.00G   263G    24K  /pool0
pool0/databases   128M   263G   128M  /pool0/databases
pool0/home       45.2M   263G  45.2M  /pool0/home
pool0/mail       4.83G   263G  4.83G  /pool0/mail

helios.mesoft.com.au : Tue May  3 08:15:49 EST 2011 : /tmp
-> 1018# zfs list -t snapshot
no datasets available

helios.mesoft.com.au : Tue May  3 08:15:53 EST 2011 : /tmp
-> 1019# zfs snapshot pool0/mail@test

helios.mesoft.com.au : Tue May  3 08:16:07 EST 2011 : /tmp
-> 1020# zfs snapshot pool0/home@test

helios.mesoft.com.au : Tue May  3 08:16:19 EST 2011 : /tmp
-> 1021# zfs list -t snapshot
NAME              USED  AVAIL  REFER  MOUNTPOINT
pool0/home@test      0      -  45.2M  -
pool0/mail@test    59K      -  4.83G  -

helios.mesoft.com.au : Tue May  3 08:16:22 EST 2011 : /tmp
-> 1022# ls -l /pool0/mail/.zfs
total 0
dr-xr-xr-x  2 root  wheel  2 Jun 16  2010 snapshot

helios.mesoft.com.au : Tue May  3 08:16:32 EST 2011 : /tmp
-> 1023# ls -l /pool0/home/.zfs
ls: snapshot: Bad file descriptor
total 0

helios.mesoft.com.au : Tue May  3 08:17:08 EST 2011 : /tmp
-> 1024# zfs destroy pool0/mail@test

helios.mesoft.com.au : Tue May  3 08:17:52 EST 2011 : /tmp
-> 1025# zfs destroy pool0/home@test

helios.mesoft.com.au : Tue May  3 08:18:05 EST 2011 : /tmp
-> 1026# zfs list -t snapshot
no datasets available

helios.mesoft.com.au : Tue May  3 08:22:13 EST 2011 : /tmp
-> 1027# zfs upgrade
This system is currently running ZFS filesystem version 4.

All filesystems are formatted with the current version.

helios.mesoft.com.au : Tue May  3 08:22:18 EST 2011 : /tmp
-> 1028# zpool upgrade
This system is currently running ZFS pool version 15.

All pools are formatted using this version.

helios.mesoft.com.au : Tue May  3 08:22:47 EST 2011 : /tmp
-> 1029# zdb  
pool0
    version=15
    name='pool0'
    state=0
    txg=908395
    pool_guid=2223392563195472406
    hostid=1798529942
    hostname='helios.mesoft.com.au'
    vdev_tree
        type='root'
        id=0
        guid=2223392563195472406
        children[0]
                type='disk'
                id=0
                guid=8062368332970890863
                path='/dev/mirror/gm0h'
                whole_disk=0
                metaslab_array=23
                metaslab_shift=31
                ashift=9
                asize=293168742400
                is_log=0
                DTL=40

helios.mesoft.com.au : Tue May  3 08:24:05 EST 2011 : /tmp
-> 1030# zpool scrub pool0

helios.mesoft.com.au : Tue May  3 08:31:22 EST 2011 : /tmp
-> 1031# zpool status -v
  pool: pool0
 state: ONLINE
 scrub: scrub completed after 0h2m with 0 errors on Tue May  3 08:26:35 2011
config:

	NAME           STATE     READ WRITE CKSUM
	pool0          ONLINE       0     0     0
	  mirror/gm0h  ONLINE       0     0     0

errors: No known data errors

----------------------------------------------------------------------
DMESG, CONFIGs, etc.  

/var/run/dmesg.boot:

Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-STABLE #1: Sun Apr 24 21:28:12 EST 2011
    root@helios.mesoft.com.au:/usr/obj/usr/src/sys/GENERIC sparc64
real memory  = 4294967296 (4096 MB)
avail memory = 4165672960 (3972 MB)
cpu0: Sun Microsystems UltraSparc-IIe Processor (648.00 MHz CPU)
ispfw: registered firmware <isp_1000>
ispfw: registered firmware <isp_1040>
ispfw: registered firmware <isp_1040_it>
ispfw: registered firmware <isp_1080>
ispfw: registered firmware <isp_1080_it>
ispfw: registered firmware <isp_12160>
ispfw: registered firmware <isp_12160_it>
ispfw: registered firmware <isp_2100>
ispfw: registered firmware <isp_2200>
ispfw: registered firmware <isp_2300>
ispfw: registered firmware <isp_2322>
ispfw: registered firmware <isp_2400>
ispfw: registered firmware <isp_2400_multi>
ispfw: registered firmware <isp_2500>
ispfw: registered firmware <isp_2500_multi>
kbd0 at kbdmux0
nexus0: <Open Firmware Nexus device>
pcib0: <U2P UPA-PCI bridge> mem 0x1fe00000000-0x1fe0000ffff,0x1fe01000000-0x1fe010000ff irq 2032,2030,2031,2021 on nexus0
pcib0: Sabre, impl 0, version 0, IGN 0x1f, bus A, 66MHz
pcib0: DVMA map: 0xc0000000 to 0xc3ffffff 8192 entries
pcib0: [FILTER]
pcib0: [FILTER]
pcib0: [GIANT-LOCKED]
pcib0: [ITHREAD]
pcib0: [FILTER]
pci0: <OFW PCI bus> on pcib0
pcib1: <APB PCI-PCI bridge> at device 1.1 on pci0
pci1: <OFW PCI bus> on pcib1
ebus0: <PCI-EBus3 bridge> mem 0xf0000000-0xf0ffffff,0xf1000000-0xf17fffff at device 12.0 on pci1
ebus0: <idprom>: incomplete
pcib2: <APB PCI-PCI bridge> at device 1.0 on pci0
pci2: <OFW PCI bus> on pcib2
ebus0: <flashprom> addr 0x1000000000-0x10000fffff (no driver attached)
eeprom0: <EEPROM/clock> addr 0x1400000000-0x1400001fff on ebus0
eeprom0: model mk48t59
ebus0: <SUNW,lomh> addr 0x1400200000-0x1400200003 irq 42 (no driver attached)
pci1: <old, non-VGA display device> at device 3.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 7.0 on pci1
isa0: <ISA bus> on isab0
gem0: <Sun ERI 10/100 Ethernet> mem 0xe0400000-0xe041ffff at device 12.1 on pci1
miibus0: <MII bus> on gem0
bmtphy0: <BCM5221 10/100baseTX PHY> PHY 1 on miibus0
bmtphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
gem0: 2kB RX FIFO, 2kB TX FIFO
gem0: Ethernet address: 00:03:ba:68:45:35
gem0: [ITHREAD]
ohci0: <Sun PCIO-2 USB controller> mem 0xe2000000-0xe2007fff at device 12.3 on pci1
ohci0: [ITHREAD]
usbus0: <Sun PCIO-2 USB controller> on ohci0
atapci0: <AcerLabs M5229 UDMA66 controller> port 0x400-0x407,0x418-0x41b,0x410-0x417,0x408-0x40b,0x420-0x42f at device 13.0 on pci1
atapci0: [ITHREAD]
atapci0: using PIO transfers above 137GB as workaround for 48bit DMA access bug, expect reduced performance
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
gem1: <Sun ERI 10/100 Ethernet> mem 0xe0440000-0xe045ffff at device 5.1 on pci1
miibus1: <MII bus> on gem1
bmtphy1: <BCM5221 10/100baseTX PHY> PHY 1 on miibus1
bmtphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
gem1: 2kB RX FIFO, 2kB TX FIFO
gem1: Ethernet address: 00:03:ba:68:45:35
gem1: [ITHREAD]
ohci1: <Sun PCIO-2 USB controller> mem 0xe5000000-0xe5007fff at device 5.3 on pci1
ohci1: [ITHREAD]
usbus1: <Sun PCIO-2 USB controller> on ohci1
sym0: <896> port 0xc00000-0xc000ff mem 0x2000-0x23ff,0x4000-0x5fff at device 8.0 on pci2
sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
sym0: [ITHREAD]
sym1: <896> port 0xc00100-0xc001ff mem 0x6000-0x63ff,0x8000-0x9fff at device 8.1 on pci2
sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking
sym1: [ITHREAD]
re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> port 0xc00200-0xc002ff mem 0xa000-0xa0ff at device 5.0 on pci2
re0: Chip rev. 0x04000000
re0: MAC rev. 0x00000000
miibus2: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus2
rgephy0:  10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 00:08:54:35:a4:97
re0: [FILTER]
nexus0: <syscons> type unknown (no driver attached)
uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 43 on isa0
uart0: [FILTER]
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> at port 0x2e8-0x2ef irq 43 on isa0
uart1: [FILTER]
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 4
ZFS storage pool version 15
Timecounter "tick" frequency 648000000 Hz quality 1000
Timecounters tick every 10.000 msec
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to deny, logging disabled
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
ugen0.1: <SUN> at usbus0
uhub0: <SUN OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <SUN> at usbus1
uhub1: <SUN OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
acd0: DVDROM <DV-28E-B/1.2B> at ata2-master UDMA33 
uhub0: 4 ports with 4 removable, self powered
uhub1: 4 ports with 4 removable, self powered
da0 at sym0 bus 0 scbus0 target 0 lun 0
da0: <ACARD SCSIDE BRIDGE320 10EP> Fixed Direct Access SCSI-3 device 
da0: 80.000MB/s transfers (40.000MHz, offset 28, 16bit)
da0: Command Queueing enabled
da0: 305245MB (625142448 512 byte sectors: 255H 63S/T 38913C)
da1 at sym0 bus 0 scbus0 target 1 lun 0
da1: <ACARD SCSIDE BRIDGE320 10EP> Fixed Direct Access SCSI-3 device 
da1: 80.000MB/s transfers (40.000MHz, offset 28, 16bit)
da1: Command Queueing enabled
da1: 305245MB (625142448 512 byte sectors: 255H 63S/T 38913C)
GEOM_MIRROR: Device mirror/gm0 launched (1/2).
GEOM_MIRROR: Device gm0: rebuilding provider da1.
GEOM: mirror/gm0: adding VTOC8 information.
GEOM_JOURNAL: Journal 4175325748: mirror/gm0e contains data.
GEOM_JOURNAL: Journal 4175325748: mirror/gm0e contains journal.
GEOM_JOURNAL: Journal mirror/gm0e clean.
GEOM_JOURNAL: Journal 2946333449: mirror/gm0g contains data.
GEOM_JOURNAL: Journal 2946333449: mirror/gm0g contains journal.
GEOM_JOURNAL: Journal mirror/gm0g clean.
Trying to mount root from ufs:/dev/mirror/gm0a

/boot/loader.conf :

ipfw_load="YES"
dummynet_load="YES"
geom_mirror_load="YES"
geom_journal_load="YES"
opensolaris_load="YES"
zfs_load="YES"
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
vfs.zfs.arc_max="128M"
vfs.zfs.vdev.cache.size="16M"
kern.hz="100"

/etc/sysctl.conf :

# $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith Exp $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
kern.ipc.shmmax=268435456
net.inet.ip.fw.verbose=1

Disk Slicing :
sunlabel da0

# /dev/da0:
text: FreeBSD298G cyl 38911 alt 2 hd 255 sec 63
bytes/sector: 512
sectors/cylinder: 16065
sectors/unit: 625105215

8 partitions:
#
# Size is in sectors.
# Offset is in cylinders.
#    size       offset
#    ---------- ----------
  a:    1060290          0
  b:    4209030         66
  c:  625105215          0
  d:    2104515        328
  e:    8401995        459
  f:    1060290        982
  g:   35664300       1048
  h:  572604795       3268

sunlabel da1
# /dev/da1:
text: FreeBSD298G cyl 38911 alt 2 hd 255 sec 63
bytes/sector: 512
sectors/cylinder: 16065
sectors/unit: 625105215

8 partitions:
#
# Size is in sectors.
# Offset is in cylinders.
#    size       offset
#    ---------- ----------
  a:    1060290          0
  b:    4209030         66
  c:  625105215          0
  d:    2104515        328
  e:    8401995        459
  f:    1060290        982
  g:   35664300       1048
  h:  572604795       3268

>How-To-Repeat:
	Allow system to run for any length of time.  
>Fix:
	Only known workaround at this time is to periodically reboot the box -
usually nightly.  
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue May 3 06:21:27 UTC 2011 
Responsible-Changed-Why:  
reclassify. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=156781 

From: Nathan Bowyer <nbowyer@gmail.com>
To: bug-followup@FreeBSD.org, bmeyer@mesoft.com.au
Cc:  
Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory,
Date: Tue, 5 Jul 2011 08:56:27 -0500

 --0015174c11fca119b604a752d913
 Content-Type: text/plain; charset=ISO-8859-1
 
 I have had this occur on amd64 with 8.2-RELEASE.  I also came to the
 conclusion that it takes a reboot to fix the issue.
 
 Is there any activity on this bug?
 
 Thanks,
 Nathan
 
 --0015174c11fca119b604a752d913
 Content-Type: text/html; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 I have had this occur on amd64 with 8.2-RELEASE. =A0I also came to the conc=
 lusion that it takes a reboot to fix the issue.<div><br></div><div>Is there=
  any activity on this bug?</div><div><br></div><div>Thanks,</div><div>Natha=
 n</div>
 
 --0015174c11fca119b604a752d913--

From: Brendon Meyer <bmeyer@mesoft.com.au>
To: Nathan Bowyer <nbowyer@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory,
Date: Wed, 6 Jul 2011 08:23:32 +1000

 On 05/07/2011, at 11:56 PM, Nathan Bowyer wrote:
 
 > I have had this occur on amd64 with 8.2-RELEASE.  I also came to the =
 conclusion that it takes a reboot to fix the issue.
 > Is there any activity on this bug?
 
 Not sure to be honest.  I'm going actually in the process of building a =
 new testing environment for a updated mail server and will probably be =
 able to give a more definitive answer a bit later (the build will be =
 done later this week and then I'll be letting my test scrips run mad on =
 it for a week or so). =20
 
 The fact that you have had the same problems with AMD means that the =
 problem isn't SPARC specific.  I am guessing that it is probably a =
 commit that was made between to the zfs or solaris code between Mon Feb =
 28 16:07:24 EST 2011 and Sun Apr 24 21:28:12 EST 2011 (those are two =
 systems I have that are very similarly configured but the earlier one =
 doesn't have the problem whereas the later one does). =20
 
 
 /BGM=

From: Patrick Proniewski <patpro@patpro.net>
To: bug-followup@FreeBSD.org,
 bmeyer@mesoft.com.au
Cc:  
Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory,
Date: Fri, 13 Apr 2012 00:15:40 +0200

 Same problem here:
 
 FreeBSD mybox 8.1-RELEASE-p5 FreeBSD 8.1-RELEASE-p5 #0: Tue Sep 27 =
 16:49:00 UTC 2011     =
 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
 
 ls of some .zfs directories returns the "Bad file descriptor" error.
 
 zfs list still shows every snapshots, and daily snapshot =
 creation/rotation works as expected.
 
 I tried to unmount the faulty zfs volume, but the system froze, I had to =
 reboot the server.=

From: Peter Maloney <peter.maloney@brockmann-consult.de>
To: bug-followup@FreeBSD.org, bmeyer@mesoft.com.au
Cc:  
Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory,
Date: Fri, 13 Apr 2012 09:25:16 +0200

 Is the faulty dataset shared over NFS? If so, is it the new NFS server 
 (default in 9.0 I think) or the old (default in 8.2)?
 
 My ZFS over NFS on 8-STABLE has slightly similar problems on the NFS 
 client side, but only when viewed by a Linux client. A simple "ls -l" on 
 the .zfs/snapshot directory on a Linux client hangs the dataset. And 
 these lines show up in /var/log/messages when reloading the NFS server:
 
 Apr 12 10:08:38 [hostname] mountd[54479]: can't delete exports for 
 /tank/[...]/.zfs/snapshot/replication-20120411112000: Invalid argument
 
 And an "ls" without "-l" on a Linux client shows that the ones mentioned 
 in messages are files and the rest are directories (Linux shells 
 generally color them blue and append a / for directories).
 
 I can only guess that they may be related.
 

From: Patrick Proniewski <patpro@patpro.net>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory,
Date: Fri, 13 Apr 2012 21:12:39 +0200

 No sharing, nothing fancy, only ZFS and snapshot on a server.

From: Patrick Proniewski <patpro@patpro.net>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory,
Date: Mon, 4 Jun 2012 23:04:07 +0200

 Well, this bug is really really annoying. I would reclassify it as =
 critical: my server is hosted in a data center, and when I want/need to =
 reboot it, it just crashes and stays frozen. I've to ask my hosting =
 provider to hard reboot the server.
 
 I wonder if I need to use `reboot -qn` until the bug is corrected, or =
 should I ditch ZFS?=

From: Nikolai Schupbach <nikolai@net24.co.nz>
To: bug-followup@FreeBSD.org
Cc: bmeyer@mesoft.com.au
Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory,
Date: Thu, 6 Sep 2012 03:28:26 +1200

 I would like to note that we are also experiencing this issue. We are =
 just using the basic core ZFS features only - snapshots and zfs =
 send/receive. There does not seem to be any set pattern as to when this =
 issue occurs (the machine below had the issue appear after 47 days) and =
 it doesn't affect all filesystems at once.
 
 It would be good if we could get some traction on this issue as it =
 appears to be a systemic issue, across all releases and architectures, =
 reported by multiple people over the last 2-3 years. As snapshots are a =
 basic core feature of ZFS, I would really like to see this bug moved to =
 a higher priority.
 
 # ls -l /mailstore/domains/.zfs
 ls: snapshot: Bad file descriptor
 total 2
 dr-xr-xr-x  2 root  wheel  2 Jul 19 15:46 shares
 
 # uname -a
 FreeBSD 9.0-RELEASE-p3 FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 02:52:29 =
 UTC 2012     =
 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
 
 # uptime
 2:45AM  up 47 days,  4:30, 1 user, load averages: 0.29, 0.29, 0.21
 
 # zfs list
 NAME                USED  AVAIL  REFER  MOUNTPOINT
 mailstore           466G  3.11T    34K  /mailstore
 mailstore/domains   464G  3.11T   247G  /mailstore/domains
 mailstore/queue    3.31M   100G   102K  /mailstore/queue
 mailstore/var      1.30G  3.11T   182M  /mailstore/var
 system             15.4G   258G    32K  none
 system/root        15.2G   258G  7.08G  legacy
 
 # zpool status
   pool: mailstore
  state: ONLINE
  scan: scrub repaired 0 in 0h11m with 0 errors on Sun Aug 19 03:13:05 =
 2012
 config:
 
 	NAME              STATE     READ WRITE CKSUM
 	mailstore         ONLINE       0     0     0
 	  mirror-0        ONLINE       0     0     0
 	    label/disk8   ONLINE       0     0     0
 	    label/disk16  ONLINE       0     0     0
 	  mirror-1        ONLINE       0     0     0
 	    label/disk9   ONLINE       0     0     0
 	    label/disk17  ONLINE       0     0     0
 	  mirror-2        ONLINE       0     0     0
 	    label/disk10  ONLINE       0     0     0
 	    label/disk18  ONLINE       0     0     0
 	  mirror-3        ONLINE       0     0     0
 	    label/disk11  ONLINE       0     0     0
 	    label/disk19  ONLINE       0     0     0
 	  mirror-4        ONLINE       0     0     0
 	    label/disk12  ONLINE       0     0     0
 	    label/disk20  ONLINE       0     0     0
 	  mirror-5        ONLINE       0     0     0
 	    label/disk13  ONLINE       0     0     0
 	    label/disk21  ONLINE       0     0     0
 	  mirror-6        ONLINE       0     0     0
 	    label/disk14  ONLINE       0     0     0
 	    label/disk22  ONLINE       0     0     0
 	  mirror-7        ONLINE       0     0     0
 	    label/disk15  ONLINE       0     0     0
 	    label/disk23  ONLINE       0     0     0
 	logs
 	  mirror-8        ONLINE       0     0     0
 	    label/disk3   ONLINE       0     0     0
 	    label/disk4   ONLINE       0     0     0
 	cache
 	  label/disk5     ONLINE       0     0     0
 	spares
 	  label/disk6     AVAIL  =20
 	  label/disk7     AVAIL  =20
 
 errors: No known data errors
 
   pool: system
  state: ONLINE
  scan: scrub repaired 0 in 0h4m with 0 errors on Mon Aug 20 03:05:52 =
 2012
 config:
 
 	NAME           STATE     READ WRITE CKSUM
 	system         ONLINE       0     0     0
 	  mirror-0     ONLINE       0     0     0
 	    gpt/disk0  ONLINE       0     0     0
 	    gpt/disk1  ONLINE       0     0     0
 	spares
 	  gpt/disk2    AVAIL  =20
 
 errors: No known data errors
 
 # zpool get all mailstore
 NAME       PROPERTY       VALUE       SOURCE
 mailstore  size           3.62T       -
 mailstore  capacity       12%         -
 mailstore  altroot        -           default
 mailstore  health         ONLINE      -
 mailstore  guid           5477043980362510628  default
 mailstore  version        28          default
 mailstore  bootfs         -           default
 mailstore  delegation     on          default
 mailstore  autoreplace    off         default
 mailstore  cachefile      -           default
 mailstore  failmode       wait        default
 mailstore  listsnapshots  off         default
 mailstore  autoexpand     off         default
 mailstore  dedupditto     0           default
 mailstore  dedupratio     1.00x       -
 mailstore  free           3.17T       -
 mailstore  allocated      466G        -
 mailstore  readonly       off     =20
 
 # zfs get all mailstore/domains
 NAME               PROPERTY              VALUE                  SOURCE
 mailstore/domains  type                  filesystem             -
 mailstore/domains  creation              Thu Jul 19 15:46 2012  -
 mailstore/domains  used                  463G                   -
 mailstore/domains  available             3.11T                  -
 mailstore/domains  referenced            247G                   -
 mailstore/domains  compressratio         1.00x                  -
 mailstore/domains  mounted               yes                    -
 mailstore/domains  quota                 none                   default
 mailstore/domains  reservation           none                   default
 mailstore/domains  recordsize            128K                   default
 mailstore/domains  mountpoint            /mailstore/domains     default
 mailstore/domains  sharenfs              off                    default
 mailstore/domains  checksum              on                     default
 mailstore/domains  compression           off                    default
 mailstore/domains  atime                 off                    local
 mailstore/domains  devices               on                     default
 mailstore/domains  exec                  on                     default
 mailstore/domains  setuid                off                    =
 inherited from mailstore
 mailstore/domains  readonly              off                    default
 mailstore/domains  jailed                off                    default
 mailstore/domains  snapdir               hidden                 default
 mailstore/domains  aclmode               discard                default
 mailstore/domains  aclinherit            restricted             default
 mailstore/domains  canmount              on                     default
 mailstore/domains  xattr                 off                    =
 temporary
 mailstore/domains  copies                1                      default
 mailstore/domains  version               5                      -
 mailstore/domains  utf8only              off                    -
 mailstore/domains  normalization         none                   -
 mailstore/domains  casesensitivity       sensitive              -
 mailstore/domains  vscan                 off                    default
 mailstore/domains  nbmand                off                    default
 mailstore/domains  sharesmb              off                    default
 mailstore/domains  refquota              none                   default
 mailstore/domains  refreservation        none                   default
 mailstore/domains  primarycache          all                    default
 mailstore/domains  secondarycache        all                    default
 mailstore/domains  usedbysnapshots       217G                   -
 mailstore/domains  usedbydataset         247G                   -
 mailstore/domains  usedbychildren        0                      -
 mailstore/domains  usedbyrefreservation  0                      -
 mailstore/domains  logbias               latency                default
 mailstore/domains  dedup                 off                    default
 mailstore/domains  mlslabel                                     -
 mailstore/domains  sync                  standard               default
 mailstore/domains  refcompressratio      1.00x                  -
 
 # zpool history
 History for 'mailstore':
 2012-07-19.15:46:37 zpool create -f mailstore mirror label/disk8 =
 label/disk16 mirror label/disk9 label/disk17 mirror label/disk10 =
 label/disk18 mirror label/disk11 label/disk19 mirror label/disk12 =
 label/disk20 mirror label/disk13 label/disk21 mirror label/disk14 =
 label/disk22 mirror label/disk15 label/disk23 log mirror label/disk3 =
 label/disk4 cache label/disk5 spare label/disk6 label/disk7
 2012-07-28.10:30:00 zfs snapshot -r mailstore@autosnap-2012-07-28_10.30
 2012-07-28.10:30:05 zfs destroy -r mailstore@autosnap-2012-07-28_08.30
 2012-07-28.10:45:01 zfs snapshot -r mailstore@autosnap-2012-07-28_10.45
 2012-07-28.10:45:06 zfs destroy -r mailstore@autosnap-2012-07-28_08.45
 2012-07-28.11:00:01 zfs snapshot -r mailstore@autosnap-2012-07-28_11.00
 2012-07-28.11:00:06 zfs destroy -r mailstore@autosnap-2012-07-27_11.00
 2012-07-28.11:15:00 zfs snapshot -r mailstore@autosnap-2012-07-28_11.15
 2012-07-28.11:15:05 zfs destroy -r mailstore@autosnap-2012-07-28_09.15
 2012-07-28.11:30:01 zfs snapshot -r mailstore@autosnap-2012-07-28_11.30
 ...
 2012-09-06.02:00:01 zfs snapshot -r mailstore@autosnap-2012-09-06_02.00
 2012-09-06.02:00:06 zfs destroy -r mailstore@autosnap-2012-09-05_02.00
 2012-09-06.02:15:00 zfs snapshot -r mailstore@autosnap-2012-09-06_02.15
 2012-09-06.02:15:05 zfs destroy -r mailstore@autosnap-2012-09-06_00.15
 2012-09-06.02:30:00 zfs snapshot -r mailstore@autosnap-2012-09-06_02.30
 2012-09-06.02:30:05 zfs destroy -r mailstore@autosnap-2012-09-06_00.30
 2012-09-06.02:45:00 zfs snapshot -r mailstore@autosnap-2012-09-06_02.45
 2012-09-06.02:45:05 zfs destroy -r mailstore@autosnap-2012-09-06_00.45
 
 # truss ls -l /mailstore/domains/.zfs
 mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D =
 34366279680 (0x80063d000)
 issetugid(0x80063e015,0x800632d7e,0x80084e7f0,0x80084e7c0,0xb297,0x0) =3D =
 0 (0x0)
 open("/etc/libmap.conf",O_RDONLY,0666)		 ERR#2 'No such file or =
 directory'
 open("/var/run/ld-elf.so.hints",O_RDONLY,057)	 =3D 3 (0x3)
 read(3,"Ehnt\^A\0\0\0\M^@\0\0\0u\0\0\0\0"...,128) =3D 128 (0x80)
 lseek(3,0x80,SEEK_SET)				 =3D 128 (0x80)
 read(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,117) =3D 117 (0x75)
 close(3)					 =3D 0 (0x0)
 access("/lib/libutil.so.9",0)			 =3D 0 (0x0)
 open("/lib/libutil.so.9",O_RDONLY,041017540)	 =3D 3 (0x3)
 fstat(3,{ mode=3D-r--r--r-- ,inode=3D10999,size=3D70168,blksize=3D70656 =
 }) =3D 0 (0x0)
 pread(0x3,0x800840f40,0x1000,0x0,0x101010101010101,0x8080808080808080) =3D=
  4096 (0x1000)
 mmap(0x0,2166784,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D =
 34368450560 (0x80084f000)
 =
 mmap(0x80084f000,61440,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCOR=
 E,3,0x0) =3D 34368450560 (0x80084f000)
 =
 mmap(0x800a5e000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0xf000)=
  =3D 34370609152 (0x800a5e000)
 =
 mmap(0x800a5f000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,=
 -1,0x0) =3D 34370613248 (0x800a5f000)
 close(3)					 =3D 0 (0x0)
 access("/lib/libncurses.so.8",0)		 =3D 0 (0x0)
 open("/lib/libncurses.so.8",O_RDONLY,041017540)	 =3D 3 (0x3)
 fstat(3,{ mode=3D-r--r--r-- ,inode=3D10996,size=3D321040,blksize=3D131072 =
 }) =3D 0 (0x0)
 pread(0x3,0x800840f40,0x1000,0x0,0x101010101010101,0x8080808080808080) =3D=
  4096 (0x1000)
 mmap(0x0,2412544,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D =
 34370617344 (0x800a60000)
 =
 mmap(0x800a60000,294912,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCO=
 RE,3,0x0) =3D 34370617344 (0x800a60000)
 =
 mmap(0x800ca8000,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x4800=
 0) =3D 34373009408 (0x800ca8000)
 close(3)					 =3D 0 (0x0)
 access("/lib/libc.so.7",0)			 =3D 0 (0x0)
 open("/lib/libc.so.7",O_RDONLY,041017540)	 =3D 3 (0x3)
 fstat(3,{ mode=3D-r--r--r-- ,inode=3D10991,size=3D1315160,blksize=3D131072=
  }) =3D 0 (0x0)
 pread(0x3,0x800840f40,0x1000,0x0,0x101010101010101,0x8080808080808080) =3D=
  4096 (0x1000)
 mmap(0x0,3432448,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D =
 34373029888 (0x800cad000)
 =
 mmap(0x800cad000,1179648,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOC=
 ORE,3,0x0) =3D 34373029888 (0x800cad000)
 =
 mmap(0x800fcd000,45056,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0x1200=
 00) =3D 34376306688 (0x800fcd000)
 =
 mmap(0x800fd8000,110592,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANO=
 N,-1,0x0) =3D 34376351744 (0x800fd8000)
 close(3)					 =3D 0 (0x0)
 munmap(0x800644000,4096)			 =3D 0 (0x0)
 mmap(0x0,40960,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D =
 34366308352 (0x800644000)
 munmap(0x80064a000,16384)			 =3D 0 (0x0)
 mmap(0x0,102400,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D =
 34366332928 (0x80064a000)
 =
 sysarch(0x81,0x7fffffffd310,0x800643188,0x0,0xffffffffff672680,0x808080808=
 0808080) =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 ioctl(1,TIOCGETA,0xffffd410)			 =3D 0 (0x0)
 ioctl(1,TIOCGWINSZ,0xffffd470)			 =3D 0 (0x0)
 getuid()					 =3D 0 (0x0)
 readlink("/etc/malloc.conf",0x7fffffffcef0,1024) ERR#2 'No such file or =
 directory'
 =
 issetugid(0x800da9bc1,0x7fffffffcef0,0xffffffffffffffff,0x0,0x2,0x7fffffff=
 d18f) =3D 0 (0x0)
 break(0x800000)					 =3D 0 (0x0)
 mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D =
 34376462336 (0x800ff3000)
 mmap(0x8013f3000,53248,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
 =3D 34380656640 (0x8013f3000)
 munmap(0x800ff3000,53248)			 =3D 0 (0x0)
 lstat("/mailstore/domains/.zfs",{ mode=3Ddr-xr-xr-x =
 ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 (0x0)
 open(".",O_RDONLY,00)				 =3D 3 (0x3)
 fchdir(0x3,0x0,0x20,0x20,0x1,0x0)		 =3D 0 (0x0)
 stat("/mailstore/domains/.zfs",{ mode=3Ddr-xr-xr-x =
 ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 (0x0)
 open("/mailstore/domains/.zfs",O_NONBLOCK|0x20000,0100401200) =3D 4 =
 (0x4)
 fstat(4,{ mode=3Ddr-xr-xr-x ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 =
 (0x0)
 fcntl(4,F_SETFD,FD_CLOEXEC)			 =3D 0 (0x0)
 fstatfs(0x4,0x7fffffffd0a0,0x0,0x26,0x606fe0,0x7fffffffc630) =3D 0 (0x0)
 fstat(4,{ mode=3Ddr-xr-xr-x ,inode=3D1,size=3D4,blksize=3D4096 }) =3D 0 =
 (0x0)
 fchdir(0x4,0x7fffffffd2b0,0x4,0x0,0x606fe0,0x7fffffffc610) =3D 0 (0x0)
 =
 getdirentries(0x4,0x801027000,0x1000,0x801025068,0x606fe0,0x7fffffffc610) =
 =3D 60 (0x3c)
 lstat("snapshot",0x801020450)			 ERR#9 'Bad file =
 descriptor'
 lstat("shares",{ mode=3Ddr-xr-xr-x ,inode=3D7,size=3D2,blksize=3D4096 }) =
 =3D 0 (0x0)
 getdirentries(0x4,0x801027000,0x1000,0x801025068,0xffffffffffff95d4,0x0) =
 =3D 0 (0x0)
 lseek(4,0x0,SEEK_SET)				 =3D 0 (0x0)
 madvise(0x801069000,0x1000,0x5,0x68,0x7fffffffca10,0x7fffffffca10) =3D 0 =
 (0x0)
 madvise(0x801026000,0x2000,0x5,0x25,0x7fffffffca10,0x7fffffffca10) =3D 0 =
 (0x0)
 madvise(0x801025000,0x1000,0x5,0x24,0x7fffffffca70,0x7fffffffca10) =3D 0 =
 (0x0)
 close(4)					 =3D 0 (0x0)
 fchdir(0x3,0x0,0x0,0x6,0x801000000,0x7fffffffca80) =3D 0 (0x0)
 stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- =
 ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0)
 open("/etc/nsswitch.conf",O_RDONLY,0666)	 =3D 4 (0x4)
 ioctl(4,TIOCGETA,0xffffcaf0)			 ERR#25 'Inappropriate =
 ioctl for device'
 fstat(4,{ mode=3D-rw-r--r-- ,inode=3D11094,size=3D323,blksize=3D4096 }) =
 =3D 0 (0x0)
 read(4,"#\n# nsswitch.conf(5) - name ser"...,4096) =3D 323 (0x143)
 read(4,0x801045000,4096)			 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 access("/lib/nss_compat.so.1",0)		 ERR#2 'No such file or =
 directory'
 access("/usr/lib/nss_compat.so.1",0)		 ERR#2 'No such file or =
 directory'
 access("/usr/lib/compat/nss_compat.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/nss_compat.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/dovecot/nss_compat.so.1",0) ERR#2 'No such file =
 or directory'
 access("/usr/local/lib/mysql/nss_compat.so.1",0) ERR#2 'No such file or =
 directory'
 access("/usr/local/libexec/openldap/nss_compat.so.1",0) ERR#2 'No such =
 file or directory'
 access("/lib/nss_compat.so.1",0)		 ERR#2 'No such file or =
 directory'
 access("/usr/lib/nss_compat.so.1",0)		 ERR#2 'No such file or =
 directory'
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 access("/lib/nss_nis.so.1",0)			 ERR#2 'No such file or =
 directory'
 access("/usr/lib/nss_nis.so.1",0)		 ERR#2 'No such file or =
 directory'
 access("/usr/lib/compat/nss_nis.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/nss_nis.so.1",0)		 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/dovecot/nss_nis.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/mysql/nss_nis.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/libexec/openldap/nss_nis.so.1",0) ERR#2 'No such file =
 or directory'
 access("/lib/nss_nis.so.1",0)			 ERR#2 'No such file or =
 directory'
 access("/usr/lib/nss_nis.so.1",0)		 ERR#2 'No such file or =
 directory'
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 access("/lib/nss_files.so.1",0)			 ERR#2 'No such file or =
 directory'
 access("/usr/lib/nss_files.so.1",0)		 ERR#2 'No such file or =
 directory'
 access("/usr/lib/compat/nss_files.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/nss_files.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/dovecot/nss_files.so.1",0) ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/mysql/nss_files.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/libexec/openldap/nss_files.so.1",0) ERR#2 'No such =
 file or directory'
 access("/lib/nss_files.so.1",0)			 ERR#2 'No such file or =
 directory'
 access("/usr/lib/nss_files.so.1",0)		 ERR#2 'No such file or =
 directory'
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 access("/lib/nss_dns.so.1",0)			 ERR#2 'No such file or =
 directory'
 access("/usr/lib/nss_dns.so.1",0)		 ERR#2 'No such file or =
 directory'
 access("/usr/lib/compat/nss_dns.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/nss_dns.so.1",0)		 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/dovecot/nss_dns.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/lib/mysql/nss_dns.so.1",0)	 ERR#2 'No such file or =
 directory'
 access("/usr/local/libexec/openldap/nss_dns.so.1",0) ERR#2 'No such file =
 or directory'
 access("/lib/nss_dns.so.1",0)			 ERR#2 'No such file or =
 directory'
 access("/usr/lib/nss_dns.so.1",0)		 ERR#2 'No such file or =
 directory'
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 ioctl(4,TIOCGETA,0xffffcb00)			 ERR#25 'Inappropriate =
 ioctl for device'
 close(4)					 =3D 0 (0x0)
 madvise(0x80108a000,0x1000,0x5,0x89,0x7fffffffc2e0,0xffffffff) =3D 0 =
 (0x0)
 madvise(0x801044000,0x2000,0x5,0x43,0x7fffffffc2e0,0xffffffff) =3D 0 =
 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 geteuid()					 =3D 0 (0x0)
 open("/etc/spwd.db",O_RDONLY,00)		 =3D 4 (0x4)
 fcntl(4,F_SETFD,FD_CLOEXEC)			 =3D 0 (0x0)
 fstat(4,{ mode=3D-rw------- ,inode=3D54894,size=3D40960,blksize=3D40960 =
 }) =3D 0 (0x0)
 read(4,"\0\^F\^Ua\0\0\0\^B\0\0\^D\M-R\0"...,260) =3D 260 (0x104)
 pread(0x4,0x801090000,0x1000,0x6000,0x1,0x0)	 =3D 4096 (0x1000)
 stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- =
 ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0)
 stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- =
 ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0)
 pread(0x4,0x801091000,0x1000,0x4000,0x1,0x0)	 =3D 4096 (0x1000)
 stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- =
 ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0)
 open("/etc/group",O_RDONLY,0666)		 =3D 5 (0x5)
 stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- =
 ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0)
 stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- =
 ,inode=3D11094,size=3D323,blksize=3D4096 }) =3D 0 (0x0)
 fstat(5,{ mode=3D-rw-r--r-- ,inode=3D11080,size=3D570,blksize=3D4096 }) =
 =3D 0 (0x0)
 lseek(5,0x0,SEEK_CUR)				 =3D 0 (0x0)
 lseek(5,0x0,SEEK_SET)				 =3D 0 (0x0)
 read(5,"# $FreeBSD: release/9.0.0/etc/gr"...,4096) =3D 570 (0x23a)
 close(5)					 =3D 0 (0x0)
 stat("/usr/share/nls/C/libc.cat",0x7fffffffc920) ERR#2 'No such file or =
 directory'
 stat("/usr/share/nls/libc/C",0x7fffffffc920)	 ERR#2 'No such file or =
 directory'
 stat("/usr/local/share/nls/C/libc.cat",0x7fffffffc920) ERR#2 'No such =
 file or directory'
 stat("/usr/local/share/nls/libc/C",0x7fffffffc920) ERR#2 'No such file =
 or directory'
 ls: write(2,"ls: ",4)				 =3D 4 (0x4)
 snapshot: Bad file descriptorwrite(2,"snapshot: Bad file descriptor",29)	=
  =3D 29 (0x1d)
 
 write(2,"\n",1)					 =3D 1 (0x1)
 fstat(1,{ mode=3Dcrw--w---- ,inode=3D185,size=3D0,blksize=3D4096 }) =3D =
 0 (0x0)
 ioctl(1,TIOCGETA,0xffffbff0)			 =3D 0 (0x0)
 total 2
 write(1,"total 2\n",8)				 =3D 8 (0x8)
 lpathconf(0x7fffffffc9e0,0x40,0x0,0x0,0x7ff7fefdc3f0,0x7fffffffc43f) =
 ERR#22 'Invalid argument'
 =
 lpathconf(0x7fffffffc9e0,0x3b,0xffffffffffffffff,0x16,0x7ff7fefdc3f0,0x7ff=
 fffffc43f) ERR#22 'Invalid argument'
 clock_gettime(13,{1346858032.000000000 })	 =3D 0 (0x0)
 access("/etc/localtime",4)			 =3D 0 (0x0)
 open("/etc/localtime",O_RDONLY,0100000700)	 =3D 5 (0x5)
 fstat(5,{ mode=3D-r--r--r-- ,inode=3D16452,size=3D2434,blksize=3D4096 }) =
 =3D 0 (0x0)
 read(5,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) =3D 2434 (0x982)
 close(5)					 =3D 0 (0x0)
 issetugid(0x800db1fd1,0x8010d6000,0x0,0xa7,0x4e,0xc) =3D 0 (0x0)
 open("/usr/share/zoneinfo/posixrules",O_RDONLY,056) =3D 5 (0x5)
 fstat(5,{ mode=3D-r--r--r-- ,inode=3D545,size=3D3519,blksize=3D4096 }) =3D=
  0 (0x0)
 read(5,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) =3D 3519 (0xdbf)
 close(5)					 =3D 0 (0x0)
 madvise(0x80111e000,0xb000,0x5,0x11d,0x7fffffffb880,0x607050) =3D 0 =
 (0x0)
 madvise(0x8010d6000,0x5000,0x5,0xd5,0x7fffffffba70,0xffffffff) =3D 0 =
 (0x0)
 madvise(0x8010d2000,0x1000,0x5,0xd1,0x7fffffffba70,0xffffffff) =3D 0 =
 (0x0)
 madvise(0x801075000,0xb000,0x5,0x74,0x7fffffffba70,0xffffffff) =3D 0 =
 (0x0)
 dr-xr-xr-x  2 root  wheel  2 Jul 19 15:46 shares
 write(1,"dr-xr-xr-x  2 root  wheel  2 Jul"...,49) =3D 49 (0x31)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 =
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTER=
 M|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIG=
 XFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)		 =3D 0 (0x0)
 process exit, rval =3D 1
 
 Regards,
 Nikolai Schupbach
 

From: Martin Matuska <mm@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/156781: [zfs] zfs is losing the snapshot directory,
Date: Mon, 10 Sep 2012 12:25:49 +0200

 As I am not experiencing this problem on my systems, it would be great
 to create a way to reproduce it.
 E.g. a script that creates/deletes 1000 snapshots, mounts/unmounts them,
 etc.
 
 -- 
 Martin Matuska
 FreeBSD committer
 http://blog.vx.sk
 

From: John Kozubik <john@kozubik.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: RE: kern/156781: zfs is losing the snapshot directory
Date: Tue, 3 Sep 2013 09:12:15 -0700 (PDT)

 I can confirm that this still occurs, this time on 8.3-RELEASE on amd64.
 
 Exactly as described by the previous submitters - nothing special going on 
 on our system (no NFS, etc.) - just plain old snapshots being created and 
 removed and ... eventually some .zfs/snapshot directories just disappear.
 
 Two things:
 
 1) In our experience, enough retrying of ls/find/etc. of the missing 
 snapshot directory will eventually lock the system up
 
 2) non-critical severity is incorrect - snapshots are a critical feature 
 and this is a critical bug.
 
 Thanks.
>Unformatted:
