From nobody@FreeBSD.org  Mon Feb 21 09:24:13 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 235F9106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 21 Feb 2011 09:24:13 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 124338FC19
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 21 Feb 2011 09:24:13 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p1L9OCrn045209
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 21 Feb 2011 09:24:12 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p1L9OC9k045208;
	Mon, 21 Feb 2011 09:24:12 GMT
	(envelope-from nobody)
Message-Id: <201102210924.p1L9OC9k045208@red.freebsd.org>
Date: Mon, 21 Feb 2011 09:24:12 GMT
From: Matthias Andree <mandree@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: ZFS: cannot delete/unlink file from full volume -> ENOSPC
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         154930
>Category:       kern
>Synopsis:       [zfs] cannot delete/unlink file from full volume -> ENOSPC
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Feb 21 09:30:08 UTC 2011
>Closed-Date:    
>Last-Modified:  Sun Jun 19 14:20:09 UTC 2011
>Originator:     Matthias Andree
>Release:        8.2-PRERELEASE
>Organization:
FreeBSD
>Environment:
FreeBSD apollo.emma.line.org 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #61: Tue Feb 15 23:03:47 CET 2011     root@apollo.emma.line.org:/usr/obj/usr/src/sys/GENERIC  amd64

>Description:
On a 100% full ZFS volume, deleting files is not possible and results
in ENOSPC (no space on device).  I suppose this is due to the write-ahead
logging nature.

Workaround: truncate files to be deleted to make room for the logs.

Suggestion: consider implicitly truncating a file to be deleted if there
is not sufficient disk space to delete the file otherwise.
>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-ports-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Feb 22 03:26:44 UTC 2011 
Responsible-Changed-Why:  
Does not appear to be a ports PR.  Reclassify. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=154930 

From: Martin Matuska <mm@FreeBSD.org>
To: bug-followup@FreeBSD.org, mandree@FreeBSD.org
Cc:  
Subject: Re: kern/154930: [zfs] cannot delete/unlink file from full volume
 -&gt; ENOSPC
Date: Tue, 22 Feb 2011 15:30:34 +0100

 I was unable to reproduce your problem.
 
 But I was able to reproduce a different situation:
 - on a dataset with one or more snapshots I am unable to delete files
 (ENOSPC) if the dataset got full.
 
 If this is your case, then:
 - deleting files does not unlink them from the snapshot.
 - you must first delete a specific snapshot (or all snapshots linking
 the file) to free space.
 
 Cheers,
 mm

From: Matthias Andree <mandree@FreeBSD.org>
To: Martin Matuska <mm@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/154930: [zfs] cannot delete/unlink file from full volume
 -&gt; ENOSPC
Date: Tue, 22 Feb 2011 16:06:47 +0100

 Am 22.02.2011 15:30, schrieb Martin Matuska:
 > I was unable to reproduce your problem.
 >
 > But I was able to reproduce a different situation:
 > - on a dataset with one or more snapshots I am unable to delete files
 > (ENOSPC) if the dataset got full.
 >
 > If this is your case, then:
 > - deleting files does not unlink them from the snapshot.
 > - you must first delete a specific snapshot (or all snapshots linking
 > the file) to free space.
 
 Hi Martin,
 
 no snapshots were ever used on the zpools or zfs volumes -- I had 
 checked that previously. Only truncation of a 20 M file would allow me 
 to delete files.
 
 Best regards
 Matthias
 
 -- 
 Matthias Andree
 ports committer

From: <soralx@cydem.org>
To: <bug-followup@FreeBSD.org>, <mm@FreeBSD.org>
Cc: <mandree@FreeBSD.org>
Subject: Re: kern/154930: [zfs] cannot delete/unlink file from full volume
 -> ENOSPC
Date: Sun, 19 Jun 2011 06:54:26 -0700

 All,
 
  I encountered a similar snag, only worse, with no solution.
 
  The server has a 7 TB ZFS volume, which consists of 5 WD2001FASS
  connected to PERC 5/i in RAID5 (originally there were 4 disks, but the
  RAID was expanded by adding another disk and rebuilding). On June 12th
  (the date of OS rebuild, both world and kernel; the last update was on
  June 6), the pool was at v15, and had 12 GB free. On June 14, the pool
  already had 0 bytes free (found out from periodic daily run output
  email). It is unknown whether the FS was accessed at all during these
  two days; however, `find ./ -newerBt 2011-06-10 -print` returns just
  one small file dated 2011-06-11, and `find ./ -newermt 2011-06-10`
  returns ~20 files with total size <1 GB (BTW, these are backups, and
  nobody could have touched them, so it's a mystery why they have a
  recent modified time). So, the question is: where could have 12 GB
  suddenly disappear to?
 
  Further, on June 18th, the pool was mounted when a `zpool upgrade tst`
  command was issued. The upgrade to v28 succeeded, but then I found out
  that deleting files, large or small, is impossible:
 
 # rm ./qemu0.raw
 rm: ./qemu0.raw: No space left on device
 # truncate -s0 ./qemu0.raw
 truncate: ./qemu0.raw: No space left on device
 # cat /dev/null > ./qemu0.raw
 ./qemu0.raw: No space left on device.
 
  Snapshots, compression, or deduplication have never been used on the volume.
 
  Also, these messages appear in dmesg:
 Solaris: WARNING: metaslab_free_dva(): bad DVA 0:5978620460544
 Solaris: WARNING: metaslab_free_dva(): bad DVA 0:5978620461568
 Solaris: WARNING: metaslab_free_dva(): bad DVA 0:5993168926208
 
  Contrary to what the pool's name might suggest, this is not test storage,
  but has valuable data on it. Help!
 
 >Environment:
 System: FreeBSD cydem.org 8.2-STABLE FreeBSD 8.2-STABLE #0: Sun Jun 12 07:55:32 PDT 2011 soralx@cydem.org:/usr/obj/usr/src/sys/CYDEM amd64
 
 `df`:
 Filesystem    1K-blocks        Used      Avail Capacity  Mounted on
 tst           7650115271 7650115271          0   100%    /stor1-tst
 
 `zpool list`:
 NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
 tst   5.44T  5.36T  80.6G    98%  1.00x  ONLINE  -
 
 `zfs list`:
 NAME   USED  AVAIL  REFER  MOUNTPOINT
 tst   7.12T      0  7.12T  /stor1-tst
 
 `zpool status`: 
   pool: tst
  state: ONLINE
  scan: scrub in progress since Sat Jun 18 06:32:37 2011
     1.82T scanned out of 5.36T at 350M/s, 2h56m to go
     0 repaired, 34.02% done
 config:
 
         NAME        STATE     READ WRITE CKSUM
         tst         ONLINE       0     0     0
           mfid1     ONLINE       0     0     0
 
 errors: No known data errors
 [this completed with 'scrub repaired 0 in 5h47m with 0 errors' at 133% done,
  i.e. it scrubbed all 7.12 TB].
 
 
 -- 
 [SorAlx]  ridin' VN2000 Classic LT
>Unformatted:
