From mwlucas@blackhelicopters.org  Wed Nov  7 06:09:23 2001
Return-Path: <mwlucas@blackhelicopters.org>
Received: from blackhelicopters.org (geburah.blackhelicopters.org [209.69.178.18])
	by hub.freebsd.org (Postfix) with ESMTP id DB52E37B405
	for <FreeBSD-gnats-submit@freebsd.org>; Wed,  7 Nov 2001 06:09:22 -0800 (PST)
Received: (from mwlucas@localhost)
	by blackhelicopters.org (8.9.3/8.9.3) id JAA01617;
	Wed, 7 Nov 2001 09:09:22 -0500 (EST)
	(envelope-from mwlucas)
Message-Id: <200111071409.JAA01617@blackhelicopters.org>
Date: Wed, 7 Nov 2001 09:09:22 -0500 (EST)
From: mwlucas@blackhelicopters.org
Reply-To: mwlucas@blackhelicopters.org
To: FreeBSD-gnats-submit@freebsd.org
Subject: new FAQ: du/df
X-Send-Pr-Version: 3.2

>Number:         31821
>Category:       docs
>Synopsis:       new FAQ: du/df
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-doc
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Wed Nov 07 06:10:00 PST 2001
>Closed-Date:    Wed Nov 7 08:28:48 PST 2001
>Last-Modified:  Wed Nov  7 17:50:01 PST 2001
>Originator:     Michael Lucas
>Release:        FreeBSD 3.5-STABLE i386
>Organization:
None
>Environment:

yesterday's -doc tree

>Description:

People have asked several times about the difference between du and
df; specifically, the example cited below keeps happening.

>How-To-Repeat:

read -questions

>Fix:

*** en_US.ISO8859-1/books/faq/book.sgml-dist	Mon Nov  5 10:49:36 2001
--- en_US.ISO8859-1/books/faq/book.sgml	Wed Nov  7 09:05:22 2001
***************
*** 5869,5874 ****
--- 5869,5921 ----
        </qandaentry>
  
        <qandaentry>
+         <question id="du-vs-df">
+           <para>The <command>du</command> and <command>df</command>
+             commands show different amounts of disk space available?
+             What's going on?</para>
+         </question>
+ 
+         <answer>
+           <para>You need to understand what <command>du</command> and
+             <command>df</command> really do.  <command>du</command>
+             shows how much of the disk is designated <emphasis>used by
+             files</emphasis>.  <command>df</command> shows how much of
+             the disk is designated <emphasis>free</emphasis>.  While
+             they seem to be opposite, that isn't exactly true.</para>
+ 
+           <para>When a program has a file open, and you delete the
+             file, it isn't really deleted until the program releases
+             the file.  You can see this easily enough with a program
+             such as <command>more</command>.  If you delete a file
+             while using <command>more</command> on it,
+             <command>more</command> doesn't immediately choke and
+             complain that it cannot view the file.  You cannot access
+             the file from anywhere except that <command>more</command>
+             window.  If the file is large enough,
+             <command>df</command> will show that it is gone.  The
+             space the file took up is now designated free.
+             <command>du</command> will show that it is still there, as
+             the space the file uses is still technically in use.  Once
+             you end the <command>more</command> session,
+             <command>du</command> and <command>df</command> will
+             agree.</para>
+ 
+           <para>Note that softupdates can delay the freeing of disk
+             space; you might need to wait up to 30 seconds for the
+             change to be visible!</para>
+ 
+           <para>This situation is common on web servers.  Many people
+             set up a FreeBSD web server and forget to rotate the log
+             files.  The access log fills up <filename>/var</filename>.
+             The new administrator deletes the file, but the system
+             still complains that the partition is full.  Stopping and
+             restarting the web server program would free the file,
+             allowing the system to release the disk space.  To prevent
+             this from happening, set up &man.newsyslog.8;.</para>
+         </answer>
+       </qandaentry>
+ 
+       <qandaentry>
          <question id="add-swap-space">
            <para>How can I add more swap space?</para>
          </question>
>Release-Note:
>Audit-Trail:

From: Dima Dorfman <dima@trit.org>
To: mwlucas@blackhelicopters.org
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: docs/31821: new FAQ: du/df 
Date: Wed, 07 Nov 2001 15:06:27 +0000

 mwlucas@blackhelicopters.org wrote:
 > *** en_US.ISO8859-1/books/faq/book.sgml-dist	Mon Nov  5 10:49:36 2001
 > --- en_US.ISO8859-1/books/faq/book.sgml	Wed Nov  7 09:05:22 2001
 > ***************
 > *** 5869,5874 ****
 > --- 5869,5921 ----
 >         </qandaentry>
 >   
 >         <qandaentry>
 > +         <question id="du-vs-df">
 > +           <para>The <command>du</command> and <command>df</command>
 > +             commands show different amounts of disk space available?
 
 Is this first sentence really a question (i.e., does the question mark
 belong there)?
 
 > +             What's going on?</para>
 > +         </question>
 > + 
 > +         <answer>
 > +           <para>You need to understand what <command>du</command> and
 > +             <command>df</command> really do.  <command>du</command>
 > +             shows how much of the disk is designated <emphasis>used by
 > +             files</emphasis>.  <command>df</command> shows how much of
 > +             the disk is designated <emphasis>free</emphasis>.  While
 > +             they seem to be opposite, that isn't exactly true.</para>
 
 I think the distinction isn't what they do, but how they do it; if one
 knows how much space is in use and the size of the disk, one can
 compute how much space is free.  On the other hand, how they do it
 matters: du traverses the directory tree and counts how much space is
 in use by each file.  If a file has no directory entry, it won't be
 counted.  On the other hand, df just asks the filesystem how much
 space it thinks is in use, and the filesystem doesn't care whether
 files have directory entires or not, just as long as they take up
 space on the disk.
 
 > + 
 > +           <para>When a program has a file open, and you delete the
 > +             file, it isn't really deleted until the program releases
 > +             the file.  You can see this easily enough with a program
 > +             such as <command>more</command>.  If you delete a file
 > +             while using <command>more</command> on it,
 > +             <command>more</command> doesn't immediately choke and
 > +             complain that it cannot view the file.  You cannot access
 > +             the file from anywhere except that <command>more</command>
 > +             window.
 
 This is a very good example.
 
 > If the file is large enough,
 
 This shouldn't (and doesn't) matter.  People just tend to notice when
 they're missing 18M instead of 18K :-).
 
 > +             <command>df</command> will show that it is gone.  The
 > +             space the file took up is now designated free.
 > +             <command>du</command> will show that it is still there, as
 > +             the space the file uses is still technically in use.  Once
 > +             you end the <command>more</command> session,
 > +             <command>du</command> and <command>df</command> will
 > +             agree.</para>
 
 This is backwards.  'du' typically reports less space is in use, not
 df (if you wrote this looking at the recent thread on -stable, note
 that the originator misnamed the output; what he claimed was from du
 was from df, and vice versa).  The reason is that 'rm' removes the
 file's directory entry, so 'du' never sees it, but 'df' does.  See
 above where I explained why 'df' seems it.
 
 Here's a small example to demonstrate my point.  (Note that I cleaned
 up the `script' output a little to make it more readable.)
 
 Script started on Wed Nov  7 14:50:19 2001
 root@mirage# mdmfs -s 32m md /mnt
 root@mirage# cd /mnt
 root@mirage# dd if=/dev/random of=fish bs=1m count=20
 20+0 records in
 20+0 records out
 20971520 bytes transferred in 27.938557 secs (750630 bytes/sec)
 root@mirage# cat>open.c
 #include <fcntl.h>
 #include <unistd.h>
 
 int
 main(void)
 {
 int d = open("fish", O_RDONLY);
 pause();
 }
 ^D
 root@mirage# make open
 cc -O -pipe   open.c  -o open
 root@mirage# ./open &
 [1] 20447
 root@mirage# df
 Filesystem          1K-blocks     Used    Avail Capacity  Mounted on
 /dev/ad0s1a            254063    68466   165272    29%    /
 devfs                       1        1        0   100%    /dev
 /dev/ad0s1f           2554468   600470  1749641    26%    /usr
 /dev/ad0s1e           1016303   226578   708421    24%    /var
 /dev/md0c               32476    20520     9360    69%    /mnt
 root@mirage# du
 20520	.
 root@mirage# rm fish
 root@mirage# df
 Filesystem          1K-blocks     Used    Avail Capacity  Mounted on
 /dev/ad0s1a            254063    68466   165272    29%    /
 devfs                       1        1        0   100%    /dev
 /dev/ad0s1f           2554468   600470  1749641    26%    /usr
 /dev/ad0s1e           1016303   226578   708421    24%    /var
 /dev/md0c               32476    20520     9360    69%    /mnt
 root@mirage# du
 16	.
 root@mirage# killall open
 [1]  + Terminated                    ./open
 root@mirage# sync
 root@mirage# df
 Filesystem          1K-blocks     Used    Avail Capacity  Mounted on
 /dev/ad0s1a            254063    68466   165272    29%    /
 devfs                       1        1        0   100%    /dev
 /dev/ad0s1f           2554468   600470  1749641    26%    /usr
 /dev/ad0s1e           1016303   226578   708421    24%    /var
 /dev/md0c               32476       16    29864     0%    /mnt
 root@mirage# du
 16	.
 root@mirage# exit
 
 Script done on Wed Nov  7 14:52:01 2001
 
 Notes:
 
  - The `open' program just opens the file and keeps it open until it's
    killed.  I didn't use `more' as you did in your example because I
    wouldn't be able to show when I started and killed it (it needs a
    tty).
 
  - The `sync' command isn't instantaneous; about 10 seconds passed
    between the sync and the subsequent df which shows the space is free.
    This is because of softupdates.

From: Michael Lucas <mwlucas@blackhelicopters.org>
To: Dima Dorfman <dima@trit.org>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: docs/31821: new FAQ: du/df
Date: Wed, 7 Nov 2001 11:10:34 -0500

 --3V7upXqbjpZ4EhLz
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 On Wed, Nov 07, 2001 at 03:06:27PM +0000, Dima Dorfman wrote:
 > This is backwards.  'du' typically reports less space is in use, not
 > df (if you wrote this looking at the recent thread on -stable, note
 > that the originator misnamed the output; what he claimed was from du
 > was from df, and vice versa).
 
 Aha!  That confused me, but I'm not going to argue with actual output.
 
 You've gone to a lot of trouble to prove me wrong, but I'm a tech
 writer.  I assume that I will be proven wrong.  :)
 
 Here's another patch, addressing your points.
 
 ==ml
 
 -- 
 Michael Lucas
 mwlucas@blackhelicopters.org
 http://www.blackhelicopters.org/~mwlucas/
 Big Scary Daemons: http://www.oreillynet.com/pub/q/Big_Scary_Daemons
 
 --3V7upXqbjpZ4EhLz
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename=du-vs-df
 
 *** en_US.ISO8859-1/books/faq/book.sgml-dist	Mon Nov  5 10:49:36 2001
 --- en_US.ISO8859-1/books/faq/book.sgml	Wed Nov  7 11:09:10 2001
 ***************
 *** 5869,5874 ****
 --- 5869,5929 ----
         </qandaentry>
   
         <qandaentry>
 +         <question id="du-vs-df">
 +           <para>The <command>du</command> and <command>df</command>
 +             commands show different amounts of disk space available.
 +             What's going on?</para>
 +         </question>
 + 
 +         <answer>
 +           <para>You need to understand what <command>du</command> and
 +             <command>df</command> really do.  <command>du</command>
 +             goes through the directory tree, measures how large each
 +             file is, and presents the totals.  <command>df</command>
 +             just asks the filesystem how much space it has left.  They
 +             seem to be the same thing, but a file without a directory
 +             entry will affect <command>df</command> but not
 +             <command>du</command>.</para>
 + 
 +           <para>When a program is using a file, and you delete the
 +             file, the file isn't really removed from the filesystem
 +             until the program stops using it.  The file is immediately
 +             deleted from the directory listing, however.  You can see
 +             this easily enough with a program such as
 +             <command>more</command>.  Assume you have a file large
 +             enough that its presence affects the output of
 +             <command>du</command> and <command>df</command>.  (Since
 +             disks can be so large today, this might be a
 +             <emphasis>very</emphasis> large file!)  If you delete this
 +             file while using <command>more</command> on it,
 +             <command>more</command> doesn't immediately choke and
 +             complain that it cannot view the file.  The entry is
 +             simply removed from the directory so no other program or
 +             user can access it.  <command>du</command> shows that it
 +             is gone -- it has walked the directory tree and the file
 +             isn't listed.  <command>df</command> shows that it is
 +             still there, as the filesystem knows that
 +             <command>more</command> is still using that space.  Once
 +             you end the <command>more</command> session,
 +             <command>du</command> and <command>df</command> will
 +             agree.</para>
 + 
 +           <para>Note that softupdates can delay the freeing of disk
 +             space; you might need to wait up to 30 seconds for the
 +             change to be visible!</para>
 + 
 +           <para>This situation is common on web servers.  Many people
 +             set up a FreeBSD web server and forget to rotate the log
 +             files.  The access log fills up <filename>/var</filename>.
 +             The new administrator deletes the file, but the system
 +             still complains that the partition is full.  Stopping and
 +             restarting the web server program would free the file,
 +             allowing the system to release the disk space.  To prevent
 +             this from happening, set up &man.newsyslog.8;.</para>
 +         </answer>
 +       </qandaentry>
 + 
 +       <qandaentry>
           <question id="add-swap-space">
             <para>How can I add more swap space?</para>
           </question>
 
 --3V7upXqbjpZ4EhLz--
State-Changed-From-To: open->closed 
State-Changed-By: dd 
State-Changed-When: Wed Nov 7 08:28:48 PST 2001 
State-Changed-Why:  
Patch applied, thanks. 

http://www.FreeBSD.org/cgi/query-pr.cgi?pr=31821 

From: Cyrille Lefevre <clefevre@citeweb.net>
To: mwlucas@blackhelicopters.org
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: docs/31821: new FAQ: du/df
Date: Thu, 8 Nov 2001 02:14:57 +0100 (CET)

 mwlucas@blackhelicopters.org wrote:
 [snip]
 > +           <para>When a program has a file open, and you delete the
 > +             file, it isn't really deleted until the program releases
 > +             the file.  You can see this easily enough with a program
 > +             such as <command>more</command>.  If you delete a file
 > +             while using <command>more</command> on it,
 > +             <command>more</command> doesn't immediately choke and
 > +             complain that it cannot view the file.  You cannot access
 > +             the file from anywhere except that <command>more</command>
 > +             window.  If the file is large enough,
 > +             <command>df</command> will show that it is gone.  The
                          ^^ du
 > +             space the file took up is now designated free.
 > +             <command>du</command> will show that it is still there, as
                          ^^ df
 > +             the space the file uses is still technically in use.  Once
 > +             you end the <command>more</command> session,
 > +             <command>du</command> and <command>df</command> will
 > +             agree.</para>
 > + 
 > +           <para>Note that softupdates can delay the freeing of disk
 > +             space; you might need to wait up to 30 seconds for the
 > +             change to be visible!</para>
 > + 
 > +           <para>This situation is common on web servers.  Many people
 > +             set up a FreeBSD web server and forget to rotate the log
 > +             files.  The access log fills up <filename>/var</filename>.
 > +             The new administrator deletes the file, but the system
 > +             still complains that the partition is full.  Stopping and
 > +             restarting the web server program would free the file,
 > +             allowing the system to release the disk space.  To prevent
 > +             this from happening, set up &man.newsyslog.8;.</para>
                                     ^
                 never delete such file, but empty it by copying /dev/null
                 to it (cp /dev/null logfile) or
 > +         </answer>
 > +       </qandaentry>
 
 Cyrille.
 -- 
 Cyrille Lefevre                 mailto:clefevre@citeweb.net

From: Wouter Van Hemel <wouter@fort-knox.rave.org>
To: <freebsd-gnats-submit@FreeBSD.org>,
	<mwlucas@blackhelicopters.org>
Cc:  
Subject: Re: docs/31821: new FAQ: du/df
Date: Thu, 8 Nov 2001 02:44:24 +0100 (CET)

 [...] but empty it by copying /dev/null to it (cp /dev/null logfile) [...]
 
 Wouldn't it be easier just HUP syslogd?
 
 
 
>Unformatted:
