From serg@ftp.citkit.ru  Tue Nov 30 23:09:38 2004
Return-Path: <serg@ftp.citkit.ru>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 9644C16A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 30 Nov 2004 23:09:38 +0000 (GMT)
Received: from ftp.citkit.ru (cit-forum0.rmt.ru [194.67.81.86])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 5B6CF43D31
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 30 Nov 2004 23:09:37 +0000 (GMT)
	(envelope-from serg@ftp.citkit.ru)
Received: from ftp.citkit.ru (localhost [127.0.0.1])
	by ftp.citkit.ru (8.13.1/8.13.1) with ESMTP id iAUN9Xih038657
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 1 Dec 2004 02:09:33 +0300 (MSK)
	(envelope-from serg@ftp.citkit.ru)
Received: (from serg@localhost)
	by ftp.citkit.ru (8.13.1/8.13.1/Submit) id iAUN9XY1038656;
	Wed, 1 Dec 2004 02:09:33 +0300 (MSK)
	(envelope-from serg)
Message-Id: <200411302309.iAUN9XY1038656@ftp.citkit.ru>
Date: Wed, 1 Dec 2004 02:09:33 +0300 (MSK)
From: Sergey Salnikov <serg@citforum.ru>
Reply-To: Sergey Salnikov <serg@citforum.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: [patch] [2TB] du doesn't handle sizes >1TB
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         74567
>Category:       bin
>Synopsis:       [2TB] [patch] du doesn't handle sizes >1TB
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Nov 30 23:10:12 GMT 2004
>Closed-Date:    Mon Mar 26 20:15:20 GMT 2007
>Last-Modified:  Mon Mar 26 20:15:20 GMT 2007
>Originator:     Sergey Salnikov <serg@citforum.ru>
>Release:        FreeBSD 5.3-RELEASE i386
>Organization:
CITForum
>Environment:
System: FreeBSD ftp.citkit.ru 5.3-RELEASE FreeBSD 5.3-RELEASE #0: Sun Nov 21 02:43:28 MSK 2004 serg@:/usr/obj/usr/src/sys/CITKIT i386
>Description:
	There is an integer overflow in du when running it on a 
	directory larger than 1TB.
>How-To-Repeat:
	Run du on any >1TB directory on a 32-bit machine.
	It reports wrong size.
>Fix:
	Here goes the patch. Modified du allocates an int64_t for
	every subdirectory and stores the size there and not in
	fts_number.
	
	Maybe fts_number just has to be 64-bit? It will make
	things faster (no need for extra malloc), but
	involves hacking libc.

--- du-2TB-patch begins here ---
--- du.c.orig	Wed Jul 28 20:03:12 2004
+++ du.c	Wed Dec  1 00:20:29 2004
@@ -82,7 +82,8 @@
 {
 	FTS		*fts;
 	FTSENT		*p;
-	long		blocksize, savednumber = 0;
+	long		blocksize;
+	int64_t		savednumber = 0;
 	int		ftsoptions;
 	int		listall;
 	int		depth;
@@ -219,24 +220,31 @@
 			case FTS_D:			/* Ignore. */
 				if (ignorep(p))
 					fts_set(fts, p, FTS_SKIP);
+				else {
+					if ((!p->fts_parent->fts_pointer
+					     && !(p->fts_parent->fts_pointer = calloc(1, sizeof(int64_t))))
+					    || !(p->fts_pointer = calloc(1, sizeof(int64_t))))
+						err(1, "calloc");
+				}
 				break;
 			case FTS_DP:
 				if (ignorep(p))
 					break;
 
-				p->fts_parent->fts_number +=
-				    p->fts_number += p->fts_statp->st_blocks;
+				*(int64_t *)p->fts_parent->fts_pointer +=
+				    *(int64_t *)p->fts_pointer += p->fts_statp->st_blocks;
 
 				if (p->fts_level <= depth) {
 					if (hflag) {
-						(void) prthumanval(howmany(p->fts_number, blocksize));
+						(void) prthumanval(howmany(*(int64_t *)p->fts_pointer, blocksize));
 						(void) printf("\t%s\n", p->fts_path);
 					} else {
-					(void) printf("%ld\t%s\n",
-					    howmany(p->fts_number, blocksize),
+					(void) printf("%lld\t%s\n",
+					    howmany(*(int64_t *)p->fts_pointer, blocksize),
 					    p->fts_path);
 					}
 				}
+				free(p->fts_pointer);
 				break;
 			case FTS_DC:			/* Ignore. */
 				break;
@@ -265,9 +273,9 @@
 					}
 				}
 
-				p->fts_parent->fts_number += p->fts_statp->st_blocks;
+				*(int64_t *)p->fts_parent->fts_pointer += p->fts_statp->st_blocks;
 		}
-		savednumber = p->fts_parent->fts_number;
+		savednumber = *(int64_t *)p->fts_parent->fts_pointer;
 	}
 
 	if (errno)
@@ -278,7 +286,7 @@
 			(void) prthumanval(howmany(savednumber, blocksize));
 			(void) printf("\ttotal\n");
 		} else {
-			(void) printf("%ld\ttotal\n", howmany(savednumber, blocksize));
+			(void) printf("%lld\ttotal\n", howmany(savednumber, blocksize));
 		}
 	}
 
--- du-2TB-patch ends here ---


>Release-Note:
>Audit-Trail:

From: Brooks Davis <brooks@one-eyed-alien.net>
To: Sergey Salnikov <serg@citforum.ru>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/74567: [patch] [2TB] du doesn't handle sizes >1TB
Date: Wed, 1 Dec 2004 08:07:10 -0800

 On Wed, Dec 01, 2004 at 02:09:33AM +0300, Sergey Salnikov wrote:
 > 	Here goes the patch. Modified du allocates an int64_t for
 > 	every subdirectory and stores the size there and not in
 > 	fts_number.
 
 It seems like off_t would be more appropriate.  Using hard coded types
 like long is what got us in to this mess in the first place. :) Have you
 done any testing to see how expensive this extra malloc is?  It might be
 cheaper to use fts_number until it overflows and then start doing
 allocations.  That would certaintly use more code though.
 
 > 	Maybe fts_number just has to be 64-bit? It will make
 > 	things faster (no need for extra malloc), but
 > 	involves hacking libc.
 
 fts_number can't change size without a library version bump, but I think
 you might be able to add a new entry (assuming there isn't any broken
 code that uses sizeof(FTSENT).
 
 -- Brooks

From: Sergey Salnikov <serg@www1.citforum.ru>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: bin/74567: [patch] [2TB] du doesn't handle sizes >1TB
Date: Fri, 03 Dec 2004 00:41:33 +0300

 I've found a serious bug in my patch - malloc was called in the
 wrong place, and du dumped core when called with no directories, but
 an ordinary file. Corrected.
 
 >It seems like off_t would be more appropriate.
 Really I simply looked at the "prthumanval(int64_t bytes)" line and
 thought that using the same typedef everywhere would be the right
 thing. But off_t is better for that, you're right.
 
 >Have you done any testing to see how expensive this extra malloc is?
 Little. "time du -sm /pub/FreeBSD" on our FTP server showed the same
 times for original and patched du, and 90% is anyway "sys".
 
 It doesn't free fts_number in the virtual top FTS_ENT. It's not a
 problem since the cycle is run just once within a process, but maybe
 that last free() should be added for the sake of cleanlyness.
 
 A new patch follows.
 
 --- du-2TB-patch-v2 begins here ---
 --- du.c.orig   Wed Jul 28 20:03:12 2004
 +++ du.c        Thu Dec  2 00:54:34 2004
 @@ -72,7 +72,7 @@
 
  static int     linkchk(FTSENT *);
  static void    usage(void);
 -void           prthumanval(int64_t);
 +void           prthumanval(off_t);
  void           ignoreadd(const char *);
  void           ignoreclean(void);
  int            ignorep(FTSENT *);
 @@ -82,7 +82,8 @@
  {
         FTS             *fts;
         FTSENT          *p;
 -       long            blocksize, savednumber = 0;
 +       long            blocksize;
 +       off_t           savednumber = 0;
         int             ftsoptions;
         int             listall;
         int             depth;
 @@ -215,8 +216,13 @@
                 err(1, "fts_open");
 
         while ((p = fts_read(fts)) != NULL) {
 +               if ((!p->fts_parent->fts_pointer
 +                    && !(p->fts_parent->fts_pointer = calloc(1, sizeof(off_t)))))
 +                       err(1, "calloc");
                 switch (p->fts_info) {
                         case FTS_D:                     /* Ignore. */
 +                               if (!(p->fts_pointer = calloc(1, sizeof(off_t))))
 +                                       err(1, "calloc");
                                 if (ignorep(p))
                                         fts_set(fts, p, FTS_SKIP);
                                 break;
 @@ -224,19 +230,20 @@
                                 if (ignorep(p))
                                         break;
 
 -                               p->fts_parent->fts_number +=
 -                                   p->fts_number += p->fts_statp->st_blocks;
 +                               *(off_t *)p->fts_parent->fts_pointer +=
 +                                   *(off_t *)p->fts_pointer +=
 p->fts_statp->st_blocks;
 
                                 if (p->fts_level <= depth) {
                                         if (hflag) {
 -                                               (void)
 prthumanval(howmany(p->fts_number, blocksize));
 +                                               (void)
 prthumanval(howmany(*(off_t *)p->fts_pointer, blocksize));
                                                 (void) printf("\t%s\n",
 p->fts_path);
                                         } else {
 -                                       (void) printf("%ld\t%s\n",
 -                                           howmany(p->fts_number, blocksize),
 +                                       (void) printf("%lld\t%s\n",
 +                                           howmany(*(off_t *)p->fts_pointer,
 blocksize),
                                             p->fts_path);
                                         }
                                 }
 +                               free(p->fts_pointer);
                                 break;
                         case FTS_DC:                    /* Ignore. */
                                 break;
 @@ -265,9 +272,9 @@
                                         }
                                 }
 
 -                               p->fts_parent->fts_number +=
 p->fts_statp->st_blocks;
 +                               *(off_t *)p->fts_parent->fts_pointer +=
 p->fts_statp->st_blocks;
                 }
 -               savednumber = p->fts_parent->fts_number;
 +               savednumber = *(off_t *)p->fts_parent->fts_pointer;
         }
 
         if (errno)
 @@ -278,7 +285,7 @@
                         (void) prthumanval(howmany(savednumber, blocksize));
                         (void) printf("\ttotal\n");
                 } else {
 -                       (void) printf("%ld\ttotal\n", howmany(savednumber,
 blocksize));
 +                       (void) printf("%lld\ttotal\n", howmany(savednumber,
 blocksize));
                 }
         }
 
 @@ -421,7 +428,7 @@
  }
 
  void
 -prthumanval(int64_t bytes)
 +prthumanval(off_t bytes)
  {
         char buf[5];
 
 --- du-2TB-patch-v2 ends here ---

From: Bruce Evans <bde@zeta.org.au>
To: Sergey Salnikov <serg@www1.citforum.ru>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: bin/74567: [patch] [2TB] du doesn't handle sizes >1TB
Date: Fri, 3 Dec 2004 22:04:25 +1100 (EST)

 On Thu, 2 Dec 2004, Sergey Salnikov wrote:
 
 >  >It seems like off_t would be more appropriate.
 >  Really I simply looked at the "prthumanval(int64_t bytes)" line and
 >  thought that using the same typedef everywhere would be the right
 >  thing. But off_t is better for that, you're right.
 
 No, off_t is worse for that.  off_t is a signed integer integer type
 suitable for representing file offsets.  It must be signed so that it
 can represent negative offsets and have a range twice as large as is
 needed for file sizes.  Using it to represent file sizes would be bogus
 since file sizes aren't offsets but would always work since off_t must
 be able to represent the offset from the beginning of a file to the
 end of the same file.  Using it to represent sums of file sizes sizs
 would be more bogus since sums may be much larger than their terms;
 in particular, it would break on systems where off_t is 32 bits but
 file systems are larger than 2^31-1 bytes.  Using it in du would be
 even more bogus since du needs to represent sums of block counts, not
 sums of file sizes.
 
 POSIX.1-2001 requires the existence of a signed integer type blkcnt_t
 which is "Used for file block counts".  It must be signed for backwards
 compatibility.  This is still not implemented in FreeBSD.  It is fuzzily
 specified in POSIX.  The spec doesn't require it to be usable for block
 counts, but it requires st_blocks in struct stat to have type blkcnt_t
 so blkcnt_t needs to be able to represent all file block counts for
 stat() to actually work.  It is fairly clear that it is not intended
 to to work for more than single files.
 
 POSIX.1-2001 requires the existence of an unsigned integer type fsblkcnt_t
 which is "Used for file system block counts".  This is implemented in
 FreeBSD but not used in the main place where something like it is needed
 (statfs()).  It must be unsigned for backwards compatibility with SYSV,
 but this makes it unsuitable for use in FreeBSD since file system block
 counts can be negative (for f_bavail).  statvfs() is unsuitable for the
 same reason.  Some BSDs have deprecated statfs() in favour of statvfs()
 despite the latter's unsuitability, but statfs() is still the primary
 interface that reports block counts in FreeBSD.  statfs() avoids some
 bugs by not using fsblkcnt_t, but has its own sign extension bugs from
 excessive use of unsigned types.
 
 Using fsblkcnt_t in du would be bogus since it is not intended to work
 for more than single file systems.
 
 Using a type related to block counts for fts_number would be bogus because
 fts_number needs to represent generic values.  fts_number needs to have a
 signed arithmetic type since generic values may be negative.  Using long
 for it was almost correct until the C standard broke the promise that
 long is the largest integer type.  Using double would have been better,
 at least on machines with IEEE floating point so that exact representability
 of all integers between about -2^53 and 2^53 is guaranteed.  Now that the C
 standard promises that intmax_t is the largest signed integer type, intmax_t
 is almost the correct type for fts_number.  Using double would still be
 safer since it has a much larger range (though less precision for large
 values).  intmax_t goes up to about 2^63 which should be enough for anyone,
 but 2^53 should also be enough for anyone.  The reason to use doubles is
 that bugs may cause preposterously large values (> 2^63) and floating
 point won't hide the bugs by truncating mod 2^63.  In theory, overflow
 can be detected for signed integer arithmetic, but no one ever turns on
 the overflow detection and excessive use of unsigned types defeats it.
 
 >  A new patch follows.
 >  ...
 >
 >  --- du-2TB-patch-v2 begins here ---
 >  --- du.c.orig   Wed Jul 28 20:03:12 2004
 >  +++ du.c        Thu Dec  2 00:54:34 2004
 >  @@ -72,7 +72,7 @@
 >
 >   static int     linkchk(FTSENT *);
 >   static void    usage(void);
 >  -void           prthumanval(int64_t);
 >  +void           prthumanval(off_t);
 
 Using int64_t in prthumanval() is as correct as possible, since
 prthumanva() is just a wrapper for humanize_number() and humanize_number()
 takes an int64_t due to previous poor choice of types.
 
 >  [... style bugs deleted ...]
 >  -                                       (void) printf("%ld\t%s\n",
 >  -                                           howmany(p->fts_number, blocksize),
 >  +                                       (void) printf("%lld\t%s\n",
 >  +                                           howmany(*(off_t *)p->fts_pointer,
 >  blocksize),
 >                                              p->fts_path);
 
 %lld is for printing long long's.  Using of for printing foo_t's is
 logically wrong and potentially gives runtime errors.  This bug is
 detected at compile time on FreeBSD's 64-bit arches because both int64_t
 and off_t are plain long (!= long long) on these arches.
 
 Another reason to always replace long by intmax_t in changes of this type
 is that by getting it right the first time you don't have to change lots
 of printfs several times.  FreeBSD already has too many %qd's and %lld's
 from intermediate changes before intmax_t became standard.  In many cases,
 compile-time detection of the type being printed being different from
 quad_t or long long, respectively, is defeated by casting the arg to
 match the format.
 
 Bruce
State-Changed-From-To: open->closed 
State-Changed-By: remko 
State-Changed-When: Mon Mar 26 20:15:16 UTC 2007 
State-Changed-Why:  
This had been comitted in du.c revision 1.37 on Fri Jan 7 00:12:24 2005 
UTC, resolve the ticket. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=74567 
>Unformatted:
