From dmitry@atlantis.dp.ua  Tue Aug 30 23:50:09 2005
Return-Path: <dmitry@atlantis.dp.ua>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id F0CEC16A41F
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 30 Aug 2005 23:50:08 +0000 (GMT)
	(envelope-from dmitry@atlantis.dp.ua)
Received: from postman.atlantis.dp.ua (postman.atlantis.dp.ua [193.108.47.1])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 167EB43D49
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 30 Aug 2005 23:50:07 +0000 (GMT)
	(envelope-from dmitry@atlantis.dp.ua)
Received: from smtp.atlantis.dp.ua (smtp.atlantis.dp.ua [193.108.46.231])
	by postman.atlantis.dp.ua (8.13.1/8.13.1) with ESMTP id j7UNnvv8024039
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 31 Aug 2005 02:49:57 +0300 (EEST)
	(envelope-from dmitry@atlantis.dp.ua)
Message-Id: <20050831024905.M23916@atlantis.atlantis.dp.ua>
Date: Wed, 31 Aug 2005 02:49:57 +0300 (EEST)
From: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
To: FreeBSD-gnats-submit@freebsd.org
Subject: panic: wrong dirclust using msdosfs in RELENG_6

>Number:         85503
>Category:       kern
>Synopsis:       [panic] wrong dirclust using msdosfs in RELENG_6
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    trhodes
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Aug 30 23:50:25 GMT 2005
>Closed-Date:    Fri Oct 14 23:20:20 GMT 2005
>Last-Modified:  Sat Oct 15 10:40:17 GMT 2005
>Originator:     Dmitry Pryanishnikov
>Release:        FreeBSD 6.0-BETA3 i386
>Organization:
Atlantis ISP
>Environment:
System:

 	FreeBSD 6.0-BETA3 #0: Mon Aug 22 22:59:46 UTC 2005
 	root@harlow.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386

 	Large enough (24Gb) FAT32 slice.

>Description:

     RELENG_6 panics during 'du' run against large FAT32 slice with many
directories. This is the specific regression in RELENG_6 against earlier
code (e.g. RELENG_5). Here is the crash dump analysis:

Unread portion of the kernel message buffer:
panic: wrong dirclust

(kgdb) where
...
#12 0xc06321f7 in panic (fmt=0x282 <Address 0x282 out of bounds>)
     at /usr/src/sys/kern/kern_shutdown.c:537
#13 0xc05eebaf in deget (pmp=0xc14d9000, dirclust=673396, diroffset=0,
     depp=0xc6ffda00) at /usr/src/sys/fs/msdosfs/msdosfs_denode.c:142
#14 0xc05f143d in msdosfs_lookup (ap=0x0)
     at /usr/src/sys/fs/msdosfs/msdosfs_lookup.c:534
#15 0xc07fe1da in VOP_CACHEDLOOKUP_APV (vop=0x0, a=0xc6ffda7c)
     at vnode_if.c:150
#16 0xc067b66e in vfs_cache_lookup (ap=0x0) at vnode_if.h:82
#17 0xc07fe123 in VOP_LOOKUP_APV (vop=0xc08b55e0, a=0xc6ffdb14)
     at vnode_if.c:99
#18 0xc067f53e in lookup (ndp=0xc6ffdba0) at vnode_if.h:56
#19 0xc067ef06 in namei (ndp=0xc6ffdba0) at /usr/src/sys/kern/vfs_lookup.c:201
#20 0xc068aea7 in kern_lstat (td=0xc1376c00, path=0x0, pathseg=UIO_USERSPACE,
     sbp=0xc6ffdc74) at /usr/src/sys/kern/vfs_syscalls.c:2102
#21 0xc068ae43 in lstat (td=0xc1376c00, uap=0xc6ffdd04)
     at /usr/src/sys/kern/vfs_syscalls.c:2086
...
(kgdb) fr 13
#13 0xc05eebaf in deget (pmp=0xc14d9000, dirclust=673396, diroffset=0,
     depp=0xc6ffda00) at /usr/src/sys/fs/msdosfs/msdosfs_denode.c:142
142                     KASSERT((*depp)->de_dirclust == dirclust, ("wrong dirclust"));
(kgdb) list
...
140             if (nvp != NULL) {
141                     *depp = VTODE(nvp);
142                     KASSERT((*depp)->de_dirclust == dirclust, ("wrong dirclust"));

Looking into the source (sys/fs/msdosfs/msdosfs_denode.c, function deget)
gives the following:

         uint64_t inode;		// Our "inode" number must have 64 bits
 	...
 	inode = pmp->pm_bpcluster * dirclust + diroffset;

This is the apparent place of the error: all members of right-hand
expression are u_long (32 bits on i386), so inode will always be
evaluated as 32-bit integer. Check this:

(kgdb) print dirclust
$2 = 673396
(kgdb) print pmp->pm_bpcluster
$3 = 16384
(kgdb) print diroffset
$4 = 0
(kgdb) print inode
$5 = 2442985472

Actually 16384 * 673396 + 0 should give us 11032920064. So vfs_hash_get()
gave us another vnode with the same low-order 32 bits of inode:

(kgdb) print **depp
$7 = {de_vnode = 0xc1805aa0, de_flag = 0, de_dev = 0x0, de_dirclust = 149108,
   de_diroffset = 0, de_fndoffset = 0, de_fndcnt = 0, de_refcnt = 1,
   de_pmp = 0xc14d9000, de_Name = ".          ", de_Attributes = 16 '\020',
   de_LowerCase = 0 '\0', de_CHun = 47 '/', de_CTime = 39991,
   de_CDate = 12972, de_ADate = 12972, de_MTime = 39991, de_MDate = 12972,
   de_StartCluster = 149108, de_FileSize = 16384, de_fc = {{fc_frcn = 0,
     fc_fsrcn = 149108}, {fc_frcn = 0, fc_fsrcn = 149108}},
   de_modrev = 140731985215, de_lockf = 0x0, de_inode = 2442985472}

Original directory cluster gives (in 32-bit expression):

(kgdb) print /x 673396*16384
$8 = 0x919d0000

This (incorrect) one gives the same:

(kgdb) print /x 149108*16384
$9 = 0x919d0000

>How-To-Repeat:

  Just install 6.0-BETA3 on i386 machine, mount large FAT32 partition with many
directories and run 'du' against this partition.

>Fix:

  The primary bug (evaluation of the inode) can easily be fixed using cast:

 	inode = (uint64_t) pmp->pm_bpcluster * dirclust + diroffset;

But the real problem is that the second argument of vfs_hash_get() also
has the same 32-bit limitation on i386 (type u_int, 32 bits), so I think
it's the author of vfs_hash_get() approach (phk) who should decide whether to
promote uint64_t to vfs_hash_get() or decouple msdosfs from this function
at all.

>Release-Note:
>Audit-Trail:

From: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
To: Poul-Henning Kamp <phk@haven.freebsd.dk>
Cc: freebsd-arch@freebsd.org, bug-followup@freebsd.org
Subject: Re: kern/85503: panic: wrong dirclust using msdosfs in RELENG_6 
Date: Mon, 5 Sep 2005 13:50:01 +0300 (EEST)

 Hello!
 
 On Fri, 2 Sep 2005, Poul-Henning Kamp wrote:
 >> found the primary error (lack of casts leaded to 32-bit result), but then
 >> we should transfer this 64-bit "inode" number to vfs_hash_get(). Oops,
 >> it also limited to u_int (32 bits on i386). Finally, I see that the
 >> primary shortcoming here: in sys/vnode.h we have
 >
 > NFS has the same sort of problem, it has 16 or 32 *bytes* filehandles
 > that need to hash to 32 bit "inode numbers".
 >
 > If you look at vfs_hash_get calls in sys/nfsclient you can see that
 > it calculates a 32bit hash but then provides a "nfs_vncmpf" function
 > to do the actual comparison to resolve hash collisions.
 
   Indeed, I've missed last 2 arguments of vfs_hash_get(). Actually it seems
 that the only error here is missing cast. After application of the following
 patch:
 
  	ftp://external.atlantis.dp.ua/FreeBSD/PR/85503/msdosfs.patch
 
 problem has gone away. Please, if possible, review and commit it. I think this
 patch is a good MFC candidate for RELENG_6 and RELENG_6_0, since it prevents 
 panic in quite common environment.
 
 Sincerely, Dmitry
 -- 
 Atlantis ISP, System Administrator
 e-mail:  dmitry@atlantis.dp.ua
 nic-hdl: LYNX-RIPE
State-Changed-From-To: open->closed 
State-Changed-By: obrien 
State-Changed-When: Wed Sep 7 10:33:04 GMT 2005 
State-Changed-Why:  
Committed to HEAD. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=85503 

From: "David O'Brien" <obrien@freebsd.org>
To: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
Cc: Poul-Henning Kamp <phk@haven.freebsd.dk>, bug-followup@freebsd.org,
        freebsd-arch@freebsd.org
Subject: Re: kern/85503: panic: wrong dirclust using msdosfs in RELENG_6
Date: Wed, 7 Sep 2005 03:34:16 -0700

 On Mon, Sep 05, 2005 at 01:50:01PM +0300, Dmitry Pryanishnikov wrote:
 > 	ftp://external.atlantis.dp.ua/FreeBSD/PR/85503/msdosfs.patch
 > 
 > problem has gone away. Please, if possible, review and commit it. I think 
 > this
 > patch is a good MFC candidate for RELENG_6 and RELENG_6_0, since it 
 > prevents panic in quite common environment.
 
 I've committed it to HEAD.  It is up to the RE's to get it into RELENG_6.
 
 -- 
 -- David  (obrien@FreeBSD.org)

From: Yar Tikhiy <yar@comp.chem.msu.su>
To: obrien@freebsd.org, Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>,
        Poul-Henning Kamp <phk@haven.freebsd.dk>, bug-followup@freebsd.org,
        freebsd-arch@freebsd.org
Cc:  
Subject: Re: kern/85503: panic: wrong dirclust using msdosfs in RELENG_6
Date: Wed, 7 Sep 2005 16:44:11 +0400

 On Wed, Sep 07, 2005 at 03:34:16AM -0700, David O'Brien wrote:
 > On Mon, Sep 05, 2005 at 01:50:01PM +0300, Dmitry Pryanishnikov wrote:
 > > 	ftp://external.atlantis.dp.ua/FreeBSD/PR/85503/msdosfs.patch
 > > 
 > > problem has gone away. Please, if possible, review and commit it. I think 
 > > this
 > > patch is a good MFC candidate for RELENG_6 and RELENG_6_0, since it 
 > > prevents panic in quite common environment.
 > 
 > I've committed it to HEAD.  It is up to the RE's to get it into RELENG_6.
 
 David, I'm afraid there is certain misunderstanding here.  First,
 the RE team will hardly take care of getting the fix to RELENG_6
 without your initiative.  Second, the PR kern/85503 shouldn't be
 closed until the problem is fixed in all relevant branches: It
 should be put in the "patched" state instead according to the
 project's well-established policy.  Thanks.
 
 -- 
 Yar
State-Changed-From-To: closed->open 
State-Changed-By: trhodes 
State-Changed-When: Wed Sep 7 14:34:30 GMT 2005 
State-Changed-Why:  
Take and spearhead movement along branches. 


Responsible-Changed-From-To: freebsd-bugs->trhodes 
Responsible-Changed-By: trhodes 
Responsible-Changed-When: Wed Sep 7 14:34:30 GMT 2005 
Responsible-Changed-Why:  
Take and spearhead movement along branches. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=85503 

From: "David O'Brien" <obrien@freebsd.org>
To: Yar Tikhiy <yar@comp.chem.msu.su>
Cc: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>,
        Poul-Henning Kamp <phk@haven.freebsd.dk>, bug-followup@freebsd.org,
        freebsd-arch@freebsd.org
Subject: Re: kern/85503: panic: wrong dirclust using msdosfs in RELENG_6
Date: Wed, 7 Sep 2005 09:27:12 -0700

 On Wed, Sep 07, 2005 at 04:44:11PM +0400, Yar Tikhiy wrote:
 > On Wed, Sep 07, 2005 at 03:34:16AM -0700, David O'Brien wrote:
 > > On Mon, Sep 05, 2005 at 01:50:01PM +0300, Dmitry Pryanishnikov wrote:
 > > > 	ftp://external.atlantis.dp.ua/FreeBSD/PR/85503/msdosfs.patch
 > > > 
 > > > problem has gone away. Please, if possible, review and commit it. I think 
 > > > this
 > > > patch is a good MFC candidate for RELENG_6 and RELENG_6_0, since it 
 > > > prevents panic in quite common environment.
 > > 
 > > I've committed it to HEAD.  It is up to the RE's to get it into RELENG_6.
 > 
 > David, I'm afraid there is certain misunderstanding here.  First,
 > the RE team will hardly take care of getting the fix to RELENG_6
 > without your initiative.
 
 There is zero misunderstanding here.  I'm not going to get into a public
 debate or fight over this.  My experiences to date is that it is too much
 effort to get all but the most critical bug fixes into a frozen branch.
 Someone else may take up the MFC during the freeze cause.
 
 > Second, the PR kern/85503 shouldn't be
 > closed until the problem is fixed in all relevant branches: It
 > should be put in the "patched" state instead according to the
 > project's well-established policy.  Thanks.
 
 I've been a FreeBSD committer for almost 10 years now - what you claim is
 "well established" isn't.  When a patch from a PR is committed to head
 the PR may be closed.  The new practice of some committers to change the
 state to patched is something some committers may like to do, but I
 don't.  The same for the MFC reminder and statement in the commit message
 is also newer thing that some like to use, but I don't.
 When I am free to commit to a branch and I have the time to adequately
 test in the branch, I look for all the commits I've done to HEAD, I do a
 code merge them and MFC them.
 
 -- 
 -- David  (obrien@FreeBSD.org)

From: des@des.no (=?iso-8859-1?q?Dag-Erling_Sm=F8rgrav?=)
To: obrien@freebsd.org
Cc: Yar Tikhiy <yar@comp.chem.msu.su>,
	  Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>,
	  Poul-Henning Kamp <phk@haven.freebsd.dk>,  bug-followup@freebsd.org,
	  freebsd-arch@freebsd.org
Subject: Re: kern/85503: panic: wrong dirclust using msdosfs in RELENG_6
Date: Thu, 08 Sep 2005 10:50:51 +0200

 "David O'Brien" <obrien@freebsd.org> writes:
 > On Wed, Sep 07, 2005 at 04:44:11PM +0400, Yar Tikhiy wrote:
 > > Second, the PR kern/85503 shouldn't be
 > > closed until the problem is fixed in all relevant branches: It
 > > should be put in the "patched" state instead according to the
 > > project's well-established policy.  Thanks.
 > I've been a FreeBSD committer for almost 10 years now - what you claim is
 > "well established" isn't.  When a patch from a PR is committed to head
 > the PR may be closed.  The new practice of some committers to change the
 > state to patched is something some committers may like to do, but I
 > don't.
 
 It is the *documented policy* of the project and has been for several
 years.  As a committer, you are expected to have read and understood
 the following document:
 
 http://www.freebsd.org/doc/en/articles/pr-guidelines/article.html
 
 DES
 --=20
 Dag-Erling Sm=F8rgrav - des@des.no
 

From: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/85503: [panic] wrong dirclust using msdosfs in RELENG_6
Date: Tue, 4 Oct 2005 22:19:02 +0300 (EEST)

 Hello!
 
 On Wed, 7 Sep 2005, Tom Rhodes wrote:
 > Responsible-Changed-By: trhodes
 > Responsible-Changed-When: Wed Sep 7 14:34:30 GMT 2005
 > Responsible-Changed-Why:
 > Take and spearhead movement along branches.
 >
 
   Maybe, it time to propagate this fix to RELENG_6? BTW, I've written
 small regression test for this problem. It's available at
 
  	ftp://external.atlantis.dp.ua/FreeBSD/PR/85503/regression/
 
 Script a.sh constructs FS image from 2 compressed parts: part1.bin.bz2
 (FAT + root dir + 1st test directory) and part2.bin.bz2 (2nd test directory).
 Just run this script, then mount_msdosfs resulting /dev/mdx device (R/O
 access is sufficient), and do 'ls -R' against the mountpoint - RELENG_6
 system will currently crash at this point.
 
 P.S. It's difficult to generate this FS "on the fly". In order to do it
 I made the following: created HDD slice with size of 4100 Mb, zeroed it,
 made FAT32 FS with large (64k) cluster and single FAT copy to reduce
 an amount of auxiliary data:
 
  	newfs_msdos -n 1 -c 128 /dev/adxsy
 
 then created file 1.bin of size (4G-64k):
 
  	truncate -s 4294901760 1.bin
 
 on another large FAT32 slice, then loaded Windows-98 and made from the command 
 line (X: - destination drive, Y: - drive with the large file 1.bin):
 
  	MKDIR X:\1
  	COPY Y:1.BIN X:\1
  	MKDIR X:\2
  	COPY CON X:\2\2.BIN
  	^Z
 
 I've used another OS since FreeBSD's msdosfs selects initial cluster randomly,
 and mtools port is very buggy: I've 'mcopy''ed 3 * 1Gb files to fresh FS and
 got totally broken FS as a result.
 
 Sincerely, Dmitry
 -- 
 Atlantis ISP, System Administrator
 e-mail:  dmitry@atlantis.dp.ua
 nic-hdl: LYNX-RIPE

From: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
To: Tom Rhodes <trhodes@FreeBSD.org>
Cc: freebsd-bugs@FreeBSD.org, bug-followup@FreeBSD.org
Subject: Re: kern/85503: [panic] wrong dirclust using msdosfs in RELENG_6
Date: Fri, 14 Oct 2005 15:26:19 +0300 (EEST)

 Hello!
 
   My patch has already been commited to RELENG_6 and propagated to RELENG_6_0.
 FreeBSD 6.0-RC1 passes my regression tests successfully. Please close
 this PR.
 
 Sincerely, Dmitry
 -- 
 Atlantis ISP, System Administrator
 e-mail:  dmitry@atlantis.dp.ua
 nic-hdl: LYNX-RIPE
State-Changed-From-To: open->closed 
State-Changed-By: trhodes 
State-Changed-When: Fri Oct 14 23:19:34 GMT 2005 
State-Changed-Why:  
Closed, code will appear in the release and 5.X won't last 
much longer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=85503 

From: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
To: Tom Rhodes <trhodes@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/85503: [panic] wrong dirclust using msdosfs in RELENG_6
Date: Sat, 15 Oct 2005 13:39:44 +0300 (EEST)

 Hello!
 
 On Fri, 14 Oct 2005, Tom Rhodes wrote:
 > Closed, code will appear in the release and 5.X won't last
 > much longer.
 >
 
   Thanks. AFAIK, this bug doesnt't affect RELENG_5 at all. Code in question
 appeared in HEAD and then in RELENG_6, and unlikely will be backported to 
 RELENG_5. Even if some brave man decides to do it, he'll backport fixed 
 version ;)
 
 Sincerely, Dmitry
 -- 
 Atlantis ISP, System Administrator
 e-mail:  dmitry@atlantis.dp.ua
 nic-hdl: LYNX-RIPE
>Unformatted:
