From nobody@FreeBSD.org  Mon Feb 18 04:25:59 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id F32AA16A417
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 18 Feb 2008 04:25:58 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id D796B13C448
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 18 Feb 2008 04:25:58 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m1I4NVc8068248
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 18 Feb 2008 04:23:31 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.2/8.14.1/Submit) id m1I4NV7p068247;
	Mon, 18 Feb 2008 04:23:31 GMT
	(envelope-from nobody)
Message-Id: <200802180423.m1I4NV7p068247@www.freebsd.org>
Date: Mon, 18 Feb 2008 04:23:31 GMT
From: Jim Bryant <freebsd@electron-tube.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Too many files in a top-level UFS-2 filesystem directory will cause a panic on mount.
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         120781
>Category:       kern
>Synopsis:       [ufs] Too many files in a top-level UFS-2 filesystem directory will cause a panic on mount.
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    secteam
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Feb 18 04:30:01 UTC 2008
>Closed-Date:    Wed Apr 23 10:25:00 UTC 2008
>Last-Modified:  Wed Apr 23 10:25:00 UTC 2008
>Originator:     Jim Bryant
>Release:        6.3-STABLE
>Organization:
myself
>Environment:
FreeBSD wahoo.sd67dfl.org 6.3-STABLE FreeBSD 6.3-STABLE #0: Sun Feb 10 21:13:39 CST 2008     jbryant@wahoo.sd67dfl.org:/usr/obj/usr/src/sys/WAHOO-SMP  i386

>Description:
I was doing a reorganization of my filesystems, and since I do offline installs, I keep a local distfiles collection (or did until yesterday when this happened), and in the process, put all of the distfiles on their own filesystem to be mounted under /usr/ports/distfiles.

All was fine until I rebooted.

On rebooting, I got a page fault panic on mount of the new distfiles filesystem.

i booted again, got it again, booted again this time into single-user, and did a fsck on the filesystem, and it only showed as being "dirty", but otherwise had no problems in the eyes of fsck.  booted again, instant panic.

i booted an older 6.2 CD and mounted the filesystem fine.  i then put that filesystem the way it was by mkdir'ing a distfiles dir and mv'ing everything into it, but on reboot it still paniced on mount.

only a newfs was able to enable the filesystem to be mounted.

today i did further research, thinking it had to do with the number of files in the top-level filesystem directory, and found that to be true.  the short c program in the next section (how to repeat the problem) contains this.

a second test shows that, after a newfs, if this done in any subdirectory of that filesystem, the panic is averted, and all is well.  apparently this bug only effects top-level directories of a UFS2 filesystem.

I have not attempted this to a non-UFS2 filesystem.

IMHO, a security advisory should be released, since any user with write access to ANY top level directory of ANY mounted filesystem (most systems have /tmp as a world writable top level filesystem directory) can create a panic situation requiring a newfs of the said filesystem.  A malicious user with root access can do this to /.  Either way, on boot, or any attempt to mount said filesystem on a running system, will cause a panic, which of course will cause an unbootable system on reboot.

>How-To-Repeat:
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv) { int i; char buf[1024]; bzero(buf, 1024); for(i = 0; i < 10000; i++) { sprintf(buf, "touch %s%05d\n", argv[1], i); system((const char *)buf);} return(0);}

/* pass a top-level mountpoint directory name of a mounted filesystem, with a trailing slash to the above as argv[1], and run.

This will create 10,000 zero-length files in the specified directory.

umount that filesystem.

perform a shitload of sync's to make sure everything outstanding is flushed to disk on all filesystems.

mount the target filesystem (preferably from a vty or serial console to catch the messages when it panics, which it will as soon as the mount is attempted).
*/

>Fix:
newfs(8)


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->secteam 
Responsible-Changed-By: remko 
Responsible-Changed-When: Mon Feb 18 06:43:33 UTC 2008 
Responsible-Changed-Why:  
Grab for secteam. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=120781 

From: Volker <volker@vwsoft.com>
To: bug-followup@FreeBSD.org, freebsd@electron-tube.net
Cc:  
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem
 directory will cause a panic on mount.
Date: Mon, 18 Feb 2008 22:18:49 +0100

 Jim,
 
 On 02/18/08 05:24, Jim Bryant wrote:
 > ...
 > Environment (output of "uname -a" on the problem machine):
 >    FreeBSD wahoo.sd67dfl.org 6.3-STABLE FreeBSD 6.3-STABLE #0: Sun Feb
 > 10 21:13:39 CST 2008
 > jbryant@wahoo.sd67dfl.org:/usr/obj/usr/src/sys/WAHOO-SMP  i386
 
 As you're not running a GENERIC kernel (I've overseen that fact
 earlier), I'm wondering if you mind to show us your kernel config?
 
 > ...
 > i booted again, got it again, booted again this time into single-user,
 > and did a fsck on the filesystem, and it only showed as being "dirty",
 > but otherwise had no problems in the eyes of fsck.  booted again,
 > instant panic.
 >
 > i booted an older 6.2 CD and mounted the filesystem fine.  i then put
 > that filesystem the way it was by mkdir'ing a distfiles dir and mv'ing
 > everything into it, but on reboot it still paniced on mount.
 
 hmm, so booting a GENERIC kernel from CD does not give you this error,
 but your own compiled one panics?
 
 Also Robert Watson was asking you for the actual panic message and a
 backtrace. As nobody else seems to reproduce your problem, this can't be
 analyzed without the requested information.
 
 In addition to that, please show us output of `fdisk' and `bsdlabel' of
 the slices and labels in question.
 
 Thanks!
 
 Volker
 

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, freebsd@electron-tube.net
Cc:  
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem
 directory will cause a panic on mount.
Date: Mon, 18 Feb 2008 21:39:26 +0000 (UTC)

 Also add a
  	dumpfs /
 output to the list of things to get.
State-Changed-From-To: open->feedback 
State-Changed-By: linimon 
State-Changed-When: Thu Feb 28 00:24:56 UTC 2008 
State-Changed-Why:  
Note that submitter has been asked for feedback. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=120781 

From: Jim Bryant <jbryant@democrats.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem
 directory will cause a panic on mount
Date: Sat, 01 Mar 2008 16:48:02 -0600

 hey, my other email addy is down at the moment... sorry for the delay
 
 in replying.
 
 after you posted on this, i found that you are right, and a 6.3-STABLE 
 GENERIC will mount it properly.
 
 So, seeing that the only options i really had on the ufs side of 
 things was the directory hash (which was the option I incorrectly 
 suspected), and the backported taxpayer funded EXTATTR and ACL options.
 
 Ummm... I think DARPA is going to be pissed.
 
 I have narrowed the culprit to one or all of:
 
 #options         UFS_EXTATTR
 #options         UFS_EXTATTR_AUTOSTART
 #options         UFS_ACL
 
 Enabling those will crate the same kind of panic (iirc the panic 
 message was regarding a garbled directory entry in inode 2 around the 
 16k point), and as before, fsck sees nothing wrong, and nor does 
 GENERIC.  This is using a cvsup from TODAY.
 
 This fsck was using the kernel that panic'ed.
 
 3:24:21pm  stimpy(1): fsck -y /dev/da0s1d
 ** /dev/da0s1d
 ** Last Mounted on /mnt
 ** Phase 1 - Check Blocks and Sizes
 ** Phase 2 - Check Pathnames
 ** Phase 3 - Check Connectivity
 ** Phase 4 - Check Reference Counts
 ** Phase 5 - Check Cyl groups
 10002 files, 80 used, 18919222 free (14 frags, 2364901 blocks, 0.0% 
 fragmentation)
 
 ***** FILE SYSTEM MARKED CLEAN *****
 
 I'm currently doing this on an external usb mounted laptop drive, on a 
 network connected older celery 333 laptop, so that i don't have to mess 
 with anything other than a pure scratch filesystem, on a system that 
 isn't my main box, I'm building the kernels on this box (a P4 3.06) for 
 speed, and doing the installkernel from the laptop with this box's 
 /usr/src and /usr/obj nfs mounted onto the same mountpoints on the 
 laptop, and am compiling with -O -march=pentiumpro -mtune=pentiumpro -pipe
 
 ufs_extattr doesn't do it on it's own.  i am compiling in the autostart option, and
 this will now determine if the problem lies in ufs_extattr (with that bug being 
 exposed by UFS_EXTATTR_AUTOSTART), or if the bug lies in UFS_ACL.
 
 I shall post a followup once sure which option is causing the problem.
 
 Please pass this on, and update the status page with this info, as I 
 may not be able to for another week or two.
 
 jim
 -- 
  "Religious fundamentalism is the biggest threat to international security that exists today."
              United Nations Secretary General B.B.Ghali, 1995
 -----------------------------------------------------------------------------------------------------
 "The Fascist State organizes the nation, but leaves a sufficient margin of liberty to the individual;
  the latter is deprived of all useless and possibly harmful freedom..." -- Benito Mussilini, 1932
    USA PATRIOT, USA PATRIOT 2, and the REALID ACT [The Mark of the Beast Act] -- The GOP agrees
 
 

From: Jim Bryant <jbryant@democrats.com>
To: Jim Bryant <jbryant@democrats.com>
Cc: bug-followup@FreeBSD.org,  volker@vwsoft.com,  secteam@FreeBSD.org
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem
 directory will cause a panic on mount
Date: Sat, 01 Mar 2008 17:13:14 -0600

 How to duplicate:
 
     1). have any kernel with the following two options enabled:
 
         options         UFS_EXTATTR
         options         UFS_EXTATTR_AUTOSTART
 
     keeping in mind that the second may only expose bugs in the first 
 that would show on a manual or scripted start.
 
 Seeing that this can be a MAJOR issue for sites that require UFS_EXTATTR 
 and UFS_ACL, and the type of usage such systems can potentially have, it 
 is my recommendation that a security bulleting be issued ASAP telling 
 admins of such systems to not allow user-level write access to any top 
 level directory of a mounted filesystem, including /tmp, and that if 
 they cannot achieve such, they should turn off UFS_EXTATTR, even if that 
 means they cannot no longer connect to a classified network, until this 
 is fixed.
 
 The TrustedBSD team should also be made aware of this problem, as IIRC, 
 this code is backported from TBSD.
 
 Jim Bryant wrote:
 > hey, my other email addy is down at the moment... sorry for the delay
 >
 > in replying.
 >
 > after you posted on this, i found that you are right, and a 6.3-STABLE 
 > GENERIC will mount it properly.
 >
 > So, seeing that the only options i really had on the ufs side of 
 > things was the directory hash (which was the option I incorrectly 
 > suspected), and the backported taxpayer funded EXTATTR and ACL options.
 >
 > Ummm... I think DARPA is going to be pissed.
 >
 > I have narrowed the culprit to one or all of:
 >
 > #options         UFS_EXTATTR
 > #options         UFS_EXTATTR_AUTOSTART
 > #options         UFS_ACL
 >
 > Enabling those will crate the same kind of panic (iirc the panic 
 > message was regarding a garbled directory entry in inode 2 around the 
 > 16k point), and as before, fsck sees nothing wrong, and nor does 
 > GENERIC.  This is using a cvsup from TODAY.
 >
 > This fsck was using the kernel that panic'ed.
 >
 > 3:24:21pm  stimpy(1): fsck -y /dev/da0s1d
 > ** /dev/da0s1d
 > ** Last Mounted on /mnt
 > ** Phase 1 - Check Blocks and Sizes
 > ** Phase 2 - Check Pathnames
 > ** Phase 3 - Check Connectivity
 > ** Phase 4 - Check Reference Counts
 > ** Phase 5 - Check Cyl groups
 > 10002 files, 80 used, 18919222 free (14 frags, 2364901 blocks, 0.0% 
 > fragmentation)
 >
 > ***** FILE SYSTEM MARKED CLEAN *****
 >
 > I'm currently doing this on an external usb mounted laptop drive, on a 
 > network connected older celery 333 laptop, so that i don't have to 
 > mess with anything other than a pure scratch filesystem, on a system 
 > that isn't my main box, I'm building the kernels on this box (a P4 
 > 3.06) for speed, and doing the installkernel from the laptop with this 
 > box's /usr/src and /usr/obj nfs mounted onto the same mountpoints on 
 > the laptop, and am compiling with -O -march=pentiumpro 
 > -mtune=pentiumpro -pipe
 >
 > ufs_extattr doesn't do it on it's own.  i am compiling in the 
 > autostart option, and
 > this will now determine if the problem lies in ufs_extattr (with that 
 > bug being exposed by UFS_EXTATTR_AUTOSTART), or if the bug lies in 
 > UFS_ACL.
 >
 > I shall post a followup once sure which option is causing the problem.
 >
 > Please pass this on, and update the status page with this info, as I 
 > may not be able to for another week or two.
 >
 > jim
 
 -- 
   "Religious fundamentalism is the biggest threat to international security that exists today."
               United Nations Secretary General B.B.Ghali, 1995
 -----------------------------------------------------------------------------------------------------
 "The Fascist State organizes the nation, but leaves a sufficient margin of liberty to the individual;
   the latter is deprived of all useless and possibly harmful freedom..." -- Benito Mussilini, 1932
     USA PATRIOT, USA PATRIOT 2, and the REALID ACT [The Mark of the Beast Act] -- The GOP agrees
 
 

From: Jim Bryant <jbryant@democrats.com>
To: Jim Bryant <jbryant@democrats.com>
Cc: bug-followup@FreeBSD.org,  volker@vwsoft.com,  secteam@FreeBSD.org
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem
 directory will cause a panic on mount
Date: Sat, 01 Mar 2008 17:28:26 -0600

 seeing that these are ufs2 issues, could this be because at some number 
 of files the extattr options are clashing with the fact that the 
 extended attributes are built into ufs2?
 
 i'm going to try this next with ONLY UFS_ACL defined in the config, and 
 see if the problem repeats - if it doesn't, then, there is a clash 
 between ufs2 native extended attributes and the explicitly compiled 
 ufs1-compat extended attributes, if it does, ACL and ufs2 have issues too.
 
 Jim Bryant wrote:
 > How to duplicate:
 >
 >    1). have any kernel with the following two options enabled:
 >
 >        options         UFS_EXTATTR
 >        options         UFS_EXTATTR_AUTOSTART
 >
 >    keeping in mind that the second may only expose bugs in the first 
 > that would show on a manual or scripted start.
 >
 > Seeing that this can be a MAJOR issue for sites that require 
 > UFS_EXTATTR and UFS_ACL, and the type of usage such systems can 
 > potentially have, it is my recommendation that a security bulleting be 
 > issued ASAP telling admins of such systems to not allow user-level 
 > write access to any top level directory of a mounted filesystem, 
 > including /tmp, and that if they cannot achieve such, they should turn 
 > off UFS_EXTATTR, even if that means they cannot no longer connect to a 
 > classified network, until this is fixed.
 >
 > The TrustedBSD team should also be made aware of this problem, as 
 > IIRC, this code is backported from TBSD.
 >
 > Jim Bryant wrote:
 >> hey, my other email addy is down at the moment... sorry for the delay
 >>
 >> in replying.
 >>
 >> after you posted on this, i found that you are right, and a 
 >> 6.3-STABLE GENERIC will mount it properly.
 >>
 >> So, seeing that the only options i really had on the ufs side of 
 >> things was the directory hash (which was the option I incorrectly 
 >> suspected), and the backported taxpayer funded EXTATTR and ACL options.
 >>
 >> Ummm... I think DARPA is going to be pissed.
 >>
 >> I have narrowed the culprit to one or all of:
 >>
 >> #options         UFS_EXTATTR
 >> #options         UFS_EXTATTR_AUTOSTART
 >> #options         UFS_ACL
 >>
 >> Enabling those will crate the same kind of panic (iirc the panic 
 >> message was regarding a garbled directory entry in inode 2 around the 
 >> 16k point), and as before, fsck sees nothing wrong, and nor does 
 >> GENERIC.  This is using a cvsup from TODAY.
 >>
 >> This fsck was using the kernel that panic'ed.
 >>
 >> 3:24:21pm  stimpy(1): fsck -y /dev/da0s1d
 >> ** /dev/da0s1d
 >> ** Last Mounted on /mnt
 >> ** Phase 1 - Check Blocks and Sizes
 >> ** Phase 2 - Check Pathnames
 >> ** Phase 3 - Check Connectivity
 >> ** Phase 4 - Check Reference Counts
 >> ** Phase 5 - Check Cyl groups
 >> 10002 files, 80 used, 18919222 free (14 frags, 2364901 blocks, 0.0% 
 >> fragmentation)
 >>
 >> ***** FILE SYSTEM MARKED CLEAN *****
 >>
 >> I'm currently doing this on an external usb mounted laptop drive, on 
 >> a network connected older celery 333 laptop, so that i don't have to 
 >> mess with anything other than a pure scratch filesystem, on a system 
 >> that isn't my main box, I'm building the kernels on this box (a P4 
 >> 3.06) for speed, and doing the installkernel from the laptop with 
 >> this box's /usr/src and /usr/obj nfs mounted onto the same 
 >> mountpoints on the laptop, and am compiling with -O -march=pentiumpro 
 >> -mtune=pentiumpro -pipe
 >>
 >> ufs_extattr doesn't do it on it's own.  i am compiling in the 
 >> autostart option, and
 >> this will now determine if the problem lies in ufs_extattr (with that 
 >> bug being exposed by UFS_EXTATTR_AUTOSTART), or if the bug lies in 
 >> UFS_ACL.
 >>
 >> I shall post a followup once sure which option is causing the problem.
 >>
 >> Please pass this on, and update the status page with this info, as I 
 >> may not be able to for another week or two.
 >>
 >> jim
 >
 
 -- 
   "Religious fundamentalism is the biggest threat to international security that exists today."
               United Nations Secretary General B.B.Ghali, 1995
 -----------------------------------------------------------------------------------------------------
 "The Fascist State organizes the nation, but leaves a sufficient margin of liberty to the individual;
   the latter is deprived of all useless and possibly harmful freedom..." -- Benito Mussilini, 1932
     USA PATRIOT, USA PATRIOT 2, and the REALID ACT [The Mark of the Beast Act] -- The GOP agrees
 
 

From: Jim Bryant <jbryant@democrats.com>
To: Jim Bryant <jbryant@democrats.com>
Cc: bug-followup@FreeBSD.org,  volker@vwsoft.com,  secteam@FreeBSD.org
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem
 directory will cause a panic on mount
Date: Sat, 01 Mar 2008 17:38:11 -0600

 WAIT.
 
 I just looked at GENERIC.  UFS_ACL is already defined by itself.  The 
 filesystem mounts under GENERIC without a panic.
 
 The problem is definitely in the ufs1 compatible optional UFS_EXTATTR 
 and UFS_EXTATTR_AUTOSTART (requiring the second option to expose the 
 bug), something in it is not in sync with the native ufs2 code.
 
 ok, so that means that the security advisory should make it clear that 
 this only effects systems with those two options enabled, but that even 
 if the large root dir is on a ufs2 filesystem, that it will cause the panic.
 
 i am about to retest and re-newfs the filesystem as a ufs1, and then, we 
 can see if that causes a panic on mount with the 10,000 zero length 
 files in the root dir.
 
 Jim Bryant wrote:
 > seeing that these are ufs2 issues, could this be because at some 
 > number of files the extattr options are clashing with the fact that 
 > the extended attributes are built into ufs2?
 >
 > i'm going to try this next with ONLY UFS_ACL defined in the config, 
 > and see if the problem repeats - if it doesn't, then, there is a clash 
 > between ufs2 native extended attributes and the explicitly compiled 
 > ufs1-compat extended attributes, if it does, ACL and ufs2 have issues 
 > too.
 >
 > Jim Bryant wrote:
 >> How to duplicate:
 >>
 >>    1). have any kernel with the following two options enabled:
 >>
 >>        options         UFS_EXTATTR
 >>        options         UFS_EXTATTR_AUTOSTART
 >>
 >>    keeping in mind that the second may only expose bugs in the first 
 >> that would show on a manual or scripted start.
 >>
 >> Seeing that this can be a MAJOR issue for sites that require 
 >> UFS_EXTATTR and UFS_ACL, and the type of usage such systems can 
 >> potentially have, it is my recommendation that a security bulleting 
 >> be issued ASAP telling admins of such systems to not allow user-level 
 >> write access to any top level directory of a mounted filesystem, 
 >> including /tmp, and that if they cannot achieve such, they should 
 >> turn off UFS_EXTATTR, even if that means they cannot no longer 
 >> connect to a classified network, until this is fixed.
 >>
 >> The TrustedBSD team should also be made aware of this problem, as 
 >> IIRC, this code is backported from TBSD.
 >>
 >> Jim Bryant wrote:
 >>> hey, my other email addy is down at the moment... sorry for the delay
 >>>
 >>> in replying.
 >>>
 >>> after you posted on this, i found that you are right, and a 
 >>> 6.3-STABLE GENERIC will mount it properly.
 >>>
 >>> So, seeing that the only options i really had on the ufs side of 
 >>> things was the directory hash (which was the option I incorrectly 
 >>> suspected), and the backported taxpayer funded EXTATTR and ACL options.
 >>>
 >>> Ummm... I think DARPA is going to be pissed.
 >>>
 >>> I have narrowed the culprit to one or all of:
 >>>
 >>> #options         UFS_EXTATTR
 >>> #options         UFS_EXTATTR_AUTOSTART
 >>> #options         UFS_ACL
 >>>
 >>> Enabling those will crate the same kind of panic (iirc the panic 
 >>> message was regarding a garbled directory entry in inode 2 around 
 >>> the 16k point), and as before, fsck sees nothing wrong, and nor does 
 >>> GENERIC.  This is using a cvsup from TODAY.
 >>>
 >>> This fsck was using the kernel that panic'ed.
 >>>
 >>> 3:24:21pm  stimpy(1): fsck -y /dev/da0s1d
 >>> ** /dev/da0s1d
 >>> ** Last Mounted on /mnt
 >>> ** Phase 1 - Check Blocks and Sizes
 >>> ** Phase 2 - Check Pathnames
 >>> ** Phase 3 - Check Connectivity
 >>> ** Phase 4 - Check Reference Counts
 >>> ** Phase 5 - Check Cyl groups
 >>> 10002 files, 80 used, 18919222 free (14 frags, 2364901 blocks, 0.0% 
 >>> fragmentation)
 >>>
 >>> ***** FILE SYSTEM MARKED CLEAN *****
 >>>
 >>> I'm currently doing this on an external usb mounted laptop drive, on 
 >>> a network connected older celery 333 laptop, so that i don't have to 
 >>> mess with anything other than a pure scratch filesystem, on a system 
 >>> that isn't my main box, I'm building the kernels on this box (a P4 
 >>> 3.06) for speed, and doing the installkernel from the laptop with 
 >>> this box's /usr/src and /usr/obj nfs mounted onto the same 
 >>> mountpoints on the laptop, and am compiling with -O 
 >>> -march=pentiumpro -mtune=pentiumpro -pipe
 >>>
 >>> ufs_extattr doesn't do it on it's own.  i am compiling in the 
 >>> autostart option, and
 >>> this will now determine if the problem lies in ufs_extattr (with 
 >>> that bug being exposed by UFS_EXTATTR_AUTOSTART), or if the bug lies 
 >>> in UFS_ACL.
 >>>
 >>> I shall post a followup once sure which option is causing the problem.
 >>>
 >>> Please pass this on, and update the status page with this info, as I 
 >>> may not be able to for another week or two.
 >>>
 >>> jim
 >>
 >
 
 -- 
   "Religious fundamentalism is the biggest threat to international security that exists today."
               United Nations Secretary General B.B.Ghali, 1995
 -----------------------------------------------------------------------------------------------------
 "The Fascist State organizes the nation, but leaves a sufficient margin of liberty to the individual;
   the latter is deprived of all useless and possibly harmful freedom..." -- Benito Mussilini, 1932
     USA PATRIOT, USA PATRIOT 2, and the REALID ACT [The Mark of the Beast Act] -- The GOP agrees
 
 

From: Robert Watson <rwatson@FreeBSD.org>
To: Jim Bryant <jbryant@democrats.com>
Cc: volker@vwsoft.com, secteam@FreeBSD.org, bug-followup@FreeBSD.org
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem
 directory will cause a panic on mount
Date: Sat, 1 Mar 2008 23:45:36 +0000 (GMT)

 On Sat, 1 Mar 2008, Jim Bryant wrote:
 
 > I just looked at GENERIC.  UFS_ACL is already defined by itself.  The 
 > filesystem mounts under GENERIC without a panic.
 >
 > The problem is definitely in the ufs1 compatible optional UFS_EXTATTR and 
 > UFS_EXTATTR_AUTOSTART (requiring the second option to expose the bug), 
 > something in it is not in sync with the native ufs2 code.
 >
 > ok, so that means that the security advisory should make it clear that this 
 > only effects systems with those two options enabled, but that even if the 
 > large root dir is on a ufs2 filesystem, that it will cause the panic.
 >
 > i am about to retest and re-newfs the filesystem as a ufs1, and then, we can 
 > see if that causes a panic on mount with the 10,000 zero length files in the 
 > root dir.
 
 It's a reasonable guess that something about UFS_EXTATTR_AUTOSTART is unhappy 
 with the size of the root directory, and that turning off 
 UFS_EXTATTR_AUTOSTART will make the panic go away.  It's fairly late at night 
 here, and I can investigate more tomorrow, but here are two thoughts:
 
 (1) Why is UFS_EXTATTR_AUTOSTART affecting the ufs2 mount?  ufs2's native EAs
      mean it shouldn't need to run during mount.  Turning that off may be an
      easy workaround for your environment, and not be a bad workaround.
 
 (2) What's actually going wrong, and is it ufs2-specific, or is it a more
      general ufs1/ufs2 issue?  Perhaps it's when we hit the first indirect
      inode pointer or the like.  In principle the auto-start code looks at
      directory contents in order to scan for specific names, but only in the
      autostart directory.  Perhaps it's the scan code, or perhaps it's the
      lookup in / that does it.
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge
 
 >
 > Jim Bryant wrote:
 >> seeing that these are ufs2 issues, could this be because at some number of 
 >> files the extattr options are clashing with the fact that the extended 
 >> attributes are built into ufs2?
 >> 
 >> i'm going to try this next with ONLY UFS_ACL defined in the config, and see 
 >> if the problem repeats - if it doesn't, then, there is a clash between ufs2 
 >> native extended attributes and the explicitly compiled ufs1-compat extended 
 >> attributes, if it does, ACL and ufs2 have issues too.
 >> 
 >> Jim Bryant wrote:
 >>> How to duplicate:
 >>>
 >>>    1). have any kernel with the following two options enabled:
 >>>
 >>>        options         UFS_EXTATTR
 >>>        options         UFS_EXTATTR_AUTOSTART
 >>>
 >>>    keeping in mind that the second may only expose bugs in the first that 
 >>> would show on a manual or scripted start.
 >>> 
 >>> Seeing that this can be a MAJOR issue for sites that require UFS_EXTATTR 
 >>> and UFS_ACL, and the type of usage such systems can potentially have, it 
 >>> is my recommendation that a security bulleting be issued ASAP telling 
 >>> admins of such systems to not allow user-level write access to any top 
 >>> level directory of a mounted filesystem, including /tmp, and that if they 
 >>> cannot achieve such, they should turn off UFS_EXTATTR, even if that means 
 >>> they cannot no longer connect to a classified network, until this is 
 >>> fixed.
 >>> 
 >>> The TrustedBSD team should also be made aware of this problem, as IIRC, 
 >>> this code is backported from TBSD.
 >>> 
 >>> Jim Bryant wrote:
 >>>> hey, my other email addy is down at the moment... sorry for the delay
 >>>> 
 >>>> in replying.
 >>>> 
 >>>> after you posted on this, i found that you are right, and a 6.3-STABLE 
 >>>> GENERIC will mount it properly.
 >>>> 
 >>>> So, seeing that the only options i really had on the ufs side of things 
 >>>> was the directory hash (which was the option I incorrectly suspected), 
 >>>> and the backported taxpayer funded EXTATTR and ACL options.
 >>>> 
 >>>> Ummm... I think DARPA is going to be pissed.
 >>>> 
 >>>> I have narrowed the culprit to one or all of:
 >>>> 
 >>>> #options         UFS_EXTATTR
 >>>> #options         UFS_EXTATTR_AUTOSTART
 >>>> #options         UFS_ACL
 >>>> 
 >>>> Enabling those will crate the same kind of panic (iirc the panic message 
 >>>> was regarding a garbled directory entry in inode 2 around the 16k point), 
 >>>> and as before, fsck sees nothing wrong, and nor does GENERIC.  This is 
 >>>> using a cvsup from TODAY.
 >>>> 
 >>>> This fsck was using the kernel that panic'ed.
 >>>> 
 >>>> 3:24:21pm  stimpy(1): fsck -y /dev/da0s1d
 >>>> ** /dev/da0s1d
 >>>> ** Last Mounted on /mnt
 >>>> ** Phase 1 - Check Blocks and Sizes
 >>>> ** Phase 2 - Check Pathnames
 >>>> ** Phase 3 - Check Connectivity
 >>>> ** Phase 4 - Check Reference Counts
 >>>> ** Phase 5 - Check Cyl groups
 >>>> 10002 files, 80 used, 18919222 free (14 frags, 2364901 blocks, 0.0% 
 >>>> fragmentation)
 >>>> 
 >>>> ***** FILE SYSTEM MARKED CLEAN *****
 >>>> 
 >>>> I'm currently doing this on an external usb mounted laptop drive, on a 
 >>>> network connected older celery 333 laptop, so that i don't have to mess 
 >>>> with anything other than a pure scratch filesystem, on a system that 
 >>>> isn't my main box, I'm building the kernels on this box (a P4 3.06) for 
 >>>> speed, and doing the installkernel from the laptop with this box's 
 >>>> /usr/src and /usr/obj nfs mounted onto the same mountpoints on the 
 >>>> laptop, and am compiling with -O -march=pentiumpro -mtune=pentiumpro 
 >>>> -pipe
 >>>> 
 >>>> ufs_extattr doesn't do it on it's own.  i am compiling in the autostart 
 >>>> option, and
 >>>> this will now determine if the problem lies in ufs_extattr (with that bug 
 >>>> being exposed by UFS_EXTATTR_AUTOSTART), or if the bug lies in UFS_ACL.
 >>>> 
 >>>> I shall post a followup once sure which option is causing the problem.
 >>>> 
 >>>> Please pass this on, and update the status page with this info, as I may 
 >>>> not be able to for another week or two.
 >>>> 
 >>>> jim
 >>> 
 >> 
 >
 > -- 
 > "Religious fundamentalism is the biggest threat to international security 
 > that exists today."
 >             United Nations Secretary General B.B.Ghali, 1995
 > -----------------------------------------------------------------------------------------------------
 > "The Fascist State organizes the nation, but leaves a sufficient margin of 
 > liberty to the individual;
 > the latter is deprived of all useless and possibly harmful freedom..." -- 
 > Benito Mussilini, 1932
 >   USA PATRIOT, USA PATRIOT 2, and the REALID ACT [The Mark of the Beast Act] 
 > -- The GOP agrees
 >
 >
 > _______________________________________________________
 > Please think twice when forwarding, cc:ing, or bcc:ing
 > security-team messages.  Ask if you are unsure.
 >

From: Jim Bryant <jbryant@democrats.com>
To: Jim Bryant <jbryant@democrats.com>
Cc: bug-followup@FreeBSD.org,  volker@vwsoft.com,  secteam@FreeBSD.org
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem
 directory will cause a panic on mount
Date: Sat, 01 Mar 2008 18:05:07 -0600

 it also panics with a UFS1 filesystem with the UFS_EXTATTR and 
 UFS_EXTATTR_AUTOSTART options enabled, and with UFS_ACL disabled.
 
 this bug is visible with BOTH UFS-1 *AND* UFS-2 filesystems.
 
 Jim Bryant wrote:
 > WAIT.
 >
 > I just looked at GENERIC.  UFS_ACL is already defined by itself.  The 
 > filesystem mounts under GENERIC without a panic.
 >
 > The problem is definitely in the ufs1 compatible optional UFS_EXTATTR 
 > and UFS_EXTATTR_AUTOSTART (requiring the second option to expose the 
 > bug), something in it is not in sync with the native ufs2 code.
 >
 > ok, so that means that the security advisory should make it clear that 
 > this only effects systems with those two options enabled, but that 
 > even if the large root dir is on a ufs2 filesystem, that it will cause 
 > the panic.
 >
 > i am about to retest and re-newfs the filesystem as a ufs1, and then, 
 > we can see if that causes a panic on mount with the 10,000 zero length 
 > files in the root dir.
 >
 > Jim Bryant wrote:
 >> seeing that these are ufs2 issues, could this be because at some 
 >> number of files the extattr options are clashing with the fact that 
 >> the extended attributes are built into ufs2?
 >>
 >> i'm going to try this next with ONLY UFS_ACL defined in the config, 
 >> and see if the problem repeats - if it doesn't, then, there is a 
 >> clash between ufs2 native extended attributes and the explicitly 
 >> compiled ufs1-compat extended attributes, if it does, ACL and ufs2 
 >> have issues too.
 >>
 >> Jim Bryant wrote:
 >>> How to duplicate:
 >>>
 >>>    1). have any kernel with the following two options enabled:
 >>>
 >>>        options         UFS_EXTATTR
 >>>        options         UFS_EXTATTR_AUTOSTART
 >>>
 >>>    keeping in mind that the second may only expose bugs in the first 
 >>> that would show on a manual or scripted start.
 >>>
 >>> Seeing that this can be a MAJOR issue for sites that require 
 >>> UFS_EXTATTR and UFS_ACL, and the type of usage such systems can 
 >>> potentially have, it is my recommendation that a security bulleting 
 >>> be issued ASAP telling admins of such systems to not allow 
 >>> user-level write access to any top level directory of a mounted 
 >>> filesystem, including /tmp, and that if they cannot achieve such, 
 >>> they should turn off UFS_EXTATTR, even if that means they cannot no 
 >>> longer connect to a classified network, until this is fixed.
 >>>
 >>> The TrustedBSD team should also be made aware of this problem, as 
 >>> IIRC, this code is backported from TBSD.
 >>>
 >>> Jim Bryant wrote:
 >>>> hey, my other email addy is down at the moment... sorry for the delay
 >>>>
 >>>> in replying.
 >>>>
 >>>> after you posted on this, i found that you are right, and a 
 >>>> 6.3-STABLE GENERIC will mount it properly.
 >>>>
 >>>> So, seeing that the only options i really had on the ufs side of 
 >>>> things was the directory hash (which was the option I incorrectly 
 >>>> suspected), and the backported taxpayer funded EXTATTR and ACL 
 >>>> options.
 >>>>
 >>>> Ummm... I think DARPA is going to be pissed.
 >>>>
 >>>> I have narrowed the culprit to one or all of:
 >>>>
 >>>> #options         UFS_EXTATTR
 >>>> #options         UFS_EXTATTR_AUTOSTART
 >>>> #options         UFS_ACL
 >>>>
 >>>> Enabling those will crate the same kind of panic (iirc the panic 
 >>>> message was regarding a garbled directory entry in inode 2 around 
 >>>> the 16k point), and as before, fsck sees nothing wrong, and nor 
 >>>> does GENERIC.  This is using a cvsup from TODAY.
 >>>>
 >>>> This fsck was using the kernel that panic'ed.
 >>>>
 >>>> 3:24:21pm  stimpy(1): fsck -y /dev/da0s1d
 >>>> ** /dev/da0s1d
 >>>> ** Last Mounted on /mnt
 >>>> ** Phase 1 - Check Blocks and Sizes
 >>>> ** Phase 2 - Check Pathnames
 >>>> ** Phase 3 - Check Connectivity
 >>>> ** Phase 4 - Check Reference Counts
 >>>> ** Phase 5 - Check Cyl groups
 >>>> 10002 files, 80 used, 18919222 free (14 frags, 2364901 blocks, 0.0% 
 >>>> fragmentation)
 >>>>
 >>>> ***** FILE SYSTEM MARKED CLEAN *****
 >>>>
 >>>> I'm currently doing this on an external usb mounted laptop drive, 
 >>>> on a network connected older celery 333 laptop, so that i don't 
 >>>> have to mess with anything other than a pure scratch filesystem, on 
 >>>> a system that isn't my main box, I'm building the kernels on this 
 >>>> box (a P4 3.06) for speed, and doing the installkernel from the 
 >>>> laptop with this box's /usr/src and /usr/obj nfs mounted onto the 
 >>>> same mountpoints on the laptop, and am compiling with -O 
 >>>> -march=pentiumpro -mtune=pentiumpro -pipe
 >>>>
 >>>> ufs_extattr doesn't do it on it's own.  i am compiling in the 
 >>>> autostart option, and
 >>>> this will now determine if the problem lies in ufs_extattr (with 
 >>>> that bug being exposed by UFS_EXTATTR_AUTOSTART), or if the bug 
 >>>> lies in UFS_ACL.
 >>>>
 >>>> I shall post a followup once sure which option is causing the problem.
 >>>>
 >>>> Please pass this on, and update the status page with this info, as 
 >>>> I may not be able to for another week or two.
 >>>>
 >>>> jim
 >>>
 >>
 >
 
 -- 
   "Religious fundamentalism is the biggest threat to international security that exists today."
               United Nations Secretary General B.B.Ghali, 1995
 -----------------------------------------------------------------------------------------------------
 "The Fascist State organizes the nation, but leaves a sufficient margin of liberty to the individual;
   the latter is deprived of all useless and possibly harmful freedom..." -- Benito Mussilini, 1932
     USA PATRIOT, USA PATRIOT 2, and the REALID ACT [The Mark of the Beast Act] -- The GOP agrees
 
 
State-Changed-From-To: feedback->analyzed 
State-Changed-By: rwatson 
State-Changed-When: Tue Mar 4 11:52:39 UTC 2008 
State-Changed-Why:  
This appears to be a bug in UFS_EXTATTR_AUTOSTART which might relate either 
to the introduction of MPSAFE file systems or GEOM.  The first ufs_lookup 
call in ufs_extattr_autostart triggers the problem (mangled directory 
offset 16384).  Performing the autostart *after* the mount has completed 
(i.e., using a system call manually) does not trigger the problem even 
with identical code.  We are still investigating the specific cause of the 
problem; in the mean time I have modified UFS so that auto-start doesn't 
run on UFS2 file systems, since that wasn't desirable anyway in change 
ufs_extattr.c:1.90: 

date: 2008/03/02 22:52:14;  author: rwatson;  state: Exp;  lines: +18 -0 
Don't auto-start or allow extattrctl for UFS2 file systems, as UFS2 has 
native extended attributes.  This didn't interfere with the operation of 
UFS2 extended attributes, but the code shouldn't be running for UFS2. 

MFC after:      2 weeks 

I'll MFC this change once it has settled, and continue to work on 
identifying the specific source of the problem. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=120781 

From: Robert Watson <rwatson@FreeBSD.org>
To: freebsd@electron-tube.net, secteam@FreeBSD.org, bug-followup@FreeBSD.org, 
    kib@FreeBSD.org
Cc:  
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem
 directory will cause a panic on mount.
Date: Tue, 4 Mar 2008 12:35:44 +0000 (GMT)

 FYI, here's the specific reproduction case I'm using:
 
 (1) Compile a kernel with UFS_EXTATTR and UFS_EXTATTR_AUTOSTART
 
 (2) Create an md device with UFS1 (substitute mdX as required)
 
      mdconfig -a -t malloc -s 32m
      newfs -O1 -i128 /dev/mdX
      mount /dev/mdX /mnt
 
 (3) Populate root directory with a lot of files and unmount
 
      cd /mnt
      jot 10000 | xargs touch
      cd /
      umount /mnt
 
 (4) Mount and panic
 
      mount /dev/mdX /mnt
 
 The actual panic message is:
 
    panic: ufs_dirbad: /mnt: bad dir ino 2 at offset 16384: mangled entry
 
 ufs_dirbad is invoked as follows: ufs_extattr_autostart -> ufs_extattr_lookup 
 -> ufs_dirbad.  This is the first call to lookup (for .attribute) in the file 
 system root so should have nothing to do with the code in UFS1 extended 
 attributes for iterating over directories.
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge

From: Kostik Belousov <kostikbel@gmail.com>
To: Robert Watson <rwatson@freebsd.org>
Cc: freebsd@electron-tube.net, secteam@freebsd.org, bug-followup@freebsd.org,
        kib@freebsd.org
Subject: Re: kern/120781: [ufs] Too many files in a top-level UFS-2 filesystem directory will cause a panic on mount.
Date: Wed, 5 Mar 2008 16:03:02 +0200

 --icp8nZRe8bNvizUl
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 On Tue, Mar 04, 2008 at 12:35:44PM +0000, Robert Watson wrote:
 >=20
 > FYI, here's the specific reproduction case I'm using:
 >=20
 > (1) Compile a kernel with UFS_EXTATTR and UFS_EXTATTR_AUTOSTART
 >=20
 > (2) Create an md device with UFS1 (substitute mdX as required)
 >=20
 >     mdconfig -a -t malloc -s 32m
 >     newfs -O1 -i128 /dev/mdX
 >     mount /dev/mdX /mnt
 >=20
 > (3) Populate root directory with a lot of files and unmount
 >=20
 >     cd /mnt
 >     jot 10000 | xargs touch
 >     cd /
 >     umount /mnt
 >=20
 > (4) Mount and panic
 >=20
 >     mount /dev/mdX /mnt
 >=20
 > The actual panic message is:
 >=20
 >   panic: ufs_dirbad: /mnt: bad dir ino 2 at offset 16384: mangled entry
 >=20
 > ufs_dirbad is invoked as follows: ufs_extattr_autostart ->=20
 > ufs_extattr_lookup -> ufs_dirbad.  This is the first call to lookup (for=
 =20
 > .attribute) in the file system root so should have nothing to do with the=
 =20
 > code in UFS1 extended attributes for iterating over directories.
 
 The problem appears to be ufs_lookup() initializing the bmask from the
 mnt_stat content at the ufs/ufs/ufs_lookup.c:183. Since mnt_stat is filled
 by the VFS_STAT() after VFS_MOUNT(), bmask is a garbage.
 
 The bmask was last touched by rev. 1.50.
 
 The following change works for me. It is the least intrusive patch.
 
 diff --git a/sys/ufs/ffs/ffs_vfsops.c b/sys/ufs/ffs/ffs_vfsops.c
 index e88a997..16784b7 100644
 --- a/sys/ufs/ffs/ffs_vfsops.c
 +++ b/sys/ufs/ffs/ffs_vfsops.c
 @@ -880,6 +880,7 @@ ffs_mountfs(devvp, mp, td)
  	 * This would all happen while the filesystem was busy/not
  	 * available, so would effectively be "atomic".
  	 */
 +	mp->mnt_stat.f_iosize =3D fs->fs_bsize;
  	(void) ufs_extattr_autostart(mp, td);
  #endif /* !UFS_EXTATTR_AUTOSTART */
  #endif /* !UFS_EXTATTR */
 
 --icp8nZRe8bNvizUl
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.8 (FreeBSD)
 
 iEYEARECAAYFAkfOqBUACgkQC3+MBN1Mb4gTngCgozUywQBtB1H5G9YplAiv72T1
 IxMAoKtYOTt+esp/lrRCGQQt/KeKJ6YI
 =DGj5
 -----END PGP SIGNATURE-----
 
 --icp8nZRe8bNvizUl--

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/120781: commit references a PR
Date: Wed,  5 Mar 2008 16:34:11 +0000 (UTC)

 kib         2008-03-05 16:34:03 UTC
 
   FreeBSD src repository
 
   Modified files:
     sys/ufs/ffs          ffs_vfsops.c 
   Log:
   Initialize mnt_stat.f_iosize before autostarting UFS1 extattrs.
   It is normally initialized by ffs_statfs() after ffs_mount finished.
   
   The extattr autostart code calls the ufs_lookup(), that uses value above
   to iterate over the directory blocks, see bmask initialization in the
   ufs_lookup() and ufsdirhash. Having the filesystem with root directory
   spanning more then one block would result in reading a random kernel
   memory.
   
   PR:     kern/120781
   Test case provided by:  rwatson
   MFC after:      1 week
   
   Revision  Changes    Path
   1.338     +1 -0      src/sys/ufs/ffs/ffs_vfsops.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: analyzed->patched 
State-Changed-By: rwatson 
State-Changed-When: Wed Mar 5 18:44:20 UTC 2008 
State-Changed-Why:  
Problem is believed to be resolved by ffs_vfsops.c:1.338, which will be 
merged to RELENG_7 and other branches after a week or so being broken 
in in HEAD.  With that fix in place, I'm unable to reproduce the panic. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=120781 
State-Changed-From-To: patched->closed 
State-Changed-By: rwatson 
State-Changed-When: Wed Apr 23 10:23:18 UTC 2008 
State-Changed-Why:  
Close -- the fix has been merged to RELENG_7. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=120781 
>Unformatted:
