From nobody@FreeBSD.org  Wed Nov 21 02:34:52 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 1411316A480
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 21 Nov 2007 02:34:52 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 1051D13C459
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 21 Nov 2007 02:34:52 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.1/8.14.1) with ESMTP id lAL2Y7pf041134
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 21 Nov 2007 02:34:07 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.1/8.14.1/Submit) id lAL2Y7cU041129;
	Wed, 21 Nov 2007 02:34:07 GMT
	(envelope-from nobody)
Message-Id: <200711210234.lAL2Y7cU041129@www.freebsd.org>
Date: Wed, 21 Nov 2007 02:34:07 GMT
From: Yuri <yuri@tsoft.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: unable to mount / rw while booting 7.0-BETA3
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         118160
>Category:       bin
>Synopsis:       mount(8): unable to mount / rw while booting 7.0-BETA3
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Nov 21 02:40:02 UTC 2007
>Closed-Date:    Sat Mar 01 01:19:53 UTC 2008
>Last-Modified:  Sat Mar 01 01:19:53 UTC 2008
>Originator:     Yuri
>Release:        7.0-BETA3
>Organization:
n/a
>Environment:
>Description:
After recompiling and reinstalling the current BETA3 my system has a
reboot problem.

While booting log says:
Starting file system checks:
<here goes the list of file systems that it reports, this is ok>
mount:  : Operation not permitted.
Mounting root file system rw failed, startup aborted.
/etc/rc: WARNING: $true is not set properly - see rc.conf(5)

and system gets to single user mode.

In single user mode / is read-only. And command 'mount -uw /' fails
Operation not permitted. I count't find the workaround so far.

'sysctl -a' shows securelevel=-1 (lowest).

The major bug seems to be in the 'mount' system call. 'man mount' says
that EPERM is returned if "The caller is neither the super-user nor the
owner of dir." I am root.

The secondary problem is this printout: WARNING: $true is not set
properly - see rc.conf(5)

It shouldn't print $true

Another secondary problem is with man mount(2). Isn't is supposed to
mention that setting securelevel also makes 'mount' return EPERM?

So now I can reboot normally only choosing "single user mode" when I
boot and running "mount -uw /" as a single user. And then continuing
the boot process.

>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:

From: Bruce Evans <brde@optusnet.com.au>
To: Yuri <yuri@tsoft.com>
Cc: freebsd-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org
Subject: Re: misc/118160: unable to mount / rw while booting 7.0-BETA3
Date: Wed, 21 Nov 2007 18:02:10 +1100 (EST)

 On Wed, 21 Nov 2007, Yuri wrote:
 
 > After recompiling and reinstalling the current BETA3 my system has a reboot problem.
 > While booting log says:
 > Starting file system checks:
 > <here goes the list of file systems that it reports, this is ok>
 > mount:  : Operation not permitted.
 
 This is probably a secondary problem.  You apparently have the root device
 mounted on "" or something like that.
 
 > Mounting root file system rw failed, startup aborted.
 > /etc/rc: WARNING: $true is not set properly - see rc.conf(5)
 
 Whatever caused this is probably the main problem.
 
 > and system gets to single user mode.
 
 > In single user mode / is read-only. And command 'mount -uw /' fails
 > Operation not permitted. I count't find the workaround so far.
 
 Please keep line lengths below 80 in mail.
 
 What does mount shouw for the root device?
 
 > The major bug seems to be in the 'mount' system call. 'man mount' says
 > that EPERM is returned if "The caller is neither the super-user nor the
 > owner of dir." I am root.
 
 THis was broken in GEOM somewhere near g_vfs_access().  g_vfs_access()
 returns EPERM for all errors involving exclusive access.  This breaks
 the documented behaviour of [n]mount() returning EBUSY for attempts to
 mount the same device more than once (unless all mounts are r/o -- multiple
 r/o mounts are broken differently, by allowing them and panicing on a
 garbage bufobj pointer later).
 
 You are apparently attempting to mount the same device twice (even though
 -u specifies an already-mounted device, the kernel is apparently confused
 about where it is mounted).
 
 > The secondary problem is this printout: WARNING: $true is not set
 > properly - see rc.conf(5).  It shouldn't print $true
 
 Fix this first.
 
 > Another secondary problem is with man mount(2). Isn't is supposed to
 > mention that setting securelevel also makes 'mount' return EPERM?
 
 I think securelevels break a lot of man pages like that.
 
 > So now I can reboot normally only choosing "single user mode" when I
 > boot and running "mount -uw /" as a single user. And then continuing
 > the boot process.
 
 Yes, it makes some sense for mounting / r/w in the right place gets it
 mounted r/w before other things mess it up.  Don't forget to run fsck -p
 manually before continuing.
 
 I can now see a plausible way to reach the bad state:
 - after booting, the root device is mounted on / r/o with no problems
 - mistype a mount command or have $true generate a wrong mount command,
    so that the root device is mounted somewhere else (I don't know how
    it can be on "", but it could be on " " or on any valid pathname).
    If you preemptively mount it r/w, then this other mount will fail
    -- look in the logs for messages about this.
 - now try to remount / r/w normally.  This will fail due to the r/o mount
    not on /.
 - if there is only 1 extra r/o mount of /, then the r/w mount should work
    after unmounting the extra.  If there are several extras, then unmounting
    them in a certain order should give the bufobj panic.
 
 The EPERM instead of EBUSY error is very confusing.  Another variation
 on it is that after shutdown to single user mode (using "kill -TERM
 1" or similar), and unmounting all devices except / an /dev, and
 remounting / r/o, "fsck -p" and "fsck /" are broken due to problems
 near g_access().  They fail with the now familiar error EPERM.  Some
 file systems have a a hack to allow them fsck to work after booting,
 but it doesn't apply later.
 
 Bruce

From: Bruce Evans <brde@optusnet.com.au>
To: Bruce Evans <brde@optusnet.com.au>
Cc: Yuri <yuri@tsoft.com>, freebsd-gnats-submit@freebsd.org,
        freebsd-bugs@freebsd.org
Subject: Re: misc/118160: unable to mount / rw while booting 7.0-BETA3
Date: Wed, 21 Nov 2007 18:42:27 +1100 (EST)

 On Wed, 21 Nov 2007, Bruce Evans wrote:
 
 > I can now see a plausible way to reach the bad state:
 > - after booting, the root device is mounted on / r/o with no problems
 > - mistype a mount command or have $true generate a wrong mount command,
 >  so that the root device is mounted somewhere else (I don't know how
 >  it can be on "", but it could be on " " or on any valid pathname).
 >  If you preemptively mount it r/w, then this other mount will fail
 >  -- look in the logs for messages about this.
 > - now try to remount / r/w normally.  This will fail due to the r/o mount
 >  not on /.
 > - if there is only 1 extra r/o mount of /, then the r/w mount should work
 >  after unmounting the extra.  If there are several extras, then unmounting
 >  them in a certain order should give the bufobj panic.
 
 The bufobj panic actually happens in the only possible order for trying to
 fix the simulated problem: after booting with -s:
 
      # mount -o ro /dev/ad0s2a /mnt     # create extra mount (ad0s2a is root dev)
      #                                  # like a runaway script might
      # umount /mnt                      # get rid of the extra mount
      # fsck -p /
      --- panic
 
 and
 
      # mount /dev/ad0s2a /mnt           # I forgot the ro
      --- this succeeds due to the hack that lets fsck succeed, but it shouldn't
 
 Bruce

From: Yuri <yuri@rawbw.com>
To: Bruce Evans <brde@optusnet.com.au>
Cc: Yuri <yuri@tsoft.com>, freebsd-gnats-submit@freebsd.org,
        freebsd-bugs@freebsd.org
Subject: Re: misc/118160: unable to mount / rw while booting 7.0-BETA3
Date: Tue, 27 Nov 2007 19:11:52 -0800

 > > While booting log says:
 > > Starting file system checks:
 > > <here goes the list of file systems that it reports, this is ok>
 > > mount:  : Operation not permitted.
 > 
 > This is probably a secondary problem.  You apparently have the root device
 > mounted on "" or something like that.
 > What does mount shouw for the root device?
 > 
 No, when I get to shell after this failure during the normal boot process
 mount shows:
 /dev/ad12s1c    /   (ufs,local,read-only)
 swapinfo shows that swap volume is /dev/ad12s1b
 
 > > The major bug seems to be in the 'mount' system call. 'man mount' says that
 > EPERM is returned if "The caller is neither the super-user nor the owner of
 > dir." I am root.
 > 
 > You are apparently attempting to mount the same device twice (even though
 > -u specifies an already-mounted device, the kernel is apparently confused
 > about where it is mounted).
 > 
 I thought that mount command is supposed to pick up the locations correctly,
 so that when I say 'mount -uw /' device should be picked up from the already
 mounted list.
 
 Also I found that swapon and mount are related in my case. Once swapon is done
 I can't remount root as r/w. And vice versa, when mount -uw is done swapon
 returns EPERM.
 
 This happens when I boot as single user. When I do swapon consecutive
 'mount -uw ' fails. When I do 'mount -uw' consecutive swapon fails.
 So I don't have swap at all since this command failed during boot.
 
 I guess 'nmount' and 'swapon' system calls are similar and somehow interfere
 with each other.
 
 So I still can't boot normally, only through single user mode and I don't
 have swap at all after this.
 
 Yuri

From: Gary Palmer <gpalmer@freebsd.org>
To: Yuri <yuri@rawbw.com>
Cc: Bruce Evans <brde@optusnet.com.au>, Yuri <yuri@tsoft.com>,
	freebsd-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org
Subject: Re: misc/118160: unable to mount / rw while booting 7.0-BETA3
Date: Tue, 27 Nov 2007 23:06:20 -0500

 On Tue, Nov 27, 2007 at 07:11:52PM -0800, Yuri wrote:
 > > > While booting log says:
 > > > Starting file system checks:
 > > > <here goes the list of file systems that it reports, this is ok>
 > > > mount:  : Operation not permitted.
 > > 
 > > This is probably a secondary problem.  You apparently have the root device
 > > mounted on "" or something like that.
 > > What does mount shouw for the root device?
 > > 
 > No, when I get to shell after this failure during the normal boot process
 > mount shows:
 > /dev/ad12s1c    /   (ufs,local,read-only)
 > swapinfo shows that swap volume is /dev/ad12s1b
  
 
 Is your root partition really on the "c" partition?  The "c" partition
 represents the whole disk slice.  Sure you don't mean ad12s1a?
 
 The conflict between using the c partition and swap on the b partion
 (which logically is a subset of the entire slice) could be the cause
 of EPERM
 
 
 > > > The major bug seems to be in the 'mount' system call. 'man mount' says that
 > > EPERM is returned if "The caller is neither the super-user nor the owner of
 > > dir." I am root.
 > > 
 > > You are apparently attempting to mount the same device twice (even though
 > > -u specifies an already-mounted device, the kernel is apparently confused
 > > about where it is mounted).
 > > 
 > I thought that mount command is supposed to pick up the locations correctly,
 > so that when I say 'mount -uw /' device should be picked up from the already
 > mounted list.
 > 
 > Also I found that swapon and mount are related in my case. Once swapon is done
 > I can't remount root as r/w. And vice versa, when mount -uw is done swapon
 > returns EPERM.
 > 
 > This happens when I boot as single user. When I do swapon consecutive
 > 'mount -uw ' fails. When I do 'mount -uw' consecutive swapon fails.
 > So I don't have swap at all since this command failed during boot.
 > 
 > I guess 'nmount' and 'swapon' system calls are similar and somehow interfere
 > with each other.
 > 
 > So I still can't boot normally, only through single user mode and I don't
 > have swap at all after this.
 > 
 > Yuri
 > _______________________________________________
 > freebsd-bugs@freebsd.org mailing list
 > http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 > To unsubscribe, send any mail to "freebsd-bugs-unsubscribe@freebsd.org"
 > 
 > 

From: Bruce Evans <brde@optusnet.com.au>
To: Yuri <yuri@rawbw.com>
Cc: Bruce Evans <brde@optusnet.com.au>, Yuri <yuri@tsoft.com>,
        freebsd-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org
Subject: Re: misc/118160: unable to mount / rw while booting 7.0-BETA3
Date: Wed, 28 Nov 2007 15:19:12 +1100 (EST)

 On Tue, 27 Nov 2007, Yuri wrote:
 
 >>> While booting log says:
 >>> Starting file system checks:
 >>> <here goes the list of file systems that it reports, this is ok>
 >>> mount:  : Operation not permitted.
 >>
 >> This is probably a secondary problem.  You apparently have the root device
 >> mounted on "" or something like that.
 >> What does mount shouw for the root device?
 >>
 > No, when I get to shell after this failure during the normal boot process
 > mount shows:
 > /dev/ad12s1c    /   (ufs,local,read-only)
 > swapinfo shows that swap volume is /dev/ad12s1b
 
 Using the 'c' partition is normally an error.  Here it is a very large
 error.  ad12s1b overlaps ad12s1c, so if swap on ad12s1b is actually
 used then it will normally clobber the file system on ad12s1c.  It
 might be possible to make the clobbering harmless, e.g., by putting
 ad12s1b at the end of ad12s1c and not using all of ad12s1c for the
 file system, but this is harder to configure than a normal configuration
 and is supposed to be disallowed.
 
 > ...
 > Also I found that swapon and mount are related in my case. Once swapon is done
 > I can't remount root as r/w. And vice versa, when mount -uw is done swapon
 > returns EPERM.
 >
 > This happens when I boot as single user. When I do swapon consecutive
 > 'mount -uw ' fails. When I do 'mount -uw' consecutive swapon fails.
 > So I don't have swap at all since this command failed during boot.
 
 This misbehaviour seems to be a result of the disallowment not being
 complete.  Mounted file systems normally have exclusive write access
 to their device.  However, for early root mounts, while the mount is
 only r/o, exclusive write access is given up so that fsck can work.
 There seems to be no exclusivity at all, so swapon on a different but
 overlapping device is granted write access.  Then while swapon is on,
 it holds exclusive write access and r/w mounts of root are denied
 (correctly except for the wrong errno and other misleading things in
 the error message).  Also, after the root mount becomes r/w, swapon
 fails (correctly except for the wrong errno).
 
 Bruce

From: Yuri <yuri@rawbw.com>
To: Bruce Evans <brde@optusnet.com.au>
Cc: Yuri <yuri@tsoft.com>, freebsd-gnats-submit@freebsd.org,
        freebsd-bugs@freebsd.org
Subject: Re: misc/118160: unable to mount / rw while booting 7.0-BETA3
Date: Tue, 27 Nov 2007 22:36:28 -0800

 Quoting Bruce Evans <brde@optusnet.com.au>:
 
 > Using the 'c' partition is normally an error.  Here it is a very large
 > error.  ad12s1b overlaps ad12s1c, so if swap on ad12s1b is actually
 > used then it will normally clobber the file system on ad12s1c.  It
 > might be possible to make the clobbering harmless, e.g., by putting
 > ad12s1b at the end of ad12s1c and not using all of ad12s1c for the
 > file system, but this is harder to configure than a normal configuration
 > and is supposed to be disallowed.
 >
 Having root mounted on ad12s1c is probably a result of my typo.
 
 > 
 > This misbehaviour seems to be a result of the disallowment not being
 > complete.  Mounted file systems normally have exclusive write access
 > to their device.  However, for early root mounts, while the mount is
 > only r/o, exclusive write access is given up so that fsck can work.
 > There seems to be no exclusivity at all, so swapon on a different but
 > overlapping device is granted write access.  Then while swapon is on,
 > it holds exclusive write access and r/w mounts of root are denied
 > (correctly except for the wrong errno and other misleading things in
 > the error message).  Also, after the root mount becomes r/w, swapon
 > fails (correctly except for the wrong errno).
 > 
 
 Should this situation generate system log message, not only EPERM?
 If it would have said: 'Attempt to mount already exclusively locked
 partition' it would have been clear what the problem is right away.
 
 So I would suggest such message to be added, otherwise this PR can be closed.
 
 Yuri
State-Changed-From-To: open->closed 
State-Changed-By: linimon 
State-Changed-When: Sat Mar 1 01:19:39 UTC 2008 
State-Changed-Why:  
Closed at submitter's request. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=118160 
>Unformatted:
