From nobody@FreeBSD.org  Sun Jan  5 03:15:34 2014
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTPS id 88166CC0
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  5 Jan 2014 03:15:34 +0000 (UTC)
Received: from oldred.freebsd.org (oldred.freebsd.org [IPv6:2001:1900:2254:206a::50:4])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id 72E0B19CD
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  5 Jan 2014 03:15:34 +0000 (UTC)
Received: from oldred.freebsd.org ([127.0.1.6])
	by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id s053FYBv000949
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 5 Jan 2014 03:15:34 GMT
	(envelope-from nobody@oldred.freebsd.org)
Received: (from nobody@localhost)
	by oldred.freebsd.org (8.14.5/8.14.5/Submit) id s053FYjQ000941;
	Sun, 5 Jan 2014 03:15:34 GMT
	(envelope-from nobody)
Message-Id: <201401050315.s053FYjQ000941@oldred.freebsd.org>
Date: Sun, 5 Jan 2014 03:15:34 GMT
From: Nathan Dorfman <na@rtfm.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: WORLDTMP first in PATH during installworld
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         185480
>Category:       misc
>Synopsis:       WORLDTMP first in PATH during installworld
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jan 05 03:20:00 UTC 2014
>Closed-Date:    
>Last-Modified:  Sun Jan  5 21:30:00 UTC 2014
>Originator:     Nathan Dorfman
>Release:        10.0-RC4 r260251
>Organization:
>Environment:
FreeBSD server 10.0-RC4 FreeBSD 10.0-RC4 #0 r260251: Sat Jan  4 15:48:21 EST 2014     na@server:/mnt/obj/usr/src/sys/GENERIC  amd64
FreeBSD hudson 10.0-RC4 FreeBSD 10.0-RC4 #0 r260251: Sat Jan  4 18:18:03 EST 2014     na@client:/mnt/obj/mips.mips64/usr/src/sys/ERL  mips

>Description:
Disclaimer: I don't know much about this stuff, but after posting it here: http://lists.freebsd.org/pipermail/freebsd-mips/2014-January/003227.html
I was asked to submit a PR, so here it is.

This only causes failures if one tries to installworld on a different platform than buildworld was run on, but I suspect it may be silently hiding unintentional behavior in the other, more usual cases.

The various subdirs of WORLDTMP (bin, usr/bin, usr/sbin, and so on) are the first elements of PATH during installkernel/installworld. Is this intentional? A couple of things suggest maybe not.

The first is that WORLDTMP contains a toolchain for MACHINE_ARCH, even if you are cross-compiling by setting TARGET_ARCH. So, if you later try to install the crossbuilt kernel/world on the TARGET_ARCH machine, it will try to execute binaries built for the build machine, and fail:

    cc: Exec format error
    make[2]: "/usr/src/share/mk/bsd.compiler.mk" line 9: warning: "cc
--version" returned non-zero status

This is easily remedied by either deleting WORLDTMP between build and install, or changing its definition in Makefile.inc1 to include ${MACHINE_ARCH} in the name before buildworld, so it won't be found by an installworld with a different MACHINE_ARCH. I'm not sure this is the right fix, though, as it leaves installkernel/world with only the existing installed root (/bin, /usr/bin, ...) in its PATH.

The second one is a bit more glaringly obvious. After the above workaround is used, installworld fails a little later:

    install -s -o root -g wheel -m 444   -fschg -S  libc.so.7 /lib
    install: exec(strip): No such file or directory

The PATH here again begins with the root at WORLDTMP, but ends not with the system root but /tmp/install.3aqj2XwQ. I noticed that installworld had just created this directory, and simply neglected to put strip in it, easily fixed by adding strip to the list of ITOOLS= in Makefile.inc1.

However, under normal circumstances, nothing in this new directory is ever used, since everything is found in WORLDTMP first. Can that be right? We just went through the trouble of creating and populating a directory we don't intend to use? It seemed more likely that the intent is to use it, and that WORLDTMP is first in the PATH by mistake.
>How-To-Repeat:
server# uname -m
amd64
server# cd /usr/src && svn diff && svn info | egrep '^URL|Revision'
URL: svn://svn0.us-east.freebsd.org/base/releng/10.0
Revision: 260251
server# export MAKEOBJDIRPREFIX=/mnt/obj
server# rm -rf $MAKEOBJDIRPREFIX
server# TARGET=mips TARGET=mips64 make KERNCONF=ERL buildworld buildkernel

client# uname -m
mips
client# mount | grep /usr
server:/usr/src on /usr/src (nfs, read-only)
server:/mnt/obj/mips.mips64 on /usr/obj (nfs, read-only)
client# cd /usr/src && make KERNCONF=ERL installkernel installworld


>Fix:
If it's intentional that WORLDTMP is first in the PATH, then:
  1. why are we bothering to build IROOT?
  2. can it at least be redefined to include ${MACHINE_ARCH} in the name, so it won't break later installs run on a different arch?
  3. if this is to be fully supported, then I guess a version of WORLDTMP needs to be compiled for the TARGET_ARCH as well when cross-building.

If it isn't intentional... well. I guess it's a lot simpler then. :)

>Release-Note:
>Audit-Trail:

From: Brooks Davis <brooks@freebsd.org>
To: Nathan Dorfman <na@rtfm.net>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/185480: WORLDTMP first in PATH during installworld
Date: Sun, 5 Jan 2014 13:30:04 -0600

 I believe that WORLDTMP is first the path to allow new versions of tools
 to be used in the install process.  It's critical that we do this or we
 could only use new tool features after multiple major releases.
 
 It is not supported to build on one system and install on another.  It
 could be, but it isn't now.  Apparently it's never been a high enough
 priority for anyone, probably because there are plenty of workaround.
 
 The simplest workaround is to just do an installworld to some arbitrary
 DESTDIR, tar up the result, remove schg flags on the target with
 "chflags -R noschg /", and extracting the tarball.  With the -DNO_ROOT
 feature I added to the install targets a while back this is easily
 accomplished even without root access on the build system.  Just do
 installworld with -DNO_ROOT and then use ${DESTDIR}/METALOG as the input
 to tar.
 
 -- Brooks

From: Nathan Dorfman <na@rtfm.net>
To: Brooks Davis <brooks@freebsd.org>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: misc/185480: WORLDTMP first in PATH during installworld
Date: Sun, 5 Jan 2014 16:03:24 -0500

 Thanks for the explanation. That just leaves one question: why are we
 bothering to create and populate ${INSTALLTMP}? Under normal
 circumstances, where ${WORLDTMP} exists, it doesn't seem to be used.
 In fact, it's missing the 'strip' binary, but it doesn't make a
 difference until I come along and try to run without ${WORLDTMP}...
 
 As for the rest, if the process is unsupported, then I guess I will
 stop doing it :) ... but I'd just like to state for the record that it
 seems to work perfectly fine here, after these workarounds. I think
 it's also possible to have a "correct" solution, by using the freshly
 built world instead of ${WORLDTMP} when ${MACHINE_ARCH} says so. If
 there is any interest at all, I can get that working and submit a
 patch.
 
 On Sun, Jan 5, 2014 at 2:30 PM, Brooks Davis <brooks@freebsd.org> wrote:
 > I believe that WORLDTMP is first the path to allow new versions of tools
 > to be used in the install process.  It's critical that we do this or we
 > could only use new tool features after multiple major releases.
 >
 > It is not supported to build on one system and install on another.  It
 > could be, but it isn't now.  Apparently it's never been a high enough
 > priority for anyone, probably because there are plenty of workaround.
 >
 > The simplest workaround is to just do an installworld to some arbitrary
 > DESTDIR, tar up the result, remove schg flags on the target with
 > "chflags -R noschg /", and extracting the tarball.  With the -DNO_ROOT
 > feature I added to the install targets a while back this is easily
 > accomplished even without root access on the build system.  Just do
 > installworld with -DNO_ROOT and then use ${DESTDIR}/METALOG as the input
 > to tar.
 >
 > -- Brooks

From: Brooks Davis <brooks@freebsd.org>
To: Nathan Dorfman <na@rtfm.net>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: misc/185480: WORLDTMP first in PATH during installworld
Date: Sun, 5 Jan 2014 15:27:01 -0600

 On Sun, Jan 05, 2014 at 04:03:24PM -0500, Nathan Dorfman wrote:
 > Thanks for the explanation. That just leaves one question: why are we
 > bothering to create and populate ${INSTALLTMP}? Under normal
 > circumstances, where ${WORLDTMP} exists, it doesn't seem to be used.
 > In fact, it's missing the 'strip' binary, but it doesn't make a
 > difference until I come along and try to run without ${WORLDTMP}...
 
 Hmm, right, I'd forgotten about INSTALLTMP.  With that in mind I think
 you may be correct.  We shouldn't need WORLDTMP at all.
 
 > As for the rest, if the process is unsupported, then I guess I will
 > stop doing it :) ... but I'd just like to state for the record that it
 > seems to work perfectly fine here, after these workarounds. I think
 > it's also possible to have a "correct" solution, by using the freshly
 > built world instead of ${WORLDTMP} when ${MACHINE_ARCH} says so. If
 > there is any interest at all, I can get that working and submit a
 > patch.
 
 If it can be done without adding much more complexity it's probably
 worth doing.  I'm a bit skeptical that it will be easy to get the built
 world to actually work in the general case where things in INSTALLTMP
 depend on a new libc but it should be possible.
 
 -- Brooks
>Unformatted:
