From nobody@FreeBSD.org  Tue Apr 23 21:30:05 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id B084F4A9
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 23 Apr 2013 21:30:05 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id A29E1105C
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 23 Apr 2013 21:30:05 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r3NLU4m0021286
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 23 Apr 2013 21:30:04 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r3NLU4nD021278;
	Tue, 23 Apr 2013 21:30:04 GMT
	(envelope-from nobody)
Message-Id: <201304232130.r3NLU4nD021278@red.freebsd.org>
Date: Tue, 23 Apr 2013 21:30:04 GMT
From: Garrett Cooper <yaneurabeya@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [md/nfs/vfs] md filesystem layering not handled correctly when stacking on top of NFS; works with tempfs
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         178091
>Category:       kern
>Synopsis:       [md/nfs/vfs] md filesystem layering not handled correctly when stacking on top of NFS; works with tempfs
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Apr 23 21:40:00 UTC 2013
>Closed-Date:    
>Last-Modified:  Wed Apr 24 01:10:00 UTC 2013
>Originator:     Garrett Cooper
>Release:        10-CURRENT
>Organization:
EMC Isilon
>Environment:
FreeBSD mina3010.west.isilon.com 10.0-CURRENT FreeBSD 10.0-CURRENT #1: Sat Apr 20 07:51:00 PDT 2013     root@mina3010.west.isilon.com:/usr/obj/usr/src/sys/ISI-GENERIC  amd64
>Description:
I intentionally layered these filesystems on a build client machine in order to avoid having to rework things with respect to MAKEOBJDIRPREFIX, but it turns out this isn't possible with memory disks mounted on NFS. Example:

Client:

# uname -a
FreeBSD mina3010.west.isilon.com 10.0-CURRENT FreeBSD 10.0-CURRENT #1: Sat Apr 20 07:51:00 PDT 2013     root@mina3010.west.isilon.com:/usr/obj/usr/src/sys/ISI-GENERIC  amd64
# stat -f %d /mnt/onefs.3/obj/
973143811
# stat -f %d /mnt/onefs.3/src/
973143811
# mount | grep /mnt
gran-tourismo:/home/gcooper/svn on /mnt (nfs, asynchronous)
/dev/md42 on /mnt/onefs.3/obj (ufs, local, soft-updates)

This however functions as I would expect with memory disks mounted on ZFS:

Client/Server:

# uname -a
FreeBSD gran-tourismo.west.isilon.com 10.0-CURRENT FreeBSD 10.0-CURRENT #2 r+ba9afe9: Fri Apr 19 20:29:10 PDT 2013     gcooper@gran-tourismo.west.isilon.com:/usr/obj/usr/src/sys/GRAN-TOURISMO  amd64
# mkdir foo; mdmfs -s 100m md4 foo
load: 12.79  cmd: mdmfs 43756 [tx->tx_quiesce_done_cv] 2.29r 0.00u 0.00s 0% 196k
# stat -f %d $PWD/foo/
124
# stat -f %d $PWD
1152244014

There are some VFS layering issues and dot-dot lookup issues with md(4)/NFS that really need to be resolved, because really the device numbers returned for both paths should be different.

tempfs works BTW in place of md(4):

# umount /mnt/onefs.3/obj
# mdconfig -d -u 42
# mount -o size=$(( 100 * 1024 * 1024 )) -t tmpfs tmpfs /mnt/onefs.3/obj
# stat -f %d /mnt/onefs.3/obj/
2264989444
# stat -f %d /mnt/onefs.3/src
973143811
>How-To-Repeat:
1. Mount an NFS share on CURRENT from a CURRENT NFS server.
2. Create a directory on the share.
3. Mount a memory disk on the directory on the share [using mdmfs].
4. Run stat -f %d on the directory on the share and the share itself. The values should be different.
>Fix:


>Release-Note:
>Audit-Trail:

From: Xin Li <delphij@delphij.net>
To: Garrett Cooper <yaneurabeya@gmail.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/178091: [md/nfs/vfs] md filesystem layering not handled
 correctly when stacking on top of NFS; works with tempfs
Date: Tue, 23 Apr 2013 17:39:21 -0700

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA512
 
 I can't reproduce this with -HEAD (r249777M).
 
 % sudo mount -o async undermydesk:/home/delphij /mnt
 % sudo mdmfs -s 1g md /mnt/base
 % mount |grep /mnt
 build:/tank/home/delphij on /mnt (nfs, asynchronous)
 /dev/md0 on /mnt/base (ufs, local, soft-updates)
 % stat -f %d /mnt/freebsd /mnt/base
 973143814
 172
 
 Cheers,
 - -- 
 Xin LI <delphij@delphij.net>    https://www.delphij.net/
 FreeBSD - The Power to Serve!           Live free or die
 -----BEGIN PGP SIGNATURE-----
 
 iQEcBAEBCgAGBQJRdym5AAoJEG80Jeu8UPuzynUIAJMBS5YOamJRMxa7R8Xhbcnp
 n9maaa7sU+oNJY06fKj24NBh9OsNATLhuQ+yfpp2rUmgiVE0FeBhNMK0CZ0V4sNG
 ur2QzEz8VJOwcY4brNzzWld6j+EjohC/sE1JgOgkscHY9blJL8oTIjvedWWXzaLV
 8/qDlzo79gRtBsIxKsvp0ywDuj3QOBh8MrpJq1KBNpySrt/5DDJ6Lpl14JCO20n+
 5Ohfn338xF1EnBpZpKXcUnsESqBKQcwNTQbPN7dLG6YhMrGcO+wj6UO9Vg45Dape
 pMrKUAFmZr8kc6dW9YSl5xN4o8z8vIvkAx/1DMTcpYrnP+Mg0vwSJFmGsOHAeSA=
 =b+n2
 -----END PGP SIGNATURE-----

From: Garrett Cooper <yaneurabeya@gmail.com>
To: d@delphij.net
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: kern/178091: [md/nfs/vfs] md filesystem layering not handled
 correctly when stacking on top of NFS; works with tempfs
Date: Tue, 23 Apr 2013 18:02:33 -0700

 On Tue, Apr 23, 2013 at 5:39 PM, Xin Li <delphij@delphij.net> wrote:
 > -----BEGIN PGP SIGNED MESSAGE-----
 > Hash: SHA512
 >
 > I can't reproduce this with -HEAD (r249777M).
 
 What's the NFS server version and what's the NFS client version? I
 assume that they're not the same machine?
 Thanks!
 -Garrett
 
 > % sudo mount -o async undermydesk:/home/delphij /mnt
 > % sudo mdmfs -s 1g md /mnt/base
 > % mount |grep /mnt
 > build:/tank/home/delphij on /mnt (nfs, asynchronous)
 > /dev/md0 on /mnt/base (ufs, local, soft-updates)
 > % stat -f %d /mnt/freebsd /mnt/base
 > 973143814
 > 172
 >
 > Cheers,
 > - --
 > Xin LI <delphij@delphij.net>    https://www.delphij.net/
 > FreeBSD - The Power to Serve!           Live free or die
 > -----BEGIN PGP SIGNATURE-----
 >
 > iQEcBAEBCgAGBQJRdym5AAoJEG80Jeu8UPuzynUIAJMBS5YOamJRMxa7R8Xhbcnp
 > n9maaa7sU+oNJY06fKj24NBh9OsNATLhuQ+yfpp2rUmgiVE0FeBhNMK0CZ0V4sNG
 > ur2QzEz8VJOwcY4brNzzWld6j+EjohC/sE1JgOgkscHY9blJL8oTIjvedWWXzaLV
 > 8/qDlzo79gRtBsIxKsvp0ywDuj3QOBh8MrpJq1KBNpySrt/5DDJ6Lpl14JCO20n+
 > 5Ohfn338xF1EnBpZpKXcUnsESqBKQcwNTQbPN7dLG6YhMrGcO+wj6UO9Vg45Dape
 > pMrKUAFmZr8kc6dW9YSl5xN4o8z8vIvkAx/1DMTcpYrnP+Mg0vwSJFmGsOHAeSA=
 > =b+n2
 > -----END PGP SIGNATURE-----
>Unformatted:
