From muir@idiom.com  Tue Mar 10 01:39:13 1998
Received: from idiom.com (idiom.com [140.174.82.4])
          by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id BAA22197
          for <FreeBSD-gnats-submit@freebsd.org>; Tue, 10 Mar 1998 01:39:13 -0800 (PST)
          (envelope-from muir@idiom.com)
Received: (from muir@localhost)
	by idiom.com (8.8.7/8.8.5) id BAA01113;
	Tue, 10 Mar 1998 01:39:12 -0800 (PST)
Message-Id: <199803100939.BAA01113@idiom.com>
Date: Tue, 10 Mar 1998 01:39:12 -0800 (PST)
From: David Muir Sharnoff <muir@idiom.com>
Reply-To: muir@idiom.com
To: FreeBSD-gnats-submit@freebsd.org
Subject: nfsd send interface selection
X-Send-Pr-Version: 3.2

>Number:         5964
>Category:       kern
>Synopsis:       nfsd send interface selection seems broken
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    peter
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Mar 10 01:40:01 PST 1998
>Closed-Date:    Sun Dec 12 21:19:59 PST 1999
>Last-Modified:  Sun Dec 12 21:21:51 PST 1999
>Originator:     David Muir Sharnoff
>Release:        FreeBSD 2.2-STABLE i386
>Organization:
Idiom
>Environment:

	FreeBSD 2.2-STABLE as of a while ago.

>Description:

	I've been trying to get NFS to work reliably.  I know that's a
	lost cause, but ...

	There are two systems involved: idiom.com and another.idiom.com.

	idiom.com is the client.  It's fairly simple: it just has
	one interface.  

	Another.idiom.com is the server.  It has three different interfaces
	over which it could possibly have a route to idiom.com.

	I don't know what's important, so I'm going to give a bit more
	detail than might be needed.

	Another.idiom.com runs gated.  When it boots, it has a default
	route out fxp0 (209.66.121.17).

	After gated starts and T1s come up etc, the best route to Idiom
	is via ETinc T1 card (eth1, 209.157.69.21).

	Another also has a lookback alias address of 209.157.69.251.

	The A records for another point to: 209.66.121.17 & 209.157.69.251.

	Idiom mounts another with:
		mount_nfs -s -b -d -c -P -i -r 1024 another:/ /net/another

	[ASIDE: How come operations hang rather than die?  I thought that
	was the whole point of soft mounts?]

	Sometimes NFS works.  Sometimes it doesn't.

	Sometimes the problem is on idiom.com's end (if I learn more, I'll
	send in a report) and somtimes it's on another.idiom.com's end.

	NFS didn't work today.

	It didn't work because another was trying to reply via the
	wrong interface.

	Here's the request from idiom:
	01:16:14.391105 140.174.82.4.109fb982 > 209.157.69.251.801: 124 access [|nfs]

	Here's the reply: 
	01:16:14.400354 209.66.121.17.801 > 140.174.82.4.109fb982: reply ok 120

	Seems, fine, but it came via the wrong interface.

	On another, netstat -rn | grep 140.174.82:

	140.174.82/27      209.157.69.22      UGc         8      837      eth1
	140.174.82.32/27   209.157.69.22      UGc         3      262      eth1
	140.174.82.66      209.157.69.22      UGH         0        0      eth1
	140.174.82.82      209.157.69.22      UGH         0        0      eth1
	140.174.82.113     209.157.69.22      UGH         0        0      eth1
	140.174.82.128/26  209.157.69.22      UGc         0        0      eth1
	140.174.82.192/26  209.157.69.22      UGc         0        0      eth1

	On another, ipfw -a list | grep 140.174.82 | grep fxp0:

	10500          0          0 deny ip from 140.174.82.0/24 to any in recv fxp0
	10658        359      63568 allow ip from 209.66.121.17 to 140.174.82.4 out xmit fxp0
	10659          7       1372 deny log ip from any to 140.174.82.0/24 out xmit fxp0
	10660         97      18992 deny ip from any to 140.174.82.0/24 out xmit fxp0
	10760          0          0 skipto 10860 ip from 140.174.82.0/24 to any out xmit fxp0

	Rule 10659 was added so that I could figure out what was going
	on:

	Mar 10 01:15:02 another /kernel: nfsd send error 13
	ipfw: 10659 Deny UDP 209.66.121.17:2049 140.174.82.4:1023 out via fxp0

	Rule 10658 was added to specifically allow the packet to go out 
	that way.

	Why is it going out that way?   The route is via eth1!  The
	route via fxp0 is not secure and holes had to be made in firewalls
	to allow the packet through.

>How-To-Repeat:

>Fix:
	
	Unknown.

	One thought though: replies should come from the same IP address
	that the request was sent to.

>Release-Note:
>Audit-Trail:

From: "Jordan K. Hubbard" <jkh@time.cdrom.com>
To: muir@idiom.com
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: kern/5964: nfsd send interface selection 
Date: Fri, 13 Mar 1998 03:17:53 -0800

 > 	Another.idiom.com is the server.  It has three different interfaces
 > 	over which it could possibly have a route to idiom.com.
 
 All of which have differently named interfaces, right?  You didn't do
 the old munchkin trick of naming all 3 interfaces idiom.com in your
 DNS so that NFS gets royally confused about where to route the packets
 back when the other end does a DNS lookup and gets a
 differently-ordered list of addresses as a reply?  I'm sure you
 didn't, but just checking..
 
 > 	[ASIDE: How come operations hang rather than die?  I thought that
 > 	was the whole point of soft mounts?]
 
 No, those are background mounts.  Soft mounts are merely interruptable.
 
 					Jordan

From: David Muir Sharnoff <muir@idiom.com>
To: "Jordan K. Hubbard" <jkh@time.cdrom.com>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: kern/5964: nfsd send interface selection 
Date: Fri, 13 Mar 1998 10:11:59 -0800 (PST)

 * All of which have differently named interfaces, right?  You didn't do
 * the old munchkin trick of naming all 3 interfaces idiom.com in your
 * DNS so that NFS gets royally confused about where to route the packets
 * back when the other end does a DNS lookup and gets a
 * differently-ordered list of addresses as a reply?  I'm sure you
 * didn't, but just checking..
 
 The interfaces all have different names.  I've tried mounting various
 interfaces by IP address.  I've mounted the T1 interface that leads 
 back to idiom.com.  I've mounted a loopback interface address.  The
 behaviour doesn't change much.  
 
 When mounting some interfaces, it takes a while before packets flow the
 wrong way.
 
 The interface I usually mount is a loopback interface.
 
 -Dave
Responsible-Changed-From-To: freebsd-bugs->peter 
Responsible-Changed-By: peter 
Responsible-Changed-When: Sat Apr 25 22:19:15 PDT 1998 
Responsible-Changed-Why:  
I'll take this.. 
State-Changed-From-To: open->closed 
State-Changed-By: dillon 
State-Changed-When: Sun Dec 12 21:19:59 PST 1999 
State-Changed-Why:  
nfsd interface binding has been added to 4.x.  While there are no current 
plans to backport it you may be able to do a poor-man's binding for 3.x 
and 2.2.x by modifying nfsd to bind to a specific IP address rather then 
INADDR_ANY.   

interface binding is not required for tcp connections, only udp connections. 
NFS TCP connections should now be reliable under both 3.x and 4.x (I don't 
know about 2.2.x but I believe some of the bug fixes were backported) 
>Unformatted:
