From per@bluetail.com  Mon Feb 11 10:47:37 2002
Return-Path: <per@bluetail.com>
Received: from mail.bluetail.com (mail.bluetail.com [195.149.129.26])
	by hub.freebsd.org (Postfix) with ESMTP id 3570437B404
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 11 Feb 2002 10:47:35 -0800 (PST)
Received: from  tordmule.bluetail.com (tordmule.bluetail.com [192.168.128.49])
 by duva.bluetail.com (BLUETAIL Mail Robustifier 2.2.2) with ESMTP
 id 409912.453253.1013duva-s1 for <FreeBSD-gnats-submit@freebsd.org>
 ; Mon, 11 Feb 2002 19:47:33 +0100
Received: (from per@localhost)
	by tordmule.bluetail.com (8.11.6/8.11.6) id g1BIlWf14160;
	Mon, 11 Feb 2002 19:47:32 +0100 (CET)
	(envelope-from per)
Message-Id: <200202111847.g1BIlWf14160@tordmule.bluetail.com>
Date: Mon, 11 Feb 2002 19:47:32 +0100 (CET)
From: Per Hedeland <per@bluetail.com>
Reply-To: Per Hedeland <per@bluetail.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: VmWare port + NIS causes "broadcast storm"
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         34842
>Category:       kern
>Synopsis:       [nis] [patch] VmWare port + NIS causes "broadcast storm"
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Feb 11 10:50:01 PST 2002
>Closed-Date:    
>Last-Modified:  Sun Nov 02 22:40:01 UTC 2008
>Originator:     Per Hedeland
>Release:        FreeBSD 4.5-RELEASE i386
>Organization:
none
>Environment:
System: FreeBSD tordmule.bluetail.com 4.5-RELEASE FreeBSD 4.5-RELEASE #0: Mon Feb 4 19:33:09 CET 2002 root@:/usr/src/sys/compile/TORDMULE i386
Vmware2 port: vmware2-2.0.4.1142

>Description:
	If running the vmware2 port in bridging mode on a system that is
	a NIS client (without specific server(s) specified to ypbind),
	and the NIS server is unavailable for some length of time while
	no vmware host is running, the system will start sending UDP
	broadcasts to port 111 at an extremely high rate. Observed with
	a Pentium-III 800Mhz on 100Mb Ethernet: 200 broadcast
	packets/sec sent - which in turn cause 200 response packets/sec,
	which in turn cause 200 ICMP port unreachable packets/sec from
	the FreeBSD, system since nothing there is listening for the
	responses - in total 600 packets/sec.

>How-To-Repeat:
	See description. Killing the ypserv process on the (only) NIS
	server will cause the problem to appear after some time (not
	measured); it can be more quickly reproduced by killing and
	restarting ypbind on the FreeBSD system while the NIS server is
	down.

>Fix:
	The above is actually caused by the interaction of a series of
	problems:

	1) When bridging is chosen at installation of the vmware2 port,
           the vmnet1 interface is still configured with a "dummy" IP
           address of its own (192.168.0.1), netmask (255.255.255.0),
           and corresponding broadcast address (192.168.0.255). As far
           as I understand, a set of bridged interfaces should have at
           most one IP address total among them.

	2) If packets are sent to *any* address in the "vmnet1 net"
           besides the configured one (192.168.0.1) when no vmware host
           is running, sendto() (or whatever) will soon return ENOBUFS,
           since the "send queue" has filled up. (Needless to say,
           nothing will ever really receive such packets - but they seem
           to "disappear" if a vmware host is running.)

	3) The RPC broadcast function (/usr/src/lib/libc/rpc/pmap_rmt.c/
           clnt_broadcast()) gives up sending immediately (returning
           RPC_CANTSEND), without even waiting for responses, if sending
           to any one of the broadcast-capable interfaces fails for
           whatever reason.

	4) Ypbind, when getting any error back from clnt_broadcast(),
           retries immediately, without any delay or backoff strategy.

	So, in this scenario, ypbind calls clnt_broadcast(), which sends
	a packet out the physical interface, then a packet on the vmnet1
	interface, gets ENOBUFS and gives up, and ypbind starts the
	process over again, ad infinitum.

	The "storm" can be prevented by fixing any one of the problems
	1)-4); a real fix (allowing ypbind to succeed) requires a fix to
	one of 1)-3). Ideally all should be fixed, of course.

	I worked around problem 1) in FreeBSD 4.2-RELEASE and an other
	version of the vmware2 port by modifying the vmware config and
	startup scripts to simply not configure an IP address on vmnet1
	when bridging is used - however this does not work in current
	versions (vmware complains about not being able to get the
	interface address), at least not the trivial way I did it.

	Instead I now looked at problem 2), and came up with the first
	patch below.  It seems reasonable to me, solves problem 2), and
	doesn't seem to affect the "normal" traffic to the vmware host
	in any way - but I guess I could be missing something... I've
	also enclosed what seems to me to be a reasonable fix for
	problem 3), however this is totally untested. I haven't looked
	for a fix for problem 4), but someone probably should...


Patch for problem 2):

--- /usr/src/sys/net/if_tap.c.ORIG	Thu Jul 27 15:57:05 2000
+++ /usr/src/sys/net/if_tap.c	Mon Feb 11 15:20:30 2002
@@ -467,10 +467,10 @@
 	/*
 	 * do not junk pending output if we are in VMnet mode.
 	 * XXX: can this do any harm because of queue overflow?
+	 * Ummm, yes...
 	 */
 
-	if (((tp->tap_flags & TAP_VMNET) == 0) && 
-	    ((tp->tap_flags & TAP_READY) != TAP_READY)) {
+	if ((tp->tap_flags & TAP_READY) != TAP_READY) {
 		struct mbuf	*m = NULL;
 
 		TAPDEBUG("%s%d not ready. minor = %#x, tap_flags = 0x%x\n",



Patch for problem 3):

--- /usr/src/lib/libc/rpc/pmap_rmt.c.ORIG	Fri Mar  3 14:04:58 2000
+++ /usr/src/lib/libc/rpc/pmap_rmt.c	Mon Feb 11 12:54:33 2002
@@ -330,15 +330,19 @@
 	 * minute or so
 	 */
 	for (t.tv_sec = 4; t.tv_sec <= 14; t.tv_sec += 2) {
+		int success = 0;
 		for (i = 0; i < nets; i++) {
 			baddr.sin_addr = addrs[i];
 			if (sendto(sock, outbuf, outlen, 0,
 				(struct sockaddr *)&baddr,
-				sizeof (struct sockaddr)) != outlen) {
-				perror("Cannot send broadcast packet");
-				stat = RPC_CANTSEND;
-				goto done_broad;
+				sizeof (struct sockaddr)) == outlen) {
+				success++;
 			}
+		}
+		if (!success) {
+			perror("Cannot send broadcast packet");
+			stat = RPC_CANTSEND;
+			goto done_broad;
 		}
 		if (eachresult == NULL) {
 			stat = RPC_SUCCESS;
>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: iedowse 
State-Changed-When: Sun Jun 30 16:36:41 PDT 2002 
State-Changed-Why:  

Thanks for the bug report and patches. Item 3 has been committed 
to -stable now (the problem was not present in -current). Your patch 
for problem 2 (if_tap.c) is probably not necessary, and other network 
drivers can get into similar states too, so keeping the changes in 
userland is better. 

May I close this report now? Maybe you would like to suggest a patch 
for ypbind to make it sleep for a while after before repeating the 
clnt_broadcast operation if it fails? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=34842 

From: Per Hedeland <per@bluetail.com>
To: freebsd-gnats-submit@FreeBSD.org, per@bluetail.com
Cc:  
Subject: Re: misc/34842: VmWare port + NIS causes "broadcast storm"
Date: Sun, 08 Sep 2002 20:35:25 +0200

 I replied directly to iedowse@FreeBSD.org on this issue, but apparently
 that didn't "take", so here's the gist of that response:
 
 -------
 > Your patch
 >for problem 2 (if_tap.c) is probably not necessary, and other network
 >drivers can get into similar states too, so keeping the changes in
 >userland is better.
 
 Yes, thinking a bit more about it, I agree - and the kernel patch is
 really a rather broken way to deal with it, since it causes information
 loss, even if the information is pretty useless in this case.
 
 >May I close this report now?
 
 Sure.
 
 > Maybe you would like to suggest a patch
 >for ypbind to make it sleep for a while after before repeating the
 >clnt_broadcast operation if it fails?
 
 Well, I looked at it a bit, and doing it "right" is rather more work
 than I'm prepared to do right now, since that really would need to be
 tested too. I.e. the main process should wait before forking off a new
 broadcaster, but I believe it shouldn't block during that wait - so it
 would need to note the failure (which would need to be returned from
 rpc_received() via handle_children()), use a shorter select() timeout,
 and then do the retry when that timeout expires, keeping in mind that
 other requests may arrive in the meantime so the timeout should really
 be re-calculated based on gettimeofday() etc...
 
 Below is a "stupid" but almost certainly "safe" patch - still untested
 though.
 
 --Per
 
 ---------------------------
 --- /usr/src/usr.sbin/ypbind/ypbind.c.ORIG      Sat Jul  7 09:30:51 2001
 +++ /usr/src/usr.sbin/ypbind/ypbind.c   Sat Jul  6 03:47:33 2002
 @@ -596,6 +596,13 @@
         struct timeval timeout;
         fd_set fds;
  
 +       if (addr->sin_addr.s_addr == (long)0) {
 +               /* Wait a bit before telling parent about failure, since it
 +                  will retry immediately - the wait should really be before
 +                  that retry in the parent, but this is simpler...         */
 +               sleep(2);
 +       }
 +
         timeout.tv_sec = 5;
         timeout.tv_usec = 0;
State-Changed-From-To: feedback->open 
State-Changed-By: ceri 
State-Changed-When: Sun Jun 8 11:05:30 PDT 2003 
State-Changed-Why:  
Feedback was requested and received. 
I can personally confirm that ypbind will still flood the network 
continuously with no backoff if the NIS server goes down on a 
June 1st, 2003 -STABLE. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=34842 
Responsible-Changed-From-To: freebsd-bugs->ceri 
Responsible-Changed-By: bms 
Responsible-Changed-When: Mon Mar 5 13:52:40 UTC 2007 
Responsible-Changed-Why:  
by request 

http://www.freebsd.org/cgi/query-pr.cgi?pr=34842 
Responsible-Changed-From-To: ceri->freebsd-bugs 
Responsible-Changed-By: ceri 
Responsible-Changed-When: Sun Nov 2 22:37:41 UTC 2008 
Responsible-Changed-Why:  
I'm not really in this project anymore. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=34842 
>Unformatted:
