From rand@meridian-enviro.com  Mon Mar 10 19:09:31 2008
Return-Path: <rand@meridian-enviro.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 8372F1065678
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 10 Mar 2008 19:09:31 +0000 (UTC)
	(envelope-from rand@meridian-enviro.com)
Received: from newman.meridian-enviro.com (newman.meridian-enviro.com [12.192.92.56])
	by mx1.freebsd.org (Postfix) with ESMTP id 1C0B88FC24
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 10 Mar 2008 19:09:30 +0000 (UTC)
	(envelope-from rand@meridian-enviro.com)
Received: from luna-0.meridian-enviro.com ([10.10.10.3])
	by newman.meridian-enviro.com (8.13.6/8.13.6) with ESMTP id m2AJ9Ubd098481
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 10 Mar 2008 14:09:30 -0500 (CDT)
	(envelope-from rand@meridian-enviro.com)
Received: (from rand@localhost)
	by luna-0.meridian-enviro.com (8.13.8/8.13.8/Submit) id m2AJ9Tmf017332;
	Mon, 10 Mar 2008 14:09:29 -0500 (CDT)
	(envelope-from rand)
Message-Id: <200803101909.m2AJ9Tmf017332@luna-0.meridian-enviro.com>
Date: Mon, 10 Mar 2008 14:09:29 -0500 (CDT)
From: "Douglas K. Rand" <rand@meridian-enviro.com>
Reply-To: "Douglas K. Rand" <rand@meridian-enviro.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: CARP hash dependent on order of IP addresses on interface
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         121574
>Category:       kern
>Synopsis:       [carp] CARP hash dependent on order of IP addresses on interface
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    mlaier
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 10 19:10:02 UTC 2008
>Closed-Date:    Mon Sep 01 23:15:20 UTC 2008
>Last-Modified:  Mon Sep 01 23:15:20 UTC 2008
>Originator:     Douglas K. Rand
>Release:        FreeBSD 6.2-RELEASE-p7 i386
>Organization:
Meridian Environmental Technology, Inc.
>Environment:
System: FreeBSD luna-0.meridian-enviro.com 6.2-RELEASE-p7 FreeBSD 6.2-RELEASE-p7 #0: Fri Aug 24 20:46:09 CDT 2007 root@:/usr/obj/usr/src/sys/LUNA i386


>Description:
Recently I had to reboot one of my redundant firewalls and
the /etc/rc.conf file had exchanged the primary and alias
addresses of the interface. (We were switching ISPs.)

If you have a carp interface with 2 IP addresses, the IP addresses
MUST be assigned in the same order on both interfaces. Luna-0 crashed
this morning, and when I brought it back up the carp1 interface on
BOTH Luna-0 and Luna-1 were in the MASTER state.

From Luna-0:
carp1: flags=49<UP,LOOPBACK,RUNNING> mtu 1500
        inet 12.192.93.2 netmask 0xffffffc0 
        inet 207.109.234.254 netmask 0xfffffff0 
        carp: MASTER vhid 2 advbase 1 advskew 0

From Luna-1:
carp1: flags=49<UP,LOOPBACK,RUNNING> mtu 1500
        inet 207.109.234.254 netmask 0xfffffff0 
        inet 12.192.93.2 netmask 0xffffffc0 
        carp: MASTER vhid 2 advbase 1 advskew 50

Both were trying to act as the master carp interface, which was
causing some strange TCP errors. (Bryan noticed his automated SSH
sessions to Mars were failing.)

Once I got it straightened out, everything worked better. And of
course, once I got the addresses added to the interfaces in the right
order I realized that since the Qwest stuff is disconnected, I didn't
need the 207.109.234.254 address anymore anyway, so I could have just
deleted that, which I've now done.

Thing thing to watch for is a LOT of "carp.: incorrect hash" messages
from the kernel. I ignored the few I saw because you ALWAYS get a few
of those at boot, but I didn't realize how many were being generated.

>How-To-Repeat:
Change the order of the IP addresses on a CARP interface from
its peer.
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Mar 10 23:32:44 UTC 2008 
Responsible-Changed-Why:  

http://www.freebsd.org/cgi/query-pr.cgi?pr=121574 

From: "Wouter de Jong" <maddog2k@maddog2k.net>
To: <bug-followup@FreeBSD.org>,
	<rand@meridian-enviro.com>
Cc:  
Subject: Re: kern/121574: [carp] CARP hash dependent on order of IPaddresses on interface
Date: Tue, 11 Mar 2008 09:58:49 +0100

 While it would be great to see carp fixed to handle this issue, 
 shouldn't the order of IP-addresses on the interface be fixed as well ?
 
 It is very random, which people won't expect ... :)

From: "Max Laier" <max@love2party.net>
To: bug-followup@freebsd.org
Cc: rand@meridian-enviro.com, maddog2k@maddog2k.net
Subject: Re: kern/121574: [carp] CARP hash dependent on order of IP 
     addresses on interface
Date: Tue, 11 Mar 2008 11:28:34 +0100 (CET)

 ------=_20080311112834_27075
 Content-Type: text/plain; charset="iso-8859-1"
 Content-Transfer-Encoding: 8bit
 
 (subject helps)
 
 Please try the attached patch (mostly OpenBSD rev. 1.114 plus follow-ups).
  This should fix the symptom.  I have no idea why your address lists would
 get skewed however.  They "should"[tm] be in the order in which you add
 aliases.
 
 -- Max
 ------=_20080311112834_27075
 Content-Type: text/x-diff; name="ip_carp.c.HEAD.diff"
 Content-Transfer-Encoding: 8bit
 Content-Disposition: attachment; filename="ip_carp.c.HEAD.diff"
 
 Index: ip_carp.c
 ===================================================================
 RCS file: /home/mlaier/fcvs/src/sys/netinet/ip_carp.c,v
 retrieving revision 1.27.2.12
 diff -u -r1.27.2.12 ip_carp.c
 --- ip_carp.c	1 Feb 2008 11:20:41 -0000	1.27.2.12
 +++ ip_carp.c	11 Mar 2008 10:16:12 -0000
 @@ -238,9 +238,12 @@
  	u_int8_t version = CARP_VERSION, type = CARP_ADVERTISEMENT;
  	u_int8_t vhid = sc->sc_vhid & 0xff;
  	struct ifaddr *ifa;
 -	int i;
 +	int i, found;
 +#ifdef INET
 +	struct in_addr last, cur, in;
 +#endif
  #ifdef INET6
 -	struct in6_addr in6;
 +	struct in6_addr last6, cur6, in6;
  #endif
  
  	if (sc->sc_carpdev)
 @@ -261,21 +264,44 @@
  	SHA1Update(&sc->sc_sha1, (void *)&type, sizeof(type));
  	SHA1Update(&sc->sc_sha1, (void *)&vhid, sizeof(vhid));
  #ifdef INET
 -	TAILQ_FOREACH(ifa, &SC2IFP(sc)->if_addrlist, ifa_list) {
 -		if (ifa->ifa_addr->sa_family == AF_INET)
 -			SHA1Update(&sc->sc_sha1,
 -			    (void *)&ifatoia(ifa)->ia_addr.sin_addr.s_addr,
 -			    sizeof(struct in_addr));
 -	}
 +	cur.s_addr = 0;
 +	do {
 +		found = 0;
 +		last = cur;
 +		cur.s_addr = 0xffffffff;
 +		TAILQ_FOREACH(ifa, &SC2IFP(sc)->if_addrlist, ifa_list) {
 +			in.s_addr = ifatoia(ifa)->ia_addr.sin_addr.s_addr;
 +			if (ifa->ifa_addr->sa_family == AF_INET &&
 +			    ntohl(in.s_addr) > ntohl(last.s_addr) &&
 +			    ntohl(in.s_addr) < ntohl(cur.s_addr)) {
 +				cur.s_addr = in.s_addr;
 +				found++;
 +			}
 +		}
 +		if (found)
 +			SHA1Update(&sc->sc_sha1, (void *)&cur, sizeof(cur));
 +	} while (found);
  #endif /* INET */
  #ifdef INET6
 -	TAILQ_FOREACH(ifa, &SC2IFP(sc)->if_addrlist, ifa_list) {
 -		if (ifa->ifa_addr->sa_family == AF_INET6) {
 +	memset(&cur6, 0, sizeof(cur6));
 +	do {
 +		found = 0;
 +		last6 = cur6;
 +		memset(&cur6, 0xff, sizeof(cur6));
 +		TAILQ_FOREACH(ifa, &SC2IFP(sc)->if_addrlist, ifa_list) {
  			in6 = ifatoia6(ifa)->ia_addr.sin6_addr;
 -			in6_clearscope(&in6);
 -			SHA1Update(&sc->sc_sha1, (void *)&in6, sizeof(in6));
 +			if (IN6_IS_SCOPE_EMBED(&in6))
 +				in6.s6_addr16[1] = 0;
 +			if (ifa->ifa_addr->sa_family == AF_INET6 &&
 +			    memcmp(&in6, &last6, sizeof(in6)) > 0 &&
 +			    memcmp(&in6, &cur6, sizeof(in6)) < 0) {
 +				cur6 = in6;
 +				found++;
 +			}
  		}
 -	}
 +		if (found)
 +			SHA1Update(&sc->sc_sha1, (void *)&cur6, sizeof(cur6));
 +	} while (found);
  #endif /* INET6 */
  
  	/* convert ipad to opad */
 ------=_20080311112834_27075
 Content-Type: text/x-diff; name="ip_carp.c.RELENG_6.diff"
 Content-Transfer-Encoding: 8bit
 Content-Disposition: attachment; filename="ip_carp.c.RELENG_6.diff"
 
 Index: ip_carp.c
 ===================================================================
 RCS file: /home/mlaier/fcvs/src/sys/netinet/ip_carp.c,v
 retrieving revision 1.27.2.12
 diff -u -r1.27.2.12 ip_carp.c
 --- ip_carp.c	1 Feb 2008 11:20:41 -0000	1.27.2.12
 +++ ip_carp.c	11 Mar 2008 10:20:42 -0000
 @@ -238,9 +238,12 @@
  	u_int8_t version = CARP_VERSION, type = CARP_ADVERTISEMENT;
  	u_int8_t vhid = sc->sc_vhid & 0xff;
  	struct ifaddr *ifa;
 -	int i;
 +	int i, found;
 +#ifdef INET
 +	struct in_addr last, cur, in;
 +#endif
  #ifdef INET6
 -	struct in6_addr in6;
 +	struct in6_addr last6, cur6, in6;
  #endif
  
  	if (sc->sc_carpdev)
 @@ -261,21 +264,45 @@
  	SHA1Update(&sc->sc_sha1, (void *)&type, sizeof(type));
  	SHA1Update(&sc->sc_sha1, (void *)&vhid, sizeof(vhid));
  #ifdef INET
 -	TAILQ_FOREACH(ifa, &SC2IFP(sc)->if_addrlist, ifa_list) {
 -		if (ifa->ifa_addr->sa_family == AF_INET)
 -			SHA1Update(&sc->sc_sha1,
 -			    (void *)&ifatoia(ifa)->ia_addr.sin_addr.s_addr,
 -			    sizeof(struct in_addr));
 -	}
 +	cur.s_addr = 0;
 +	do {
 +		found = 0;
 +		last = cur;
 +		cur.s_addr = 0xffffffff;
 +		TAILQ_FOREACH(ifa, &SC2IFP(sc)->if_addrlist, ifa_list) {
 +			in.s_addr = ifatoia(ifa)->ia_addr.sin_addr.s_addr;
 +			if (ifa->ifa_addr->sa_family == AF_INET &&
 +			    ntohl(in.s_addr) > ntohl(last.s_addr) &&
 +			    ntohl(in.s_addr) < ntohl(cur.s_addr)) {
 +				cur.s_addr = in.s_addr;
 +				found++;
 +			}
 +		}
 +		if (found)
 +			SHA1Update(&sc->sc_sha1, (void *)&cur, sizeof(cur));
 +	} while (found);
  #endif /* INET */
  #ifdef INET6
 -	TAILQ_FOREACH(ifa, &SC2IFP(sc)->if_addrlist, ifa_list) {
 -		if (ifa->ifa_addr->sa_family == AF_INET6) {
 +	memset(&cur6, 0, sizeof(cur6));
 +	do {
 +		found = 0;
 +		last6 = cur6;
 +		memset(&cur6, 0xff, sizeof(cur6));
 +		TAILQ_FOREACH(ifa, &SC2IFP(sc)->if_addrlist, ifa_list) {
  			in6 = ifatoia6(ifa)->ia_addr.sin6_addr;
 -			in6_clearscope(&in6);
 -			SHA1Update(&sc->sc_sha1, (void *)&in6, sizeof(in6));
 +			if (IN6_IS_SCOPE_LINKLOCAL(&in6) ||
 +			    IN6_IS_ADDR_MC_INTFACELOCAL(&in6))
 +				in6.s6_addr16[1] = 0;
 +			if (ifa->ifa_addr->sa_family == AF_INET6 &&
 +			    memcmp(&in6, &last6, sizeof(in6)) > 0 &&
 +			    memcmp(&in6, &cur6, sizeof(in6)) < 0) {
 +				cur6 = in6;
 +				found++;
 +			}
  		}
 -	}
 +		if (found)
 +			SHA1Update(&sc->sc_sha1, (void *)&cur6, sizeof(cur6));
 +	} while (found);
  #endif /* INET6 */
  
  	/* convert ipad to opad */
 ------=_20080311112834_27075--
 
State-Changed-From-To: open->feedback 
State-Changed-By: mlaier 
State-Changed-When: Tue Mar 11 11:03:51 UTC 2008 
State-Changed-Why:  
Patch needs testing. 


Responsible-Changed-From-To: freebsd-net->mlaier 
Responsible-Changed-By: mlaier 
Responsible-Changed-When: Tue Mar 11 11:03:51 UTC 2008 
Responsible-Changed-Why:  
Take this one. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121574 

From: "Douglas K. Rand" <rand@meridian-enviro.com>
To: "Max Laier" <max@love2party.net>
Cc: bug-followup@freebsd.org, maddog2k@maddog2k.net
Subject: Re: kern/121574: [carp] CARP hash dependent on order of IP       addresses on interface
Date: Tue, 11 Mar 2008 08:11:57 -0500

 Max> Please try the attached patch (mostly OpenBSD rev. 1.114 plus
 Max> follow-ups).
 
 OK.
 
 Max> This should fix the symptom.  I have no idea why your address
 Max> lists would get skewed however.  They "should"[tm] be in the
 Max> order in which you add aliases.
 
 Sorry, perhaps my PR wasn't clear. The addresses ARE in the order in
 which they were added. My rc.conf changed such that the addresses were
 added in a different order. (The ifconfig_fxp1 and ifconfig_fxp1_alias0 
 lines were exchanged because we were migrating from one ISP to
 another.) 
 
 I was surprised that the order of the addresses caused our CARP
 problem. If this is the expected behaviour, thats fine, I know better
 now.  :)
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/121574: commit references a PR
Date: Mon,  2 Jun 2008 18:58:28 +0000 (UTC)

 mlaier      2008-06-02 18:58:23 UTC
 
   FreeBSD src repository
 
   Modified files:
     sys/netinet          ip_carp.c 
   Log:
   SVN rev 179490 on 2008-06-02 18:58:07Z by mlaier
   
   Sort IP addresses before hashing them for the signature.  Otherwise carp is
   sensitive to address configuration order.
   
   PR:             kern/121574
   Reported by:    Douglas K. Rand, Wouter de Jong
   Obtained from:  OpenBSD (rev 1.114 + fixes)
   MFC after:      2 weeks
   
   Revision  Changes    Path
   1.54      +39 -13    src/sys/netinet/ip_carp.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: feedback->patched 
State-Changed-By: mlaier 
State-Changed-When: Mon Jun 2 19:00:31 UTC 2008 
State-Changed-Why:  
Committed to head, MFC timeout in two weeks. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121574 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/121574: commit references a PR
Date: Tue, 24 Jun 2008 23:25:45 +0000 (UTC)

 mlaier      2008-06-24 23:24:38 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_7)
     sys/dev/ciss         ciss.c 
     sys/netinet          ip_carp.c 
     sys/pci              nfsmb.c 
   Log:
   SVN rev 179986 on 2008-06-24 23:24:38Z by mlaier
   
   MFC: r179490
   
     Sort IP addresses before hashing them for the signature.  Otherwise carp
     is sensitive to address configuration order.
   
     PR:                   kern/121574
     Reported by:          Douglas K. Rand, Wouter de Jong
     Obtained from:        OpenBSD (rev 1.114 + fixes)
   
   Revision   Changes    Path
   1.81.2.11  +0 -0      src/sys/dev/ciss/ciss.c
   1.52.2.1   +39 -13    src/sys/netinet/ip_carp.c
   1.6.2.9    +0 -0      src/sys/pci/nfsmb.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/121574: commit references a PR
Date: Mon,  1 Sep 2008 23:05:57 +0000 (UTC)

 mlaier      2008-09-01 23:05:41 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_6)
     sys/netinet          ip_carp.c 
   Log:
   SVN rev 182652 on 2008-09-01 23:05:41Z by mlaier
   
   MFC: r179490
   
    Sort IP addresses before hashing them for the signature. Otherwise carp
    is sensitive to address configuration order.
   
   PR:             kern/121574
   Forgotten by:   mlaier
   
   Revision   Changes    Path
   1.27.2.13  +39 -13    src/sys/netinet/ip_carp.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: mlaier 
State-Changed-When: Mon Sep 1 23:14:30 UTC 2008 
State-Changed-Why:  
Fix is MFCed to stable/{6,7} to be part of the upcoming releases. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121574 
>Unformatted:
