From nobody@FreeBSD.org  Wed Feb 25 18:50:44 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id B6F71106574A
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 25 Feb 2009 18:50:42 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 7FB7B8FC1B
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 25 Feb 2009 18:50:42 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n1PIog22011610
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 25 Feb 2009 18:50:42 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n1PIog3K011609;
	Wed, 25 Feb 2009 18:50:42 GMT
	(envelope-from nobody)
Message-Id: <200902251850.n1PIog3K011609@www.freebsd.org>
Date: Wed, 25 Feb 2009 18:50:42 GMT
From: Andrew Daugherity <adaugherity@tamu.edu>
To: freebsd-gnats-submit@FreeBSD.org
Subject: carp(4) advskew setting ignored when carp IP used on a gif(4) interface
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         132107
>Category:       kern
>Synopsis:       [carp] carp(4) advskew setting ignored when carp IP used on a gif(4) interface
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    glebius
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 25 19:00:09 UTC 2009
>Closed-Date:    
>Last-Modified:  Sat Dec 17 03:30:11 UTC 2011
>Originator:     Andrew Daugherity
>Release:        7.1
>Organization:
Texas A&M University
>Environment:
FreeBSD imslb4.tamu.edu 7.1-RELEASE-p1 FreeBSD 7.1-RELEASE-p1 #0: Wed Jan 14 15:03:53 CST 2009     root@imslb4.tamu.edu:/usr/obj/usr/src/sys/IMS  amd64
>Description:
For background, see my post to the freebsd-pf mailing list:
http://docs.freebsd.org/cgi/mid.cgi?38ce25da0901271659m2b6d8a1fg2e425df93781f6f5

When an IP configured on a carp(4) interface is used on a gif(4) tunnel, in order that the endpoint of the tunnel fails over, the advskew setting for this carp interface is ignored and the interface is brought up with advskew 0.  This obviously prevents proper operation of a "preferred master" configuration.

After configuring devd to ignore attach events for carp interfaces (devd automatically runs "/etc/rc.d/netif start carp0" after you do an "ifconfig carp0 create", and does so needlessly after carp0 is already configured at boot time; that's a separate bug I shall file shortly), I have determined that the "start all interfaces" mode of /etc/rc.d/netif (i.e. "rc.d/netif start", without specifying an interface) configures the advskew correctly.  It's the "rc.d/netif start carp0" mode that loses the advskew setting.

In the standard configuration, "rc.d/netif start" is called and carp interfaces are brought up (correctly).  A bit later in the bootup, devd starts, and thinks it needs to configure carpN, so it calls "rc.d/netif start carp0", which resets advskew to 0.

I'm thinking it's a kernel bug since enabling rc_debug and logging the outputs of "sh -x /etc/rc.d/netif start" vs. "ifconfig carp0 create; sh -x /etc/rc.d/netif start carp0" shows that the exact same sequence of "ifconfig carp0 ..." commands is run.  The only difference is that in the global start mode, other interfaces (including gif0) are ifconfig'd between "ifconfig carp0 create" and the rest of the "ifconfig carp0 ..." commands.

I have observed this problem on both 7.1/amd64 and 6.2/i386.
>How-To-Repeat:
Assign the same IP to a carp interface and gif interface, e.g.:
cloned_interfaces="carp0 carp1"
gif_interfaces="gif0"
carp0_advskew="advskew 100"
ifconfig_carp0="vhid 16 $carp_advskew pass XXXXXX 10.95.8.1/21"
ifconfig_gif0="10.95.8.1 10.95.0.1 netmask 255.255.255.255"

The name "carp0" is nothing special; I can call it carp3 and the behavior is the same.  Only the 'ifconfig_gif0' setting matters; the gifconfig_gif0 setting doesn't (an IP I used for that is configured on carp1, which has no problems).  Also, the mere presence of gif0 doesn't matter -- when gif0 is configured with IPs that are NOT assigned to a carp interface, carp0 comes up with the correct advskew setting.
>Fix:
A workaround is to configure devd to ignore attach events for carp interfaces, like so:
====
# don't have devd call network_start on carp0; for some reason this
#   loses the advskew setting (besides being entirely superfluous).

notify 100 {
	match "system" "IFNET";
	match "subsystem" "carp[0-9]+";
	match "type"	"ATTACH";
	action "echo devd called on $subsystem, type $type";
};
====
This overrides the following section of devd.conf:
notify 0 {
        match "system"          "IFNET";
        match "type"            "ATTACH";
        action "/etc/pccard_ether $subsystem start";
};


After doing this, carp interfaces are brought up correctly by "rc.d/netif start" on boot, and devd does not call "rc.d/netif start carp0", so the advskew setting remains correct.  This doesn't fix the underlying problem but it is an acceptable workaround to me.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Feb 25 22:07:24 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=132107 

From: Daniel Duerr <dd@justlinuxhosting.com>
To: bug-followup@FreeBSD.org,
 adaugherity@tamu.edu
Cc:  
Subject: Re: kern/132107: [carp] carp(4) advskew setting ignored when carp IP used on a gif(4) interface
Date: Thu, 18 Jun 2009 23:51:09 -0700

 Having the same issue here and the workaround isn't working for me --  
 it preserves the advskew but the vpn doesn't seem to work without the  
 devd attach event.  Hope someone will fix this soon.
 
 
 
Responsible-Changed-From-To: freebsd-net->glebius 
Responsible-Changed-By: glebius 
Responsible-Changed-When: Sat Oct 15 09:00:31 UTC 2011 
Responsible-Changed-Why:  
I'm now working on major rewrite of CARP for FreeBSD 10, and 
I'd like to take all related PRs.  

http://www.freebsd.org/cgi/query-pr.cgi?pr=132107 

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Andrew Daugherity <adaugherity@tamu.edu>
Cc: bug-followup@FreeBSD.org
Subject: kern/132107: [carp] carp(4) advskew setting ignored when carp IP
 used on a gif(4) interface
Date: Sat, 15 Oct 2011 13:00:11 +0400

   Andrew,
 
   can you please share the entire configuration on all involved
 boxes: local and remote side of tunnel, the CARP master and slave
 boxes?
 
   I'd like to reproduce your configuration and check whether my
 new CARP implementation is subject to your problem or not. For more
 info, see: http://people.freebsd.org/~glebius/newcarp/
 
 -- 
 Totus tuus, Glebius.
State-Changed-From-To: open->closed 
State-Changed-By: glebius 
State-Changed-When: Fri Dec 16 12:59:09 UTC 2011 
State-Changed-Why:  
Feedback timeout. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=132107 

From: "Daugherity, Andrew W" <adaugherity@tamu.edu>
To: "bug-followup@FreeBSD.org" <bug-followup@FreeBSD.org>
Cc:  
Subject: Re: kern/132107: [carp] carp(4) advskew setting ignored when carp
 IP used on a gif(4) interface
Date: Fri, 16 Dec 2011 17:13:34 +0000

 Sorry, I missed your earlier email.  I am sending you my config off-list.
 
 
 
 
 
State-Changed-From-To: closed->open 
State-Changed-By: glebius 
State-Changed-When: Sat Dec 17 03:29:29 UTC 2011 
State-Changed-Why:  
Feedback received. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=132107 

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: kern/132107
Date: Sat, 17 Dec 2011 07:29:21 +0400

 --djJN5oi3zFpblwUd
 Content-Type: text/plain; charset=koi8-r
 Content-Disposition: inline
 
   Add to Audit-Trail.
 
 ----- Forwarded message from "Daugherity, Andrew W" <adaugherity@tamu.edu> -----
 
 Date: Fri, 16 Dec 2011 17:37:34 +0000
 From: "Daugherity, Andrew W" <adaugherity@tamu.edu>
 To: "glebius@FreeBSD.org" <glebius@FreeBSD.org>
 Subject: carp config (kern/132107)
 
 Sorry for not responding earlier.  Here's my config for a carp failover pair, which is one end of the gif tunnel.  The other end of the tunnel is another pair configured very similarly.  I abstracted as much as possible into rc.common.conf which I keep synced between the two.
 
 ipsec etc. settings have no effect on the this bug and can be ignored.
 
 I am now running 8.2 all around (this pair is amd64, the other i386); I haven't checked whether it behave any differently than 6.x/7.x did without my workaround (I would presume so, unless there are some carp changes in the kernel I don't know about) but the workaround still works and doesn't cause any problems.
 
 Kernel config:
 ====
 % uname -a
 FreeBSD imslb3.tamu.edu 8.2-RELEASE-p4 FreeBSD 8.2-RELEASE-p4 #9: Mon Oct 31 15:27:42 CDT 2011     root@imslb3.tamu.edu:/usr/obj/usr/src/sys/IMS  amd64
 % cat /usr/src/sys/amd64/conf/IMS
 # include the GENERIC config (as of 7.0, GENERIC has SMP support, and there
 #  are no more separate GENERIC and SMP kernels)
 include GENERIC
 
 ident           VPR-SMP-GENERIC
 
 device          carp
 device          pf
 device          pflog
 device          pfsync
 
 # In order to enable IPSEC you MUST also add device crypto to
 # your kernel configuration
 options         IPSEC                   #IP security (requires device crypto)
 #options        IPSEC_DEBUG             #debug for IP security
 device          crypto
 ====
 rc.confs attached.  It's nearly completely intact for your benefit, but please scrub public IPs and hostnames (including in comments) before sharing publicly.
 
 
 
 
 Thanks,
 
 
 Andrew Daugherity
 Systems Analyst
 Division of Research, Texas A&M University
 adaugherity@tamu.edu
 
 
 
 ----- End forwarded message -----
 
 -- 
 Totus tuus, Glebius.
 
 --djJN5oi3zFpblwUd
 Content-Type: application/octet-stream; name="rc.common.conf"
 Content-Description: rc.common.conf
 Content-Disposition: attachment; filename="rc.common.conf"
 Content-Transfer-Encoding: base64
 
 IyBzdGFuZGFyZCBMQiBjb25maWcNCmRlZmF1bHRyb3V0ZXI9IjEyOC4xOTQuMTAyLjEyOSIN
 Cm50cGRfZW5hYmxlPSJZRVMiDQpudHBkX3N5bmNfb25fc3RhcnQ9IllFUyINCnNzaGRfZW5h
 YmxlPSJZRVMiDQpwZl9lbmFibGU9IllFUyINCnBmbG9nX2VuYWJsZT0iWUVTIg0KcmVsYXlk
 X2VuYWJsZT0iWUVTIg0KdG1vbl9lbmFibGU9IllFUyIJCSMgaG9zdCBtb25pdG9yIFBlcmwg
 c2NyaXB0LCB1cGRhdGVzIHBmIHRhYmxlcw0Kc2VuZG1haWxfZW5hYmxlPSJZRVMiDQpuZ2lu
 eF9lbmFibGU9IllFUyINCg0KIyBDQVJQIGZhaWxvdmVyOiBSVEZNIGNhcnAoNCkNCmNhc2Ug
 IiRjYXJwX2VuYWJsZSIgaW4NCiAgICBbWXldW0VlXVtTc10pDQoJIyBjcmVhdGUgY2FycDAg
 aW50ZXJmYWNlDQoJY2xvbmVkX2ludGVyZmFjZXM9ImNhcnAwIGNhcnAxIg0KDQoJIyBwcm92
 aWRlIGFuIGVhc3kga25vYiBmb3IgcmMuY29uZiBmb3IgZGVzaWduYXRpbmcgdGhlIGxiIGNs
 dXN0ZXINCgkjICAgbWFzdGVyIG5vZGUNCgljYXNlICIkY2FycDBfbWFzdGVyIiBpbg0KCSAg
 ICBbWXldW0VlXVtTc10pDQoJCWNhcnBfYWR2c2tldz0iIjsNCgkJOzsNCgkgICAgKikNCgkJ
 Y2FycF9hZHZza2V3PSJhZHZza2V3IDEwMCI7DQoJZXNhYw0KDQoJIyBtYWluIGNhcnAwIElQ
 ID0gaW1zbGIudGFtdS5lZHUNCglpZmNvbmZpZ19jYXJwMD0idmhpZCAxICRjYXJwX2FkdnNr
 ZXcgcGFzcyBYWFhYWFggMTI4LjE5NC4xMDIuMTQ2LzI3Ig0KCSMgMTI4LjE5NC4xMDIuMTM1
 ID0gaW1zcm4udGFtdS5lZHUNCglpZmNvbmZpZ19jYXJwMF9hbGlhczA9IjEyOC4xOTQuMTAy
 LjEzNS8zMiINCgkjIDEyOC4xOTQuMTAyLjE0MSA9IGltc3dlYi50YW11LmVkdQ0KCWlmY29u
 ZmlnX2NhcnAwX2FsaWFzMT0iMTI4LjE5NC4xMDIuMTQxLzMyIg0KCSMgMTI4LjE5NC4xMDIu
 MTU4ID0gaW1zZmxhc2gudGFtdS5lZHUNCglpZmNvbmZpZ19jYXJwMF9hbGlhczI9IjEyOC4x
 OTQuMTAyLjE1OC8zMiINCgkjIDEyOC4xOTQuMTAyLjEzOSA9IG1tLWRhdi50YW11LmVkdQ0K
 CWlmY29uZmlnX2NhcnAwX2FsaWFzMz0iMTI4LjE5NC4xMDIuMTM5LzMyIg0KCSMgMTI4LjE5
 NC4xMDIuMTQwID0gbW0tZGF2LWxvY2FsLnRhbXUuZWR1DQoJaWZjb25maWdfY2FycDBfYWxp
 YXM0PSIxMjguMTk0LjEwMi4xNDAvMzIiDQoNCgkjIG1haW4gY2FycDEgSVAgPSBpbXNyb3V0
 ZXIudnByLmludA0KCWlmY29uZmlnX2NhcnAxPSJ2aGlkIDExICRjYXJwX2FkdnNrZXcgcGFz
 cyBYWFhYWFggMTAuOTUuOC4xLzIxIg0KCTs7DQogICAgKikNCgk7Ow0KZXNhYw0KDQpjYXNl
 ICIkdHVubmVsX2VuYWJsZSIgaW4NCiAgICBbWXldW0VlXVtTc10pDQoJIyBjb25maWd1cmUg
 Z2lmKDQpIHR1bm5lbCBjb25uZWN0aW5nIDEwLjk1LzIxIGFuZCAxMC45NS44LzIxDQoJIyBh
 bGwgdGhlIElQcyBzaG93biBoZXJlIF9zaG91bGRfIGZhaWwgb3Zlcg0KCWdpZl9pbnRlcmZh
 Y2VzPSJnaWYwIg0KCSMgZXh0ZXJuYWwgSVBzOiBpbXNsYi50YW11LmVkdSA8PT0+IHNzbC0w
 MS5pc2MudGFtdS5lZHUNCglnaWZjb25maWdfZ2lmMD0iMTI4LjE5NC4xMDIuMTQ2IDE2NS45
 MS4xMTguMjMzIg0KCSMgaW50ZXJuYWwgSVBzOiBpbXNyb3V0ZXIudnByLmludCA8PT0+IHJv
 dXRlci52cHIuaW50DQoJaWZjb25maWdfZ2lmMD0iMTAuOTUuOC4xIDEwLjk1LjAuMSBuZXRt
 YXNrIDI1NS4yNTUuMjU1LjI1NSINCg0KCXN0YXRpY19yb3V0ZXM9ImdpZiB2cG4iDQoJcm91
 dGVfZ2lmPSItbmV0IDEwLjk1LzIxIDEwLjk1LjAuMSINCgkjIHdlIGFsc28gbmVlZCBhIHN0
 YXRpYyByb3V0ZSBmb3IgdGhlIE9wZW5WUE4gbmV0d29yayBvbiB0aGUgb3RoZXIgZW5kDQoJ
 cm91dGVfdnBuPSItbmV0IDEwLjk1LjE2LzI0IDEwLjk1LjAuMSINCg0KICAgICAgICAjIGlw
 c2VjIHN0dWZmDQogICAgICAgIGlwc2VjX2VuYWJsZT0iWUVTIg0KICAgICAgICBpcHNlY19w
 cm9ncmFtPSIvdXNyL2xvY2FsL3NiaW4vc2V0a2V5Ig0KICAgICAgICBpcHNlY19maWxlPSIv
 dXNyL2xvY2FsL2V0Yy9yYWNvb24vc2V0a2V5LmNvbmYiDQogICAgICAgIHJhY29vbl9lbmFi
 bGU9IllFUyINCgk7Ow0KICAgICopDQoJOzsNCmVzYWMNCg==
 
 --djJN5oi3zFpblwUd
 Content-Type: application/octet-stream; name="rc.conf-lb3"
 Content-Description: rc.conf-lb3
 Content-Disposition: attachment; filename="rc.conf-lb3"
 Content-Transfer-Encoding: base64
 
 aG9zdG5hbWU9Imltc2xiMy50YW11LmVkdSINCmlwdjRfYWRkcnNfYmdlMD0iMTI4LjE5NC4x
 MDIuMTU2LzI3Ig0KaXB2NF9hZGRyc19iZ2UxPSIxMC45NS44LjEzLzIxIg0KDQojIHRoZSBm
 b2xsb3dpbmcgYXJlIGxvY2FsIGtub2JzLCBwYXJzZWQgaW4gcmMuY29tbW9uLmNvbmYNCmNh
 cnBfZW5hYmxlPSJZRVMiDQpjYXJwMF9tYXN0ZXI9IllFUyINCnR1bm5lbF9lbmFibGU9IllF
 UyINCg0KLiAvZXRjL3JjLmNvbW1vbi5jb25mDQo=
 
 --djJN5oi3zFpblwUd
 Content-Type: application/octet-stream; name="rc.conf-lb4"
 Content-Description: rc.conf-lb4
 Content-Disposition: attachment; filename="rc.conf-lb4"
 Content-Transfer-Encoding: base64
 
 I3JjX2RlYnVnPSJZRVMiDQpob3N0bmFtZT0iaW1zbGI0LnRhbXUuZWR1Ig0KaXB2NF9hZGRy
 c19iZ2UwPSIxMjguMTk0LjEwMi4xNTcvMjciDQppcHY0X2FkZHJzX2JnZTE9IjEwLjk1Ljgu
 MTQvMjEiDQoNCiMgdGhlIGZvbGxvd2luZyBhcmUgbG9jYWwga25vYnMsIHBhcnNlZCBpbiBy
 Yy5jb21tb24uY29uZg0KY2FycF9lbmFibGU9IllFUyINCmNhcnAwX21hc3Rlcj0iTk8iDQp0
 dW5uZWxfZW5hYmxlPSJZRVMiDQoNCi4gL2V0Yy9yYy5jb21tb24uY29uZg0K
 
 --djJN5oi3zFpblwUd--
>Unformatted:
