From nobody@FreeBSD.org  Thu Jun 28 13:26:02 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 7C25D16A400
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 28 Jun 2007 13:26:02 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id 6003F13C44C
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 28 Jun 2007 13:26:02 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l5SDQ1GM006202
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 28 Jun 2007 13:26:01 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id l5SDQ1gH006171;
	Thu, 28 Jun 2007 13:26:01 GMT
	(envelope-from nobody)
Message-Id: <200706281326.l5SDQ1gH006171@www.freebsd.org>
Date: Thu, 28 Jun 2007 13:26:01 GMT
From: Nerijus Ambrazas <nerijus.ambrazas@ktu.lt>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Carp+pf delay with high state limit
X-Send-Pr-Version: www-3.0

>Number:         114095
>Category:       kern
>Synopsis:       [carp] carp+pf delay with high state limit
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-pf
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jun 28 13:30:06 GMT 2007
>Closed-Date:    Sat Oct 15 14:20:57 UTC 2011
>Last-Modified:  Mon Oct 17 19:20:10 UTC 2011
>Originator:     Nerijus Ambrazas
>Release:        FreeBSD 6.2-RELEASE amd64
>Organization:
Kaunas University of Technology
>Environment:
FreeBSD ad-baltas.ktu.lt 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Wed Jun 27 11:07:42 EEST 2007     root@:/usr/src/sys/amd64/compile/geras  amd64

>Description:
CARP takes a long time to assign master and backup nodes after restart.
One node initially becomes master on all vhids, not dependant on the
value of advskew value. It might take a few minutes or more for the
master/slave nodes to be set properly. The wait time seems to be linked
to the the state limit set in pf. The higher the limit the longer it
takes CARP to properly initialize.


The kernel config is as follows :

include GENERIC

options SMP

#PF
device pf
device pflog
device pfsync
device carp

#ALTQ
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_CDNR
options ALTQ_PRIQ
options ALTQ_NOPCC





pf.conf used for testing is


set limit {states 50000, frags 50000}
nat on em0 from 10.0.0.0/8 to !10.0.0.0/8 -> 193.219.184.77

pass quick all



rc.conf is as follows :

# -- sysinstall generated deltas -- # Fri Jun 22 15:57:33 2007
# Created: Fri Jun 22 15:57:33 2007
# Enable network daemons for user convenience.
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.
check_quotas="NO"
hostname="ad-baltas.ktu.lt"
ifconfig_em0="inet 193.219.184.74 netmask 255.255.255.240"
ifconfig_em1="up"
defaultrouter="193.219.184.65"

cloned_interfaces="lo0 vlan251 vlan252 vlan253 vlan254 vlan255 vlan256 vlan257 vlan258 vlan259 vlan260 vlan261 vlan262 vlan26
3 vlan264 vlan265 carp0 carp251 carp252 carp253 carp254 carp255 carp256 carp257 carp258 carp259 carp260 carp261 carp262 carp2
63 carp264 carp4 carp265"

ifconfig_carp0="up 193.219.184.77 netmask 255.255.255.240 vhid 1 advskew 100"

#Tarptautiniu studiju centras
ifconfig_vlan251="inet 10.12.7.252 netmask 255.255.248.0 vlan 251 vlandev em1 up"
ifconfig_carp251="up 10.12.7.254 netmask 255.255.248.0 vhid 2 advskew 100 "

#ITPI 2a
ifconfig_vlan252="inet 10.0.15.252 netmask 255.255.248.0 vlan 252 vlandev em1 up"
ifconfig_carp252="up 10.0.15.254 netmask 255.255.248.0 vhid 3 advskew 100 "

#ITPI 103  
ifconfig_vlan253="inet 10.0.23.252 netmask 255.255.248.0 vlan 253 vlandev em1 up"
ifconfig_carp253="up 10.0.23.254 netmask 255.255.248.0 vhid 4 advskew 100 "

#Dizainas  
ifconfig_vlan254="inet 10.2.7.252 netmask 255.255.248.0 vlan 254 vlandev em1 up"
ifconfig_carp254="up 10.2.7.254 netmask 255.255.248.0 vhid 5 advskew 100 "

#Chemija
ifconfig_vlan255="inet 10.1.7.252 netmask 255.255.248.0 vlan 255 vlandev em1 up"
ifconfig_carp255="up 10.1.7.254 netmask 255.255.248.0 vhid 6 advskew 100 "

#Mechanikos
ifconfig_vlan256="inet 10.10.7.252 netmask 255.255.248.0 vlan 256 vlandev em1 up"
ifconfig_carp256="up 10.10.7.254 netmask 255.255.248.0 vhid 7 advskew 100"

#Elektros?
ifconfig_vlan257="inet 10.5.7.252 netmask 255.255.248.0 vlan 257 vlandev em1 up"
ifconfig_carp257="up 10.5.7.254 netmask 255.255.248.0 vhid 8 advskew 100 "

#Ekonomikos ir vadybos
ifconfig_vlan258="inet 10.3.7.252 netmask 255.255.248.0 vlan 258 vlandev em1 up"
ifconfig_carp258="up 10.3.7.254 netmask 255.255.248.0 vhid 9 advskew 100 "

#Humanitariniu
ifconfig_vlan259="inet 10.9.7.252 netmask 255.255.248.0 vlan 259 vlandev em1 up"
ifconfig_carp259="up 10.9.7.254 netmask 255.255.248.0 vhid 10 advskew 100 "

#Europos
ifconfig_vlan260="inet 10.13.7.252 netmask 255.255.248.0 vlan 260 vlandev em1 up"
ifconfig_carp260="up 10.13.7.254 netmask 255.255.248.0 vhid 11 advskew 100"

#Telekomunikaciju
ifconfig_vlan261="inet 10.7.7.252 netmask 255.255.248.0 vlan 261 vlandev em1 up"
ifconfig_carp261="up 10.7.7.254 netmask 255.255.248.0 vhid 12 advskew 100"

#Serverynas
ifconfig_vlan262="inet 10.0.7.252 netmask 255.255.248.0 vlan 262 vlandev em1 up"
ifconfig_carp262="up 10.0.7.254 netmask 255.255.248.0 vhid 13 advskew 0 "

#Socialiniu
ifconfig_vlan263="inet 10.11.7.252 netmask 255.255.248.0 vlan 263 vlandev em1 up"
ifconfig_carp263="up 10.11.7.254 netmask 255.255.248.0 vhid 14 advskew 100"

#Informatikos
ifconfig_vlan264="inet 10.6.7.252 netmask 255.255.248.0 vlan 264 vlandev em1 up"
ifconfig_carp264="up 10.6.7.254 netmask 255.255.248.0 vhid 15 advskew 100 "
#ifconfig_carp265="up 10.6.7.254 netmask 255.255.248.0 vhid 16 "

#AD Serveris
ifconfig_vlan265="inet 10.255.0.252 netmask 255.255.248.0 vlan 265 vlandev em1 up"
ifconfig_carp265="up 10.255.0.254 netmask 255.255.248.0 vhid 16 advskew 100"


ifconfig_em2="inet 172.16.0.1 netmask 255.255.255.0"
ifconfig_carp4="up 172.16.0.200 netmask 255.255.255.0 vhid 100"
ifconfig_pfsync0="up syncif em2"

usbd_enable="YES"
pf_enable="YES"
gateway_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
sshd_enable="YES"
ntpd_enable="YES"

>How-To-Repeat:
Set the state limit in pf.conf to 100000 or something like that.
>Fix:
not known

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Thu Jun 28 17:30:15 UTC 2007 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=114095 
Responsible-Changed-From-To: freebsd-net->pf 
Responsible-Changed-By: kmacy 
Responsible-Changed-When: Thu Nov 15 22:56:24 UTC 2007 
Responsible-Changed-Why:  

not clear that this is actually a bug with the state limit set that high 

http://www.freebsd.org/cgi/query-pr.cgi?pr=114095 
Responsible-Changed-From-To: pf->freebsd-pf 
Responsible-Changed-By: remko 
Responsible-Changed-When: Fri Nov 16 07:13:21 UTC 2007 
Responsible-Changed-Why:  
reassign to "standard" group 

http://www.freebsd.org/cgi/query-pr.cgi?pr=114095 
State-Changed-From-To: open->closed 
State-Changed-By: glebius 
State-Changed-When: Sat Oct 15 14:20:00 UTC 2011 
State-Changed-Why:  
Not a bug. This is a feature. pfsync(4) suppresses carp(4) 
preemption until new recently booted node downloads full 
table of pf(4) states from its peer. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=114095 

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Ermal Lu?i <eri@FreeBSD.org>
Cc: nerijus.ambrazas@ktu.lt, freebsd-pf@FreeBSD.org, bug-followup@FreeBSD.org
Subject: Re: kern/114095: [carp] carp+pf delay with high state limit
Date: Mon, 17 Oct 2011 17:17:42 +0400

 On Mon, Oct 17, 2011 at 02:18:38PM +0200, Ermal Lu?i wrote:
 E> On Sat, Oct 15, 2011 at 4:20 PM,  <glebius@freebsd.org> wrote:
 E> > Synopsis: [carp] carp+pf delay with high state limit
 E> >
 E> > State-Changed-From-To: open->closed
 E> > State-Changed-By: glebius
 E> > State-Changed-When: Sat Oct 15 14:20:00 UTC 2011
 E> > State-Changed-Why:
 E> > Not a bug. This is a feature. pfsync(4) suppresses carp(4)
 E> > preemption until new recently booted node downloads full
 E> > table of pf(4) states from its peer.
 E> 
 E> This is not true on FreeBSD.
 E> The issue might be from other reasons.
 
 This is a surprise for me that this feature had been removed!
 
 It used to be in stable/6:
 
 http://fxr.watson.org/fxr/ident?v=FREEBSD60;i=carp_suppress_preempt
 
 And I always treated that variable in CARP as shared with pf. Why did
 they removed this feature from pfsync?
 
 P.S. Since PR is about 6.2-RELEASE, then I have closed it correctly.
 
 -- 
 Totus tuus, Glebius.

From: =?ISO-8859-1?Q?Ermal_Lu=E7i?= <eri@freebsd.org>
To: Gleb Smirnoff <glebius@freebsd.org>
Cc: nerijus.ambrazas@ktu.lt, freebsd-pf@freebsd.org, bug-followup@freebsd.org
Subject: Re: kern/114095: [carp] carp+pf delay with high state limit
Date: Mon, 17 Oct 2011 20:47:31 +0200

 2011/10/17 Gleb Smirnoff <glebius@freebsd.org>:
 > On Mon, Oct 17, 2011 at 02:18:38PM +0200, Ermal Lu?i wrote:
 > E> On Sat, Oct 15, 2011 at 4:20 PM, =A0<glebius@freebsd.org> wrote:
 > E> > Synopsis: [carp] carp+pf delay with high state limit
 > E> >
 > E> > State-Changed-From-To: open->closed
 > E> > State-Changed-By: glebius
 > E> > State-Changed-When: Sat Oct 15 14:20:00 UTC 2011
 > E> > State-Changed-Why:
 > E> > Not a bug. This is a feature. pfsync(4) suppresses carp(4)
 > E> > preemption until new recently booted node downloads full
 > E> > table of pf(4) states from its peer.
 > E>
 > E> This is not true on FreeBSD.
 > E> The issue might be from other reasons.
 >
 > This is a surprise for me that this feature had been removed!
 >
 > It used to be in stable/6:
 >
 > http://fxr.watson.org/fxr/ident?v=3DFREEBSD60;i=3Dcarp_suppress_preempt
 >
 > And I always treated that variable in CARP as shared with pf. Why did
 > they removed this feature from pfsync?
 >
 
 OpenBSD has it but FreeBSD is SMP capable and global vars without
 synchronization do not work well.
 To support that you have to add cross-dependencies and synchronization
 between the two.
 
 Not only synchronization though even some housekeeping around....
 I will probably give a look at this again after 9.0.
 
 > P.S. Since PR is about 6.2-RELEASE, then I have closed it correctly.
 >
 > --
 > Totus tuus, Glebius.
 >
 
 
 
 --=20
 Ermal

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Ermal Lu?i <eri@FreeBSD.org>
Cc: nerijus.ambrazas@ktu.lt, freebsd-pf@FreeBSD.org, bug-followup@FreeBSD.org
Subject: Re: kern/114095: [carp] carp+pf delay with high state limit
Date: Mon, 17 Oct 2011 23:13:48 +0400

 On Mon, Oct 17, 2011 at 08:47:31PM +0200, Ermal Lu?i wrote:
 E> > This is a surprise for me that this feature had been removed!
 E> >
 E> > It used to be in stable/6:
 E> >
 E> > http://fxr.watson.org/fxr/ident?v=FREEBSD60;i=carp_suppress_preempt
 E> >
 E> > And I always treated that variable in CARP as shared with pf. Why did
 E> > they removed this feature from pfsync?
 E> 
 E> OpenBSD has it but FreeBSD is SMP capable and global vars without
 E> synchronization do not work well.
 E> To support that you have to add cross-dependencies and synchronization
 E> between the two.
 E> 
 E> Not only synchronization though even some housekeeping around....
 E> I will probably give a look at this again after 9.0.
 
 Well, a possible race when pfsync clears its increment to
 carp_suppress_preempt but the CPU where carp callout is running
 doesn't notice it doesn't see it due to cache is harmless.
 It just mean that preemption would happen not right after
 pfsync has finished downloading states, but a couple of seconds
 later.
 
 -- 
 Totus tuus, Glebius.
>Unformatted:
