From nobody@FreeBSD.org  Sat Mar 20 23:06:36 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id CAD311065672
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 20 Mar 2010 23:06:36 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id A01028FC14
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 20 Mar 2010 23:06:36 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o2KN6a5Q038745
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 20 Mar 2010 23:06:36 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o2KN6Z55038744;
	Sat, 20 Mar 2010 23:06:35 GMT
	(envelope-from nobody)
Message-Id: <201003202306.o2KN6Z55038744@www.freebsd.org>
Date: Sat, 20 Mar 2010 23:06:35 GMT
From: Doychin Dokov <root@net1.cc>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Flowtable crashes system 
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         144917
>Category:       kern
>Synopsis:       [flowtable] [panic] flowtable crashes system [regression]
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-net
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Mar 20 23:10:02 UTC 2010
>Closed-Date:    
>Last-Modified:  Mon Jan 23 21:00:16 UTC 2012
>Originator:     Doychin Dokov
>Release:        8.0-RELEASE-p2
>Organization:
Data Syst Ltd.
>Environment:
FreeBSD wilma.net1.cc 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #0: Tue Jan  5 21:11:58 UTC 2010     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

>Description:
It seems like flowtable has been merged and enabled by default in 8.0.... which is a really really bad idea.
On a system which handles two full BGP tables it makes one of the CPU cores run at 100% right after most of the prefixes get installed in the routing table. Performance is seriously harmed, as well is ZFS speed  - it's dramatically slow (sometimes 3x-4x). The system crashes in random time - between 1 and 10 minutes right after it has booted and started installing the routes in the kernel routing table. Observations with top -PS show that a process called 'flowcontroller' is overwhelming the CPU.
Adding net.inet.flowtable.enable=0 to /etc/sysctl.conf AND rebooting the system fixes the problem. Changing the sysctl after the problem has appeared does not seem to fix it - the system still crashes in a while. 
This is very very embarassing when upgrading from 7.x, and is also very undocumented:
wilma# man flowcontroller
No manual entry for flowcontroller
wilma# man flowtable
No manual entry for flowtable
I think this feature should be disabled by default (according to what i've found about that, it helps with heavy traffic like 10G links - it's far more frequent for one to use lots of prefixes than 10G), or - at least - better documented and maybe automatically disabled by packages which deal with lots of prefixes (e.g. quagga, openbgpd, etc.) 
>How-To-Repeat:
1. Install 8.0-RELEASE (does not matter if you update to the latest patchlevel with freebsd-update)
2. Install quagga or openbgpd or some other routing daemon
3. Feed it with lots of routes (e.g. full internet routing table)
4. Grab a coffee
5. It'll crash until your cup is still smoking.

P.S. I've only tested with FreeBSD installed on ZFS root
>Fix:
Add this to /etc/sysctl.conf:
net.inet.flowtable.enable=0

>Release-Note:
>Audit-Trail:

From: NetOne - Doychin Dokov <root@net1.cc>
To: bug-followup@FreeBSD.org, root@net1.cc
Cc:  
Subject: Re: kern/144917: Flowtable crashes system
Date: Sun, 21 Mar 2010 01:50:53 +0200

 Forgot some details:
 The system was forwarding about 3-4Kpps for about 150-180 different IP 
 addresses during those crashes experienced. Don't know if it's related 
 (it's very far from  high traffic), but maybe the total combination of 
 Kpps x IPs x Prefixes makes the flowtable behave like this.

From: Mikolaj Golub <to.my.trociny@gmail.com>
To: Doychin Dokov <root@net1.cc>
Cc: freebsd-bugs@FreeBSD.org, bug-followup@FreeBSD.org, "Bjoern A. Zeeb" <bz@FreeBSD.org>
Subject: Re: kern/144917: Flowtable crashes system
Date: Sun, 21 Mar 2010 16:24:11 +0200

 Are you  sure about process' name 'flowcontroller'?
 
 zhuzha:~% grep -R flowcontroller /usr/src/sys 
 zhuzha:~% 
 
 May be you mean 'flowcleaner'? If so it might be related to the issue
 discussed on freebsd-net@ in February (Subject: mpd has hung):
 
 http://lists.freebsd.org/pipermail/freebsd-net/2010-February/024642.html
 
 You could try patches provided by Bjoern there to see if it helps.
 
 -- 
 Mikolaj Golub

From: NetOne - Doychin Dokov <root@net1.cc>
To: Mikolaj Golub <to.my.trociny@gmail.com>
Cc: freebsd-bugs@FreeBSD.org, bug-followup@FreeBSD.org, 
 "Bjoern A. Zeeb" <bz@FreeBSD.org>
Subject: Re: kern/144917: Flowtable crashes system
Date: Sun, 21 Mar 2010 17:37:54 +0200

 I'm not sure, really, as it was a long night of debugging the problem....
 
 Unfortunately, this machine is in production, and that was a scheduled 
 update, so I'm unable to test the patches provided. I'll try in a day or 
 two to do a similar setup on a test machine and throw some traffic at it 
 to see if it behaves the same, then will try the patches provided.
 
 Anyways, what's that flowtable stuff, and what's the benefit of using 
 it? Is there any clear information about that which I have missed?
 
 Kind regards,
 Doychin

From: NetOne - Doychin Dokov <root@net1.cc>
To: Mikolaj Golub <to.my.trociny@gmail.com>
Cc: freebsd-bugs@FreeBSD.org, bug-followup@FreeBSD.org, 
 "Bjoern A. Zeeb" <bz@FreeBSD.org>
Subject: Re: kern/144917: Flowtable crashes system
Date: Sun, 21 Mar 2010 17:44:16 +0200

 You are probably right about the process name, googling some more about 
 this reveals that topic:
 http://old.nabble.com/Quggaa-locking-hard.-td26637507.html
 
 That's exactly the issue I've observed, resolved by the exactly same 
 sysctl, with reboot only - 100% the same issue.
 I think that Kip Macy's post clears it all.
 
 Kind regards,
 Doychin

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, root@net1.cc
Cc:  
Subject: Re: kern/144917: Flowtable crashes system
Date: Sun, 21 Mar 2010 15:48:24 +0000 (UTC)

 On Sun, 21 Mar 2010, NetOne - Doychin Dokov wrote:
 
 > I'm not sure, really, as it was a long night of debugging the problem....
 
 I wonder if you still have "logs" from that debugging session - as in
 what exactly you had seen, top or ddb output, panic strings, backtraces, ...?
 
 -- 
 Bjoern A. Zeeb         It will not break if you know what you are doing.

From: NetOne -Doychin Dokov <root@net1.cc>
To: "Bjoern A. Zeeb" <bz@FreeBSD.org>, <bug-followup@FreeBSD.org>
Cc:  
Subject: RE: kern/144917: Flowtable crashes system
Date: Sun, 21 Mar 2010 20:56:00 +0200

 No, actually I was trying to bring back the machine in working condition as=
  soon as possible.
 If it is of interest, i'll try to replicate the problem on another machine =
 and provide what you want me to.
 Kind regards,
 Doychin
 
 -----Original Message-----
 From: Bjoern A. Zeeb <bz@FreeBSD.org>
 Sent: 21 March 2010 17:48
 To: bug-followup@FreeBSD.org; root@net1.cc
 Subject: Re: kern/144917: Flowtable crashes system
 
 On Sun, 21 Mar 2010, NetOne - Doychin Dokov wrote:
 
 > I'm not sure, really, as it was a long night of debugging the problem....
 
 I wonder if you still have "logs" from that debugging session - as in
 what exactly you had seen, top or ddb output, panic strings, backtraces, ..=
 .?
 
 --=20
 Bjoern A. Zeeb         It will not break if you know what you are doing.
 
 
State-Changed-From-To: open->feedback 
State-Changed-By: bz 
State-Changed-When: Sat Mar 19 15:27:52 UTC 2011 
State-Changed-Why:  
The PR has been waiting for feedback for almost a year; keep 
it open along with kern/146792 as soem changes to the code 
had only happened lately. 


Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: bz 
Responsible-Changed-When: Sat Mar 19 15:27:52 UTC 2011 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144917 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/144917: commit references a PR
Date: Sat, 19 Mar 2011 15:50:49 +0000 (UTC)

 Author: bz
 Date: Sat Mar 19 15:50:34 2011
 New Revision: 219775
 URL: http://svn.freebsd.org/changeset/base/219775
 
 Log:
   For now remove options FLOWTABLE from the remaining GENERIC kernel
   configurations and make it opt-in for those who want it.  LINT will
   still build it.
   
   While it may be a perfect win in some scenarios, it still troubles users
   (see PRs) in general cases.  In addition we are still allocating resources
   even if disabled by sysctl and still leak arp/nd6 entries in case of
   interface destruction.
   
   Discussed with:	qingli (2010-11-24, just never executed)
   Discussed with: juli (OCTEON1)
   PR:		kern/148018, kern/155604, kern/144917, kern/146792
   MFC after:	2 weeks
 
 Modified:
   head/sys/amd64/conf/GENERIC
   head/sys/i386/conf/GENERIC
   head/sys/ia64/conf/GENERIC
   head/sys/mips/conf/OCTEON1
 
 Modified: head/sys/amd64/conf/GENERIC
 ==============================================================================
 --- head/sys/amd64/conf/GENERIC	Sat Mar 19 14:44:24 2011	(r219774)
 +++ head/sys/amd64/conf/GENERIC	Sat Mar 19 15:50:34 2011	(r219775)
 @@ -61,7 +61,6 @@ options 	KBD_INSTALL_CDEV	# install a CD
  options 	HWPMC_HOOKS		# Necessary kernel hooks for hwpmc(4)
  options 	AUDIT			# Security event auditing
  options 	MAC			# TrustedBSD MAC Framework
 -options 	FLOWTABLE		# per-cpu routing cache
  #options 	KDTRACE_FRAME		# Ensure frames are compiled in
  #options 	KDTRACE_HOOKS		# Kernel DTrace hooks
  options 	INCLUDE_CONFIG_FILE     # Include this file in kernel
 
 Modified: head/sys/i386/conf/GENERIC
 ==============================================================================
 --- head/sys/i386/conf/GENERIC	Sat Mar 19 14:44:24 2011	(r219774)
 +++ head/sys/i386/conf/GENERIC	Sat Mar 19 15:50:34 2011	(r219775)
 @@ -62,7 +62,6 @@ options 	KBD_INSTALL_CDEV	# install a CD
  options 	HWPMC_HOOKS		# Necessary kernel hooks for hwpmc(4)
  options 	AUDIT			# Security event auditing
  options 	MAC			# TrustedBSD MAC Framework
 -options 	FLOWTABLE		# per-cpu routing cache
  #options 	KDTRACE_HOOKS		# Kernel DTrace hooks
  options 	INCLUDE_CONFIG_FILE     # Include this file in kernel
  
 
 Modified: head/sys/ia64/conf/GENERIC
 ==============================================================================
 --- head/sys/ia64/conf/GENERIC	Sat Mar 19 14:44:24 2011	(r219774)
 +++ head/sys/ia64/conf/GENERIC	Sat Mar 19 15:50:34 2011	(r219775)
 @@ -31,8 +31,6 @@ options 	COMPAT_FREEBSD7	# Compatible wi
  options 	DDB		# Support DDB
  options 	DEADLKRES	# Enable the deadlock resolver
  options 	FFS		# Berkeley Fast Filesystem
 -#options 	FLOWTABLE	# per-cpu routing cache (removed due to
 -				# misaligned access -- see kern/148018)
  options 	GDB		# Support remote GDB
  options 	GEOM_LABEL	# Provides labelization
  options 	INCLUDE_CONFIG_FILE     # Include this file in kernel
 
 Modified: head/sys/mips/conf/OCTEON1
 ==============================================================================
 --- head/sys/mips/conf/OCTEON1	Sat Mar 19 14:44:24 2011	(r219774)
 +++ head/sys/mips/conf/OCTEON1	Sat Mar 19 15:50:34 2011	(r219775)
 @@ -73,7 +73,6 @@ options 	PRINTF_BUFR_SIZE=128	# Prevent 
  options 	HWPMC_HOOKS		# Necessary kernel hooks for hwpmc(4)
  options 	AUDIT			# Security event auditing
  options 	MAC			# TrustedBSD MAC Framework
 -options 	FLOWTABLE		# per-cpu routing cache
  #options 	KDTRACE_FRAME		# Ensure frames are compiled in
  #options 	KDTRACE_HOOKS		# Kernel DTrace hooks
  options 	INCLUDE_CONFIG_FILE     # Include this file in kernel
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/144917: commit references a PR
Date: Sat,  9 Apr 2011 12:04:46 +0000 (UTC)

 Author: bz
 Date: Sat Apr  9 12:04:35 2011
 New Revision: 220486
 URL: http://svn.freebsd.org/changeset/base/220486
 
 Log:
   MFC r219775:
   
     For now remove options FLOWTABLE from the remaining GENERIC kernel
     configurations and make it opt-in for those who want it.  LINT will
     still build it.
   
     While it may be a perfect win in some scenarios, it still troubles users
     (see PRs) in general cases.  In addition we are still allocating resources
     even if disabled by sysctl and still leak arp/nd6 entries in case of
     interface destruction.
   
     Discussed with:	qingli (2010-11-24, just never executed)
     Discussed with:	juli (OCTEON1)
   PR:			kern/148018, kern/155604, kern/144917, kern/146792
 
 Modified:
   stable/8/sys/amd64/conf/GENERIC
   stable/8/sys/i386/conf/GENERIC
   stable/8/sys/ia64/conf/GENERIC
   stable/8/sys/mips/conf/OCTEON1
 Directory Properties:
   stable/8/sys/   (props changed)
   stable/8/sys/amd64/include/xen/   (props changed)
   stable/8/sys/cddl/contrib/opensolaris/   (props changed)
   stable/8/sys/contrib/dev/acpica/   (props changed)
   stable/8/sys/contrib/pf/   (props changed)
 
 Modified: stable/8/sys/amd64/conf/GENERIC
 ==============================================================================
 --- stable/8/sys/amd64/conf/GENERIC	Sat Apr  9 11:03:04 2011	(r220485)
 +++ stable/8/sys/amd64/conf/GENERIC	Sat Apr  9 12:04:35 2011	(r220486)
 @@ -72,7 +72,6 @@ options 	KBD_INSTALL_CDEV	# install a CD
  options 	HWPMC_HOOKS		# Necessary kernel hooks for hwpmc(4)
  options 	AUDIT			# Security event auditing
  options 	MAC			# TrustedBSD MAC Framework
 -options 	FLOWTABLE		# per-cpu routing cache
  #options 	KDTRACE_FRAME		# Ensure frames are compiled in
  #options 	KDTRACE_HOOKS		# Kernel DTrace hooks
  options 	INCLUDE_CONFIG_FILE     # Include this file in kernel
 
 Modified: stable/8/sys/i386/conf/GENERIC
 ==============================================================================
 --- stable/8/sys/i386/conf/GENERIC	Sat Apr  9 11:03:04 2011	(r220485)
 +++ stable/8/sys/i386/conf/GENERIC	Sat Apr  9 12:04:35 2011	(r220486)
 @@ -73,7 +73,6 @@ options 	KBD_INSTALL_CDEV	# install a CD
  options 	HWPMC_HOOKS		# Necessary kernel hooks for hwpmc(4)
  options 	AUDIT			# Security event auditing
  options 	MAC			# TrustedBSD MAC Framework
 -options 	FLOWTABLE		# per-cpu routing cache
  #options 	KDTRACE_HOOKS		# Kernel DTrace hooks
  options 	INCLUDE_CONFIG_FILE     # Include this file in kernel
  
 
 Modified: stable/8/sys/ia64/conf/GENERIC
 ==============================================================================
 --- stable/8/sys/ia64/conf/GENERIC	Sat Apr  9 11:03:04 2011	(r220485)
 +++ stable/8/sys/ia64/conf/GENERIC	Sat Apr  9 12:04:35 2011	(r220486)
 @@ -31,7 +31,6 @@ options 	COMPAT_43TTY	# BSD 4.3 TTY comp
  options 	COMPAT_FREEBSD6	# Compatible with FreeBSD6
  options 	COMPAT_FREEBSD7	# Compatible with FreeBSD7
  options 	FFS		# Berkeley Fast Filesystem
 -options 	FLOWTABLE	# per-cpu routing cache
  options 	GDB		# Support remote GDB
  options 	GEOM_LABEL	# Provides labelization
  options 	INCLUDE_CONFIG_FILE     # Include this file in kernel
 
 Modified: stable/8/sys/mips/conf/OCTEON1
 ==============================================================================
 --- stable/8/sys/mips/conf/OCTEON1	Sat Apr  9 11:03:04 2011	(r220485)
 +++ stable/8/sys/mips/conf/OCTEON1	Sat Apr  9 12:04:35 2011	(r220486)
 @@ -76,7 +76,6 @@ options 	PRINTF_BUFR_SIZE=128	# Prevent 
  options 	HWPMC_HOOKS		# Necessary kernel hooks for hwpmc(4)
  options 	AUDIT			# Security event auditing
  options 	MAC			# TrustedBSD MAC Framework
 -options 	FLOWTABLE		# per-cpu routing cache
  #options 	KDTRACE_FRAME		# Ensure frames are compiled in
  #options 	KDTRACE_HOOKS		# Kernel DTrace hooks
  options 	INCLUDE_CONFIG_FILE     # Include this file in kernel
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: "K. Macy" <kmacy@freebsd.org>
To: root@net1.cc
Cc: bug-followup@freebsd.org
Subject: kern/144917: [flowtable] [panic] flowtable crashes system [regression]
Date: Mon, 23 Jan 2012 21:50:08 +0100

 Have you tested this workload on 9.0 without flowtable removed /
 disabled? Is the problem still there or reduced?
 
 Thanks
>Unformatted:
