From nobody@FreeBSD.org  Sat Oct 22 13:15:29 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 5C552106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 22 Oct 2011 13:15:29 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 4B6E58FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 22 Oct 2011 13:15:29 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p9MDFSOq066936
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 22 Oct 2011 13:15:28 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p9MDFSTN066935;
	Sat, 22 Oct 2011 13:15:28 GMT
	(envelope-from nobody)
Message-Id: <201110221315.p9MDFSTN066935@red.freebsd.org>
Date: Sat, 22 Oct 2011 13:15:28 GMT
From: Steven Hartland <killing@multiplay.co.uk>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Repeating RTM_MISS packets causing high CPU load for ntpd
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         161899
>Category:       kern
>Synopsis:       [route] ntpd(8): Repeating RTM_MISS packets causing high CPU load for ntpd
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-rc
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Oct 22 13:20:08 UTC 2011
>Closed-Date:    Thu Dec 13 22:19:52 UTC 2012
>Last-Modified:  Thu Dec 13 22:19:52 UTC 2012
>Originator:     Steven Hartland
>Release:        8.2-STABLE
>Organization:
Multiplay
>Environment:
FreeBSD db4.multiplay.co.uk 8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct  4 09:53:17 UTC 2011     root@db3-old.multiplay.co.uk:/usr/obj/usr/src/sys/MULTIPLAY  amd64
>Description:
We just updated a machine to 8-STABLE and I've noticed
that ntpd is using notible amounts of CPU 5-7% which
is very high for such a trivial daemon.

8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct  4 09:53:17 UTC 2011

truss indicates its constantly checking and reading
from a socket

0.047297485 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1)
0.047513160 clock_gettime(0,{1317770389.969538247 }) = 0 (0x0)
0.047604515 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1)
0.047668212 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8)
0.049395293 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1)
0.049503689 clock_gettime(0,{1317770389.971526820 }) = 0 (0x0)
0.049606219 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1)
0.049669916 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8)
0.049809882 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1)
..

running with debug enabled it sits looping outputting:-
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
..

It seems socket 28 is a duplicate of an internal routing socket
as seen here in the trace:-
0.044544269 socket(PF_ROUTE,SOCK_RAW,0)      = 4 (0x4)
0.044595394 fcntl(4,F_DUPFD,0x14)        = 28 (0x1c)
0.044645960 close(4)                 = 0 (0x0)
0.044695968 fcntl(28,F_SETFL,O_NONBLOCK)     = 0 (0x0)

Now this looks like its RTM_MISS as defined:-
sys/net/route.h:#define RTM_MISS        0x7     /* Lookup failed on this address */

So the question was why is PF_ROUTE socket constantly
spamming RTM_MISS?

route -n monitor on this machines shows:-
got message of size 184 on Tue Oct  4 23:46:36 2011
RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags:<DONE>
locks:  inits: 
sockaddrs: <DST>
 ::A.B.C.D

got message of size 184 on Tue Oct  4 23:46:36 2011
RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags:<DONE>
locks:  inits: 
sockaddrs: <DST>
 ::A.B.C.D

This seems very much like the following pr which was fixed:-
"Remove a bogusly introduced rtalloc_ign() in rev. 1.335/SVN 178029,
generating an RTM_MISS for every IP packet forwarded making user space
routing daemons unhappy":-
http://www.freebsd.org/cgi/query-pr.cgi?pr=124540

The box is doing no routing, its fairly basic install with
1 main IP on em0 + 1 alias + gw addres and 1 private ip on em1.

Its running mysql and thats about it.

Additional discussion can be found in the the thread topic freebsd-stable list
"serious packet routing issue causing ntpd high load?"
>How-To-Repeat:
According to Vlad Galu, which Li Qing confirms on the original discussion thread:
"serious packet routing issue causing ntpd high load?"

1. Start listening for routing messages on a non-default FIB, e.g.
setfib 2 route monitor
2. Add any static route within that FIB.
3. The machine I run the test on is a heavy DNS client, it generates a
few dozen requests per second. The monitor process starts getting
RTM_MISS messages for each outgoing DNS request (the dst sockaddr is
the same as my first resolv.conf entry, seq is always 0).

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sat Oct 22 16:05:57 UTC 2011 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=161899 

From: "Steven Hartland" <killing@multiplay.co.uk>
To: <bug-followup@freebsd.org>
Cc:  
Subject: Re: kern/161899: [route] ntpd(8): Repeating RTM_MISS packets causing high CPU load for ntpd
Date: Tue, 7 Feb 2012 09:24:47 -0000

 Any update on this, would have been nice to see a fix hit before
 9.0. If you need any more information please let me know.
 

From: Eugene Grosbein <egrosbein@rdtc.ru>
To: Steven Hartland <killing@multiplay.co.uk>
Cc: freebsd-net@freebsd.org, bug-followup@freebsd.org
Subject: Re: kern/161899: [route] ntpd(8): Repeating RTM_MISS packets	causing
 high CPU load for ntpd
Date: Wed, 08 Feb 2012 16:13:27 +0700

 08.02.2012 03:30, Steven Hartland :
 > The following reply was made to PR kern/161899; it has been noted by GNATS.
 > 
 > From: "Steven Hartland" <killing@multiplay.co.uk>
 > To: <bug-followup@freebsd.org>
 > Cc:  
 > Subject: Re: kern/161899: [route] ntpd(8): Repeating RTM_MISS packets causing high CPU load for ntpd
 > Date: Tue, 7 Feb 2012 09:24:47 -0000
 > 
 >  Any update on this, would have been nice to see a fix hit before
 >  9.0. If you need any more information please let me know.
 
 This is known problem. You should remove "options FLOWTABLE"
 from your kernel configuration, as it was removed from GENERIC
 for such misbehaviours. That had fixed same problem for me.
 
 Eugene Grosbein

From: "Steven Hartland" <killing@multiplay.co.uk>
To: "Eugene Grosbein" <egrosbein@rdtc.ru>
Cc: <freebsd-net@freebsd.org>,
	<bug-followup@freebsd.org>
Subject: Re: kern/161899: [route] ntpd(8): Repeating RTM_MISS packets causing high CPU load for ntpd
Date: Wed, 8 Feb 2012 09:53:03 -0000

 ----- Original Message ----- 
 From: "Eugene Grosbein" <egrosbein@rdtc.ru>
 
 > This is known problem. You should remove "options FLOWTABLE"
 > from your kernel configuration, as it was removed from GENERIC
 > for such misbehaviours. That had fixed same problem for me.
 
 We already have this removed due to the issues it causes with IP
 changes, unfortunately it has no impact on this behaviour here.
 
     Regards
     Steve
 
 ================================================
 This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 
 
 In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
 or return the E.mail to postmaster@multiplay.co.uk.
 

From: Dmitrij Tejblum <tejblum@yandex-team.ru>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/161899: [route] ntpd(8): Repeating RTM_MISS packets	causing
 high CPU load for ntpd
Date: Wed, 08 Feb 2012 14:30:51 +0400

 I would suggest to remove RTM_MISS messages at all. I believe that there 
 is no sofware that actually use it. OTOH, in some cases RTM_MISS 
 messages are really disturbing.
 
 E.g., a router without default route (e.g. runnung BGP) will always 
 generate some amount of RTM_MISS messages. They have no use, but require 
 daemons to parse them, and could result in overflow on routing socket 
 queue and, in turn, cause some important routing messages to be dropped.
 
 I have a patch that add a sysctl to turn off RTM_MISS messages, but 
 since no one use them, it would be easier to just remove them entirely.
 
 -- 
 Dmitry

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Steven Hartland <killing@multiplay.co.uk>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/161899: Repeating RTM_MISS packets causing high CPU load
 for ntpd
Date: Wed, 8 Feb 2012 16:59:44 +0400

 > Any update on this, would have been nice to see a fix hit before
 > 9.0. If you need any more information please let me know.
 
 AFAIK, this is no longer a problem in 9.0-RELEASE or in HEAD.
 
 The cause for this number of misses is absense of a route for
 IPv4 mapped block in IPv6 routing table.
 
 Here it is:
 
 # netstat -rn -f inet6 
 Routing tables
 
 Internet6:
 Destination                       Gateway                       Flags      Netif Expire
 ::/96                             ::1                           UGRS        lo0
 
 Some rc.d script installs this prefix in 9.0 and 10.0. If it hasn't
 been merged to stable/8, then it needs to be found and merged.
 
 -- 
 Totus tuus, Glebius.
Responsible-Changed-From-To: freebsd-net->freebsd-rc 
Responsible-Changed-By: glebius 
Responsible-Changed-When: Wed Feb 8 13:55:07 UTC 2012 
Responsible-Changed-Why:  
I think, that solution to the problem lives somewhere in th rc-land. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=161899 
State-Changed-From-To: open->patched 
State-Changed-By: glebius 
State-Changed-When: Wed Feb 8 13:55:37 UTC 2012 
State-Changed-Why:  
Not an issue on head/, however I don't know which revision needs to be merged :) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=161899 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/161899: commit references a PR
Date: Thu, 16 Feb 2012 01:41:48 +0000 (UTC)

 Author: hrs
 Date: Thu Feb 16 01:41:34 2012
 New Revision: 231803
 URL: http://svn.freebsd.org/changeset/base/231803
 
 Log:
   Add static routes to ::ffff:0.0.0.0/96 and ::0.0.0.0/96 unconditionally when
   the kernel supports PF_INET6.
   
   PR:	kern/161899
 
 Modified:
   stable/8/etc/rc.d/network_ipv6
   stable/8/etc/rc.d/routing
 
 Modified: stable/8/etc/rc.d/network_ipv6
 ==============================================================================
 --- stable/8/etc/rc.d/network_ipv6	Thu Feb 16 01:32:23 2012	(r231802)
 +++ stable/8/etc/rc.d/network_ipv6	Thu Feb 16 01:41:34 2012	(r231803)
 @@ -41,10 +41,6 @@ start_cmd="network_ipv6_start"
  
  network_ipv6_start()
  {
 -	# disallow "internal" addresses to appear on the wire
 -	route add -inet6 ::ffff:0.0.0.0 -prefixlen 96 ::1 -reject
 -	route add -inet6 ::0.0.0.0 -prefixlen 96 ::1 -reject
 -
  	case ${ipv6_network_interfaces} in
  	[Aa][Uu][Tt][Oo])
  		# Get a list of network interfaces
 
 Modified: stable/8/etc/rc.d/routing
 ==============================================================================
 --- stable/8/etc/rc.d/routing	Thu Feb 16 01:32:23 2012	(r231802)
 +++ stable/8/etc/rc.d/routing	Thu Feb 16 01:41:34 2012	(r231803)
 @@ -57,6 +57,14 @@ static_start()
  			atmconfig natm add ${route_args}
  		done
  	fi
 +
 +	# Disallow "internal" addresses to appear on the wire if inet6
 +	# is enabled.
 +	if afexists inet6; then
 +		# disallow "internal" addresses to appear on the wire
 +		route add -inet6 ::ffff:0.0.0.0 -prefixlen 96 ::1 -reject
 +		route add -inet6 ::0.0.0.0 -prefixlen 96 ::1 -reject
 +	fi
  }
  
  _ropts_initdone=
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: smh 
State-Changed-When: Thu Dec 13 22:19:51 UTC 2012 
State-Changed-Why:  
patched in stable/8/ with r231803 

http://www.freebsd.org/cgi/query-pr.cgi?pr=161899 
>Unformatted:
