From nobody@FreeBSD.org  Sun Sep 21 20:19:40 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 356CE1065672
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 21 Sep 2008 20:19:40 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 254838FC19
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 21 Sep 2008 20:19:40 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id m8LKJdFd076838
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 21 Sep 2008 20:19:39 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id m8LKJdqf076837;
	Sun, 21 Sep 2008 20:19:39 GMT
	(envelope-from nobody)
Message-Id: <200809212019.m8LKJdqf076837@www.freebsd.org>
Date: Sun, 21 Sep 2008 20:19:39 GMT
From: Seth Mos <seth.mos@xs4all.nl>
To: freebsd-gnats-submit@FreeBSD.org
Subject: icmp socket receives icmp replies not owned by the process.
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         127528
>Category:       kern
>Synopsis:       [icmp]: icmp socket receives icmp replies not owned by the process.
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-net
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Sep 21 20:20:01 UTC 2008
>Closed-Date:    
>Last-Modified:  Thu May 14 03:40:01 UTC 2009
>Originator:     Seth Mos
>Release:        7.0p4
>Organization:
pfSense
>Environment:
# uname -a
FreeBSD beerme.iserv.nl 7.0-RELEASE-p3 FreeBSD 7.0-RELEASE-p3 #0: Fri Aug 29 05:17:50 EDT 2008     sullrich@builder7.bgn.pfsense.org:/usr/obj.pfSense/usr/src/sys/pfSense_wrap.7  i386

>Description:
When running simultaneous ping processes from the host to the same target on a FreeBSD host where 2 different processes ping the same host, one process will see the echo replies from the other process and fail.

We came to see this on pfSense 1.2.1 which is based on FreeBSD 7.0p4 and is using fping to monitor the gateways.

People in our forum started complaining that gateways were invalidly marked as down. More investigation from the users led to the discovery that out of sequence replies were causing fping to fail.
>How-To-Repeat:
This can be demonstrated in a number of ways. When running apinger in the foreground with apinger -c /var/etc/apinger.conf -df it is pinging a gateway once a second it works fine.
Start up a seperate ping process in another terminal and apinger will start logging "Alien icmp echo replies". These replies belong to the other process.

Another example is where the icmp replies are received out of sequence. This can be demonstrated using fping. When using fping to monitor the gateway we use a initial timeout value of 400ms. When a reply is received after 400ms the reply upsets the fping socket and starts failing replies.

This was previously never a issue on FreeBSD 6(1,2,3).

A tcpdump on the wire confirms that both processes are actively sending out icmp requests and receiving icmp echo replies.

It also shows that both echo requests have different identifiers in the id field which should keep the icmp streams seperated.
>Fix:
Would the network stack not keep unique icmp streams apart?

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: remko 
Responsible-Changed-When: Sun Sep 21 21:03:41 UTC 2008 
Responsible-Changed-Why:  
Reassign to networking team 

http://www.freebsd.org/cgi/query-pr.cgi?pr=127528 

From: "Bruce M. Simpson" <bms@FreeBSD.org>
To: remko@FreeBSD.org
Cc: freebsd-net@FreeBSD.org, freebsd-bugs@FreeBSD.org
Subject: Re: kern/127528: [icmp]: icmp socket receives icmp replies not	owned
 by the process.
Date: Sun, 21 Sep 2008 23:12:30 +0100

 remko@FreeBSD.org wrote:
 > Old Synopsis: icmp socket receives icmp replies not owned by the process.
 > New Synopsis: [icmp]: icmp socket receives icmp replies not owned by the process.
 >   
 
 This PR is bogus because:
 ICMP has no concept of datagrams being "owned" by a process. There is no 
 field in the ICMP protocol which differentiates ICMP "sessions" on a 
 per-process basis, and this is because ICMP has no concept of "sessions" 
 -- ICMP messages are directed at IP endpoints.
 
 The networking stack will only selectively dispatch ICMP traffic based 
 on two conditions:
  1. ip_proto number (raw sockets may selectively bind to a protocol) and
  2. multicast group membership (not applicable in this instance).
 
  > It also shows that both echo requests have different identifiers in 
 the id field which should keep the icmp streams seperated.
 
 There is absolutely no requirement for the kernel code to look at the ID 
 field, beyond reporting it to consumers of the SOCK_RAW interface.
 
 This PR can be closed, the submitter should consult the pfSense maintainers.
 
 thanks
 BMS
 
 
 
 
 
 _______________________________________________
 freebsd-net@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-net
 To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
 

From: Chris Buechler <cmb@pfsense.org>
To: "Bruce M. Simpson" <bms@FreeBSD.org>
Cc: freebsd-net@FreeBSD.org, remko@FreeBSD.org, freebsd-bugs@FreeBSD.org
Subject: Re: kern/127528: [icmp]: icmp socket receives icmp replies not	owned
 by the process.
Date: Sun, 21 Sep 2008 19:11:05 -0400

 Bruce M. Simpson wrote:
 > remko@FreeBSD.org wrote:
 >> Old Synopsis: icmp socket receives icmp replies not owned by the 
 >> process.
 >> New Synopsis: [icmp]: icmp socket receives icmp replies not owned by 
 >> the process.
 >>   
 >
 > This PR is bogus because:
 > ICMP has no concept of datagrams being "owned" by a process. There is 
 > no field in the ICMP protocol which differentiates ICMP "sessions" on 
 > a per-process basis, and this is because ICMP has no concept of 
 > "sessions" -- ICMP messages are directed at IP endpoints.
 
 ICMP echo and echo replies do have "sessions" of sorts, at least unique 
 identifying fields - identifier and sequence number.
 
 This was opened by a pfSense maintainer because it's a change in 
 behavior from 6.x releases where this was never an issue, and is 
 something we feel is a regression.
 
 Ideally you don't want to be pinging the same host from two different 
 processes, but it's difficult to avoid in some circumstances and it's 
 something that always worked fine prior to FreeBSD 7.0.
 
 Thanks,
 Chris
 
 _______________________________________________
 freebsd-bugs@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 To unsubscribe, send any mail to "freebsd-bugs-unsubscribe@freebsd.org"
 

From: Robert Watson <rwatson@FreeBSD.org>
To: Chris Buechler <cmb@pfsense.org>
Cc: freebsd-net@FreeBSD.org, remko@FreeBSD.org,
	"Bruce M. Simpson" <bms@FreeBSD.org>, freebsd-bugs@FreeBSD.org
Subject: Re: kern/127528: [icmp]: icmp socket receives icmp replies not	owned
 by the process.
Date: Mon, 22 Sep 2008 09:56:45 +0100 (BST)

 On Sun, 21 Sep 2008, Chris Buechler wrote:
 
 >> This PR is bogus because: ICMP has no concept of datagrams being "owned" by 
 >> a process. There is no field in the ICMP protocol which differentiates ICMP 
 >> "sessions" on a per-process basis, and this is because ICMP has no concept 
 >> of "sessions" -- ICMP messages are directed at IP endpoints.
 >
 > ICMP echo and echo replies do have "sessions" of sorts, at least unique 
 > identifying fields - identifier and sequence number.
 >
 > This was opened by a pfSense maintainer because it's a change in behavior 
 > from 6.x releases where this was never an issue, and is something we feel is 
 > a regression.
 >
 > Ideally you don't want to be pinging the same host from two different 
 > processes, but it's difficult to avoid in some circumstances and it's 
 > something that always worked fine prior to FreeBSD 7.0.
 
 As far as I'm aware, IP raw sockets have never supported using the ID field to 
 identify sessions or direct packets to specific sockets.  This means that a 
 kernel regression in that session semantic is unlikely.
 
 However, it could be that you're seeing the impact of another regression 
 relating to other behavior for restricting ICMP received on the raw socket. 
 The kernel code in question is in raw_ip.c:rip_input(), which performs the 
 following checks for each raw IP socket it considers a candidate for delivery:
 
 - If a non-zero 'protocol' argument was specified in the socket() call, is the
    protocol of the packet the same as the protocol on the socket?
 
 - If a local address is bound on the raw socket, is it the same as the
    destination address on the packet?
 
 - If a forieng address is bound on the raw socket, is it the same as the
    source address on the packet?
 
 - If the socket was created by a process in a jail, is the jail IP the same as
    the destination address on the packet?
 
 With these in mind, I'd look for one of two things that might have lead to the 
 symptoms you're seeing: a change in the way your application is written or run 
 such that the address limits on packets received are no longer requested or 
 may now be ambiguous, or a change in the way our system works such that it's 
 no longer implementing the above checks correctly.
 
 Ping applications are, btw, supposed to select unique identifiers (typically 
 the pid is used) and then ignore replies with a different identifier, and last 
 I checked ping(8) did that.  Could it be that your application is not doing 
 that?
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge
 _______________________________________________
 freebsd-net@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-net
 To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->feedback 
State-Changed-By: rwatson 
State-Changed-When: Mon Sep 22 14:22:16 UTC 2008 
State-Changed-Why:  
Request feedback based on correspondence to date -- the delivery of ICMP 
messages to multiple raw sockets is historical behavior from the original 
BSD network stack, and not considered a bug.  However, clearly something 
has changed in the software configuration leading to the reported 
symptoms, and a FreeBSD kernel bug is not precluded as perhaps some other 
portion of raw IP socket input processing has changed and/or been broken. 

Please let us know what further investigation reveals about the source of 
the problem. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=127528 

From: "Seth Mos" <seth.mos@xs4all.nl>
To: bug-followup@FreeBSD.org, seth.mos@xs4all.nl
Cc:  
Subject: kern/127528: [icmp]: icmp socket receives icmp replies not owned 
     by the process.
Date: Wed, 13 May 2009 21:58:40 +0200 (CEST)

 Hi there again, it's been a while.
 
 We are currently on a pfSense 1.2.3-RC1 release and we are now getting a
 host of people on the forum and in the mailing lists that are complaining
 about the load balancer flapping.
 
 Some debugging turned out some very weird behaviour.
 
 Every now and then, depending on the concurrenty of icmp traffic
 originating from the FreeBSD host, ping will miss icmp replies.
 
 Here are the steps performed to debug this issue.
 
 For example the output
 tcpdump -ni vlan0 -t icmp
 Where vlan0 is the external interface. The IP address being pinged is a
 local gateway connected by ethernet so there is nothing but the switch in
 between.
 http://pastebin.com/f6608c875
 
 The ping command issued is
 /sbin/ping -t 4 -oqc 5 -i 0.7 213.23.199.249
 
 This command line will ping the target and exit with a return status of 0
 when it receives a reply which can be any of the 5 attempts within the
 maximum of 4 seconds.
 
 The output clearly shows that all attempts have unique process IDs, which
 is good. It also shows that all attempt have a sequence number of 0 which
 is the 1st attempt to ping.
 
 There is however one attempt where this is different, on attempt with pid
 46060 you will see that it attempted 4 times to ping the target, it got
 valid responses to all 5 requests!.
 
 However, ping exited with a return status of 1 which would imply a timeout.
 
 So now it is worse then the originally reported issue we had on 7.0. We
 can now make the FreeBSD distributed ping fail.
 
 Specifically, it fails to see the valid replies.
 
 For your reference I also pasted the tcpdump below. I can also reproduce
 this exact same behaviour by using fping. So I would not consider this a
 bug in the ping binary itself.
 
 Since the original PR had this issue with apinger, which is different from
 both ping and fping I am quite confident there is a serious regression in
 here.
 
 # uname -a
 FreeBSD noord.coltex.nl 7.1-RELEASE-p4 FreeBSD 7.1-RELEASE-p4 #0: Tue Mar
 24 01:18:19 EDT 2009    
 sullrich@RELENG_1_2-snapshots.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.7
  i386
 
 Kind regards,
 
 Seth Mos
 pfSense Developer
 
 
 --------------
    1.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 25068, seq
 0, length 64
    2.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 25068, seq
 0, length 64
    3.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 27372, seq
 0, length 64
    4.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 27372, seq
 0, length 64
    5.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 29676, seq
 0, length 64
    6.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 29676, seq
 0, length 64
    7.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 31468, seq
 0, length 64
    8.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 31468, seq
 0, length 64
    9.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 33772, seq
 0, length 64
   10.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 33772, seq
 0, length 64
   11.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 36076, seq
 0, length 64
   12.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 36076, seq
 0, length 64
   13.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 0, length 64
   14.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 0, length 64
   15.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 1, length 64
   16.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 1, length 64
   17.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 12525, seq
 0, length 64
   18.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 12525, seq
 0, length 64
   19.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 14829, seq
 0, length 64
   20.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 14829, seq
 0, length 64
   21.
       IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.120.254
 unreachable - admin prohibited filter, length 60
   22.
       IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.148.1
 unreachable - admin prohibited filter, length 60
   23.
       IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.112.1
 unreachable - admin prohibited filter, length 60
   24.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 2, length 64
   25.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 2, length 64
   26.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 17645, seq
 0, length 64
   27.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 17645, seq
 0, length 64
   28.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 3, length 64
   29.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 3, length 64
   30.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 4, length 64
   31.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 4, length 64
 

From: Chris Buechler <cmb@pfsense.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/127528: [icmp]: icmp socket receives icmp replies not owned 
	by the process.
Date: Wed, 13 May 2009 23:08:05 -0400

 A brief addition to Seth's update on this issue, which should provide
 adequate detail. We switched from fping to the stock FreeBSD ping, and
 it made the issue better, but we're still seeing problems with
 FreeBSD's ping. In many circumstances it never appears, in others it's
 very easy to replicate. There isn't any commonality between
 environments that can replicate it, it happens across many types of
 NICs, using VLANs and not, etc. We should be able to arrange access to
 one of the systems that can routinely replicate the problem for an
 interested FreeBSD developer. We're in a bit over our heads here and
 not sure how to troubleshoot further.
 
 thanks,
 Chris
>Unformatted:
