From bw@exile.desync.com  Fri Feb 14 07:51:37 2014
Return-Path: <bw@exile.desync.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTPS id CA70992C
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 14 Feb 2014 07:51:37 +0000 (UTC)
Received: from exile.desync.com (exile.desync.com [IPv6:2607:f178::167])
	by mx1.freebsd.org (Postfix) with ESMTP id 8679D19CE
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 14 Feb 2014 07:51:37 +0000 (UTC)
Received: by exile.desync.com (Postfix, from userid 666)
	id B72493861; Fri, 14 Feb 2014 02:51:27 -0500 (EST)
Message-Id: <20140214075127.B72493861@exile.desync.com>
Date: Fri, 14 Feb 2014 02:51:27 -0500 (EST)
From: Ben Wilber <ben@desync.com>
Reply-To: Ben Wilber <ben@desync.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: ipsec tunnels don't work with pf or ipfw
X-Send-Pr-Version: 3.114
X-GNATS-Notify:

>Number:         186755
>Category:       kern
>Synopsis:       ipsec tunnels don't work with pf or ipfw
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    glebius
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Feb 14 08:00:00 UTC 2014
>Closed-Date:    Tue Mar 18 16:56:23 UTC 2014
>Last-Modified:  Tue Mar 18 17:00:01 UTC 2014
>Originator:     Ben Wilber
>Release:        FreeBSD 10.0-STABLE amd64
>Organization:
>Environment:
System: FreeBSD server 10.0-STABLE FreeBSD 10.0-STABLE #2 r261258M: Fri Jan 31 17:36:49 EST 2014     bw@exodus:/usr/obj/factory/stable/sys/COMRADE  amd64


	
>Description:
Traffic forwarded from an IPsec SA in tunnel mode appears to bypass pf and ipfw.

Last tested working in 9.2-STABLE around December 2013.
>How-To-Repeat:
Quick lab with two instances of bhyve, server and client.

Set up basic connectivity:

root@server:~ # ifconfig vtnet0 208.83.20.170/27
root@server:~ # route add default 208.83.20.161
add net default: gateway 208.83.20.161 fib 0

root@client:~ # ifconfig vtnet0 208.83.20.171/27

Create an IPsec tunnel.  On server:

echo add 208.83.20.170 208.83.20.171 esp 31337 -m tunnel -E des-cbc 0xc001c001c001c001\; | setkey -c
echo add 208.83.20.171 208.83.20.170 esp 31338 -m tunnel -E des-cbc 0xc001c001c001c001\; | setkey -c
echo 'spdadd 0.0.0.0/0[any] 10.0.0.1/32[any] any -P out ipsec esp/tunnel/208.83.20.170-208.83.20.171/require;' | setkey -c
echo 'spdadd 10.0.0.1/32[any] 0.0.0.0/0[any] any -P in ipsec esp/tunnel/208.83.20.171-208.83.20.170/require;' | setkey -c

On client:

echo add 208.83.20.170 208.83.20.171 esp 31337 -m tunnel -E des-cbc 0xc001c001c001c001\; | setkey -c
echo add 208.83.20.171 208.83.20.170 esp 31338 -m tunnel -E des-cbc 0xc001c001c001c001\; | setkey -c
echo 'spdadd 10.0.0.1/32[any] 0.0.0.0/0[any] any -P out ipsec esp/tunnel/208.83.20.171-208.83.20.170/require;' | setkey -c
echo 'spdadd 0.0.0.0/0[any] 10.0.0.1/32[any] any -P in ipsec esp/tunnel/208.83.20.170-208.83.20.171/require;' | setkey -c

Start a running ping from client and verify that traffic is arriving on server over the tunnel:

root@client:~ # ifconfig lo0 inet alias 10.0.0.1/32
root@client:~ # ping -S 10.0.0.1 8.8.8.8

root@server:~ # ifconfig enc0 up && tcpdump -ni enc0
tcpdump: WARNING: enc0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 65535 bytes
capability mode sandbox enabled
01:54:35.376875 (confidential): SPI 0x00007a6a: IP 208.83.20.171 > 208.83.20.170: IP 10.0.0.1 > 8.8.8.8: ICMP echo request, id 60677, seq 0, length 64 (ipip-proto-4)
01:54:36.434174 (confidential): SPI 0x00007a6a: IP 208.83.20.171 > 208.83.20.170: IP 10.0.0.1 > 8.8.8.8: ICMP echo request, id 60677, seq 1, length 64 (ipip-proto-4)
01:54:37.450706 (confidential): SPI 0x00007a6a: IP 208.83.20.171 > 208.83.20.170: IP 10.0.0.1 > 8.8.8.8: ICMP echo request, id 60677, seq 2, length 64 (ipip-proto-4)
01:54:38.472992 (confidential): SPI 0x00007a6a: IP 208.83.20.171 > 208.83.20.170: IP 10.0.0.1 > 8.8.8.8: ICMP echo request, id 60677, seq 3, length 64 (ipip-proto-4)

Enable IP forwarding on server and verify that the tunneled traffic is being forwarded:

root@server:~ # sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 0 -> 1

root@server:~ # tcpdump -n
listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 bytes
capability mode sandbox enabled
01:48:32.311286 IP 208.83.20.171 > 208.83.20.170: ESP(spi=0x00007a6a,seq=0x2a), length 104
01:48:32.311367 IP 10.0.0.1 > 8.8.8.8: ICMP echo request, id 59141, seq 41, length 64
01:48:33.332164 IP 208.83.20.171 > 208.83.20.170: ESP(spi=0x00007a6a,seq=0x2b), length 104
01:48:33.332246 IP 10.0.0.1 > 8.8.8.8: ICMP echo request, id 59141, seq 42, length 64
01:48:34.352446 IP 208.83.20.171 > 208.83.20.170: ESP(spi=0x00007a6a,seq=0x2c), length 104
01:48:34.352516 IP 10.0.0.1 > 8.8.8.8: ICMP echo request, id 59141, seq 43, length 64

Enable NAT via pf:

root@server:~ # kldload pf
root@server:~ # echo "nat on vtnet0 inet all -> vtnet0" | pfctl -ef -
No ALTQ support in kernel
ALTQ related functions disabled
pf enabled

No states are created for the tunneled traffic and NAT does not function.
Similarly, it doesn't appear possible to construct a filter rule that applies
to the tunneled traffic.  The same happens with ipfw using kernel NAT.

>Fix:

	


>Release-Note:
>Audit-Trail:

From: Nicolas DEFFAYET <nicolas@deffayet.com>
To: bug-followup@FreeBSD.org, ben@desync.com, andre@freebsd.org
Cc:  
Subject: Re: kern/186755: ipsec tunnels don't work with pf or ipfw
Date: Sun, 23 Feb 2014 14:34:00 +0100

 After very long testing, i have discovered the route cause.
 
 The revision 254519 break the firewall with IPsec.
 http://svnweb.freebsd.org/base?view=revision&revision=254519
 
 "Move the global M_SKIP_FIREWALL mbuf flags to a protocol layer specific
 flag instead.  The flag is only used within the IP and IPv6 layer 3
 protocols.
 
 Because some firewall packages treat IPv4 and IPv6 packets the same the
 flag should have the same value for both."
 
 It seem that some code doesn't have been updated for allow firewall to
 work with IPsec.
 
 -- 
 Nicolas DEFFAYET
 
State-Changed-From-To: open->patched 
State-Changed-By: glebius 
State-Changed-When: Wed Mar 12 14:29:44 UTC 2014 
State-Changed-Why:  
Fixed in head/. 


Responsible-Changed-From-To: freebsd-bugs->glebius 
Responsible-Changed-By: glebius 
Responsible-Changed-When: Wed Mar 12 14:29:44 UTC 2014 
Responsible-Changed-Why:  
Fixed in head/. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=186755 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/186755: commit references a PR
Date: Wed, 12 Mar 2014 14:29:23 +0000 (UTC)

 Author: glebius
 Date: Wed Mar 12 14:29:08 2014
 New Revision: 263091
 URL: http://svnweb.freebsd.org/changeset/base/263091
 
 Log:
   Since both netinet/ and netinet6/ call into netipsec/ and netpfil/,
   the protocol specific mbuf flags are shared between them.
   
   - Move all M_FOO definitions into a single place: netinet/in6.h, to
     avoid future  clashes.
   - Resolve clash between M_DECRYPTED and M_SKIP_FIREWALL which resulted
     in a failure of operation of IPSEC and packet filters.
   
   Thanks to Nicolas and Georgios for all the hard work on bisecting,
   testing and finally finding the root of the problem.
   
   PR:			kern/186755
   PR:			kern/185876
   In collaboration with:	Georgios Amanakis <gamanakis gmail.com>
   In collaboration with:	Nicolas DEFFAYET <nicolas-ml deffayet.com>
   Sponsored by:		Nginx, Inc.
 
 Modified:
   head/sys/netinet/ip_input.c
   head/sys/netinet/ip_var.h
   head/sys/netinet6/in6.h
   head/sys/netinet6/ip6_var.h
 
 Modified: head/sys/netinet/ip_input.c
 ==============================================================================
 --- head/sys/netinet/ip_input.c	Wed Mar 12 12:27:13 2014	(r263090)
 +++ head/sys/netinet/ip_input.c	Wed Mar 12 14:29:08 2014	(r263091)
 @@ -702,6 +702,7 @@ ours:
  	 * ip_reass() will return a different mbuf.
  	 */
  	if (ip->ip_off & htons(IP_MF | IP_OFFMASK)) {
 +		/* XXXGL: shouldn't we save & set m_flags? */
  		m = ip_reass(m);
  		if (m == NULL)
  			return;
 @@ -794,6 +795,8 @@ SYSCTL_PROC(_net_inet_ip, OID_AUTO, maxf
      NULL, 0, sysctl_maxnipq, "I",
      "Maximum number of IPv4 fragment reassembly queue entries");
  
 +#define	M_IP_FRAG	M_PROTO9
 +
  /*
   * Take incoming datagram fragment and try to reassemble it into
   * whole datagram.  If the argument is the first fragment or one
 
 Modified: head/sys/netinet/ip_var.h
 ==============================================================================
 --- head/sys/netinet/ip_var.h	Wed Mar 12 12:27:13 2014	(r263090)
 +++ head/sys/netinet/ip_var.h	Wed Mar 12 14:29:08 2014	(r263091)
 @@ -162,15 +162,6 @@ void	kmod_ipstat_dec(int statnum);
  #define IP_ROUTETOIF		SO_DONTROUTE	/* 0x10 bypass routing tables */
  #define IP_ALLOWBROADCAST	SO_BROADCAST	/* 0x20 can send broadcast packets */
  
 -/*
 - * IPv4 protocol layer specific mbuf flags.
 - */
 -#define	M_FASTFWD_OURS		M_PROTO1	/* changed dst to local */
 -#define	M_IP_NEXTHOP		M_PROTO2	/* explicit ip nexthop */
 -#define	M_SKIP_FIREWALL		M_PROTO3	/* skip firewall processing,
 -						   keep in sync with IP6 */
 -#define	M_IP_FRAG		M_PROTO4	/* fragment reassembly */
 -
  #ifdef __NO_STRICT_ALIGNMENT
  #define IP_HDR_ALIGNED_P(ip)	1
  #else
 
 Modified: head/sys/netinet6/in6.h
 ==============================================================================
 --- head/sys/netinet6/in6.h	Wed Mar 12 12:27:13 2014	(r263090)
 +++ head/sys/netinet6/in6.h	Wed Mar 12 14:29:08 2014	(r263091)
 @@ -622,13 +622,18 @@ struct ip6_mtuinfo {
  #endif /* __BSD_VISIBLE */
  
  /*
 - * Redefinition of mbuf flags
 + * Since both netinet/ and netinet6/ call into netipsec/ and netpfil/,
 + * the protocol specific mbuf flags are shared between them.
   */
 -#define	M_AUTHIPHDR	M_PROTO2
 -#define	M_DECRYPTED	M_PROTO3
 -#define	M_LOOP		M_PROTO4
 -#define	M_AUTHIPDGM	M_PROTO5
 -#define	M_RTALERT_MLD	M_PROTO6
 +#define	M_FASTFWD_OURS		M_PROTO1	/* changed dst to local */
 +#define	M_IP6_NEXTHOP		M_PROTO2	/* explicit ip nexthop */
 +#define	M_IP_NEXTHOP		M_PROTO2	/* explicit ip nexthop */
 +#define	M_SKIP_FIREWALL		M_PROTO3	/* skip firewall processing */
 +#define	M_AUTHIPHDR		M_PROTO4
 +#define	M_DECRYPTED		M_PROTO5
 +#define	M_LOOP			M_PROTO6
 +#define	M_AUTHIPDGM		M_PROTO7
 +#define	M_RTALERT_MLD		M_PROTO8
  
  #ifdef _KERNEL
  struct cmsghdr;
 
 Modified: head/sys/netinet6/ip6_var.h
 ==============================================================================
 --- head/sys/netinet6/ip6_var.h	Wed Mar 12 12:27:13 2014	(r263090)
 +++ head/sys/netinet6/ip6_var.h	Wed Mar 12 14:29:08 2014	(r263091)
 @@ -293,13 +293,6 @@ struct ip6aux {
  #define	IPV6_FORWARDING		0x02	/* most of IPv6 header exists */
  #define	IPV6_MINMTU		0x04	/* use minimum MTU (IPV6_USE_MIN_MTU) */
  
 -/*
 - * IPv6 protocol layer specific mbuf flags.
 - */
 -#define	M_IP6_NEXTHOP		M_PROTO2	/* explicit ip nexthop */
 -#define	M_SKIP_FIREWALL		M_PROTO3	/* skip firewall processing,
 -						   keep in sync with IPv4 */
 -
  #ifdef __NO_STRICT_ALIGNMENT
  #define IP6_HDR_ALIGNED_P(ip)	1
  #else
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: glebius 
State-Changed-When: Tue Mar 18 16:56:06 UTC 2014 
State-Changed-Why:  
Merged to stable/10. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=186755 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/186755: commit references a PR
Date: Tue, 18 Mar 2014 16:56:15 +0000 (UTC)

 Author: glebius
 Date: Tue Mar 18 16:56:05 2014
 New Revision: 263307
 URL: http://svnweb.freebsd.org/changeset/base/263307
 
 Log:
   Merge r263091: fix mbuf flags clash that lead to failure of operation
   of IPSEC and packet filters.
   
   PR:		kern/185876
   PR:		kern/186755
 
 Modified:
   stable/10/sys/netinet/ip_input.c
   stable/10/sys/netinet/ip_var.h
   stable/10/sys/netinet6/in6.h
   stable/10/sys/netinet6/ip6_var.h
 Directory Properties:
   stable/10/   (props changed)
 
 Modified: stable/10/sys/netinet/ip_input.c
 ==============================================================================
 --- stable/10/sys/netinet/ip_input.c	Tue Mar 18 16:41:32 2014	(r263306)
 +++ stable/10/sys/netinet/ip_input.c	Tue Mar 18 16:56:05 2014	(r263307)
 @@ -707,6 +707,7 @@ ours:
  	 * ip_reass() will return a different mbuf.
  	 */
  	if (ip->ip_off & htons(IP_MF | IP_OFFMASK)) {
 +		/* XXXGL: shouldn't we save & set m_flags? */
  		m = ip_reass(m);
  		if (m == NULL)
  			return;
 @@ -799,6 +800,8 @@ SYSCTL_PROC(_net_inet_ip, OID_AUTO, maxf
      NULL, 0, sysctl_maxnipq, "I",
      "Maximum number of IPv4 fragment reassembly queue entries");
  
 +#define	M_IP_FRAG	M_PROTO9
 +
  /*
   * Take incoming datagram fragment and try to reassemble it into
   * whole datagram.  If the argument is the first fragment or one
 
 Modified: stable/10/sys/netinet/ip_var.h
 ==============================================================================
 --- stable/10/sys/netinet/ip_var.h	Tue Mar 18 16:41:32 2014	(r263306)
 +++ stable/10/sys/netinet/ip_var.h	Tue Mar 18 16:56:05 2014	(r263307)
 @@ -162,15 +162,6 @@ void	kmod_ipstat_dec(int statnum);
  #define IP_ROUTETOIF		SO_DONTROUTE	/* 0x10 bypass routing tables */
  #define IP_ALLOWBROADCAST	SO_BROADCAST	/* 0x20 can send broadcast packets */
  
 -/*
 - * IPv4 protocol layer specific mbuf flags.
 - */
 -#define	M_FASTFWD_OURS		M_PROTO1	/* changed dst to local */
 -#define	M_IP_NEXTHOP		M_PROTO2	/* explicit ip nexthop */
 -#define	M_SKIP_FIREWALL		M_PROTO3	/* skip firewall processing,
 -						   keep in sync with IP6 */
 -#define	M_IP_FRAG		M_PROTO4	/* fragment reassembly */
 -
  #ifdef __NO_STRICT_ALIGNMENT
  #define IP_HDR_ALIGNED_P(ip)	1
  #else
 
 Modified: stable/10/sys/netinet6/in6.h
 ==============================================================================
 --- stable/10/sys/netinet6/in6.h	Tue Mar 18 16:41:32 2014	(r263306)
 +++ stable/10/sys/netinet6/in6.h	Tue Mar 18 16:56:05 2014	(r263307)
 @@ -622,13 +622,18 @@ struct ip6_mtuinfo {
  #endif /* __BSD_VISIBLE */
  
  /*
 - * Redefinition of mbuf flags
 + * Since both netinet/ and netinet6/ call into netipsec/ and netpfil/,
 + * the protocol specific mbuf flags are shared between them.
   */
 -#define	M_AUTHIPHDR	M_PROTO2
 -#define	M_DECRYPTED	M_PROTO3
 -#define	M_LOOP		M_PROTO4
 -#define	M_AUTHIPDGM	M_PROTO5
 -#define	M_RTALERT_MLD	M_PROTO6
 +#define	M_FASTFWD_OURS		M_PROTO1	/* changed dst to local */
 +#define	M_IP6_NEXTHOP		M_PROTO2	/* explicit ip nexthop */
 +#define	M_IP_NEXTHOP		M_PROTO2	/* explicit ip nexthop */
 +#define	M_SKIP_FIREWALL		M_PROTO3	/* skip firewall processing */
 +#define	M_AUTHIPHDR		M_PROTO4
 +#define	M_DECRYPTED		M_PROTO5
 +#define	M_LOOP			M_PROTO6
 +#define	M_AUTHIPDGM		M_PROTO7
 +#define	M_RTALERT_MLD		M_PROTO8
  
  #ifdef _KERNEL
  struct cmsghdr;
 
 Modified: stable/10/sys/netinet6/ip6_var.h
 ==============================================================================
 --- stable/10/sys/netinet6/ip6_var.h	Tue Mar 18 16:41:32 2014	(r263306)
 +++ stable/10/sys/netinet6/ip6_var.h	Tue Mar 18 16:56:05 2014	(r263307)
 @@ -293,13 +293,6 @@ struct ip6aux {
  #define	IPV6_FORWARDING		0x02	/* most of IPv6 header exists */
  #define	IPV6_MINMTU		0x04	/* use minimum MTU (IPV6_USE_MIN_MTU) */
  
 -/*
 - * IPv6 protocol layer specific mbuf flags.
 - */
 -#define	M_IP6_NEXTHOP		M_PROTO2	/* explicit ip nexthop */
 -#define	M_SKIP_FIREWALL		M_PROTO3	/* skip firewall processing,
 -						   keep in sync with IPv4 */
 -
  #ifdef __NO_STRICT_ALIGNMENT
  #define IP6_HDR_ALIGNED_P(ip)	1
  #else
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
>Unformatted:
