From nobody@FreeBSD.org  Wed Mar 12 11:48:51 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 3E0CB1065673
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 12 Mar 2008 11:48:51 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 2EFD48FC32
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 12 Mar 2008 11:48:51 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m2CBjYGn067924
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 12 Mar 2008 11:45:34 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.2/8.14.1/Submit) id m2CBjYor067923;
	Wed, 12 Mar 2008 11:45:34 GMT
	(envelope-from nobody)
Message-Id: <200803121145.m2CBjYor067923@www.freebsd.org>
Date: Wed, 12 Mar 2008 11:45:34 GMT
From: Alexander Zagrebin <alexz@visp.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: When using IPSec, tcpdump shows non-decapsulated packets on gif interface
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         121642
>Category:       kern
>Synopsis:       [ipsec] [gif]: When using IPSec, tcpdump shows non-decapsulated packets on gif interface
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    ae
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Mar 12 11:50:00 UTC 2008
>Closed-Date:    
>Last-Modified:  Fri Apr 04 09:36:28 UTC 2014
>Originator:     Alexander Zagrebin
>Release:        7.0-RELEASE
>Organization:
>Environment:
FreeBSD hostname 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Thu Mar  6 14:32:06 MSK 2008     alex@hostname:/usr/src/sys/i386/compile/KERNEL  i386
>Description:
I'm using ipsec-protected tunnel between 2 hosts.
The first one (HostA) is running 6.3-RELEASE and second one (HostB) - 7.0-RELEASE.
The network topology:

     ------- public network ------- 
        |                      |
        |                      |
     A.A.A.A                B.B.B.B
      Host A                 Host B
   6.3-RELEASE            7.0-RELEASE
   192.168.0.2            192.168.1.1
        |                      |
        |                      |
     private                private
net 192.168.0.0/24    net 192.168.1.0/24

After creating the tunnel using gif interfaces without IPSec, I can successfully
ping host 192.168.0.10 from 192.168.1.2.
tcpdump on gif interfaces shows:

HostA# tcpdump -ni gif0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
21:21:20.861465 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 49921, length 40
21:21:20.862060 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 49921, length 40

HostB# tcpdump -ni gif0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
21:21:25.012390 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 49921, length 40
21:21:25.014457 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 49921, length 40

All right.

Now I'm starting IPSec.

ipsec.conf on HostA:
spdadd B.B.B.B/32 A.A.A.A/32 ipencap
    -P in ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;
spdadd A.A.A.A/32 B.B.B.B/32 ipencap
    -P out ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;

ipsec.conf on HostB:
spdadd A.A.A.A/32 B.B.B.B/32 ipencap
    -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
spdadd B.B.B.B/32 A.A.A.A/32 ipencap
    -P out ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;

Pinging 192.168.0.10 from 192.168.1.2 is successful again, but
tcpdump shows:

HostA# tcpdump -ni gif0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
21:24:12.429778 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
21:24:12.430246 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 51969, length 40

HostB# tcpdump -ni gif0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
21:24:16.577374 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
21:24:16.580876 IP 10.2.222.2 > 10.255.255.2: IP 192.168.0.10 > 192.168.1.2: 
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ICMP echo reply, id 512, seq 51969, length 40 (ipip-proto-4)

Oops!
tcpdump shows non-decapsulated packets, but tunnel works fine.
>How-To-Repeat:
See above
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->gnn 
Responsible-Changed-By: remko 
Responsible-Changed-When: Wed Mar 12 12:24:14 UTC 2008 
Responsible-Changed-Why:  
Hi George, can you have a look at this one please? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121642 
Responsible-Changed-From-To: gnn->bz 
Responsible-Changed-By: remko 
Responsible-Changed-When: Wed Mar 12 12:42:07 UTC 2008 
Responsible-Changed-Why:  
Reassign to Bjoern on his request 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121642 

From: Volker <volker@vwsoft.com>
To: bug-followup@FreeBSD.org, alexz@visp.ru
Cc:  
Subject: Re: kern/121642: [ipsec] [gif]: When using IPSec, tcpdump shows non-decapsulated
 packets on gif interface
Date: Wed, 12 Mar 2008 17:52:15 +0100

 > Now I'm starting IPSec.
 > 
 > ipsec.conf on HostA:
 > spdadd B.B.B.B/32 A.A.A.A/32 ipencap
 > -P in ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;
 > spdadd A.A.A.A/32 B.B.B.B/32 ipencap
 > -P out ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
 > 
 > ipsec.conf on HostB:
 > spdadd A.A.A.A/32 B.B.B.B/32 ipencap
 > -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
 > spdadd B.B.B.B/32 A.A.A.A/32 ipencap
 > -P out ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;
 > 
 > Pinging 192.168.0.10 from 192.168.1.2 is successful again, but
 > tcpdump shows:
 > 
 > HostA# tcpdump -ni gif0
 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 > listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
 > 21:24:12.429778 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
 > 21:24:12.430246 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 51969, length 40
 > 
 > HostB# tcpdump -ni gif0
 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 > listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
 > 21:24:16.577374 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
 > 21:24:16.580876 IP 10.2.222.2 > 10.255.255.2: IP 192.168.0.10 > 192.168.1.2:
 
 hmm, I'm wondering if `ping -S A.A.A.A B.B.B.B' (executed at Host A) may
 serve your expectations?
 
 Actually I can smell some misconfigurations here. ;)
 
 Volker
 

From: Volker <volker@vwsoft.com>
To: bug-followup@FreeBSD.org, alexz@visp.ru
Cc:  
Subject: Re: kern/121642: [ipsec] [gif]: When using IPSec, tcpdump shows non-decapsulated
 packets on gif interface
Date: Wed, 12 Mar 2008 18:24:49 +0100

 Sorry, I've been giving just half of an answer with my previous post (as
 you're trying to use gif).
 
 Say, you're using a policy like:
 
 spdadd 192.168.0.0/24 192.168.1.0/24 ipencap
  -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
 
 then trying if `ping -S 192.168.0.1 192.168.1.1' may magically let you
 see encrypted traffic.

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, alexz@visp.ru
Cc:  
Subject: Re: kern/121642: When using IPSec, tcpdump shows non-decapsulated
 packets on gif interface
Date: Wed, 12 Mar 2008 23:42:41 +0000 (UTC)

 On Wed, 12 Mar 2008, Alexander Zagrebin wrote:
 
 Hi,
 
 > I'm using ipsec-protected tunnel between 2 hosts.
 > The first one (HostA) is running 6.3-RELEASE and second one (HostB) - 7.0-RELEASE.
 > The network topology:
 >
 >     ------- public network -------
 >        |                      |
 >        |                      |
 >     A.A.A.A                B.B.B.B
 >      Host A                 Host B
 >   6.3-RELEASE            7.0-RELEASE
 >   192.168.0.2            192.168.1.1
 >        |                      |
 >        |                      |
 >     private                private
 > net 192.168.0.0/24    net 192.168.1.0/24
 >
 > After creating the tunnel using gif interfaces without IPSec, I can successfully
 > ping host 192.168.0.10 from 192.168.1.2.
 > tcpdump on gif interfaces shows:
 >
 > HostA# tcpdump -ni gif0
 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 > listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
 > 21:21:20.861465 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 49921, length 40
 > 21:21:20.862060 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 49921, length 40
 >
 > HostB# tcpdump -ni gif0
 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 > listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
 > 21:21:25.012390 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 49921, length 40
 > 21:21:25.014457 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 49921, length 40
 >
 > All right.
 >
 > Now I'm starting IPSec.
 >
 > ipsec.conf on HostA:
 > spdadd B.B.B.B/32 A.A.A.A/32 ipencap
 >    -P in ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;
 > spdadd A.A.A.A/32 B.B.B.B/32 ipencap
 >    -P out ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
 >
 > ipsec.conf on HostB:
 > spdadd A.A.A.A/32 B.B.B.B/32 ipencap
 >    -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/require;
 > spdadd B.B.B.B/32 A.A.A.A/32 ipencap
 >    -P out ipsec esp/tunnel/B.B.B.B-A.A.A.A/require;
 
 
 
 If you are only using IPsec to protect the 'gif tunnel'
 you could use transport mode.
 
 If you do not need the real interface for a link state
 routing protocol or similar you could simply use tunnel
 mode for the entire 'private nets' without the gif tunnel.
 
 
 Now that's not your problem, let's see..
 
 
 > Pinging 192.168.0.10 from 192.168.1.2 is successful again, but
 > tcpdump shows:
 >
 > HostA# tcpdump -ni gif0
 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 > listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
 > 21:24:12.429778 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
 > 21:24:12.430246 IP 192.168.0.10 > 192.168.1.2: ICMP echo reply, id 512, seq 51969, length 40
 >
 > HostB# tcpdump -ni gif0
 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 > listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
 > 21:24:16.577374 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
 > 21:24:16.580876 IP 10.2.222.2 > 10.255.255.2: IP 192.168.0.10 > 192.168.1.2:
 >                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 > ICMP echo reply, id 512, seq 51969, length 40 (ipip-proto-4)
 >
 > Oops!
 > tcpdump shows non-decapsulated packets, but tunnel works fine.
 
 ok, so that's the input path where you are seeing this. let's see what
 happens there... ( I simplified things the further on I got as it's
 long ...)
 
 ip_input (IP, ESP)
 ip_ipsec_filtertunnel() false -> cont.
 pfil_run_hooks
 ip_ipsec_input
 ipsec_in_reject (cont)
 dispatch(ESPv4)
 esp4_input
 ipsec4_common_input
 ipsec_common_input
 dispatch(XF_ESP)
 esp_input
 crypto_dispatch
 esp_input_cb	[ strip esp header -> outer IPIP header now ]
 ipsec_common_input_cb
 ipsec4_common_input_cb
 enc(BEFORE)
 netisr_queue(NETISR_IP)
 ip_input
 filter (here we see the ipencap from the ipsec on the incoming if)
 dispatch(ipencap)
 encap4_input
 ip_input
 _ipip_input (remove outer IP header (the one from ouside ESP) ->
               inner ESP IP header + ULP which is inner gif IP header left)
               -> IP:ULP == IP:IP again)
 enc(AFTER)
 netisr_queue
 ip_input
 filter
 dispatch(upper layer protocol == IP -> ipencap)
 encap4_input
 in_gif_input
 m_adj(m, off) [should cut off the first IP header -> IP:ULP == IP:ICMP left]
 gif_input
 BPF << there you should see it with one IP header left.
 dispatch(IP)
 ip_input
 ...
 
 
 I wonder what I am missing atm, it's a bit late here. I'll try to find
 the clue tomo^H^H^H^Hlater today.
 
 I knew there are things that happen differently, especially the IPIP
 seen in ip_input before the call to _ipip_input but I wasn't aware
 that there were later problems as well and basically should not be.
 
 
 -- 
 Bjoern A. Zeeb                                 bzeeb at Zabbadoz dot NeT
 Software is harder than hardware  so better get it right the first time.
State-Changed-From-To: open->feedback 
State-Changed-By: bz 
State-Changed-When: Thu Mar 13 09:58:00 UTC 2008 
State-Changed-Why:  
Wait for feedback whether things go through gif/bpf twice or not. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121642 

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, alexz@visp.ru
Cc:  
Subject: Re: kern/121642: When using IPSec, tcpdump shows non-decapsulated
 packets on gif interface
Date: Thu, 13 Mar 2008 09:55:23 +0000 (UTC)

 On Wed, 12 Mar 2008, Bjoern A. Zeeb wrote:
 
 Hi,
 
 I had suspected the right thing while thinking about this after
 writing the reply but I now also read the code to validate this.
 
 Basically things seem to work as expected.
 
 I wonder if you are also seeing the inner packet only as well
 (two packets in the outgoing direction)?
 
 Can you confirm this?
 
 
 
 I'll add comments inline further down.
 
 
 > > HostB# tcpdump -ni gif0
 > > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 > > listening on gif0, link-type NULL (BSD loopback), capture size 96 bytes
 > > 21:24:16.577374 IP 192.168.1.2 > 192.168.0.10: ICMP echo request, id 512, seq 51969, length 40
 > > 21:24:16.580876 IP 10.2.222.2 > 10.255.255.2: IP 192.168.0.10 > 192.168.1.2:
 > >                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 > > ICMP echo reply, id 512, seq 51969, length 40 (ipip-proto-4)
 > >
 > > Oops!
 > > tcpdump shows non-decapsulated packets, but tunnel works fine.
 >
 > ok, so that's the input path where you are seeing this. let's see what
 > happens there... ( I simplified things the further on I got as it's
 > long ...)
 >
 > ip_input (IP, ESP)
 > ip_ipsec_filtertunnel() false -> cont.
 > pfil_run_hooks
 > ip_ipsec_input
 > ipsec_in_reject (cont)
 > dispatch(ESPv4)
 > esp4_input
 > ipsec4_common_input
 > ipsec_common_input
 > dispatch(XF_ESP)
 > esp_input
 > crypto_dispatch
 > esp_input_cb	[ strip esp header -> outer IPIP header now ]
 > ipsec_common_input_cb
 > ipsec4_common_input_cb
 > enc(BEFORE)
 > netisr_queue(NETISR_IP)
 > ip_input
 > filter (here we see the ipencap from the ipsec on the incoming if)
 > dispatch(ipencap)
 > encap4_input
 
 This is walking through checking all possible candidates, which would
 be:
 
 - ipe4_protosw (_ipip_input from xform_ipip.c)
 - in_gif_protosw (gif_encapcheck from in_gif.c/if_gif.c)
 
 the gif is returning the higher priority thus it gets the packet.
 
 so from here on it goes like this:
 
 in_gif_input
 m_adj(m, off) [should cut off the first IP header -> IP:ULP == IPIP left]
 gif_input
 BPF << here you are now seeing the IPIP as above
 dispatch(IP)
 ip_input
 filter (ipencap again)
 dispatch(ipencap)	with just esxactly the same IPIP headers again as above
 and to my understanding it now should go through 
 encap4_input
 in_gif_input
 ...
 again and you should see it a 2nd time in BPF.
 
 Can you confirm this?
 
 
 
 
 So basically this is a general problem with the IPsec/ipencap code.
 
 I have on my todo list:
 9 ipsec improvements: do not go through ip_input three times
 
 
 There are multiple solutions to fix that but all have to be thought of
 carefully to not break other assumptions.
 
 
 The most important problem with the current code is that for
 if_enc you will not see the packet for filtering but you will see it
 in ip_input more often.
 
 
 
 > ip_input
    ^^ this should have been ip4_input (from xform_ipip.c btw)
 > _ipip_input (remove outer IP header (the one from ouside ESP) ->
 >               inner ESP IP header + ULP which is inner gif IP header left)
 >               -> IP:ULP == IP:IP again)
 > enc(AFTER)
 > netisr_queue
 > ip_input
 > filter
 > dispatch(upper layer protocol == IP -> ipencap)
 > encap4_input
 > in_gif_input
 > m_adj(m, off) [should cut off the first IP header -> IP:ULP == IP:ICMP left]
 > gif_input
 > BPF << there you should see it with one IP header left.
 > dispatch(IP)
 > ip_input
 > ...
 
 -- 
 Bjoern A. Zeeb                                 bzeeb at Zabbadoz dot NeT
 Software is harder than hardware  so better get it right the first time.

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, alexz@visp.ru
Cc:  
Subject: RE: kern/121642: When using IPSec, tcpdump shows non-decapsulated
 packets on gif interface
Date: Thu, 13 Mar 2008 14:30:59 +0000 (UTC)

 On Thu, 13 Mar 2008, Alexander Zagrebin wrote:
 
 Hi,
 
 >> I wonder if you are also seeing the inner packet only as well
 >> (two packets in the outgoing direction)?
 >>
 >> Can you confirm this?
 >
 > If I have correctly understood, you suppose, that tcpdump may show 2
 > outgoing packets instead 1?
 > No. It shows one packet on each outgoing one.
 
 hmm, ok , then something else is going on still. I guess I need to
 really trace packets.
 
 
 > If it can help, after switching the tunnel to transport mode,
 > tcpdump shows traffic via gif absolutely correctly.
 > So, the problem exists when using gif and ipsec in tunnel mode only.
 
 Makes perfectly sense.
 
 With tunnel mode you end up with:
 
 IP | ESP | IP | IP | ULP
 reducing to
 IP | IP | IP | ULP (1st ipip decap|gif, gif wins)
                     (in_gif_input removes the first header)
 BPF happens
 IP | IP | ULP      (2nd ipip decap|gif --- I thought gif wins
  			again and we'd see it a 2nd time in bpf)
 IP | ULP
 
 
 With transport mode it's like:
 IP | ESP | IP | ULP
 IP | IP | ULP      (1st ipip decap|gif, gif wins)
 IP | ULP           (in_gif_input removes the first header)
 BPF happens
 
 -- 
 Bjoern A. Zeeb                                 bzeeb at Zabbadoz dot NeT
 Software is harder than hardware  so better get it right the first time.
State-Changed-From-To: feedback->open 
State-Changed-By: bz 
State-Changed-When: Thu Mar 13 14:41:43 UTC 2008 
State-Changed-Why:  
Feedback recevied. Needs for analysing. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121642 
Responsible-Changed-From-To: bz->ae 
Responsible-Changed-By: ae 
Responsible-Changed-When: Fri Apr 4 09:35:45 UTC 2014 
Responsible-Changed-Why:  
I'm working on this now. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121642 
>Unformatted:
