From sa@nkz.delikates-nk.ru  Mon Sep 11 13:41:54 2006
Return-Path: <sa@nkz.delikates-nk.ru>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 91A8016A49E;
	Mon, 11 Sep 2006 13:41:54 +0000 (UTC)
	(envelope-from sa@nkz.delikates-nk.ru)
Received: from nkz.delikates-nk.ru (nkz.delikates-nk.ru [81.16.143.102])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 92B7B43D49;
	Mon, 11 Sep 2006 13:41:53 +0000 (GMT)
	(envelope-from sa@nkz.delikates-nk.ru)
Received: from nkz.delikates-nk.ru (localhost [127.0.0.1])
	by nkz.delikates-nk.ru (8.13.8/8.13.8) with ESMTP id k8BDfogL020222;
	Mon, 11 Sep 2006 21:41:50 +0800 (KRAST)
	(envelope-from sa@nkz.delikates-nk.ru)
Received: (from sa@localhost)
	by nkz.delikates-nk.ru (8.13.8/8.13.8/Submit) id k8BDfneZ020221;
	Mon, 11 Sep 2006 21:41:49 +0800 (KRAST)
	(envelope-from sa)
Message-Id: <200609111341.k8BDfneZ020221@nkz.delikates-nk.ru>
Date: Mon, 11 Sep 2006 21:41:49 +0800 (KRAST)
From: Eugene Grosbein <eugen@grosbein.pp.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc: net@freebsd.org
Subject: ipsec with ipfw divert (not NAT) encodes a packet twice breaking PMTUD
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         103135
>Category:       kern
>Synopsis:       [ipsec] ipsec with ipfw divert (not NAT) encodes a packet twice breaking PMTUD
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-net
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Sep 11 13:50:16 GMT 2006
>Closed-Date:    
>Last-Modified:  Wed Jan 14 22:24:54 UTC 2009
>Originator:     Eugene Grosbein
>Release:        FreeBSD 6.1-STABLE i386
>Organization:
Svyaz Service JSC
>Environment:
System: FreeBSD nkz.delikates-nk.ru 6.1-STABLE FreeBSD 6.1-STABLE #1: Thu Sep 7 13:31:53 KRAST 2006 root@nkz.delikates-nk.ru:/home/obj/home/src/sys/NKZ i386
	options IPDIVERT
	options IPSEC
	options IPSEC_ESP

>Description:
	When outgoing packet encoded due to corresponding IPSEC policy
	is passed to divert socket (f.e. to ipacctd for accounting),
	it is encoded second time with IPSEC then. Besides obvious
	logic error, this also results in broken Path MTU Discovery.

>How-To-Repeat:

	Use a kernel with options IPDIVERT, IPSEC, IPSEC_ESP
	(my kernel also contains IPSEC_FILTERGIF, but this should not matter).

	Suppose there are two local nets numbered 192.168.1.0/24
	and 192.168.2.0/24, each has a FreeBSD router
	(192.168.1.1 and 192.168.2.1). Routers make gif(4) tunnel between
	and use IPSEC transport mode to encrypt its contents.
	Their external IP addresses are 1.1.1.1 and 2.2.2.2

	Here is /etc/ipsec.conf:

add 1.1.1.1  81.16.143.102 esp 1007 -m transport -E blowfish-cbc "xxx";
add 2.2.2.2 1.1.1.1 esp 2007 -m transport -E blowfish-cbc "yyy";

spdadd 1.1.1.1/32 2.2.2.2/32 any -P in  ipsec esp/transport/1.1.1.1-2.2.2.2/require;
spdadd 2.2.2.2/32 1.1.1.1/32 any -P out ipsec esp/transport/2.2.2.2-1.1.1.1/require;

	Another router has similar /etc/ipsec.conf.

	Use this script to prepare IPSEC keys for tcpdump:

#!/bin/sh

setkey -D | awk '
  /^[1-9]/  { ip=$2; }
  $1=="esp" { 
		sub(/spi=[^(]+\(/, "", $3);
		sub(/\)/, "", $3);
		printf"%s@%s ",$3,ip;
	    }
  $1=="E:"  { printf "%s:0x%s%s%s%s\n", $2, $3, $4, $5, $6; }
' > /tmp/keys.txt

	When we use one of the routers to run
	"tcpdump -s0 -n -p -i $iface -E /tmp/keys.txt esp",
	we see that traffic is encoded with ESP and decoded properly,
	still good.

	Now install ports/net-mgmt/ipacctd, run it manually with
	"ipacctd -p 4000", then command:
	"ipfw add 10 divert 4000 ip from any to any out"
	and rerun tcpdump command shown above. You'll see that
	outgoing packets are encapsulated with ESP twice.
	The same you'll see at the other side for incoming packets.

>Fix:

	Unknown. The only known workaround is to avoid diverting ESP.
	This workaroung is not always acceptable.


Eugene Grosbein
>Release-Note:
>Audit-Trail:
Adding to audit trail from misfiled PR kern/103163:

Date: Mon, 11 Sep 2006 14:27:26 -0700
From: Julian Elischer <julian@elischer.org>

 Eugene Grosbein wrote:
 
 >	When outgoing packet encoded due to corresponding IPSEC policy
 >	is passed to divert socket (f.e. to ipacctd for accounting),
 >	it is encoded second time with IPSEC then. Besides obvious
 >	logic error, this also results in broken Path MTU Discovery.
 
 unfortunatly this comes from the fact that divert returns packets to the
 kernel by passing them to the IP stack and letting them be processed 
 again. 
 There is a flag that is set to allow the ipfw to know that they
 have been seen before, and it is possible that one could make IPSEC notice
 that flag as well but it would be pretty hacky.  One other solution 
 would be
 to make some way in which ipdivert can really inject a packet back at the
 point where it was extracted but that would probably require spliting 
 ip_output()
 into two functions, (as was done in ether_output() ) but that would 
 probably
 not 'fly' very well.

From: Eugene Grosbein <eugen@kuzbass.ru>
To: bug-followup@freebsd.org
Cc: julian@elischer.org
Subject: Re: kern/103135: ipsec with ipfw divert (not NAT) encodes a packet twice 
 breaking PMTUD
Date: Mon, 01 Jan 2007 15:52:26 +0700

 Hi!
 
 I've found that when DUMMYNET reinjects a packet to the stack
 to pass it over next ipfw rules, it is processed with IPSEC second time too.
 And it is encapsulated with ESP sencond time breaking PMTUD, again.
 
 I've found acceptable workaround: we need to say IPSEC code
 not to process already encapsulated packets:
 
 spdadd 1.1.1.1/32 2.2.2.2/32 esp -P out none;
 
 Sadly, setkey(8) parser has a bug preventing us from using this workaround.
 See http://www.freebsd.org/cgi/query-pr.cgi?pr=107392
 for details and trivial patch against setkey.
 
 Eugene
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: vwe 
Responsible-Changed-When: Wed Jan 14 22:24:42 UTC 2009 
Responsible-Changed-Why:  

Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=103135 
>Unformatted:
