From nobody@FreeBSD.org  Fri Jan  6 18:39:10 2006
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id AE04816A41F
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  6 Jan 2006 18:39:10 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 7223143D48
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  6 Jan 2006 18:39:10 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k06IdAJE068156
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 6 Jan 2006 18:39:10 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id k06Id98m068155;
	Fri, 6 Jan 2006 18:39:09 GMT
	(envelope-from nobody)
Message-Id: <200601061839.k06Id98m068155@www.freebsd.org>
Date: Fri, 6 Jan 2006 18:39:09 GMT
From: Nate Nielsen <nielsen@memberwebs.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Problem with PMTU Discovery / DF / IPSEC / GIF Tunnels
X-Send-Pr-Version: www-2.3

>Number:         91412
>Category:       kern
>Synopsis:       [netinet] [patch] Problem with PMTU Discovery / DF / IPSEC / GIF Tunnels
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    bz
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Jan 06 18:40:04 GMT 2006
>Closed-Date:    Sun Nov 19 10:12:23 GMT 2006
>Last-Modified:  Tue Aug 28 07:30:02 GMT 2007
>Originator:     Nate Nielsen
>Release:        FreeBSD 6.0
>Organization:
>Environment:
FreeBSD northstar-link.ws.local 6.0-RELEASE FreeBSD 6.0-RELEASE #34: Tue Dec 13 00:40:26 UTC 2005 nate@mesh-master.ws.local:/usr/src/sys/i386/compile/NETSOEKRIS  i386

>Description:
I encountered a strange problem with PMTU discovery not working properly
on various machines when the packets were tunneled over a GIF / IPSEC
Transport type tunnel (both ends running FreeBSD 6.0). Configuration
files attached.

Various older FreeBSD systems (it seemed systems that had jails running)
and also Windows Virtual Machines running in Microsoft's Virtual Server
2005 system, did not perform PMTU discovery properly.

The FreeBSD 6.0 routers were sending out ICMP host-unreachable
need-fragment packets without an MTU hint. Most machines handle this
fine, but the ones noted above did not decrease PMTU for the connection.

The attached patch makes sure that the FreeBSD 6.0 router will include
an MTU hint in the ICMP packet. The problem was caused by the IPSec
lookup in ip_forward() returning an secpolicy pointer, but then that
pointer having no details (such as request, etc...) contained in it. The
attached patch (against 6.0) covers that eventuality.

The 'bug' is obviously in the machines that don't handle the missing MTU
hint properly, but since we can't patch Windows, this patch helps
alleviate the problem from the other side.
>How-To-Repeat:
Will attach configuration files in a follow-up.
>Fix:
Will attach patch in a follow-up.
>Release-Note:
>Audit-Trail:

From: Nate Nielsen <nielsen@memberwebs.com>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/91412: Problem with PMTU Discovery / DF / IPSEC / GIF Tunnels
Date: Fri,  6 Jan 2006 20:13:52 +0000 (GMT)

 This is a multi-part message in MIME format.
 --------------070507060309070107090700
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: 7bit
 
 
 
 --------------070507060309070107090700
 Content-Type: application/x-gzip;
  name="pmtu-gif-ipsec-config.tar.gz"
 Content-Transfer-Encoding: base64
 Content-Disposition: inline;
  filename="pmtu-gif-ipsec-config.tar.gz"
 
 H4sICCX8u0MAA2ZyLjE0MTUyLjAucG10dS1naWYtaXBzZWMtY29uZmlnLnRhcgDtltuOmzAQ
 hrktT2H5HoPNaaUol7ndVupVpUqRAQNWwLDYdLVvXw67DVGrkN2o2R7miyLMMPkdxfNPpq1N
 7xQyd2SrReqkjcpl4XZNb0TncHeKkjFqvR1vIAqC8Urj0FteJ4KAWtTzwyiOwziOrOGpH0QW
 8q7Y82J6bXiHkKUFV+fy1p7/peRVr8uNrdvseWXzLEPUI4zcEfqyYEjoFrHh3JCzQ0nVPOZS
 l06apAjzJM1EXpSU+cFwhHd4s5RgR61nCXaRxPiNRhmPTC/XW6xkGyDnE5IKTeU5Crum40q3
 TWdctxMPvezE5gKJocxXNOz3PqDfTHve/93V5rfW/R/G/tH/wej/KAoZ+P8W2AU34pE/7YXi
 SSW2+MvuM7bLRhvF6+HW9EqJyqmkOjicGKENqZqUV9hOq0aJbC/VUCk5T4Xe4qGOPGzLfK6i
 /Xi7xVIJMzaBmFA6d5RpxdAQr7k+IBaG5OXtoa/2BzQzb/1zL1rsUKn0dIeQUJcFpxn0mDEJ
 TQn2QTzVvN3iXpPsW9PxA7a1LrPT30ELldVcVj+i9x/vd8OHH3peFPw0dzLQr0K5HAOuMOni
 7xT/IZ3lrP+Td/C/H87+p+D/W/Aa/yfX+58tOsHr/H8cJM75P1r3PwP/L1jx/63n/8hj/jT/
 ++D/mwDz//89/wMAAAAAAAAAAAAAAAAAAAAA8O/xHfPkVqoAKAAA
 --------------070507060309070107090700
 Content-Type: text/x-patch;
  name="pmtu-gif-ipsec.patch"
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline;
  filename="pmtu-gif-ipsec.patch"
 
 --- sys/netinet/ip_input.c.orig	Wed Jan  4 10:18:01 2006
 +++ sys/netinet/ip_input.c	Wed Jan  4 10:39:35 2006
 @@ -1918,29 +1918,29 @@ ip_forward(struct mbuf *m, int srcrt)
  					}
  				}
  
  #ifdef IPSEC
  				key_freesp(sp);
  #else /* FAST_IPSEC */
  				KEY_FREESP(&sp);
  #endif
 -				ipstat.ips_cantfrag++;
 -				break;
 -			} else 
 +			}
  #endif /*IPSEC || FAST_IPSEC*/
 -		/*
 -		 * When doing source routing 'ia' can be NULL.  Fall back
 -		 * to the minimum guaranteed routeable packet size and use
 -		 * the same hack as IPSEC to setup a dummyifp for icmp.
 -		 */
 -		if (ia == NULL)
 -			mtu = IP_MSS;
 -		else
 -			mtu = ia->ia_ifp->if_mtu;
 +		if (!mtu) {
 +			/*
 +			 * When doing source routing 'ia' can be NULL.  Fall back
 +			 * to the minimum guaranteed routeable packet size and use
 +			 * the same hack as IPSEC to setup a dummyifp for icmp.
 +			 */
 +			if (ia == NULL)
 +				mtu = IP_MSS;
 +			else
 +				mtu = ia->ia_ifp->if_mtu;
 +		}
  #if defined(IPSEC) || defined(FAST_IPSEC)
  		}
  #endif /*IPSEC || FAST_IPSEC*/
  		ipstat.ips_cantfrag++;
  		break;
  
  	case ENOBUFS:
  		/*
 
 --------------070507060309070107090700--
 
Responsible-Changed-From-To: freebsd-bugs->andre 
Responsible-Changed-By: andre 
Responsible-Changed-When: Tue Jan 24 17:42:36 UTC 2006 
Responsible-Changed-Why:  
take over. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=91412 
State-Changed-From-To: open->patched 
State-Changed-By: andre 
State-Changed-When: Tue Jan 24 17:57:46 UTC 2006 
State-Changed-Why:  
Fixed more throughout in rev. 1.312 of ip_input.c.  Nonetheless many thanks 
to the submitter for the good patch as starting point. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=91412 

From: Tom Judge <tom@tomjudge.com>
To: bug-followup@FreeBSD.org,  nielsen@memberwebs.com
Cc:  
Subject: Re: kern/91412: [netinet] [patch] Problem with PMTU Discovery / DF
 / IPSEC / GIF Tunnels
Date: Wed, 08 Nov 2006 17:25:20 +0000

 Hi,
 
 The patch that was applied to sys/netinet/ip_input.c (1.312) does not 
 fully correct the problem as there are some states that will case the 
 MTU to remain 0 if the ipsec related code fails to correctly set the 
 mtu.  In the patch submitted by Nate the ip statistics update and break 
 statement where removed from the ipsec code, in turn meaning that if the 
 mtu was failed to be set by ipsec then the mtu would be guaranteed to be 
 set by the following code block. 
 
 I have been testing problems with this for the last few days and found 
 that by applying the following section of the patch the problems with 
 ICMP Host Unreachable (DF Set, Need to Frag)'s MTU hint being set to 0 
 are fixed:
 
 --- sys/netinet/ip_input.c.orig  Wed Jan  4 10:18:01 2006
 +++ sys/netinet/ip_input.c       Wed Jan  4 10:39:35 2006
 @@ -1918,29 +1918,29 @@ ip_forward(struct mbuf *m, int srcrt)
                                          }
                                  }
  
  #ifdef IPSEC
                                  key_freesp(sp);
  #else /* FAST_IPSEC */
                                  KEY_FREESP(&sp);
  #endif
 -                                ipstat.ips_cantfrag++;
 -                                break;
 
 
 I will be putting this patch into production tomorrow if you feel 
 further testing is required.
 
 Tom J

From: Nate Nielsen <nielsen@memberwebs.com>
To: Tom Judge <tom@tomjudge.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/91412: [netinet] [patch] Problem with PMTU Discovery / DF
 / IPSEC / GIF Tunnels
Date: Thu,  9 Nov 2006 09:43:16 +0000 (GMT)

 Tom Judge wrote:
 > -                                ipstat.ips_cantfrag++;
 > -                                break;
 > 
 > 
 > I will be putting this patch into production tomorrow if you feel
 > further testing is required.
 
 It would seem that removing the 'break;' line would have unwanted side
 effects.
 
 Cheers,
 Nate
 
 
 

From: Tom Judge <tom@tomjudge.com>
To: Nate Nielsen <nielsen@memberwebs.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/91412: [netinet] [patch] Problem with PMTU Discovery / DF
 / IPSEC / GIF Tunnels
Date: Thu, 09 Nov 2006 11:42:14 +0000

 Nate Nielsen wrote:
 > Tom Judge wrote:
 >> -                                ipstat.ips_cantfrag++;
 >> -                                break;
 >>
 >>
 >> I will be putting this patch into production tomorrow if you feel
 >> further testing is required.
 > 
 > It would seem that removing the 'break;' line would have unwanted side
 > effects.
 > 
 > Cheers,
 > Nate
 > 
 > 
 > 
 
 After applying the attached patch I have found that the code that is 
 designed to compute the MTU from the ipsec security policy fails as 
 sp->req == NULL.  Which means that the mtu is always left at 0 in this case.
 
 The question that I have to ask is, is this code even required when the 
 non ipsec code can handle ipsec links (more reliably) as well?
 
 Cheers
 
 Tom J
 
 
 
 --- Test Packet Generation ---
 
 root@bob '11:25:40' '~'
  > $ ping -s 1280 -D mecca
 PING mecca (172.31.255.29): 1280 data bytes
 36 bytes from 10.0.0.102: frag needed and DF set (MTU 1280)
 Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
   4  5  00 1c05 a700   0 0000  40  01 d958 10.0.0.75  172.31.255.29
 
 36 bytes from 10.0.0.102: frag needed and DF set (MTU 1280)
 Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
   4  5  00 1c05 a704   0 0000  40  01 d954 10.0.0.75  172.31.255.29
 
 
 --- Log from router ---
 
 Nov  9 11:35:26 rodney kernel: forward: src 4b00000a dst 1dff1fac ttl 40
 Nov  9 11:35:26 rodney kernel: forward mtu mismatch (IPSEC): src 
 4b00000a dst 1dff1fac ttl 63
 Nov  9 11:35:26 rodney kernel: forward mtu mismatch (IPSEC): ipsechdr 0
 Nov  9 11:35:26 rodney kernel: forward mtu mismatch (IPSEC): sp->req == NULL
 Nov  9 11:35:26 rodney kernel: forward mtu mismatch: src 4b00000a dst 
 1dff1fac ttl 63
 Nov  9 11:35:26 rodney kernel: forward mtu mismatch: mtu - 0
 Nov  9 11:35:26 rodney kernel: forward mtu mismatch: ia->ia_ifp->if_mtu 
 - 1280
 
 
 
 
 
 Index: sys/netinet/ip_input.c
 ===================================================================
 --- sys/netinet/ip_input.c      (revision 24)
 +++ sys/netinet/ip_input.c      (working copy)
 @@ -86,6 +86,7 @@
   #include <netipsec/ipsec.h>
   #include <netipsec/key.h>
   #endif
 +#define DIAGNOSTIC
 
   int rsvp_on = 0;
 
 @@ -147,7 +148,7 @@
       &ip_checkinterface, 0, "Verify packet arrives on correct interface");
 
   #ifdef DIAGNOSTIC
 -static int     ipprintfs = 0;
 +static int     ipprintfs = 1;
   #endif
 
   struct pfil_head inet_pfil_hook;       /* Packet filter hooks */
 @@ -1948,7 +1949,7 @@
                  {
                          struct secpolicy *sp = NULL;
                          int ipsecerror;
 -                       int ipsechdr;
 +                       int ipsechdr =0;
                          struct route *ro;
 
   #ifdef IPSEC
 @@ -1962,6 +1963,57 @@
                                                     IP_FORWARDING,
                                                     &ipsecerror);
   #endif
 +#ifdef DIAGNOSTIC
 +            if (ipprintfs) {
 +                printf("forward mtu mismatch (IPSEC): src %lx dst %lx 
 ttl %u\n",
 +                    (u_long)ip->ip_src.s_addr, (u_long)ip->ip_dst.s_addr,
 +                    ip->ip_ttl);
 +                if (sp != NULL) {
 +                                   printf("forward mtu mismatch 
 (IPSEC): ipsechdr %u\n",ipsec4_hdrsiz(mcopy,
 +                                                        IPSEC_DIR_OUTBOUND,
 +                                                        NULL));
 +                    if (sp->req == NULL) {
 +                        printf("forward mtu mismatch (IPSEC): sp->req 
 == NULL\n");
 +                    } else if (sp->req->sav == NULL) {
 +                        printf("forward mtu mismatch (IPSEC): 
 sp->req->sav == NULL\n");
 +                    } else if (sp->req->sav->sah == NULL) {
 +                        printf("forward mtu mismatch (IPSEC): 
 sp->req->sav->sah == NULL\n");
 +                    } else {
 +                        ro = &sp->req->sav->sah->sa_route;
 +                        if (!ro->ro_rt) {
 +                            printf("forward mtu mismatch (IPSEC): 
 ro->ro_rt == false\n");
 +                        } else if (!ro->ro_rt->rt_ifp) {
 +                            printf("forward mtu mismatch (IPSEC): 
 ro->ro_rt->rt_ifp == false\n");
 +                        }
 +                        if (ro->ro_rt && ro->ro_rt->rt_ifp) {
 +                            printf("forward mtu mismatch (IPSEC): 
 ro->ro_rt->rt_rmx.rmx_mtu %lu\n",ro->ro_rt->rt_rmx.rmx_mtu);
 +                            printf("forward mtu mismatch (IPSEC): 
 ro->ro_rt->rt_ifp->if_mtu %lu\n",ro->ro_rt->rt_ifp->if_mtu);
 +
 +                            mtu =
 +                                ro->ro_rt->rt_rmx.rmx_mtu ?
 +                                ro->ro_rt->rt_rmx.rmx_mtu :
 +                                ro->ro_rt->rt_ifp->if_mtu;
 +
 +                            printf("forward mtu mismatch (IPSEC): mtu 
 before ipsec header - %u\n", mtu);
 +                            mtu -= ipsechdr;
 +                            printf("forward mtu mismatch (IPSEC): mtu 
 after ipsec header - %u\n", mtu);
 +
 +                        } else {
 +                            printf("forward mtu mismatch (IPSEC): 
 ro->ro_rt && ro->ro_rt->rt_if =\n");
 +                        }
 +                    }
 +#ifdef IPSEC
 +                               key_freesp(sp);
 +#else /* FAST_IPSEC */
 +                               KEY_FREESP(&sp);
 +#endif
 +
 +                } else {
 +                    printf("forward mtu mismatch (IPSEC): Security 
 Policy is NULL\n");
 +                }
 +            }
 +#else
 +
                          if (sp != NULL) {
                                  /* count IPsec header size */
                                  ipsechdr = ipsec4_hdrsiz(mcopy,
 @@ -1984,17 +2036,33 @@
                                                  mtu -= ipsechdr;
                                          }
                                  }
 -
   #ifdef IPSEC
                                  key_freesp(sp);
   #else /* FAST_IPSEC */
                                  KEY_FREESP(&sp);
   #endif
 -                               ipstat.ips_cantfrag++;
 -                               break;
                          }
 +#endif
                  }
   #endif /*IPSEC || FAST_IPSEC*/
 +#ifdef DIAGNOSTIC
 +        if (ipprintfs) {
 +            printf("forward mtu mismatch: src %lx dst %lx ttl %u\n",
 +                (u_long)ip->ip_src.s_addr, (u_long)ip->ip_dst.s_addr,
 +                ip->ip_ttl);
 +            printf("forward mtu mismatch: mtu - %u\n", mtu);
 +            if (ia == NULL) {
 +                printf("forward mtu mismatch: ia == NULL");
 +                printf("forward mtu mismatch: ip->ip_len - %u\n", 
 ip->ip_len);
 +                printf("forward mtu mismatch: ip_next_mtu - %u\n", 
 ip_next_mtu(ip->ip_len,0));
 +
 +            } else {
 +                printf("forward mtu mismatch: ia->ia_ifp->if_mtu - 
 %lu\n", ia->ia_ifp->if_mtu);
 +
 +            }
 +        }
 +#endif
 +
                  /*
                   * If the MTU wasn't set before use the interface mtu or
                   * fall back to the next smaller mtu step compared to the
 
 
Responsible-Changed-From-To: andre->bz 
Responsible-Changed-By: bz 
Responsible-Changed-When: Wed Nov 15 21:15:43 UTC 2006 
Responsible-Changed-Why:  
I'll handle this after 6.2-RELEASE. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=91412 

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, nielsen@memberwebs.com,
	Tom Judge <tom@tomjudge.com>
Cc:  
Subject: Re: kern/91412: [netinet] [patch] Problem with PMTU Discovery / DF
 / IPSEC / GIF Tunnels
Date: Thu, 16 Nov 2006 21:33:15 +0000 (UTC)

 Your patch seems to be correct. I am trying to get it in asap - but as
 said before I don't know if it can make it to 6.2-RELEASE.
 
 ===================================================================
 RCS file: /local/mirror/FreeBSD/r/ncvs/src/sys/netinet/ip_input.c,v
 retrieving revision 1.301.2.10
 diff -u -p -r1.301.2.10 ip_input.c
 --- ip_input.c  4 Sep 2006 10:39:47 -0000       1.301.2.10
 +++ ip_input.c  16 Nov 2006 21:24:51 -0000
 @@ -2000,8 +2000,6 @@ ip_forward(struct mbuf *m, int srcrt)
   #else /* FAST_IPSEC */
                                  KEY_FREESP(&sp);
   #endif
 -                               ipstat.ips_cantfrag++;
 -                               break;
                          }
                  }
   #endif /*IPSEC || FAST_IPSEC*/
 
 
 -- 
 Bjoern A. Zeeb				bzeeb at Zabbadoz dot NeT

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/91412: commit references a PR
Date: Sun, 19 Nov 2006 10:07:22 +0000 (UTC)

 bz          2006-11-19 10:07:08 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_6)
     sys/netinet          ip_input.c 
   Log:
   Fix PMTU discovery in IPsec case by using an MTU hint in ICMP unreachable
   fragmentation needed other then 0 when we cannot get a security policy.
   This changes the code path to match what we have had in HEAD since
   rev. 1.312.
   
   PR:             kern/91412
   Submitted by:   Tom Judge <tom tomjudge.com>
   
   Revision    Changes    Path
   1.301.2.11  +0 -2      src/sys/netinet/ip_input.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: bz 
State-Changed-When: Sun Nov 19 10:11:23 UTC 2006 
State-Changed-Why:  
A patch was committed to RELENG_6 too now. 
Thanks for reporting, thanks for the patches and testing. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=91412 

From: Sergey Svishchev <svs@ropnet.ru>
To: bug-followup@FreeBSD.org,  nielsen@memberwebs.com,  bz@freebsd.org
Cc:  
Subject: Re: kern/91412
Date: Tue, 28 Aug 2007 11:21:22 +0400

 Any chance of MFC to RELENG_5?
 
 -- 
 Sergey Svishchev
>Unformatted:
