From paul@megaptera.semiocast.net  Fri Jun  7 08:19:42 2013
Return-Path: <paul@megaptera.semiocast.net>
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id 5E22BBB7
	for <FreeBSD-gnats-submit@freebsd.org>; Fri,  7 Jun 2013 08:19:42 +0000 (UTC)
	(envelope-from paul@megaptera.semiocast.net)
Received: from megaptera.semiocast.net (87-231-1-249.rev.numericable.fr [87.231.1.249])
	by mx1.freebsd.org (Postfix) with ESMTP id 28F3C1F9D
	for <FreeBSD-gnats-submit@freebsd.org>; Fri,  7 Jun 2013 08:19:41 +0000 (UTC)
Received: from megaptera.semiocast.net (localhost [127.0.0.1])
	by megaptera.semiocast.net (Postfix) with ESMTP id C67B7964
	for <FreeBSD-gnats-submit@freebsd.org>; Fri,  7 Jun 2013 08:10:01 +0000 (UTC)
Received: (from paul@localhost)
	by megaptera.semiocast.net (8.14.5/8.14.5/Submit) id r578A16J072183;
	Fri, 7 Jun 2013 08:10:01 GMT
	(envelope-from paul)
Message-Id: <201306070810.r578A16J072183@megaptera.semiocast.net>
Date: Fri, 7 Jun 2013 08:10:01 GMT
From: Paul Guyot <paul@semiocast.com>
Reply-To: Paul Guyot <paul@semiocast.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: [pf] [ip6] Incorrect TCP checksums in rdr return packets
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         179392
>Category:       kern
>Synopsis:       [pf] [ip6] Incorrect TCP checksums in rdr return packets
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-pf
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Jun 07 08:20:00 UTC 2013
>Closed-Date:    
>Last-Modified:  Fri May 23 03:30:01 UTC 2014
>Originator:     Paul Guyot
>Release:        FreeBSD 9.1-RELEASE amd64
>Organization:
Semiocast
>Environment:
System: FreeBSD megaptera.semiocast.net 9.1-RELEASE FreeBSD 9.1-RELEASE #1 r250855M: Tue May 21 15:14:41 UTC 2013 root@megaptera.semiocast.net:/usr/obj/usr/src/sys/GENERIC amd64

Also happens on:
FreeBSD fangsiao.semiocast.net 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #0: Sat Nov 17 23:54:19 UTC 2012     root@fangsiao.semiocast.net:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
PF incorrectly computes the checksum of IPv6 TCP packets that it rewrites as part of a simple rdr rule that just changes the port number. Maybe it just does not update the checksum...

The following IPv4 rule works properly:
rdr pass inet proto tcp from any to 188.165.36.95 port 443 -> 188.165.36.95 port 8443

The following IPv6 rule does not work:
rdr pass inet6 proto tcp from any to 2001:41d0:8:8822::1 port 443 -> 2001:41d0:8:8822::1 port 8443

Ports redirection happens properly, but on their way out, packets do not have the proper checksum and are eventually dropped by the client. Since this happens during TCP connection handshake, the only rewritten packets on their way out are SYN, ACK.

This happens with and without scrub rules.

The problem, or a similar problem, is discussed on mailing lists. Yet no PR seems to have been filed, probably because it was insufficiently documented.

In both threads, it is linked with jails as jails are one of the use-case of rdr/nat with IPv6 :
http://lists.freebsd.org/pipermail/freebsd-stable/2012-July/068987.html
http://lists.freebsd.org/pipermail/freebsd-pf/2012-August/006710.html

and
http://lists.freebsd.org/pipermail/freebsd-pf/2012-December/006916.html

Also, this bug might or might not be related to kern/172648 or kern/171733

However, unlike kern/172648, this bug has no known workaround. Indeed, the problem here is to be able to rewrite incoming packets on a privileged port and forward them to a local server, then get the response from the local server and forward it to the client after rewriting the source port.

>How-To-Repeat:
Create a simple rdr rule on IPv6 and let everything else pass.

rdr pass inet6 proto tcp from any to [IPv6 External IP] port 443 -> [IPv6 External IP] port 8443

Start a server on this port, for example with nc.
Try to connect from outside on port 8443. It should work.
Restart nc. Try to connect from outside port 443. The client will eventually timeout.

Eventually, use tcpdump to record the packets and notice the checksums are incorrect on received SYN, ACK packets from port 443.

>Fix:

	


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-pf 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Thu Jun 27 18:03:00 UTC 2013 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=179392 

From: Martin Sugioarto <martin@sugioarto.com>
To: bug-followup@FreeBSD.org, paul@semiocast.com
Cc:  
Subject: Re: kern/179392: [pf] [ip6] Incorrect TCP checksums in rdr return
 packets
Date: Wed, 15 Jan 2014 22:36:53 +0100

 Hello FreeBSD team, hello Paul,
 
 I would like to confirm this. The original PR says it all. I have also
 reproduced it on FreeBSD 9.2R.
 
 FreeBSD 9.2-RELEASE-p2 #3 r258725 on amd64
 
 The checksum is not updated, tcpdump says it clearly. The packet is
 being quietly dropped and never arrives at the service listening socket.
 
 Here the comparison for two rules for an intercepting HTTP proxy:
 
 # works (IPv4)
 rdr on $if_int inet proto tcp \
 	from any to !$net_int port www -> 127.0.0.1 port 8118 
 
 # incorrect checksum (IPv6)
 rdr on $if_int inet6 proto tcp \
 	from any to !$net_int port www -> ::1 port 8118
 
 
 In my opinion, this is quite important. It costed me a day to find out
 what is going on and I have come to the same conclusion as Paul
 (independently).
 
 
 Yours
 Martin Sugioarto

From: David Thiel <lx@freebsd.org>
To: bug-followup@FreeBSD.org, paul@semiocast.com
Cc:  
Subject: Re: kern/179392: [pf] [ip6] Incorrect TCP checksums in rdr return
 packets
Date: Mon, 10 Feb 2014 17:23:49 -0800

 I've replicated this issue as well, on 10.0-RELEASE, amd64. With jails
 running on a cloned lo1, outbound IPv6 works fine, but pf redirect
 traffic gets results in incorrect checksums and traffic being dropped.
 Loopback interfaces no longer seem to support the -txcsum6 or -rxcsum6
 flags. Would love to have a fix for this, as it kind of breaks the
 "service jail" model for IPv6.
 
 Cheers,
 David

From: Ari Suutari <ari@stonepile.fi>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/179392: [pf] [ip6] Incorrect TCP checksums in rdr return
 packets
Date: Fri, 16 May 2014 09:12:52 +0300

 I was hit by this problem also, on 9.2-RELEASE.
 I think it might be caused by the workaround
 introduced in kern/170070.
 
     Ari S.
 

From: J David <j.david.lists@gmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/179392: [pf] [ip6] Incorrect TCP checksums in rdr return packets
Date: Thu, 22 May 2014 23:22:35 -0400

 We also have encountered this issue.
 
 PF not working properly with IPv6 seems like a very serious problem.
 
 What needs to happen in order to advance this issue?  How can we help?
 
 Thanks!
>Unformatted:
