From nobody@FreeBSD.org  Wed Jun 30 12:10:14 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 21F51106566C
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 30 Jun 2010 12:10:14 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 107F48FC14
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 30 Jun 2010 12:10:14 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o5UCADwR042398
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 30 Jun 2010 12:10:13 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o5UCADUN042375;
	Wed, 30 Jun 2010 12:10:13 GMT
	(envelope-from nobody)
Message-Id: <201006301210.o5UCADUN042375@www.freebsd.org>
Date: Wed, 30 Jun 2010 12:10:13 GMT
From: Alexey Guskov <adg@a-real.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: pf rdr incompatible with dummynet
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         148260
>Category:       kern
>Synopsis:       [pf] [patch] pf rdr incompatible with dummynet
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-pf
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jun 30 12:20:07 UTC 2010
>Closed-Date:    
>Last-Modified:  Fri Mar 25 22:30:11 UTC 2011
>Originator:     Alexey Guskov
>Release:        7.2-RELEASE
>Organization:
Areal Company
>Environment:
FreeBSD test.local 7.2-RELEASE FreeBSD 7.2-RELEASE #1: Tue Jun 29 09:04:57 UTC 2010     root@tamaez.office1.a-real.ru:/usr/src/sys/i386/compile/GATE  i386
>Description:
The problem is the same as described here: http://lists.freebsd.org/pipermail/freebsd-net/2006-July/011076.html

I'm using two firewalls: ipfw for dummynet traffic shaping and pf for rules, forwards etc. When a dummynet pipe is used and rdr rule exists pf (i.e. for transparent proxy) tcp connections doesn't get established.

For example: we have a pipe for traffic going to user, and rdr rule for trasparent proxy. Ipfw processes incoming packets before pf. Here's how traffic goes thru kernel:

SYN packet:
[network interface] -> ... -> pfil_run_hooks() -> ipfw_check_in() -> pf_check_in() /the packet is being redirected and pf state entry is created/ -> ... -> [squid]

The packet is successfully delivered.

ACK packet:
[squid] -> ... -> ip_output() -> ... -> pfil_run_hooks() -> pf_check_out() /the packet is being unredirected using state table entry/ -> ipfw_check_out() -> ip_dn_io_ptr()

NB: ipfw stops processing the packet after it bin injected into dummynet. Heres what happens next:

ip_dn_ip_ptr() -> /some pipe vodoo magic/ -> dummynet_send() -> ip_output()

TADA! After coming out from pipe the packet is being re-injected into main packet-processing routine! Let's see what's gonna happen:

ip_output() -> ... -> pfil_run_hooks() -> pf_check_out() /pf cannot find entry in state table because the packet has already been unredirected and violently kills it/ -> OH SHI--

So, tcp connection never gets established.
>How-To-Repeat:
pf.conf
--
rdr on le1 proto tcp from { 192.168.1.0/24 } to any port { 80 } -> (le1) port 3128 
--

ipfw.conf
--
ipfw pipe 1 config bw 16KByte/s
ipfw 100 add pipe 1 all from any to 192.168.1.101
--

make ipfw process incoming packets before pf 
--- 
# ipfw disable firewall
# ipfw enable firewall 
--
>Fix:
When ipfw send a packet to dummynet it tags it with PACKET_TAG_DUMMYNET and saves rule number that triggered this action. When the packet comes to ipfw for the second time, ipfw can see that it already has been checked and continues processing from the next rule.

The problem is pf doesn't know that the packet already has been processed by him. So we'll teach him:

--- sys/contrib/pf/net/pf_ioctl.c	2010-06-14 06:09:06.000000000 +0400
+++ nsys/contrib/pf/net/pf_ioctl.c	2010-06-30 14:20:55.000000000 +0400
@@ -3636,6 +3637,7 @@
 	 */
 	struct ip *h = NULL;
 	int chk;
+  struct m_tag *dn_tag;
 
 	if ((*m)->m_pkthdr.len >= (int)sizeof(struct ip)) {
 		/* if m_pkthdr.len is less than ip header, pf will handle. */
@@ -3643,7 +3645,13 @@
 	        HTONS(h->ip_len);
 	        HTONS(h->ip_off);
 	}
-	chk = pf_test(PF_IN, ifp, m, NULL, inp);
+	
+	dn_tag = m_tag_find(*m, PACKET_TAG_DUMMYNET, NULL);
+	if (dn_tag == NULL)
+    chk = pf_test(PF_IN, ifp, m, NULL, inp);
+	else
+    chk = PF_PASS;
+
 	if (chk && *m) {
 		m_freem(*m);
 		*m = NULL;
@@ -3671,6 +3679,7 @@
 	 */
 	struct ip *h = NULL;
 	int chk;
+	struct m_tag *dn_tag;
 
 	/* We need a proper CSUM befor we start (s. OpenBSD ip_output) */
 	if ((*m)->m_pkthdr.csum_flags & CSUM_DELAY_DATA) {
@@ -3683,7 +3692,13 @@
 	        HTONS(h->ip_len);
 	        HTONS(h->ip_off);
 	}
-	chk = pf_test(PF_OUT, ifp, m, NULL, inp);
+	
+	dn_tag = m_tag_find(*m, PACKET_TAG_DUMMYNET, NULL);
+	if (dn_tag == NULL)
+	  chk = pf_test(PF_OUT, ifp, m, NULL, inp);
+	else
+    chk = PF_PASS;
+	
 	if (chk && *m) {
 		m_freem(*m);
 		*m = NULL;


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-pf 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sat Jul 3 11:17:25 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=148260 

From: =?UTF-8?Q?Marcin_Wi=C5=9Bnicki?= <mwisnicki@gmail.com>
To: bug-followup <bug-followup@FreeBSD.org>, adg <adg@a-real.ru>
Cc:  
Subject: Re: kern/148260: [pf] [patch] pf rdr incompatible with dummynet
Date: Fri, 25 Mar 2011 22:52:22 +0100

 How about a more generic solution:
 
 Add new mbuf tag PACKET_TAG_PFIL_RESUME_FROM that contains address of
 a function registered with pfil_add_hook (ipfw_check_hook in this
 case) and modify pfil_run_hooks() to skip all hooks until that one (if
 such tag is present).
 
 Before reinjecting packet into ip_output by dummynet, prepend this
 m_tag to mbuf (also strip that tag if it ever comes back?).
 
 I don't know if mbuf api allows it but such tag could theoretically
 have just one instance (created on dummynet module load) to avoid
 allocation costs.
 
 This way you don't have to put ugly workaround in every pfil consumer.
>Unformatted:
