From dave@zinc.org.uk  Wed Oct 27 15:23:18 2004
Return-Path: <dave@zinc.org.uk>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id A38A816A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 27 Oct 2004 15:23:18 +0000 (GMT)
Received: from zinc.org.uk (zinc.org.uk [193.201.200.225])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 602F643D55
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 27 Oct 2004 15:23:18 +0000 (GMT)
	(envelope-from dave@zinc.org.uk)
Received: from dave by zinc.org.uk with local (Exim 4.43 (FreeBSD))
	id 1CMpdq-0002b4-BF
	for FreeBSD-gnats-submit@freebsd.org; Wed, 27 Oct 2004 16:23:14 +0100
Message-Id: <E1CMpdq-0002b4-BF@zinc.org.uk>
Date: Wed, 27 Oct 2004 16:23:14 +0100
From: David Haworth <dave@zinc.org.uk>
Reply-To: David Haworth <dave@zinc.org.uk>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: IPF causing major tcp problems with 3rd party apps (apache, exim etc)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         73202
>Category:       kern
>Synopsis:       IPF causing major tcp problems with 3rd party apps (apache, exim etc)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    darrenr
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 27 15:30:22 GMT 2004
>Closed-Date:    Mon Dec 20 08:09:19 GMT 2004
>Last-Modified:  Mon Dec 20 08:09:19 GMT 2004
>Originator:     David Haworth
>Release:        FreeBSD 5.3-RELEASE i386
>Organization:
>Environment:
System: FreeBSD zinc.org.uk 5.3-RELEASE FreeBSD 5.3-RELEASE #2: Mon Oct 25 23:17:28 BST 2004 root@zinc.org.uk:/usr/obj/usr/src/sys/ZINC i386


	

>Description:

Server was running 5.1-stable, I cvsupped to 5.3-Release, configured a
custom kernel (generic plus all three firewalls, some power management
devices, ipsec), rebuilt world (make buildworld; make buildkernel;
make installkernel; reboot; mergemaster -p; make installworld;
mergemaster; reboot) and sshd back into the machine. 

I then started checking the server to make sure everything was alright
and I immediately found major problems, apache and courier weren't
working properly, exim seemed to be working intermittantly. example
with apache: I could see the processes running, no errors logged, and
it was logging requests, but was sending back no data. exim was
spawnign new processes for inbound mail, but the processes were
stalled, and doing nothing. yet, sshd worked fine (thankfully), as did
bind9 from ports (compiled previous to the upgrade).

having read UPDATING, I remembered the comment about the upgrade to
gcc 3.4.2 and that some apps may have to be recompiled. this made
sense in light of sshd, so I recompiled my major apps, and then the
apps they depend on, but to no avail.

I eventually determined it to be a network issue. using tethereal, I
could see a http connection come in, the three way handshake would be
completed (syn, synack, ack) and then the server would simply stop
responding. the client would keep retrying until it gave up. the
server process was obviously getting the request (ie the apache
logging and the exim process spawning) but could not reply for some
reason.

I recompiled a new kernel, with only ipf of the firewalls in and
installed and rebooted. no effect. I then compiled a kernel with none
of the firewalls in and rebooted without a firewall. the machine
worked fine and all process transmitted data to their clients.

>How-To-Repeat:
>Fix:

I used a workaround, rather than a fix. I had wanted to transition to pf anyway and this forced my hand. I loaded pf up as a kernel module and configured it to match the older ipf config. the machine reacts as one would expect and I am experiencing no further problems. as this box is a colocated server in production, I can't go back and keep trying new kernels and options to see if there is a problem, but this seemed like a possible show-stopper for others so I thought it worth flagging.

dave




>Release-Note:
>Audit-Trail:

From: Kris Kennaway <kris@obsecurity.org>
To: David Haworth <dave@zinc.org.uk>
Cc: FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: kern/73202: IPF causing major tcp problems with 3rd party apps (apache, exim etc)
Date: Wed, 27 Oct 2004 10:38:06 -0700

 On Wed, Oct 27, 2004 at 04:23:14PM +0100, David Haworth wrote:
 
 > I eventually determined it to be a network issue. using tethereal, I
 > could see a http connection come in, the three way handshake would
 > be completed (syn, synack, ack) and then the server would simply
 > stop responding. the client would keep retrying until it gave
 > up. the server process was obviously getting the request (ie the
 > apache logging and the exim process spawning) but could not reply
 > for some reason.
 
 First guess would be that your ipf ruleset was wrong.  Can you please
 include it for verification?
 
 Kris

From: David Haworth <dave@fyonn.net>
To: Kris Kennaway <kris@obsecurity.org>
Cc: FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: kern/73202: IPF causing major tcp problems with 3rd party apps (apache, exim etc)
Date: Wed, 27 Oct 2004 23:18:14 +0100

 --Apple-Mail-18-349210026
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=US-ASCII;
 	format=flowed
 
 > First guess would be that your ipf ruleset was wrong.  Can you please
 > include it for verification?
 
 you're quite right, I should have pointed out that the firewall ruleset 
 was completely unchanged from the 5.1 config. I don't really want to 
 post my firewall config to a public forum so I'll enclose a suitably 
 edited version.
 
 this config worked fine with 5.1 and caused no problems.
 
 dave
 
 
 
 # deny by default
 block in log on vr0
 
 pass in quick on lo0
 pass out quick on lo0
 
 # get rid of unwanted and unexpected networks
 block in quick on vr0 from 192.168.0.0/16 to any
 block in quick on vr0 from 172.16.0.0/12 to any
 block in quick on vr0 from 10.0.0.0/8 to any
 block in quick on vr0 from 127.0.0.0/8 to any
 block in quick on vr0 from 0.0.0.0/8 to any
 block in quick on vr0 from 169.254.0.0/16 to any
 block in quick on vr0 from 192.0.2.0/24 to any
 block in quick on vr0 from 204.152.64.0/23 to any
 block in quick on vr0 from 224.0.0.0/3 to any
 
 #Rule to block nmap fingerprinting attempts
 block in quick on vr0 proto tcp all flags FUP
 
 #block all packets with ip options.
 block in log quick all with ipopts
 
 #block all fragmented and short packets
 block in quick all with frag
 block in quick all with short
 
 # block silently netbios/msds/mssql traffic from the local lan
 block in quick on vr0 proto tcp from any to any port = 135
   <more like this>
 
 # allow mail/web traffic
 pass in quick on vr0 proto tcp from any to $local_ip1 port = smtp
 pass in quick on vr0 proto tcp from any to $local_ip1 port = http
 pass in quick on vr0 proto tcp from any to $local_ip2 port = http
   <more like this>
 
 # allow pings and traceroutes
 pass in quick proto icmp from any to $local_ip1 icmp-type 8      # echo 
 request
 pass in quick proto udp  from any to $local_ip1 port 33434 >< 33690 
 keep state
 
 #allow anyone to ssh in
 pass in quick on vr0 proto tcp from any to any port = 22 flags S keep 
 state
 
 # stateful allowing of internal traffic and replies
 pass out quick on vr0 proto tcp/udp from any to any keep state keep 
 frags
 pass out quick on vr0 proto icmp    from any to any keep state
 
 
 --Apple-Mail-18-349210026
 Content-Transfer-Encoding: base64
 Content-Type: application/pkcs7-signature;
 	name=smime.p7s
 Content-Disposition: attachment;
 	filename=smime.p7s
 
 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGEDCCAskw
 ggIyoAMCAQICAwuGHzANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhh
 d3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVt
 YWlsIElzc3VpbmcgQ0EwHhcNMDQwMTIxMTgxMzI1WhcNMDUwMTIwMTgxMzI1WjBAMR8wHQYDVQQD
 ExZUaGF3dGUgRnJlZW1haWwgTWVtYmVyMR0wGwYJKoZIhvcNAQkBFg5kYXZlQGZ5b25uLm5ldDCC
 ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALV37S70FvBLzigYBYNcLSI6mKRp2MH7+k5h
 28Tk78FDRrIgTD0gvABODQ7Iqc/eaAuN3iZ6MplgVdCnL1tIolNE+xeRAop8yT224RgBSwBxAwrT
 yDruf3TG0OrLs9hLvGHqkBgUVf7jiKP646Gy86AoaATLpD2D43dbUf/uJxiFJEhNauxgEJbL5UHu
 Im0vE5t7IejnKlpeVV6lppMcI8ZF2OsFb7TuCXfN05eef7xqIOmNG8YfNX5Sja+xLnvYFZqhy/HG
 tL1XbZqj530GBK9bbNL/bQ5Panw7h6eUKK92cXcM/z01jXgb+jtqLdKWu2H0iiOlyhEgJ8q6Fp9Y
 8pUCAwEAAaMrMCkwGQYDVR0RBBIwEIEOZGF2ZUBmeW9ubi5uZXQwDAYDVR0TAQH/BAIwADANBgkq
 hkiG9w0BAQQFAAOBgQATQm5+ArByLY6kAHmYYPHYTHPay7bAlAJaRfGYZLh+zefKqMkD9IyceJjh
 SnVqdDgtx4g+h/exeskdgudr9rtcH4dzvE6PLQ3rEE0uTcNtl4ou7Ax+0FHk6R6Zl/Yg0sf78yfe
 7Z76OjoD3hmvhaRyTlPin65LRd9picnphhuOqzCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEF
 BQAwgdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUg
 VG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24g
 U2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTEr
 MCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0wMzA3MTcwMDAw
 MDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3Vs
 dGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWlu
 ZyBDQTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV+065yplaHmjAdQRwnd/p/6Me
 7L3N9VvyGna9fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfArhVqqP3FWy688Cwfn8R+RNiQqE88r
 1fOCdz0Dviv+uxg+B79AgAJk16emu59l0cUqVIUPSAR/p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCB
 kTASBgNVHRMBAf8ECDAGAQH/AgEAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3Rl
 LmNvbS9UaGF3dGVQZXJzb25hbEZyZWVtYWlsQ0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAg
 pB4wHDEaMBgGA1UEAxMRUHJpdmF0ZUxhYmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPq
 Cy7MDaNmrGcPf6+svsIXoUOWlJ1/TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUa
 C4J0niVQlGLH2ydxVyWN3amcOY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx
 0x1G/11fZU8xggLnMIIC4wIBATBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29u
 c3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNz
 dWluZyBDQQIDC4YfMAkGBSsOAwIaBQCgggFTMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJ
 KoZIhvcNAQkFMQ8XDTA0MTAyNzIyMTgxNFowIwYJKoZIhvcNAQkEMRYEFInkUARDPLU+ub1uoa4k
 BS+/HlyMMHgGCSsGAQQBgjcQBDFrMGkwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBD
 b25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJ
 c3N1aW5nIENBAgMLhh8wegYLKoZIhvcNAQkQAgsxa6BpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQK
 ExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwg
 RnJlZW1haWwgSXNzdWluZyBDQQIDC4YfMA0GCSqGSIb3DQEBAQUABIIBAGD3cE6hhUU9aq12oVQ4
 DGfUS1AgR6u5AKQqXVCEO1IDEn4vlczzvWye0oQGDdFHUNradirJzZvk2UzcQZaN2Zy4iyzrFRNm
 Z6/BID7/ccmSq+KeZ3oEeMjHLDq+USQEq0kAG15FFHkVO3hiBLDUXywfGmO6lbUfd89LjlpQnd36
 XRBUolhucVVWhH9fU7kWiBL1b9kiuOwh4+FfHCXFt6w5+OXoGExgesCuZNRD1dQj9CloUPL9reeY
 7g3tAF/zVCno1vhCOypvjbvnbM/iYtv1QVKPT9vSDPcrws7rrGSanqkZzkctiunzO36PH1c4kBVE
 4uFg4Yln4COx3Q5Qc1YAAAAAAAA=
 
 --Apple-Mail-18-349210026--
 

From: Giorgos Keramidas <keramida@freebsd.org>
To: David Haworth <dave@fyonn.net>
Cc: bug-followup@freebsd.org
Subject: Re: kern/73202: IPF causing major tcp problems with 3rd party apps (apache, exim etc)
Date: Thu, 28 Oct 2004 06:34:57 +0300

 On 2004-10-27 22:20, David Haworth <dave@fyonn.net> wrote:
 > You're quite right, I should have pointed out that the firewall ruleset was
 > completely unchanged from the 5.1 config. I don't really want to post my
 > firewall config to a public forum so I'll enclose a suitably edited version.
 >
 >  this config worked fine with 5.1 and caused no problems.
 
 I think you have problems because of the unmatched `in' rules for some
 services that you make visible from outside.  I call these rules `unmatched'
 because there is no matching `out' rule to let the replies get out too:
 
 > block in log on vr0
 > block in log quick all with ipopts
 > block in quick all with frag
 > block in quick all with short
 > block in quick on vr0 proto tcp from any to any port = 135
 > [...]
 > pass in quick on vr0 proto tcp from any to $local_ip1 port = smtp
 > pass in quick on vr0 proto tcp from any to $local_ip1 port = http
 > pass in quick on vr0 proto tcp from any to $local_ip2 port = http
 > [...]
 
 This means that incoming packets for these ports are unconditionally allowed
 to pass through.  Nothing is said about outgoing packets, so the default
 policy is assumed.  You haven't set the default `out' policy for interface vr0
 in your ruleset so this can be either `pass' (the default) or `block' (if you
 have compiled your kernel with IPFILTER_DEFAULT_BLOCK).
 
 A few rules further down you have:
 
 > pass in quick on vr0 proto tcp from any to any port = 22 flags S keep state
 > pass out quick on vr0 proto tcp/udp from any to any keep state keep frags
 
 Your problems are very probably caused by this mixing of stateless and
 stateful rules.  Combined with the fact that you don't cover *all* possible
 cases of packets, this can be tricky.
 
 Try converting the stateless rules to stateful, i.e. replace this:
 
 > pass in quick on vr0 proto tcp from any to $local_ip1 port = smtp
 > pass in quick on vr0 proto tcp from any to $local_ip1 port = http
 > pass in quick on vr0 proto tcp from any to $local_ip2 port = http
 
 with something like this:
 
 > pass in quick on vr0 proto tcp from any to $local_ip1 port = smtp keep state
 > pass in quick on vr0 proto tcp from any to $local_ip1 port = http keep state
 > pass in quick on vr0 proto tcp from any to $local_ip2 port = http keep state
 
 Let us know if that fixes the problems you're seeing.
 
 Regards,
 Giorgos.
 

From: "David Haworth" <dave@fyonn.net>
To: "Giorgos Keramidas" <keramida@freebsd.org>
Cc: bug-followup@freebsd.org
Subject: Re: kern/73202: IPF causing major tcp problems with 3rd party apps 
     (apache, exim etc)
Date: Fri, 29 Oct 2004 14:17:42 +0100 (BST)

 > I think you have problems because of the unmatched `in' rules for some
 > services that you make visible from outside.  I call these rules
 > `unmatched' because there is no matching `out' rule to let the replies
 > get out too:
 
 well, there is an allow all out rule at the bottom, but my thought was
 that it worked absolutely fine when I was running 5.1, if ipf has become
 more strict about it's syntax then fair enough. to be honest, I thought it
 unlikely that such a showstopper could exist this close to release so if
 it's just me writing some slightly off colour rules then fair enough, we
 can close the bug. I just wanted to flag it if it wasn't.
 
 > Let us know if that fixes the problems you're seeing.
 
 well, I've transitioned the ruleset to pf now which is working fine and
 it's a production box in colo, so I can't keep swapping kernels in and
 out. I am happy to accept that you're above suggestion is correct.
 
 dave
 
 
 
Responsible-Changed-From-To: freebsd-bugs->darrenr 
Responsible-Changed-By: keramida 
Responsible-Changed-When: Fri Oct 29 13:32:23 GMT 2004 
Responsible-Changed-Why:  
Darren is our IPF guru. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=73202 

From: Giorgos Keramidas <keramida@freebsd.org>
To: David Haworth <dave@fyonn.net>
Cc: bug-followup@freebsd.org
Subject: Re: kern/73202: IPF causing major tcp problems with 3rd party apps
Date: Fri, 29 Oct 2004 16:39:43 +0300

 On 2004-10-29 14:17, David Haworth <dave@fyonn.net> wrote:
 > > I think you have problems because of the unmatched `in' rules for some
 > > services that you make visible from outside.  I call these rules
 > > `unmatched' because there is no matching `out' rule to let the replies
 > > get out too:
 >
 > well, there is an allow all out rule at the bottom, but my thought was
 > that it worked absolutely fine when I was running 5.1, if ipf has become
 > more strict about it's syntax then fair enough.
 
 Well, you have a ""keep state" rule, but AFAIK these work well when
 combined with "flags S" for TCP.  Which might be what's bitting you.
 
 I remember having to tweak my rulesets when one of the ipf upgrades came
 into the 5.X tree.  I am not an IPF source code expert though, so I've
 assigned this PR to Darren Reed, who is the author of IPF.
 
 I think it's the rules, but he will certainly know better.
 
 - Giorgos
 
State-Changed-From-To: open->closed 
State-Changed-By: darrenr 
State-Changed-When: Mon Dec 20 08:09:04 GMT 2004 
State-Changed-Why:  
user solved the problem for themselves 

http://www.freebsd.org/cgi/query-pr.cgi?pr=73202 
>Unformatted:
