From nobody@FreeBSD.org  Tue Oct 13 21:34:32 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 652F31065692
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 13 Oct 2009 21:34:32 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 540CF8FC19
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 13 Oct 2009 21:34:32 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n9DLYWam041206
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 13 Oct 2009 21:34:32 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n9DLYVsl041205;
	Tue, 13 Oct 2009 21:34:31 GMT
	(envelope-from nobody)
Message-Id: <200910132134.n9DLYVsl041205@www.freebsd.org>
Date: Tue, 13 Oct 2009 21:34:31 GMT
From: alexus <freebsd@alexus.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: ipfw pipe
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         139581
>Category:       kern
>Synopsis:       [ipfw] "ipfw pipe" not limiting bandwidth
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-ipfw
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Oct 13 21:40:02 UTC 2009
>Closed-Date:    Mon Jun 06 07:15:10 UTC 2011
>Last-Modified:  Mon Jun 06 07:15:10 UTC 2011
>Originator:     alexus
>Release:        7.2
>Organization:
alexusbiz corp.
>Environment:
FreeBSD dd.alexus.org 7.2-RELEASE-p1 FreeBSD 7.2-RELEASE-p1 #7: Sat Jun 27 02:42:30 UTC 2009     alexus@dd.alexus.org:/usr/obj/usr/src/sys/GENERIC  amd64

>Description:
su-3.2# cat /etc/ipfw.rules 
flush
pipe flush
pipe 1 config bw 2Mbit/s
add 100 allow ip from any to any via lo0
add 200 deny ip from any to 127.0.0.0/8
add 300 deny ip from 127.0.0.0/8 to any
add 8380 pipe 1 tcp from any to any src-port www uid daemon
add 8380 pipe 1 tcp from any to any dst-port www uid daemon
add 65000 pass all from any to any
su-3.2# ipfw show
00100 1249368  205115325 allow ip from any to any via lo0
00200       0          0 deny ip from any to 127.0.0.0/8
00300       0          0 deny ip from 127.0.0.0/8 to any
08380 2838075 3586421013 pipe 1 tcp from any 80 to any uid daemon
08380 2097473  136454502 pipe 1 tcp from any to any dst-port 80 uid daemon
65000 5740679 4716157064 allow ip from any to any
65535       0          0 deny ip from any to any
su-3.2# ipfw pipe show
00001:   2.000 Mbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp     64.237.55.83/59388    208.80.152.3/80    4936077 3723134341  0    0 30179
su-3.2# ps auxwww | grep ^daemon
daemon  81736  0.7  0.3 77768 26460  ??  SJ    9:28PM   0:00.60 /usr/local/apache2/bin/httpd -k start
daemon  81244  0.0  0.3 76744 23860  ??  SJ    9:27PM   0:00.23 /usr/local/apache2/bin/httpd -k start
daemon  81253  0.0  0.3 75720 23628  ??  SJ    9:27PM   0:00.34 /usr/local/apache2/bin/httpd -k start
daemon  81624  0.0  0.3 76744 25184  ??  SJ    9:27PM   0:00.52 /usr/local/apache2/bin/httpd -k start
daemon  81625  0.0  0.3 75720 23640  ??  SJ    9:27PM   0:00.15 /usr/local/apache2/bin/httpd -k start
daemon  81678  0.0  0.3 75720 23672  ??  SJ    9:28PM   0:00.24 /usr/local/apache2/bin/httpd -k start
daemon  81929  0.0  0.3 75720 23564  ??  SJ    9:29PM   0:00.25 /usr/local/apache2/bin/httpd -k start
daemon  81930  0.0  0.3 75720 23484  ??  SJ    9:29PM   0:00.13 /usr/local/apache2/bin/httpd -k start
daemon  81931  0.0  0.3 75720 23616  ??  SJ    9:29PM   0:00.14 /usr/local/apache2/bin/httpd -k start
daemon  81938  0.0  0.3 76744 23912  ??  SJ    9:29PM   0:00.14 /usr/local/apache2/bin/httpd -k start
daemon  82710  0.0  0.3 75720 23468  ??  SJ    9:30PM   0:00.07 /usr/local/apache2/bin/httpd -k start
daemon  82747  0.0  0.3 75720 23492  ??  SJ    9:30PM   0:00.04 /usr/local/apache2/bin/httpd -k start
daemon  82748  0.0  0.3 75720 23604  ??  SJ    9:30PM   0:00.04 /usr/local/apache2/bin/httpd -k start
daemon  82749  0.0  0.3 76744 23808  ??  SJ    9:30PM   0:00.06 /usr/local/apache2/bin/httpd -k start
daemon  82758  0.0  0.3 75720 23448  ??  SJ    9:31PM   0:00.02 /usr/local/apache2/bin/httpd -k start
daemon  82759  0.0  0.3 75720 23460  ??  SJ    9:31PM   0:00.02 /usr/local/apache2/bin/httpd -k start
su-3.2# 

I'm trying to limit my apache that runs under daemon to up 2Mbit/s

when I do "ipfw pipe show" I don't see anything in my slots other then very first entry that never chage, nor does it limits my traffic, as if I look at my MRTG i see way more traffic then 2Mbit/s
>How-To-Repeat:
su-3.2# cat /etc/ipfw.rules 
flush
pipe flush
pipe 1 config bw 2Mbit/s
add 100 allow ip from any to any via lo0
add 200 deny ip from any to 127.0.0.0/8
add 300 deny ip from 127.0.0.0/8 to any
add 8380 pipe 1 tcp from any to any src-port www uid daemon
add 8380 pipe 1 tcp from any to any dst-port www uid daemon
add 65000 pass all from any to any
su-3.2# /etc/rc.d/ipfw restart
/etc/rc.d/ipfw: DEBUG: checkyesno: firewall_enable is set to YES.
/etc/rc.d/ipfw: DEBUG: checkyesno: firewall_enable is set to YES.
/etc/rc.d/ipfw: DEBUG: run_rc_command: doit: ipfw_stop 
net.inet.ip.fw.enable: 1 -> 0
/etc/rc.d/natd: DEBUG: checkyesno: natd_enable is set to NO.
/etc/rc.d/ipfw: DEBUG: checkyesno: firewall_enable is set to YES.
/etc/rc.d/ipfw: DEBUG: run_rc_command: start_precmd: ipfw_prestart 
/etc/rc.d/ipfw: DEBUG: checkyesno: dummynet_enable is set to NO.
/etc/rc.d/ipfw: DEBUG: checkyesno: firewall_nat_enable is set to NO.
/etc/rc.d/ipfw: DEBUG: load_kld: ipfw kernel module already loaded.
/etc/rc.d/ipfw: DEBUG: run_rc_command: doit: ipfw_start 
/etc/rc.d/natd: DEBUG: checkyesno: natd_enable is set to NO.
Firewall rules loaded.
/etc/rc.d/ipfw: DEBUG: checkyesno: firewall_logging is set to YES.
Firewall logging enabled.
net.inet.ip.fw.enable: 0 -> 1
su-3.2# 

>Fix:
beats me! i post question on freebsd mailing list, freebsd forums asked same question on other websites no one seems to know...

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-ipfw 
Responsible-Changed-By: gavin 
Responsible-Changed-When: Wed Oct 14 20:17:06 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=139581 

From: Ian Smith <smithi@nimnet.asn.au>
To: bug-followup@FreeBSD.org, freebsd@alexus.org
Cc:  
Subject: Re: kern/139581: [ipfw] "ipfw pipe" not limiting bandwidth
Date: Tue, 20 Oct 2009 01:24:17 +1100

 May be a usage issue; I'll have a go.  Partial quoting, out of order.
 
 : I'm trying to limit my apache that runs under daemon to up 2Mbit/s
 : when I do "ipfw pipe show" I don't see anything in my slots other then
 : very first entry that never chage, nor does it limits my traffic, as
 : if I look at my MRTG i see way more traffic then 2Mbit/s
 
 Unless you specify masks on your pipes you'll only ever see the first
 connection that used that pipe, that's normal.
 
 MRTG sees all traffic on an interface, and your ipfw stats indicate at
 least 25% more traffic than that due to your webserver, so it's not
 clear how you could tell if your pipe was exceeding 2Mbit/s or not?
 
 Also, it's recommended not to run your inbound and outbound traffic
 through the one pipe, unless simulating half-duplex connections; see
 explanation in ipfw(8), EXAMPLES section under TRAFFIC SHAPING.
 
 : su-3.2# ipfw show
 : 00100 1249368 205115325 allow ip from any to any via lo0
 : 00200 0 0 deny ip from any to 127.0.0.0/8
 : 00300 0 0 deny ip from 127.0.0.0/8 to any
 : 08380 2838075 3586421013 pipe 1 tcp from any 80 to any uid daemon
 : 08380 2097473 136454502 pipe 1 tcp from any to any dst-port 80 uid daemon
 : 65000 5740679 4716157064 allow ip from any to any
 : 65535 0 0 deny ip from any to any
 
 3.586 GiB outbound from the webserver (served data)
 0.136 GiB inbound to the webserver (requests, acks)
 + ---
 3.722 GiB through the pipe.
 but
 4.716 GiB passed from any to any, either way.
 
 So there's about 1 Gig of extra traffic shown here, assuming you have
 net.inet.ip.fw.one_pass=0 and all traffic eventually hits rule 65000
 (and 4.7G extra traffic if net.inet.ip.fw.one_pass=1) but there's not
 enough info to see whether or not it's on the interface MRTG watches?
 
 : su-3.2# ipfw pipe show
 : 00001: 2.000 Mbit/s 0 ms 50 sl. 1 queues (1 buckets) droptail
 : mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 : BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
 Pkt/Byte Drp
 : 0 tcp 64.237.55.83/59388 208.80.152.3/80 4936077 3723134341 0 0 30179
 
 Total packets and bytes match the above, indicating that this was done
 just after the ipfw show.  0.6% dropped packets indicates some limiting
 happening, but with a shared in/outbound pipe, not in which direction.
 
 If this is still an issue, please:
 
 . be more precise than "way more traffic" if you have more data?
 . say whether the extra ~25% traffic shown is on the same interface
    as the webserver, ie the interface MRTG monitors, or not?
 . the value of sysctl net.inet.ip.fw.one_pass ?
 
 cheers, Ian
 

From: alexus <alexus@alexus.org>
To: Ian Smith <smithi@nimnet.asn.au>
Cc: bug-followup@FreeBSD.org,
 freebsd@alexus.org
Subject: Re: kern/139581: [ipfw] "ipfw pipe" not limiting bandwidth
Date: Mon, 19 Oct 2009 11:58:41 -0400

 On Oct 19, 2009, at 10:24 AM, Ian Smith wrote:
 
 > May be a usage issue; I'll have a go.  Partial quoting, out of order.
 >
 > : I'm trying to limit my apache that runs under daemon to up 2Mbit/s
 > : when I do "ipfw pipe show" I don't see anything in my slots other  
 > then
 > : very first entry that never chage, nor does it limits my traffic, as
 > : if I look at my MRTG i see way more traffic then 2Mbit/s
 >
 > Unless you specify masks on your pipes you'll only ever see the first
 > connection that used that pipe, that's normal.
 
 ok
 
 new set of rules
 
 su-3.2# cat /etc/ipfw.rules
 flush
 pipe flush
 pipe 1 config bw 1Mbit/s mask src-port www
 pipe 2 config bw 1Mbit/s mask src-port www
 add 100 allow ip from any to any via lo0
 add 200 deny ip from any to 127.0.0.0/8
 add 300 deny ip from 127.0.0.0/8 to any
 add 8381 pipe 1 tcp from any to any dst-port www uid daemon
 add 8382 pipe 2 tcp from any to any src-port www uid daemon
 add 65000 pass all from any to any
 su-3.2# ipfw show
 00100 1476  230632 allow ip from any to any via lo0
 00200    0       0 deny ip from any to 127.0.0.0/8
 00300    0       0 deny ip from 127.0.0.0/8 to any
 08381  482   36368 pipe 1 tcp from any to any dst-port 80 uid daemon
 08382  620  743113 pipe 2 tcp from any 80 to any uid daemon
 65000 6832 5040856 allow ip from any to any
 65535    0       0 deny ip from any to any
 su-3.2# ipfw pipe show
 00001:   1.000 Mbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
      mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/ 
 Byte Drp
    0 tcp     64.237.55.83/49492   66.230.133.69/80     509    38156   
 0    0   0
 00002:   1.000 Mbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
      mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/ 
 Byte Drp
    0 tcp    66.230.133.69/80       64.237.55.83/49492  656   785292  1  
 1500   1
 su-3.2# ipfw pipe show
 00001:   1.000 Mbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
      mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/ 
 Byte Drp
    0 tcp     64.237.55.83/49492   66.230.133.69/80    1247    98023   
 0    0   0
 00002:   1.000 Mbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
      mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/ 
 Byte Drp
    0 tcp    66.230.133.69/80       64.237.55.83/49492 1475  1453606   
 0    0   1
 su-3.2#
 
 in this case i did specify mask for pipe, yet when I'm issuing ipfw  
 pipe show I still don't see anything in terms of slots that being in use
 
 su-3.2# sysctl net.inet.ip.dummynet.pipe_slot_limit
 net.inet.ip.dummynet.pipe_slot_limit: 100
 su-3.2#
 
 seems like at all time I see only 1 slot being utilized and as I  
 mention before it never changes.
 
 
 >
 > MRTG sees all traffic on an interface, and your ipfw stats indicate at
 > least 25% more traffic than that due to your webserver, so it's not
 > clear how you could tell if your pipe was exceeding 2Mbit/s or not?
 >
 
 I obviously do have other traffic then www, but majority of it is www.
 but I see why you coming with this, so let me just give you an example.
 if I at peak time shutdown my apache, my traffic drops dramatically  
 and by dramatically i mean at least 90% (and in most cases more)
 my traffic went to as much as 10mbps with supposedly limited pipe of  
 2mbps, when I set it to 1mbps it seems to be almost there...
 
 > Also, it's recommended not to run your inbound and outbound traffic
 > through the one pipe, unless simulating half-duplex connections; see
 > explanation in ipfw(8), EXAMPLES section under TRAFFIC SHAPING.
 
 i thought about that and as you suggested i did separate them into 2  
 separate pipes (see on top)
 
 >
 > : su-3.2# ipfw show
 > : 00100 1249368 205115325 allow ip from any to any via lo0
 > : 00200 0 0 deny ip from any to 127.0.0.0/8
 > : 00300 0 0 deny ip from 127.0.0.0/8 to any
 > : 08380 2838075 3586421013 pipe 1 tcp from any 80 to any uid daemon
 > : 08380 2097473 136454502 pipe 1 tcp from any to any dst-port 80 uid  
 > daemon
 > : 65000 5740679 4716157064 allow ip from any to any
 > : 65535 0 0 deny ip from any to any
 >
 > 3.586 GiB outbound from the webserver (served data)
 > 0.136 GiB inbound to the webserver (requests, acks)
 > + ---
 > 3.722 GiB through the pipe.
 > but
 > 4.716 GiB passed from any to any, either way.
 >
 > So there's about 1 Gig of extra traffic shown here, assuming you have
 > net.inet.ip.fw.one_pass=0 and all traffic eventually hits rule 65000
 > (and 4.7G extra traffic if net.inet.ip.fw.one_pass=1) but there's not
 > enough info to see whether or not it's on the interface MRTG watches?
 >
 > : su-3.2# ipfw pipe show
 > : 00001: 2.000 Mbit/s 0 ms 50 sl. 1 queues (1 buckets) droptail
 > : mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 > : BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
 > Pkt/Byte Drp
 > : 0 tcp 64.237.55.83/59388 208.80.152.3/80 4936077 3723134341 0 0  
 > 30179
 >
 > Total packets and bytes match the above, indicating that this was done
 > just after the ipfw show.  0.6% dropped packets indicates some  
 > limiting
 > happening, but with a shared in/outbound pipe, not in which direction.
 >
 > If this is still an issue, please:
 >
 > . be more precise than "way more traffic" if you have more data?
 > . say whether the extra ~25% traffic shown is on the same interface
 >  as the webserver, ie the interface MRTG monitors, or not?
 > . the value of sysctl net.inet.ip.fw.one_pass ?
 >
 > cheers, Ian
 >
 

From: Ian Smith <smithi@nimnet.asn.au>
To: alexus <alexus@alexus.org>
Cc: bug-followup@FreeBSD.org, freebsd@alexus.org
Subject: Re: kern/139581: [ipfw] "ipfw pipe" not limiting bandwidth
Date: Thu, 22 Oct 2009 23:17:23 +1100 (EST)

 On Mon, 19 Oct 2009, alexus wrote:
 
  > new set of rules
 
  > pipe 1 config bw 1Mbit/s mask src-port www
  > pipe 2 config bw 1Mbit/s mask src-port www
 
 Wrong mask syntax entirely.  You can see from your pipe masks as shown, 
 it's taken as meaning no mask at all:
 
  > mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
 
 Anyway, masking pipes creates dynamic pipes per masked flow, each of 
 which gets ALL of the specified bandwidth.  If you want to limit total 
 bandwidth to 1Mbit/s, you likely want to use dynamic queues instead.
 
 ipfw(8) is a precise reference, but very terse.  Suggested reading:
 
  http://info.iet.unipi.it/~luigi/dummynet/
 
 and especially the last link from that page:
 
  http://info.iet.unipi.it/~luigi/ip_dummynet/original.html
 
 for clear examples of sharing evenly a single link - though noting 
 that page is outdated re the sysctls for dummynet, bridging etc.
 
 Still looking more like a usage issue than describing a bug, but: 
 
  > > If this is still an issue, please:
 
  > > . say whether the extra ~25% traffic shown is on the same interface
  > >   as the webserver, ie the interface MRTG monitors, or not?
  > > . the value of sysctl net.inet.ip.fw.one_pass ?
 
 cheers, Ian
State-Changed-From-To: open->closed 
State-Changed-By: ae 
State-Changed-When: Mon Jun 6 07:12:25 UTC 2011 
State-Changed-Why:  
Seems like a problem in the incorrect usage. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=139581 
>Unformatted:
