From sa264@cam.ac.uk  Sun Feb 10 11:34:13 2002
Return-Path: <sa264@cam.ac.uk>
Received: from lilac.csi.cam.ac.uk (lilac.csi.cam.ac.uk [131.111.8.44])
	by hub.freebsd.org (Postfix) with ESMTP id E66DD37B404
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 10 Feb 2002 11:34:12 -0800 (PST)
Received: from m218-3.phy.cam.ac.uk ([131.111.79.217])
	by lilac.csi.cam.ac.uk with esmtp (Exim 3.34 #1)
	id 16Zzjo-0000Gb-00
	for FreeBSD-gnats-submit@freebsd.org; Sun, 10 Feb 2002 19:34:12 +0000
Received: from sa264 by m218-3.phy.cam.ac.uk with local (Exim 3.34 #1)
	id 16Zzjo-0006YJ-00
	for FreeBSD-gnats-submit@freebsd.org; Sun, 10 Feb 2002 19:34:12 +0000
Message-Id: <E16Zzjo-0006YJ-00@m218-3.phy.cam.ac.uk>
Date: Sun, 10 Feb 2002 19:34:12 +0000
From: AMAKAWA Shuhei <sa264@cam.ac.uk>
Sender: AMAKAWA Shuhei <sa264@cam.ac.uk>
Reply-To: AMAKAWA Shuhei <sa264@cam.ac.uk>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: TCP window size bug (afflicting IP Filter keep state)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         34801
>Category:       kern
>Synopsis:       TCP window size bug (afflicting IP Filter keep state)
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    darrenr
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Feb 10 11:40:01 PST 2002
>Closed-Date:    Fri Apr 23 19:15:17 PDT 2004
>Last-Modified:  Fri Apr 23 19:15:17 PDT 2004
>Originator:     AMAKAWA Shuhei
>Release:        FreeBSD 4.5-STABLE i386
>Organization:
>Environment:
System: FreeBSD m218-3.phy.cam.ac.uk 4.5-STABLE FreeBSD 4.5-STABLE #4: Fri Feb 1 20:30:18 GMT 2002 sa264@m218-3.phy.cam.ac.uk:/usr/obj/usr/src/sys/KERNCONF i386


	
>Description:

The default value of net.inet.tcp.recvspace was raised to 65536 a
few months ago.  So, the max TCP window size is 65536 now.  But this
amounts to 0 in the 16-bit world.  The valid max window size is
65535 (forget about RFC 1323 here).

Apparently a number of IP Filter users have been afflicted by this.
IP Filter's stateful inspection code uses the window size
information to decide whether a packet can be regarded as valid or
not.  IP Filter's state entry treats the window size as 16-bit.

>How-To-Repeat:

tcpdump.

Use IP Filter, keep state flags S.
Then transfer a big file.
Packet loss will lead to the following valid packets being blocked
and the connection will be lost.

>Fix:

Workaround:

sysctl net.inet.tcp.recvspace=65535
or less.

Fix:

--- /usr/src/sys/netinet/tcp_usrreq.c	Sun Dec 16 18:16:51 2001
+++ tcp_usrreq.c	Sun Feb 10 18:43:44 2002
@@ -995,7 +995,7 @@
 u_long	tcp_sendspace = 1024*32;
 SYSCTL_INT(_net_inet_tcp, TCPCTL_SENDSPACE, sendspace, CTLFLAG_RW, 
     &tcp_sendspace , 0, "Maximum outgoing TCP datagram size");
-u_long	tcp_recvspace = 1024*64;
+u_long	tcp_recvspace = 1024*64-1;
 SYSCTL_INT(_net_inet_tcp, TCPCTL_RECVSPACE, recvspace, CTLFLAG_RW, 
     &tcp_recvspace , 0, "Maximum incoming TCP datagram size");
 
>Release-Note:
>Audit-Trail:

From: AMAKAWA Shuhei <sa264@cam.ac.uk>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Tue, 12 Feb 2002 01:19:00 +0000

 kern/34801 is obviously incorrect.  In fact, RFC 1323 comes into
 play.  The default value of net.inet.tcp.recvspace of 65536 requires
 that the RFC 1323 window scale extention be used.
 
 However, IP Filter apparently doesn't support RFC 1323 extensions
 and takes the window size field literally.  You could blame IP
 Filter for this, but I think lowering the default tcp.recvspace to
 65535 is a reasonable concession to make, to make IP Filter work
 without tweaking anything.  The original aim of raising it to 64K
 must not have been to activate window scaling.  So, please apply the
 patch.

From: Andrew Gallatin <gallatin@cs.duke.edu>
To: freebsd-gnats-submit@FreeBSD.org, sa264@cam.ac.uk
Cc:  
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter
Date: Tue, 12 Feb 2002 13:36:51 -0500 (EST)

 keep state)
 X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid
 
 
 I'd greatly prefer we keep the default tcp_recvspace a multiple of
 MCLBYTES, or better yet, a multiple of PAGE_SIZE.
 
 So if its decided that reducing tcp_recvspace is the expediant fix
 (rather than fixing ipfiler, which would be much better), set it to
 56k or 32k or something.  Don't set it to 64k-1.
 
 Drew
 
 
 

From: AMAKAWA Shuhei <sa264@cam.ac.uk>
To: Andrew Gallatin <gallatin@cs.duke.edu>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter
Date: Tue, 12 Feb 2002 21:01:37 +0000

 At Tue, 12 Feb 2002 13:36:51 -0500 (EST),
 Andrew Gallatin wrote:
 > I'd greatly prefer we keep the default tcp_recvspace a multiple of
 > MCLBYTES, or better yet, a multiple of PAGE_SIZE.
 > 
 > So if its decided that reducing tcp_recvspace is the expediant fix
 > (rather than fixing ipfiler, which would be much better), set it to
 > 56k or 32k or something.  Don't set it to 64k-1.
 
 Sounds fine to me.  As a matter of fact, I use 32768.
 The question is what the folks who decided to make this change have
 to say.
 I don't know why the particular value of 64k was chosen.
 
 I'm interested to hear the reason behind Matt Dillon's reservation
 about going beyond 32k, noted in the CVS log (if he had such
 reservation at all).
 
 http://www.FreeBSD.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_usrreq.c
 revision 1.64
 
 --
 Shuhei

From: Sheldon Hearn <sheldonh@starjuice.net>
To: AMAKAWA Shuhei <sa264@cam.ac.uk>
Cc: obrien@FreeBSD.org, bug-followup@freebsd.org
Subject: Re: kern/34801
Date: Tue, 19 Mar 2002 11:36:00 +0200

 On Tue, 19 Mar 2002 09:04:32 GMT, AMAKAWA Shuhei wrote:
 
 > Incidentally, are you interested in looking at
 > 
 > http://www.FreeBSD.org/cgi/query-pr.cgi?pr=kern=34801
 
 Seems odd to me that nobody seems to have called for comment from David
 O'Brien who made the change that bumped net.inet.tcp.recvspace to 2^16.
 
 David, lots of IPFilter users are complaining that this value breaks
 IPFilter, because that software treats the window size as a 16-bit
 value, which can't represent anything more than 65535.
 
 However, we don't really want to drop the value to 65535, because we
 really want it to be a multiple of PAGE_SIZE.
 
 So various folks have suggested dropping it back down to 32768.
 
 In your commit log for the change, you said that dillon had agreed in
 principle, but only "on the 32k level".  If dillon had reservations
 about "the 64k level", can you remember why you ignored them?
 
 Alternatively, are you happy with us just going ahead and dropping the
 value for the sake of software that never dreamed the default would
 become so large?
 
 Seems pretty sane to me, and it's still an improvement over 16k.
 
 Ciao,
 Sheldon.

From: "David O'Brien" <obrien@freebsd.org>
To: Sheldon Hearn <sheldonh@starjuice.net>,
	Andrew Gallatin <gallatin@cs.duke.edu>
Cc: AMAKAWA Shuhei <sa264@cam.ac.uk>, bug-followup@freebsd.org
Subject: Re: kern/34801
Date: Tue, 19 Mar 2002 09:19:02 -0800

 On Tue, Mar 19, 2002 at 11:36:00AM +0200, Sheldon Hearn wrote:
 > In your commit log for the change, you said that dillon had agreed in
 > principle, but only "on the 32k level".  If dillon had reservations
 > about "the 64k level", can you remember why you ignored them?
 
 There was a long thread in ?-current? (?-hackers?) about this.  It was
 felt that 64k should be OK.  IIRC Dillon's reservations might have been
 because that was the highest he runs at.
 
 > Alternatively, are you happy with us just going ahead and dropping the
 > value for the sake of software that never dreamed the default would
 > become so large?
 > 
 > Seems pretty sane to me, and it's still an improvement over 16k.
 
 It sounds like we can get away with any large value less than 64k.  So
 lets try keeping it as high as possible.
 
 Andrew, just so I don't mess this up, is the 56k value you would like
 "57344".
 
 -- 
 -- David  (obrien@FreeBSD.org)
Responsible-Changed-From-To: freebsd-bugs->darrenr 
Responsible-Changed-By: silby 
Responsible-Changed-When: Sun May 19 18:42:07 PDT 2002 
Responsible-Changed-Why:  
Darren, I think I read somewhere that ipfilter has been 
updated to handle RFC 1323, and hence this issue 
is fixed.  If such a newer version of ipfilter has been 
imported into -stable, could you close the PR? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=34801 

From: Stas Kisel <stas_k_freebsd@tiger.unisquad.com>
To: freebsd-gnats-submit@FreeBSD.org, sa264@cam.ac.uk
Cc:  
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Tue, 28 May 2002 11:11:53 +0300

 Hi.
 
 Just in case you'll decide to use 65535 or think that issue can be fixed
 in ipfilter.
 
 I'm not using ipfilter, but have trouble communicating 4.5-RELEASE with
 other OS (4.2-RELEASE and linux-2.2.21rc3), looks like due to default
 window size of 65535.
 
 1. tcpdump shows window size of 65535, not 65536. While sysctl shows 65536.
 2. Trouble occurs when sending large file to 4.5-RELEASE machine
 over lossy link (2-3%).
 3. Trouble does not occur if recv buffer size is set to 32k. I did not try
 other values.
 4. ipfilter is not compiled into kernel, I use ipfw instead.
 5. tcpdump log:
 
 09:19:01.523854 45.45.45.45.22 > 42.42.42.42.1037: . ack 589048 win 65535 (DF) [tos 0x8]
 09:19:02.513711 42.42.42.42.1037 > 45.45.45.45.22: . 589048:590508(1460) ack 3205 win 17520 (DF) [tos 0x8]
 09:19:02.516098 45.45.45.45.22 > 42.42.42.42.1037: . ack 593428 win 61320 (DF) [tos 0x8]
 09:19:02.516183 42.42.42.42.1037 > 45.45.45.45.22: P 593428:594788(1360) ack 3205 win 17520 (DF) [tos 0x8]
 09:19:04.513727 42.42.42.42.1037 > 45.45.45.45.22: P 593428:594788(1360) ack 3205 win 17520 (DF) [tos 0x8]
 09:19:04.611854 45.45.45.45.22 > 42.42.42.42.1037: . ack 594788 win 65535 (DF) [tos 0x8]
 09:20:05.472890 45.45.45.45.22 > 42.42.42.42.1037: R 3253:3253(0) ack 594788 win 65535 (DF) [tos 0x8]
 
 I've tried several (~5) times and always see:
 1. PUSH data
 2. PUSH retransmit data
 3. ACK
 4. 4.2 or linux machine does not send any data after ACK for some reason
 5. delay and RST
 
 -- 
 Stas Kysel, UNIX System Administrator of Yessoftware Inc
 http://www.codecharge.com/
 

From: AMAKAWA Shuhei <sa264@cam.ac.uk>
To: Stas Kisel <stas_k_freebsd@tiger.unisquad.com>
Cc: freebsd-gnats-submit@FreeBSD.org, obrien@freebsd.org,
	gallatin@cs.duke.edu, sheldonh@freebsd.org, dillon@freebsd.org
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Tue, 28 May 2002 18:02:34 +0100

 Hi.
 
 At Tue, 28 May 2002 11:11:53 +0300,
 Stas Kisel wrote:
 > 
 > Just in case you'll decide to use 65535 or think that issue can be fixed
 > in ipfilter.
 >
 > I'm not using ipfilter, but have trouble communicating 4.5-RELEASE with
 > other OS (4.2-RELEASE and linux-2.2.21rc3), looks like due to default
 > window size of 65535.
 
 Please carefully distinguish 65535 and 65536!
 The default is 65536, which activates the window size scaling.
 This may be the cause of your trouble.
 I suggested to lower the default value (to 65535 or 32768 or
 something).
 
 Try
 sysctl net.inet.tcp.recvspace=65535
 This will most clearly show you if window scaling is doing you harm.
 
 
 I think it's wiser not to activate the window size scaling by using
 65536 (off by one).  The benefit is minimal (unless you are trying
 to find bugs in TCP implementations).
 
 Can anybody lower it to a more sensible value?
 I will be happy with anything >= 32768.
 I wanted to see this resolved before 4.6-RELEASE.
 
 --
 Shuhei

From: Sheldon Hearn <sheldonh@starjuice.net>
To: AMAKAWA Shuhei <sa264@cam.ac.uk>
Cc: Stas Kisel <stas_k_freebsd@tiger.unisquad.com>,
	freebsd-gnats-submit@FreeBSD.org, obrien@freebsd.org,
	gallatin@cs.duke.edu, dillon@freebsd.org
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state) 
Date: Tue, 28 May 2002 19:23:44 +0200

 On Tue, 28 May 2002 18:02:34 +0100, AMAKAWA Shuhei wrote:
 
 > Can anybody lower it to a more sensible value?
 > I will be happy with anything >= 32768.
 > I wanted to see this resolved before 4.6-RELEASE.
 
 The number I've seen suggested by quite a few seemingly sensible folks
 was 57344.  It should be a multiple of the page size and it should be
 expressable as an unsigned, 16-bit value.
 
 Ciao,
 Sheldon.

From: Matthew Dillon <dillon@apollo.backplane.com>
To: AMAKAWA Shuhei <sa264@cam.ac.uk>
Cc: re@freebsd.org, Stas Kisel <stas_k_freebsd@tiger.unisquad.com>,
	freebsd-gnats-submit@freebsd.org, obrien@freebsd.org,
	gallatin@cs.duke.edu, sheldonh@freebsd.org,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Tue, 28 May 2002 10:25:57 -0700 (PDT)

     The receive window change from 16K -> 64K in -stable was MFC'd by Alfred
     in November 2001, so I am adding him to the list.  I am also adding the
     release engineers so they can make a final determination on what to do
     about the default for the 4.6 release.
 
     In anycase, a value of 65536 will only activate window scaling if
     the RFC 1323 is turned on (net.inet.tcp.rfc1323).  However, rfc1323
     *IS* turned on by default.
 
     It would probably be safer for the release to set the maximum receive
     window to 65535 rather then 65536.  I'm on the fence, though, because
     what we really ought to be doing is tracking down the interoperability
     problem between Linux and FreeBSD instead of doing an end-run around
     the whole thing by disabling window scaling.  Both sides must negotiate
     the option so either the linux implementation is broken, or ours is.
 
     I believe FreeBSD 4.2 had bugs related to window scaling and that would
     not be expected to work but it should be a simple matter to turn off
     rfc1323 in 4.2 to solve the problem (sysctl -w net.inet.tcp.rfc1323=0).
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>
 
 
 :Hi.
 :
 :At Tue, 28 May 2002 11:11:53 +0300,
 :Stas Kisel wrote:
 :> 
 :> Just in case you'll decide to use 65535 or think that issue can be fixed
 :> in ipfilter.
 :>
 :> I'm not using ipfilter, but have trouble communicating 4.5-RELEASE with
 :> other OS (4.2-RELEASE and linux-2.2.21rc3), looks like due to default
 :> window size of 65535.
 :
 :Please carefully distinguish 65535 and 65536!
 :The default is 65536, which activates the window size scaling.
 :This may be the cause of your trouble.
 :I suggested to lower the default value (to 65535 or 32768 or
 :something).
 :
 :Try
 :sysctl net.inet.tcp.recvspace=65535
 :This will most clearly show you if window scaling is doing you harm.
 :
 :
 :I think it's wiser not to activate the window size scaling by using
 :65536 (off by one).  The benefit is minimal (unless you are trying
 :to find bugs in TCP implementations).
 :
 :Can anybody lower it to a more sensible value?
 :I will be happy with anything >= 32768.
 :I wanted to see this resolved before 4.6-RELEASE.
 :
 :--
 :Shuhei

From: Matthew Dillon <dillon@apollo.backplane.com>
To: Sheldon Hearn <sheldonh@starjuice.net>
Cc: AMAKAWA Shuhei <sa264@cam.ac.uk>,
	Stas Kisel <stas_k_freebsd@tiger.unisquad.com>,
	freebsd-gnats-submit@FreeBSD.ORG, obrien@FreeBSD.ORG,
	gallatin@cs.duke.edu, re@FreeBSD.ORG,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state) 
Date: Tue, 28 May 2002 10:50:59 -0700 (PDT)

 :On Tue, 28 May 2002 18:02:34 +0100, AMAKAWA Shuhei wrote:
 :
 :> Can anybody lower it to a more sensible value?
 :> I will be happy with anything >= 32768.
 :> I wanted to see this resolved before 4.6-RELEASE.
 :
 :The number I've seen suggested by quite a few seemingly sensible folks
 :was 57344.  It should be a multiple of the page size and it should be
 :expressable as an unsigned, 16-bit value.
 :
 :Ciao,
 :Sheldon.
 
     Seems reasonable to me, though I don't think there is any particular
     page-size-alignment issue.  TCP handles silly-window-syndrome and 
     neither packets nor MTUs are page-aligned, typically.  I would be
     happy with anything over 32768 as a default.
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>

From: Stas Kisel <stas_k_freebsd@tiger.unisquad.com>
To: AMAKAWA Shuhei <sa264@cam.ac.uk>
Cc: freebsd-gnats-submit@FreeBSD.org, obrien@FreeBSD.org,
	gallatin@cs.duke.edu, sheldonh@FreeBSD.org, dillon@FreeBSD.org
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Tue, 28 May 2002 20:54:49 +0300

 On Tue, May 28, 2002 at 06:02:34PM +0100, AMAKAWA Shuhei wrote:
 > Stas Kisel wrote:
 > > I'm not using ipfilter, but have trouble communicating 4.5-RELEASE with
 > > other OS (4.2-RELEASE and linux-2.2.21rc3), looks like due to default
 > > window size of 65535.
 > 
 > Please carefully distinguish 65535 and 65536!
 
 > The default is 65536, which activates the window size scaling.
 
 I'm not sure that window scaling had place, because there were no special tcp
 options in handshake, and window only was 65535 or a bit less, never
 more than 65535 and never less that 32k.
 
 09:04:32.098188 42.42.42.42.1037 > 45.45.45.45.22: S 4290239031:4290239031(0) win 16384 <mss 1460> (DF)
 09:04:32.099060 45.45.45.45.22 > 42.42.42.42.1037: S 2313172213:2313172213(0) ack 4290239032 win 65535 <mss 1460>
 
 Probably window scaling did not take place because 4.2 side started
 connection, so 4.5 just used minimum 16bit value (I did not check source,
 it is just guess). My 4.2 machine does not support window scaling:
 
 # sysctl -a | grep net.inet.tcp.rfc1323
 net.inet.tcp.rfc1323: 0
 
 Probably there is some sign error or another integer math error in 4.2 and
 Linux both, but they both stopped sending traffic at some point after packet
 loss/retransmit and getting ACK with window of 65535.
 
 Unfortunately, I have no couple non-production machines to check 57344
 and other values, and also to reproduce this in clean environment,
 without natd, ipfw, etc.
 
 -- 
 Stas Kysel, UNIX System Administrator of Yessoftware Inc
 http://www.codecharge.com/
 

From: Stas Kisel <stas_k_freebsd@tiger.unisquad.com>
To: Matthew Dillon <dillon@apollo.backplane.com>
Cc: Sheldon Hearn <sheldonh@starjuice.net>,
	AMAKAWA Shuhei <sa264@cam.ac.uk>, freebsd-gnats-submit@FreeBSD.ORG,
	obrien@FreeBSD.ORG, gallatin@cs.duke.edu, re@FreeBSD.ORG,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Tue, 28 May 2002 21:02:12 +0300

 Probabaly you remember 4-5-year old bug in Linux when uid of nobody
 (65535) was handled as uid of root (0) under some circumstances.
 It was exploited via apache and cgi, you know.
 I mean this kind of bug - win=65535 is transformed to win=0 under
 some circumstances. Just guess.
 
 \bye
 Stas
 

From: AMAKAWA Shuhei <sa264@cam.ac.uk>
To: Matthew Dillon <dillon@apollo.backplane.com>
Cc: re@freebsd.org, Stas Kisel <stas_k_freebsd@tiger.unisquad.com>,
	freebsd-gnats-submit@freebsd.org, obrien@freebsd.org,
	gallatin@cs.duke.edu, sheldonh@freebsd.org,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Wed, 29 May 2002 18:25:20 +0100

 At Tue, 28 May 2002 10:25:57 -0700 (PDT),
 Matthew Dillon wrote:
 > 
 >     The receive window change from 16K -> 64K in -stable was MFC'd by Alfred
 >     in November 2001, so I am adding him to the list.  I am also adding the
 >     release engineers so they can make a final determination on what to do
 >     about the default for the 4.6 release.
 > 
 >     In anycase, a value of 65536 will only activate window scaling if
 >     the RFC 1323 is turned on (net.inet.tcp.rfc1323).  However, rfc1323
 >     *IS* turned on by default.
 > 
 >     It would probably be safer for the release to set the maximum receive
 >     window to 65535 rather then 65536.  I'm on the fence, though, because
 >     what we really ought to be doing is tracking down the interoperability
 >     problem between Linux and FreeBSD instead of doing an end-run around
 >     the whole thing by disabling window scaling.  Both sides must negotiate
 >     the option so either the linux implementation is broken, or ours is.
 
 Do the release engineers have anything to say about this?
 I am for lowering the default value for the release.
 
 --
 Shuhei

From: Murray Stokely <murray@FreeBSD.org>
To: AMAKAWA Shuhei <sa264@cam.ac.uk>
Cc: Matthew Dillon <dillon@apollo.backplane.com>, re@freebsd.org,
	Stas Kisel <stas_k_freebsd@tiger.unisquad.com>,
	freebsd-gnats-submit@freebsd.org, obrien@freebsd.org,
	gallatin@cs.duke.edu, sheldonh@freebsd.org,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Wed, 29 May 2002 13:02:46 -0700

 On Wed, May 29, 2002 at 06:25:20PM +0100, AMAKAWA Shuhei wrote:
 > Do the release engineers have anything to say about this?
 > I am for lowering the default value for the release.
 
   I would also like to lower the default value for the release, as
 long as we can put out another RC with the change to give it some
 testing.  In particular, I would like to get a commitment from some of
 the people involved in this thread (IPFilter users, and others) to
 actually download and test out our next RC before we release.  If we
 can have such commitment to test the change in real-world settings
 then I'd be happy with any of the proposed changes (57344 or 65535).
 
   - Murray

From: Matthew Dillon <dillon@apollo.backplane.com>
To: Murray Stokely <murray@FreeBSD.org>
Cc: AMAKAWA Shuhei <sa264@cam.ac.uk>, re@FreeBSD.org,
	Stas Kisel <stas_k_freebsd@tiger.unisquad.com>,
	freebsd-gnats-submit@FreeBSD.org, obrien@FreeBSD.org,
	gallatin@cs.duke.edu, sheldonh@FreeBSD.org,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Wed, 29 May 2002 13:23:40 -0700 (PDT)

 :
 :On Wed, May 29, 2002 at 06:25:20PM +0100, AMAKAWA Shuhei wrote:
 :> Do the release engineers have anything to say about this?
 :> I am for lowering the default value for the release.
 :
 :  I would also like to lower the default value for the release, as
 :long as we can put out another RC with the change to give it some
 :testing.  In particular, I would like to get a commitment from some of
 :the people involved in this thread (IPFilter users, and others) to
 :actually download and test out our next RC before we release.  If we
 :can have such commitment to test the change in real-world settings
 :then I'd be happy with any of the proposed changes (57344 or 65535).
 :
 :  - Murray
 
     I can make that commitment.   I have two DELL2550's I can commit to
     running tests w/ the RCs.  Are there are any particular tests you would
     like me to do?  Do you want me to burn a CD and do a from-scratch
     install?
 
     In regards to a TCP recvpipe default, either 57344 or 65535 works for
     me.  I'll leave it to the RE's to make the final decision & commit
     the change if approved.
 
     I will add a note, however, that the original bug report that prompted
     this thread was followed up by the original author who noted that he did
     not observe any window scaling options in the connections he was having
     problems with, so the original bug report may not actually be related
     to window scaling.  There still could be other issues involved in a
     default of 65536 so it may still be a good idea to reduce the default
     maximum for the release.
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>

From: AMAKAWA Shuhei <sa264@cam.ac.uk>
To: Matthew Dillon <dillon@apollo.backplane.com>
Cc: Murray Stokely <murray@FreeBSD.org>, re@FreeBSD.org,
	Stas Kisel <stas_k_freebsd@tiger.unisquad.com>,
	freebsd-gnats-submit@FreeBSD.org, obrien@FreeBSD.org,
	gallatin@cs.duke.edu, sheldonh@FreeBSD.org,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Thu, 30 May 2002 07:12:41 +0100

 I could try installing the RC on a machine and use it, but I won't
 be able to dedicate the machine to extensive testing.
 
 Lowering the default should not cause any trouble (in theory, of
 course).  It used to be 16384.
 
 On the other hand, 65536 does badly interfere with IP Filter's
 stateful inspection (before 3.2.26) when rfc1323 is enabled as per
 FreeBSD's default.  IP Filter 3.2.27, which is in FreeBSD, has a
 window scaling code incorporated, but I suspect it has some bugs.
 So anyway 65536 is not good for IP Filter.
 
 Stas's report is indicative of another issue (not window size
 scaling) with 65536.
 
 --
 Shuhei

From: Stas Kysel <stas_k_freebsd@tiger.unisquad.com>
To: AMAKAWA Shuhei <sa264@cam.ac.uk>
Cc: Matthew Dillon <dillon@apollo.backplane.com>,
	Murray Stokely <murray@FreeBSD.org>, re@FreeBSD.org,
	freebsd-gnats-submit@FreeBSD.org, obrien@FreeBSD.org,
	gallatin@cs.duke.edu, sheldonh@FreeBSD.org,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Thu, 30 May 2002 21:33:25 +0300

 Hi.
 
 After some testing I found out some details about issue with
 large windows (the one which does not involve window scaling):
 1. It is only possible to reproduce it with scp, not with netcat
 2. It is only possible to reproduce it with net.inet.tcp.recvspace=65535 or
 net.inet.tcp.recvspace=65536 (in both cases win=65535).
 3. I cannot reproduce it with net.inet.tcp.recvspace=57344
 4. There were another troubles when using recvspace of 57344 and 32768,
 resulting in very slow scp, probably hardware is concerned.
 
 Machines used for testing were 4.2-RELEASE and 4.5-RELEASE
 
 -- 
 Stas Kysel, UNIX System Administrator of Yessoftware Inc
 http://codecharge.com/
 

From: Matthew Dillon <dillon@apollo.backplane.com>
To: Stas Kysel <stas_k_freebsd@tiger.unisquad.com>
Cc: AMAKAWA Shuhei <sa264@cam.ac.uk>,
	Murray Stokely <murray@FreeBSD.org>, re@FreeBSD.org,
	freebsd-gnats-submit@FreeBSD.org, obrien@FreeBSD.org,
	gallatin@cs.duke.edu, sheldonh@FreeBSD.org,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Thu, 30 May 2002 13:15:54 -0700 (PDT)

 :
 :Hi.
 :
 :After some testing I found out some details about issue with
 :large windows (the one which does not involve window scaling):
 :1. It is only possible to reproduce it with scp, not with netcat
 :2. It is only possible to reproduce it with net.inet.tcp.recvspace=65535 or
 :net.inet.tcp.recvspace=65536 (in both cases win=65535).
 :3. I cannot reproduce it with net.inet.tcp.recvspace=57344
 :4. There were another troubles when using recvspace of 57344 and 32768,
 :resulting in very slow scp, probably hardware is concerned.
 :
 :Machines used for testing were 4.2-RELEASE and 4.5-RELEASE
 :
 :-- 
 :Stas Kysel, UNIX System Administrator of Yessoftware Inc
 :http://codecharge.com/
 
     That is very odd.  recvspace should have no effect on the mtu and
     mtu is all I can think of which would have a hardware effect.
 
     However, it does appear that you may have hardware issues unrelated
     to the TCP stack.  You should not get any stalls at all even with
     a small window size.  It kinda sounds like you are getting packet 
     loss or collisions.  It may be worthwhile to some serious network
     testing and tcpdump's on server and client to track down the cause of
     the poor performance / stalls.  It is usually fairly easy using tcpdump
     on both client and server to track down the reason for a protocol stall.
     It takes a little getting used to the massive amount of data you have
     to sift through, but it's actually not so bad if a problem can be
     reproduced within a few seconds.
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>

From: Stas Kysel <stas_k_freebsd@tiger.unisquad.com>
To: Matthew Dillon <dillon@apollo.backplane.com>
Cc: AMAKAWA Shuhei <sa264@cam.ac.uk>,
	Murray Stokely <murray@FreeBSD.org>, re@FreeBSD.org,
	freebsd-gnats-submit@FreeBSD.org, obrien@FreeBSD.org,
	gallatin@cs.duke.edu, sheldonh@FreeBSD.org,
	Alfred Perlstein <bright@mu.org>
Subject: Re: kern/34801: TCP window size bug (afflicting IP Filter keep state)
Date: Fri, 31 May 2002 21:06:28 +0300

 Matthew,
 
 Thank you for advices. I was using exactly the same method you supposed,
 because packet loss of 2% (meausured with ping -f -s) should not cause
 connection timeout after 300-900 kb of data.
 Results are in my first comment on this PR. full tcpdump log is too big to
 post it here (60kb), so I gzipped it and put on
 http://codecharge.com/~stask/scp.txt.gz (non-standart port is used for
 sshd because I didn't want to restart real sshd), so you can check
 that I did not miss something.
 
 This silence after having ACKed last packet might mean:
 1. scp did not provide more data (BTW, last packet size is reduced)
 2. kernel uses win 65535 as 0 (lowering default window fixes it)
 3. something else
 But packet loss does not deal with it, because as long as server where
 tcpdump is running got ACK, it should send data. And it sends it in
 most cases, but sometimes it does not. Packet loss may have only indirect
 influence on this, because data is not sent only if previous packet
 was resent before it was ACKed.
 
 After I seen that this only affects scp, and does not affect netcat,
 I agree with you that it is very odd, and all my input to this PR
 should be probably ignored if someone will try and fail to reproduce this
 with ssh on link with similar packet loss.
 
 -- 
 Stas Kysel, UNIX System Administrator of Yessoftware Inc
 http://codecharge.com/
State-Changed-From-To: open->feedback 
State-Changed-By: darrenr 
State-Changed-When: Tue Apr 20 06:03:58 PDT 2004 
State-Changed-Why:  
after all the feedback, this should be closable by now with the 
current version of ipfilter already in the tree? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=34801 
State-Changed-From-To: feedback->closed 
State-Changed-By: darrenr 
State-Changed-When: Fri Apr 23 19:15:07 PDT 2004 
State-Changed-Why:  
problem fixed 

http://www.freebsd.org/cgi/query-pr.cgi?pr=34801 
>Unformatted:
