From brian@apocalypse.saturn.net  Wed Jun 25 01:11:07 1997
Received: from apocalypse.saturn.net ([131.128.23.140])
          by hub.freebsd.org (8.8.5/8.8.5) with ESMTP id BAA09269
          for <FreeBSD-gnats-submit@freebsd.org>; Wed, 25 Jun 1997 01:11:03 -0700 (PDT)
Received: (from brian@localhost)
	by apocalypse.saturn.net (8.8.5/8.8.5) id EAA01275;
	Wed, 25 Jun 1997 04:09:20 -0400 (EDT)
Message-Id: <199706250809.EAA01275@apocalypse.saturn.net>
Date: Wed, 25 Jun 1997 04:09:20 -0400 (EDT)
From: Brian Mitchell <brian@firehouse.net>
Reply-To: brian@firehouse.net
To: FreeBSD-gnats-submit@freebsd.org
Subject: nonworking t/tcp server side
X-Send-Pr-Version: 3.2

>Number:         3948
>Category:       kern
>Synopsis:       nonworking t/tcp server side
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    jlemon
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jun 25 01:20:01 PDT 1997
>Closed-Date:    Wed Feb 19 13:20:46 PST 2003
>Last-Modified:  Wed Feb 19 13:20:46 PST 2003
>Originator:     Brian Mitchell
>Release:        FreeBSD 2.2.1-RELEASE i386
>Organization:
firehouse.net
>Environment:

486 running 2.2.1-RELEASE

>Description:

t/tcp seems not to work on server side. To test, I wrote a
simple t/tcp server and sniffed with tcpdump. The packet
traces do not match the description in the manpage. Client
side appears to work, server side does not.


>How-To-Repeat:
(this is somewhat long, but I believe most of it is relevant)

TRANSACTION MODEL
     The expected model of a ``transaction'' as used by T/TCP is a fairly sim-
     ple one:

     1.   A client program generates a request to be sent to the server, which
          is small enough to fit in a single TCP segment, and sends a SYN PUSH
          FIN segment with options and data to the server.

     2.   The server program accepts the request in the same manner as for
          regular TCP connections, interprets it, and generates a reply which
          may be small enough to fit in a single segment.  If it is, the reply
          is sent in a single SYN PUSH FIN ACK segment with (different) op-
          tions and data back to the client.  If not, then the connection de-
          generates into (almost) the usual case for TCP. The server then
          closes its socket.

     3.   The client reads the reply and closes its socket.


My understanding of this is the server gets a SPF segment (as it does) and 
replies with a SPFA segment if it is small enough to fit in a single
packet. This is where things appear to fall apart, as is shown in the 
following tcpdump log.


03:48:31.553801 localhost.1056 > localhost.finger: SFP 319123411:319123420(9) 
win 57344 <mss 16344,nop,wscale 0,nop,nop,timestamp 48396 
0,nop,nop,cc[|tcp]> (DF)

03:48:31.555366 localhost.finger > localhost.1056: S 319212645:319212645(0) 
ack 319123422 win 57344 <mss 16344,nop,wscale 0,nop,nop,timestamp 48396
48396,nop,nop,cc[|tcp]> (DF)

03:48:31.555998 localhost.1056 > localhost.finger: . ack 1 win 57344 
<nop,nop,timestamp 48396 48396,nop,nop,cc 42> (DF)

03:48:31.560896 localhost.finger > localhost.1056: FP 1:7(6) ack 1 win 57344 
<nop,nop,timestamp 48396 48396,nop,nop,cc 43> (DF)

03:48:31.561525 localhost.1056 > localhost.finger: . ack 8 win 57338 
<nop,nop,timestamp 48396 48396,nop,nop,cc 42> (DF)


Here is the execution of the server program. I'm using port 79 since
finger already has t/tcp support.
# ./ttcp1 79
test123
 

Here is execution of finger, and the response.
> finger test123@localhost
[localhost]
REPLY


Here I make sure t/tcp is indeed enabled (although there was little doubt
judging from the initial segment from the client).
> /usr/sbin/sysctl net.inet.tcp.rfc1644
net.inet.tcp.rfc1644: 1

Here is the output of uname -r
> uname -r
2.2.1-RELEASE




Here is the code. Perhaps my interpretation of the manpage was incorrect?

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
main(int argc, char **argv)
{
	u_short port = 79;
	struct sockaddr_in sin;
	int s;
	int i;
	char buffer[1024];

	if(argc > 1)
		port = atoi(argv[1]);
	if(port < 1)
		port = 79;
	s = socket(PF_INET, SOCK_STREAM, 0);
	if(s < 0)
	{
		perror("socket");
		exit(1);
	}
	if(setsockopt(s, IPPROTO_TCP, TCP_NOPUSH, &s, sizeof(s)) < 0)
	{
		perror("setsockopt");
		exit(1);
	}
	sin.sin_family = PF_INET;
	sin.sin_port = htons(port);
	sin.sin_addr.s_addr = INADDR_ANY;
	if(bind(s, (struct sockaddr *)&sin, sizeof(sin)) < 0)
	{
		perror("bind");
		exit(1);
	}
	if(listen(s, 5) < 0)
	{
		perror("listen");
		exit(1);
	}
	i = sizeof(sin);
	s = accept(s, (struct sockaddr *)&sin, &i);
	if(s < 0)
	{
		perror("accept");
		exit(1);
	}
	i = read(s, buffer, 1024);
	if(i < 0)
	{
		perror("read");
		exit(1);
	}
	else if(!i)
	{
		fprintf(stderr, "peer closed connection\n");
		exit(1);
	}
	else
	{
		write(1, buffer, i);
		write(s, "REPLY\n", 6);
		exit(0);
	}
}

>Fix:
	

>Release-Note:
>Audit-Trail:

From: Garrett Wollman <wollman@khavrinen.lcs.mit.edu>
To: brian@firehouse.net
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: kern/3948: nonworking t/tcp server side
Date: Wed, 25 Jun 1997 12:45:38 -0400 (EDT)

 <<On Wed, 25 Jun 1997 04:09:20 -0400 (EDT), Brian Mitchell <brian@firehouse.net> said:
 
 > 03:48:31.553801 localhost.1056 > localhost.finger: SFP 319123411:319123420(9) 
 > win 57344 <mss 16344,nop,wscale 0,nop,nop,timestamp 48396 
 > 0,nop,nop,cc[|tcp]> (DF)
 
 > 03:48:31.555366 localhost.finger > localhost.1056: S 319212645:319212645(0) 
 > ack 319123422 win 57344 <mss 16344,nop,wscale 0,nop,nop,timestamp 48396
 > 48396,nop,nop,cc[|tcp]> (DF)
 
 You're right that this does appear to be an error, assuming that this
 is /not/ the first T/TCP connection on localhost.  I doubt that anyone
 will have the time to figure this out any time soon, since T/TCP is
 not widely used, but if you're motivated to look into it I'm certain
 any further diagnosis you can make will be helpful if the problem is
 to be eventually solved.
 
 Note that it certainly is possible for the syn-ack to get sent without
 waiting for data, but only in the particular case of the 200-ms
 delayed-ack timer firing between the acceptance of the connection and
 the server's presentation of new data to send.  (In this
 implementation, the fast timer will always force out any unsent data,
 including the return syn, if a delayed ack is pending, as in this
 case.)  However, this does not strike me as a particularly likely
 occurrence, and certainly should not occur every time unless you have
 a particularly slow machine.
 
 You might start looking for this problem at this point (in
 tcp_input.c):
                         if ((tiflags & TH_FIN) || (ti->ti_len != 0 &&
                             in_localaddr(inp->inp_faddr)))
                                 tp->t_flags |= (TF_DELACK | TF_NEEDSYN);
                         else
                                 tp->t_flags |= (TF_ACKNOW | TF_NEEDSYN);
 
 The condition in the if statement should be true; you might add some
 debugging statements around this point to see whether TF_ACKNOW is
 being set.  You could also try using the TCP debugging features (set
 SO_DEBUG and use trpt(8) on a kernel with the TCPDEBUG option); this
 should definitively tell you whether a timer is firing.
 
 -GAWollman
 
 --
 Garrett A. Wollman   | O Siem / We are all family / O Siem / We're all the same
 wollman@lcs.mit.edu  | O Siem / The fires of freedom 
 Opinions not those of| Dance in the burning flame
 MIT, LCS, CRS, or NSA|                     - Susan Aglukark and Chad Irschick

From: Bill Fenner <fenner@parc.xerox.com>
To: freebsd-gnats-submit@freebsd.org
Cc: brian@firehouse.net
Subject: Re: kern/3948: nonworking t/tcp server side
Date: Wed, 25 Jun 1997 11:36:20 PDT

 Using your test program, I was able to perform a T/TCP transaction on an
 unmodified 2.2 kernel:
 
 17:51:07.632196 sundae.parc.xerox.com.1555 > sundae.parc.xerox.com.finger: SFP 4072296610:4072296612(2) win 17280 <mss 16344,nop,wscale 0,nop,nop,timestamp 4642723 0,nop,nop,cc 1161> (DF)
 17:51:07.635814 sundae.parc.xerox.com.finger > sundae.parc.xerox.com.1555: SFP 4072370814:4072370820(6) ack 4072296614 win 17280 <mss 16344,nop,wscale 0,nop,nop,timestamp 4642723 4642723,nop,nop,cc 1162,nop,nop,ccecho 1161> (DF)
 17:51:07.636017 sundae.parc.xerox.com.1555 > sundae.parc.xerox.com.finger: . ack 8 win 17274 <nop,nop,timestamp 4642723 4642723,nop,nop,cc 1161> (DF)
 pwd
 
 However, the ack when using "localhost" instead of my host's address
 doesn't appear to be delayed so the data doesn't get a chance to
 piggyback on the SYN/ACK:
 
 18:21:33.530418 localhost.1562 > localhost.finger: SFP 127085611:127085622(11) win 57344 <mss 16344,nop,wscale 0,nop,nop,timestamp 4646375 0,nop,nop,cc 1175> (DF)
 18:21:33.531323 localhost.finger > localhost.1562: S 127209529:127209529(0) ack 127085624 win 57344 <mss 16344,nop,wscale 0,nop,nop,timestamp 4646375 4646375,nop,nop,cc 1176,nop,nop,ccecho 1175> (DF)
 18:21:33.531466 localhost.1562 > localhost.finger: . ack 1 win 57344 <nop,nop,timestamp 4646375 4646375,nop,nop,cc 1175> (DF)
 18:21:33.534860 localhost.finger > localhost.1562: FP 1:7(6) ack 1 win 57344 <nop,nop,timestamp 4646375 4646375,nop,nop,cc 1176> (DF)
 18:21:33.535015 localhost.1562 > localhost.finger: . ack 8 win 57338 <nop,nop,timestamp 4646375 4646375,nop,nop,cc 1175> (DF)
 
 Both transactions used the "lo0" interface, and both returning syn's
 included the "ccecho" option, indicating that TAO should have succeeded.
 Perhaps there is some interaction with the larger window used when
 talking to localhost, or perhaps the TAO data doesn't get cached in the
 route to localhost for some reason.
 
   Bill

From: Martin Kammerhofer <dada@localhost.tu-graz.ac.at>
To: FreeBSD problems <freebsd-gnats-submit@freebsd.org>
Cc: brian@firehouse.net
Subject: Re: kern/3948: nonworking t/tcp server side
Date: Tue, 30 Mar 1999 00:24:06 +0200 (CEST)

 The problem is a benign bug in tcp_output.c. It has nothing to do
 with delayed acks. It shows up when two conditions hold:
  1. The accepting server reads it's input before sending to the socket.
     (If the server writes a greeting message just after the accept, it
     will be piggy backed on the SYN/ACK segment as expected.)
  2. The server has a large socket buffer space compared to the
     MSS obtained from the clients initial packet.
 
 Here is in detail was goes wrong:
  1. The clients sends a SFP segment with request data to the server port.
  2. The server decides in tcp_input() do delay the ACK. (Until the
     TCP fast timer runs).
  3. The server accepts the connection.
  4. The server reads from the socket. tcp_usr_rcvd() (in tcp_usrreq.c)
     calls tcp_output() to check if a window update should be sent.
  5. It passes the following test [snippet from tcp_output.c]:
 	/*
 	 * Compare available window to amount of window
 	 * known to peer (as advertised window less
 	 * next expected input).  If the difference is at least two
 	 * max size segments, or at least 50% of the maximum possible
 	 * window, then want to send a window update to peer.
 	 */
 	if (win > 0) {
 		/*
 		 * "adv" is the amount we can increase the window,
 		 * taking into account that we are limited by
 		 * TCP_MAXWIN << tp->rcv_scale.
 		 */
 		long adv = min(win, (long)TCP_MAXWIN << tp->rcv_scale) -
 			(tp->rcv_adv - tp->rcv_nxt);
 
 		if (adv >= (long) (2 * tp->t_maxseg))
 			goto send;
 		if (2 * adv >= (long) so->so_rcv.sb_hiwat)
 			goto send;
 	}
 	[end of snippet]
 	
     win is the amount of available space in the socket and is
     typically large because only one segment has been received.
  6. The "goto send" does what you expect from the label's name.
 
 Doing that test here in the context of a half synchronized T/TCP
 connection is a bug. The idea is to let the peer know that a
 considerable larger window is available now. But in T/TCP's half
 synchronized CLOSE-WAIT* state there is no window advertised to
 the client - the client has not seen any packet yet.
 The fix is simple: change the test from (win > 0) to
 (win > 0 && !(tp->t_flags & TF_NEEDSYN)). See below for a patch.
 
 Although this bug has been reported for FreeBSD 2.2.1 and I verified
 my fix with 2.2.8-STABLE it affects the MAIN branch too. (I looked
 into the repository version.)
 
 With this tiny patch you could get what rfc1644 promises on page 1:
 [quote]
 
    (d)  The minimum transaction latency for a client should be RTT +
         SPT, where RTT is the round-trip time and SPT is the server
         processing time.
 
    (e)  In favorable circumstances, a reliable request/response
         handshake should be achievable with exactly one packet in each
         direction.
 [quote off]
 
 What a pity that almost nobody uses T/TCP :(
 
 Regards,
   Martin
 --
 Kommunikation lt sich durch nichts ersetzen.
 
 <=========== cut here ==============
 --- /src/sys/netinet/tcp_output.c.orig	Tue Mar 30 00:19:16 1999
 +++ /src/sys/netinet/tcp_output.c	Mon Mar 29 22:52:57 1999
 @@ -241,15 +241,15 @@
  	/*
  	 * Compare available window to amount of window
  	 * known to peer (as advertised window less
  	 * next expected input).  If the difference is at least two
  	 * max size segments, or at least 50% of the maximum possible
  	 * window, then want to send a window update to peer.
  	 */
 -	if (win > 0) {
 +	if (win > 0 && !(tp->t_flags & TF_NEEDSYN)) {
  		/*
  		 * "adv" is the amount we can increase the window,
  		 * taking into account that we are limited by
  		 * TCP_MAXWIN << tp->rcv_scale.
  		 */
  		long adv = min(win, (long)TCP_MAXWIN << tp->rcv_scale) -
  			(tp->rcv_adv - tp->rcv_nxt);
 
 
 
Responsible-Changed-From-To: freebsd-bugs->jlemon 
Responsible-Changed-By: phk 
Responsible-Changed-When: Wed Mar 28 10:49:33 PST 2001 
Responsible-Changed-Why:  
If this is still relevant, jlemon will deal with it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=3948 

From: Joseph Holland King <gte743n@cad.gatech.edu>
To: freebsd-gnats-submit@freebsd.org
Cc:  
Subject: Re: kern/3948: nonworking t/tcp server side
Date: Wed, 20 Nov 2002 18:48:08 -0500

 this is over a year stale, and there is a patch there, could this be
 closed?
 
 -- 
 Holland King        
 gte743n@cad.gatech.edu

From: Joseph Holland King <gte743n@cad.gatech.edu>
To: freebsd-gnats-submit@freebsd.org
Cc:  
Subject: Re: kern/3948: nonworking t/tcp server side
Date: Mon, 17 Feb 2003 00:15:40 -0500

 This PR can be closed
 -- 
 Holland King        
 gte743n@cad.gatech.edu
State-Changed-From-To: open->closed 
State-Changed-By: jlemon 
State-Changed-When: Wed Feb 19 13:20:26 PST 2003 
State-Changed-Why:  
Patch applied to 5.0, thanks. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=3948 
>Unformatted:
