From rob.zietlow@gmail.com  Mon Nov 12 14:40:47 2007
Return-Path: <rob.zietlow@gmail.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id C4E3716A417
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 12 Nov 2007 14:40:47 +0000 (UTC)
	(envelope-from rob.zietlow@gmail.com)
Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.234])
	by mx1.freebsd.org (Postfix) with ESMTP id 5F46113C4B3
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 12 Nov 2007 14:40:47 +0000 (UTC)
	(envelope-from rob.zietlow@gmail.com)
Received: by nz-out-0506.google.com with SMTP id l8so808982nzf
        for <FreeBSD-gnats-submit@freebsd.org>; Mon, 12 Nov 2007 06:40:36 -0800 (PST)
Received: by 10.114.124.1 with SMTP id w1mr723914wac.1194876908687;
        Mon, 12 Nov 2007 06:15:08 -0800 (PST)
Received: by 10.114.94.19 with HTTP; Mon, 12 Nov 2007 06:15:08 -0800 (PST)
Message-Id: <bf64a0fe0711120615t75947f79ge041fe41965fdebb@mail.gmail.com>
Date: Mon, 12 Nov 2007 08:15:08 -0600
From: "Rob Zietlow" <rob.zietlow@gmail.com>
To: FreeBSD-gnats-submit@freebsd.org
Subject: Can no longer SSH into 7.0 Beta Host.

>Number:         118005
>Category:       kern
>Synopsis:       [tcp] Can No Longer SSH into 7.0 host
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    andre
>State:          patched
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Nov 12 14:50:01 UTC 2007
>Closed-Date:    
>Last-Modified:  Sat Aug 14 23:01:31 UTC 2010
>Originator:     Rob.Zietlow@gmail.com
>Release:        FreeBSD 7.0-BETA2 i386
>Organization:
>Environment:
System: FreeBSD voltron.example.com 7.0-BETA2 FreeBSD 7.0-BETA2 #3: Thu Nov
8 15:08:45 CST 2007 root@voltron.example.com:/usr/src/sys/i386/compile/GENERIC
i386


>Description:
        Since upgrading to 7.0 I am no longer able to SSH into my server.  I
cvsup'ed to 7.0 code and rebuild world and since then I have had this
issue.  I have rebuilt multiple times in beta 1, 1.5 and 2. I can SSH into
my host from some hosts within the local LAN. Some machines from outside my
LAN I cannot ssh into this host.  Hosts on my lan I have ssh'ed into this
host with are windows(putty), Linux, and Solaris.  From outside my LAN I
cannot ssh into my host from Freebsd 6.2, Openbsd 4.1, and Linux(RHEL 4U4).
Freebsd & Openbsd machines are on my home network. However my OSX laptop and
windows machine, from my home network, can SSH into the host without a
problem.

From the hosts that get denied I get the following message:
"ssh_exchange_identification: read: Connection reset by peer"
On the server I see the following in /var/log/auth.log: "Nov  9 10:45:10
voltron sshd[15867]: Did not receive identification string from
192.168.3.132"

No other information.  I currently have no firewall running on the host.
voltron# pfctl -si
pfctl: /dev/pf: No such file or directory
You have new mail.
voltron#

/etc/hosts.allow is allowing everything
voltron# cat /etc/hosts.allow
# Wrapping sshd(8) is not normally a good idea, but if you
#sshd : .evil.cracker.example.com : deny
ALL : ALL : allow
voltron#

No special settings in /etc/ssh/sshd_config. I have copied over the sshd
from an existing host and this still doesn't seem to help. Here are my
current settings.
voltron# grep -v \# /etc/ssh/sshd_config
Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_dsa_key
SyslogFacility AUTH
LogLevel DEBUG
Subsystem       sftp    /usr/libexec/sftp-server
DSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys

When I telnet to the port from a host that has issues I immediately get
disconnected.  When I telnet from an allowed machine I get a banner.
.ssh]$ telnet 192.168.8.163 22
Trying 192.168.8.163...
Connected to 192.168.8.163.
Escape character is '^]'.
Connection closed by foreign host.

Banner:   SSH-2.0-OpenSSH_4.5p1 FreeBSD-20061110

Verbose output from a problem host:

[user@bastion .ssh]$ ssh -vvv 192.168.8.163
OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.8.163 [192.168.8.163] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/identity type -1
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
ssh_exchange_identification: read: Connection reset by peer

Debugging from the server:
voltron# /usr/sbin/sshd -ddd
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 332
debug2: parse_server_config: config /etc/ssh/sshd_config len 332
debug3: /etc/ssh/sshd_config:19 setting Port 22
debug3: /etc/ssh/sshd_config:20 setting Protocol 2
debug3: /etc/ssh/sshd_config:28 setting HostKey /etc/ssh/ssh_host_dsa_key
debug3: /etc/ssh/sshd_config:36 setting SyslogFacility AUTH
debug3: /etc/ssh/sshd_config:37 setting LogLevel DEBUG
debug3: /etc/ssh/sshd_config:111 setting Subsystem sftp
/usr/libexec/sftp-server
debug3: /etc/ssh/sshd_config:118 setting DSAAuthentication yes
debug3: /etc/ssh/sshd_config:119 setting PubkeyAuthentication yes
debug3: /etc/ssh/sshd_config:120 setting AuthorizedKeysFile
.ssh/authorized_keys
debug1: sshd version OpenSSH_4.5p1 FreeBSD-20061110
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #0 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: fd 4 clearing O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 7 config len 332
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
debug1: res_init()
Connection from 192.168.3.132 port 41916
Did not receive identification string from 192.168.3.132


tcpdump (does show an incorrect checksum, and broken apart for easier
reading)
voltron# tcpdump -e -vvnn port 22 and host 192.168.3.132
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 68
bytes
08:09:55.816411 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4
(0x0800), length 74: (tos 0x0, ttl 61, id 56887, offset 0, flags [DF], proto
TCP (6), length 60) 192.168.3.132.41922 > 192.168.8.163.22: S
722288481:722288481(0) win 5840 <mss 1460,sackOK,timestamp 1350033750[|tcp]>

08:09:55.816432 00:18:fe:67:54:76 > 00:00:0c:07:ac:09, ethertype IPv4
(0x0800), length 74: (tos 0x0, ttl 64, id 27230, offset 0, flags [DF], proto
TCP (6), length 60) 192.168.8.163.22 > 192.168.3.132.41922: S
2406244836:2406244836(0) ack 722288482 win 65535 <mss 1460,nop,wscale
3,nop,nop,timestamp[|tcp]>

08:09:55.816925 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4
(0x0800), length 60: (tos 0x0, ttl 58, id 0, offset 0, flags [none], proto
TCP (6), length 40) 192.168.3.132.41922 > 192.168.8.163.22: ., cksum 0x6872
(correct), 1:1(0) ack 1 win 0

08:09:55.816933 00:18:fe:67:54:76 > 00:00:0c:07:ac:09, ethertype IPv4
(0x0800), length 54: (tos 0x0, ttl 64, id 27231, offset 0, flags [DF], proto
TCP (6), length 40) 192.168.8.163.22 > 192.168.3.132.41922: R, cksum 0x47e3
(incorrect (-> 0xd2ed), 2406244837:2406244837(0) win 0

08:09:55.817215 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4
(0x0800), length 66: (tos 0x0, ttl 61, id 56889, offset 0, flags [DF], proto
TCP (6), length 52) 192.168.3.132.41922 > 192.168.8.163.22: ., cksum 0x8036
(correct), 1:1(0) ack 1 win 1460 <nop,nop,timestamp 1350033751 1692996280>

08:09:55.833093 00:18:fe:67:54:76 > 00:00:0c:07:ac:09, ethertype IPv4
(0x0800), length 105: (tos 0x0, ttl 64, id 27232, offset 0, flags [DF],
proto TCP (6), length 91) 192.168.8.163.22 > 192.168.3.132.41922: P 1:40(39)
ack 1 win 8326 <nop,nop,timestamp 1692996295 1350033751>

08:09:55.833929 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4
(0x0800), length 60: (tos 0x0, ttl 61, id 8446, offset 0, flags [DF], proto
TCP (6), length 40) 192.168.3.132.41922 > 192.168.8.163.22: R, cksum 0x59d0
(correct), 722288482:722288482(0) win 0




>How-To-Repeat:
       ssh into the host from certain machines.
>Fix:

        None at this time.

>Release-Note:
>Audit-Trail:
From RFC1323:

        A TCP may send the Timestamps option (TSopt) in an initial
        <SYN> segment (i.e., segment containing a SYN bit and no ACK
        bit), and may send a TSopt in other segments only if it re-
        ceived a TSopt in the initial <SYN> segment for the connection.


That means that the following check in tcp_syncache.c:



       /*
        * If timestamps were present in the SYN and we accepted
        * them in our SYN|ACK we require them to be present from
        * now on.  And vice versa.
        */
       if ((sc->sc_flags & SCF_TIMESTAMP) && !(to->to_flags & TOF_TS)) {
               if ((s = tcp_log_addrs(inc, th, NULL, NULL)))
                       log(LOG_DEBUG, "%s; %s: Timestamp missing, "
                           "segment rejected\n", s, __func__);
               goto failed;
       }



is in fact, not valid.

A patch has been submitted to re@. 
State-Changed-From-To: open->analyzed 
State-Changed-By: kmacy 
State-Changed-When: Thu Nov 15 21:09:05 UTC 2007 
State-Changed-Why:  

The bug was found and a patch is pending. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=118005 
Responsible-Changed-From-To: freebsd-bugs->silby 
Responsible-Changed-By: kmacy 
Responsible-Changed-When: Thu Nov 15 23:12:21 UTC 2007 
Responsible-Changed-Why:  

I analyzed it but silby has taken responsibility for it being MFC'd 

http://www.freebsd.org/cgi/query-pr.cgi?pr=118005 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/118005: commit references a PR
Date: Tue, 20 Nov 2007 06:56:14 +0000 (UTC)

 silby       2007-11-20 06:56:04 UTC
 
   FreeBSD src repository
 
   Modified files:
     sys/netinet          tcp_syncache.c 
   Log:
   Comment out the syncache's test which ensures that hosts which negotiate TCP
   timestamps in the initial SYN packet actually use them in the rest of the
   connection.  Unfortunately, during the 7.0 testing cycle users have already
   found network devices that violate this constraint.
   
   RFC 1323 states 'and may send a TSopt in other segments' rather than
   'and MUST send', so we must allow it.
   
   Discovered by: Rob Zietlow
   Tracked down by: Kip Macy
   PR: bin/118005
   
   Revision  Changes    Path
   1.134     +6 -0      src/sys/netinet/tcp_syncache.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: analyzed->closed 
State-Changed-By: kmacy 
State-Changed-When: Tue Nov 20 21:41:43 UTC 2007 
State-Changed-Why:  

Fix committed by silby. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=118005 
State-Changed-From-To: closed->open 
State-Changed-By: andre 
State-Changed-When: Thu Jan 24 10:51:56 UTC 2008 
State-Changed-Why:  
The analysis and the fix seem incorrect.  A proper analysis of the 
supplied information in the PR will follow shortly. 


Responsible-Changed-From-To: silby->andre 
Responsible-Changed-By: andre 
Responsible-Changed-When: Thu Jan 24 10:51:56 UTC 2008 
Responsible-Changed-Why:  
Take over. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=118005 
State-Changed-From-To: open->patched 
State-Changed-By: linimon 
State-Changed-When: Fri Feb 29 02:06:52 UTC 2008 
State-Changed-Why:  
A patch has been committed, but andre apparently disagrees with it. 
Change the state to flag that at least something got committed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=118005 
>Unformatted:
