From nobody@FreeBSD.org  Fri Nov 11 19:11:21 2005
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 5F39016A420
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 11 Nov 2005 19:11:19 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id EB21343D60
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 11 Nov 2005 19:11:13 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id jABJBD7U032342
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 11 Nov 2005 19:11:13 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id jABJBDgk032341;
	Fri, 11 Nov 2005 19:11:13 GMT
	(envelope-from nobody)
Message-Id: <200511111911.jABJBDgk032341@www.freebsd.org>
Date: Fri, 11 Nov 2005 19:11:13 GMT
From: Susan LoVerso <sue@sleepycat.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: XDR/RPC breaks when sending msgs > 9000 bytes
X-Send-Pr-Version: www-2.3

>Number:         88856
>Category:       kern
>Synopsis:       [nfs] XDR/RPC breaks when sending msgs > 9000 bytes
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    rwatson
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Nov 11 19:20:15 GMT 2005
>Closed-Date:    Wed Jul 05 11:40:38 GMT 2006
>Last-Modified:  Wed Jul 05 11:40:38 GMT 2006
>Originator:     Susan LoVerso
>Release:        5.4
>Organization:
Sleepycat Software
>Environment:
FreeBSD tiffany.bostic.com 5.4-RELEASE FreeBSD 5.4-RELEASE #0: Sun May  8 07:00:26 UTC 2005     root@portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64                           
>Description:
 The addition of the non-blocking code to XDR/RPC breaks some
operations, in particular those that have a message that exceeds
9000 bytes (RPC_MAXDATASIZE) when being sent from the client to the
server.  Sending from the server to the client large messages does
not have this problem and succeeds.  The client is sending the full
amount and the server is returning XPRT_DIED from __xdrrec_getrec
because fraglen is the full message size and it is > in_maxrec.

This results in the server closing its end and the client getting ECONNRESET.

             
>How-To-Repeat:
 There is a simple program that simulates what our software is doing.
You can retrieve it from 
     ftp://ftp.sleepycat.com/hidden/rpcbug.tar.gz.
This contains sources and some ktrace output from an earlier failed
run from both client and server halves.

After unpacking it you should just be able to:
% make
% ./file_svc&
% ./rls localhost test/*

The server outputs messages like this :
SVC: Sending file test/file1: size 85
SVC: Received file test/file1: size 85
..

The client program outputs messages like this:
% ./rls localhost test/*
Recv test/file1: 85 bytes
Sent test/file1: 85 bytes
Recv test/file2: 1360 bytes
Sent test/file2: 1360 bytes
Recv test/file3: 8925 bytes
Sent test/file3: 8925 bytes
Recv test/file4: 9010 bytes
localhost: RPC: Unable to receive; errno = Connection reset by peer

You can see that file4 is able to be transferred from server TO
client, with size 9010, but when the client tries to send file4,
it gets ECONNRESET.

You can also use 'Makefile.dbg' which will build with local versions
of instrumented copies of xdr_rec.c and clnt_vc.c which have print
messages in interesting places.

             
>Fix:
no              
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->rwatson 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Thu Mar 9 01:34:29 UTC 2006 
Responsible-Changed-Why:  
Grab ownership of this PR. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=88856 
State-Changed-From-To: open->analyzed 
State-Changed-By: rwatson 
State-Changed-When: Mon Mar 13 12:16:12 UTC 2006 
State-Changed-Why:  
After fixing the compile of the sample program, I am able to reproduce this 
problem on FreeBSD.  Now working to debug and fix. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=88856 
State-Changed-From-To: analyzed->feedback 
State-Changed-By: rwatson 
State-Changed-When: Tue Mar 14 18:40:20 UTC 2006 
State-Changed-Why:  
This behavior is now understood to be a property of rpc_svcout.c:1.15, 
which overrides libc's rpc defaults for maximum record sizes.  The 
submitter has confirmed that a patch backing out portions of this change 
restores the ability to use larger RPC sizes over TCP.  I have sent 
e-mail to mbr, who committed the change, to inquire about why it was 
made, and if we can safely remove those changes.  If I don't hear back 
in a few days, I will go ahead and commit the change.  Note: although 
the commit comment states that the change originated in NetBSD, there is 
no sign of code in the current NetBSD revision. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=88856 
State-Changed-From-To: feedback->patched 
State-Changed-By: rwatson 
State-Changed-When: Wed May 17 23:19:01 UTC 2006 
State-Changed-Why:  
I have gone ahead and merged the patch, as it appears to resolve the 
problem, after feedback timeouts from alfred and mbr.  I will leave it 
in this state for two weeks, then merge to the RELENG_6 and RELENG_5 
branches.  The change is rpc_svcout.c:1.23.  Commit message attached. 

revision 1.23 
date: 2006/05/17 23:18:17;  author: rwatson;  state: Exp;  lines: +0 -11 
In change 1.15, an arbitrary setting of the max RPC record size to 
RPC_MAXDATASIZE was introduced.  This is believed to have been debugging 
code committed accidentally, although I've been unable to reach the 
committer to confirm this.  The effect was to limit the size of RPCs on 
TCP and UDP to 9k, well below the default protocol limits in the libc 
rpc code.  This change simply removes these introduced limits, falling 
back on the libc definitions. 

PR:             88856 
Reported by:    Keith Bostic <bostic at sleepycat dot com> 
Testing by:     Susan LoVerso <sue at loverso dot southborough dot ma dot us> 
Reveiwed by:    cel, rees 
Review timeout: alfred, mbr 
MFC after:      2 weeks 

Please let me know if there are any problems. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=88856 
State-Changed-From-To: patched->closed 
State-Changed-By: rwatson 
State-Changed-When: Wed Jul 5 11:39:47 UTC 2006 
State-Changed-Why:  
This change has now been merged to FreeBSD RELENG_6 as rpc_svcout.c:1.16.2.1 
and RELENG_5 1.15.6.1, and will appear in FreeBSD 6.2. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=88856 
>Unformatted:
