From dart@nersc.gov  Thu Mar 24 22:25:16 2005
Return-Path: <dart@nersc.gov>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 1D63F16A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 24 Mar 2005 22:25:16 +0000 (GMT)
Received: from mx2.nersc.gov (mx2.nersc.gov [128.55.6.22])
	by mx1.FreeBSD.org (Postfix) with ESMTP id BD75943D39
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 24 Mar 2005 22:25:15 +0000 (GMT)
	(envelope-from dart@nersc.gov)
Received: by mx2.nersc.gov (Postfix, from userid 4002)
	id CD55C7786; Thu, 24 Mar 2005 14:21:26 -0800 (PST)
Received: from mx2.nersc.gov (localhost [127.0.0.1])
	by localhost.nersc.gov (Postfix) with ESMTP
	id 20F2977B1; Thu, 24 Mar 2005 14:21:21 -0800 (PST)
Received: from omega.nersc.gov (omega.nersc.gov [128.55.14.26])
	by mx2.nersc.gov (Postfix) with ESMTP
	id 59EAC7786; Thu, 24 Mar 2005 14:17:31 -0800 (PST)
Received: by omega.nersc.gov (Postfix, from userid 31586)
	id 1F7CF11469; Thu, 24 Mar 2005 14:17:31 -0800 (PST)
Message-Id: <20050324221731.1F7CF11469@omega.nersc.gov>
Date: Thu, 24 Mar 2005 14:17:31 -0800 (PST)
From: Eli dart <dart@nersc.gov>
Reply-To: Eli dart <dart@nersc.gov>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: iozone hurts tcp-based NFS
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         79214
>Category:       kern
>Synopsis:       [nfs] iozone hurts tcp-based NFS
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Mar 24 22:30:02 GMT 2005
>Closed-Date:    Thu Nov 15 22:28:45 UTC 2007
>Last-Modified:  Thu Nov 15 22:28:45 UTC 2007
>Originator:     Eli dart
>Release:        FreeBSD 5.4-PRERELEASE i386
>Organization:
NERSC
>Environment:
System: FreeBSD omega.nersc.gov 5.4-PRERELEASE FreeBSD 5.4-PRERELEASE #0: Mon Mar 21 10:48:01 PST 2005 nickb@omega.nersc.gov:/usr/obj/usr/src/sys/OMEGA-STABLE-SMP i386

>Description:

iozone causes NFS server problems when using TCP-based NFS:

Mar 23 17:10:49 epsilon-new nfs server omega-nfs1:/data/flow: not responding
Mar 23 17:10:49 epsilon-new nfs send error 35 for server omega-nfs1:/data/flow
Mar 23 17:10:49 epsilon-new nfs server omega-nfs1:/data/syslog: not responding
Mar 23 17:10:49 epsilon-new nfs send error 35 for server omega-nfs1:/data/flow
Mar 23 17:10:49 epsilon-new nfs send error 35 for server omega-nfs1:/data/flow
Mar 23 17:10:49 epsilon-new nfs send error 35 for server omega-nfs1:/data/flow
Mar 23 17:10:49 epsilon-new nfs server omega-nfs1:/data/syslog: is alive again
Mar 23 17:10:49 epsilon-new nfs server omega-nfs1:/data/flow: is alive again
Mar 23 17:10:49 epsilon-new nfs server omega-nfs1:/data/flow: is alive again
Mar 23 17:10:49 epsilon-new nfs server omega-nfs1:/data/flow: not responding
Mar 23 17:10:49 epsilon-new nfs server omega-nfs1:/data/flow: not responding
Mar 23 17:10:49 epsilon-new nfs server omega-nfs1:/data/flow: is alive again

>How-To-Repeat:

client and server are connected via back-to-back em interfaces
(crossover cable, no switch).

em driver modified to increase transmit and receive descriptor counts.
Chipset version supports this.
#define EM_MAX_TXD                      4096
#define EM_MAX_RXD                      4096


set up NFS as follows:

===== client /etc/fstab =====
omega-nfs1:/data/flow   /data/flow      nfs     rw,nodev,noexec,nosuid,noinet6,nfsv3,intr,-
U,-b,-r=32768,-w=65535  0       0

===== Server config =====
# nfs
rpcbind_enable="YES"
rpcbind_flags="-h ${nfs_addr1} -h ${nfs_addr2}"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-t -n 8 -h ${nfs_addr1} -h ${nfs_addr2}"
mountd_enable="YES"

===== /etc/exports =====
/data/flow -maproot=nfsnobody:nogroup client-host

Run iozone on client as follows:
iozone 2048 32768 /data/flow/iozone/crunch

iozone version is 2.01

>Fix:

Use UDP for NFS.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->cel 
Responsible-Changed-By: cel 
Responsible-Changed-When: Fri May 12 21:05:14 UTC 2006 
Responsible-Changed-Why:  
"Use UDP for NFS." That's a workaround, not a fix. 

TCP should work better than UDP in this case. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=79214 
Responsible-Changed-From-To: cel->freebsd-bugs 
Responsible-Changed-By: cel 
Responsible-Changed-When: Mon Mar 12 15:30:34 UTC 2007 
Responsible-Changed-Why:  
Back to the public pool. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=79214 
State-Changed-From-To: open->feedback 
State-Changed-By: kmacy 
State-Changed-When: Thu Nov 15 21:11:47 UTC 2007 
State-Changed-Why:  


I followed up with dart@. I don't think this is an issue any more, but if  
it is it should be looked into given that TCP will be the default soon. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=79214 
State-Changed-From-To: feedback->closed 
State-Changed-By: kris 
State-Changed-When: Thu Nov 15 22:27:02 UTC 2007 
State-Changed-Why:  
I am unable to reproduce this on 7.0, but with all of the Giant 
locking in 5.4 it was very plausible that was the cause 
(i.e. everything fighting for Giant and starving NFS until the timer 
fired).  Please let us know if it recurs on 6.x or 7.x. 

P.S. The example fstab has -U for UDP mounts, I assume that was an 
oversight (but tried both). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=79214 
>Unformatted:
