From nobody@FreeBSD.org  Tue Nov 20 11:01:13 2001
Return-Path: <nobody@FreeBSD.org>
Received: from freefall.freebsd.org (freefall.FreeBSD.org [216.136.204.21])
	by hub.freebsd.org (Postfix) with ESMTP id F2DC037B419
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 20 Nov 2001 11:01:12 -0800 (PST)
Received: (from nobody@localhost)
	by freefall.freebsd.org (8.11.4/8.11.4) id fAKJ1C283740;
	Tue, 20 Nov 2001 11:01:12 -0800 (PST)
	(envelope-from nobody)
Message-Id: <200111201901.fAKJ1C283740@freefall.freebsd.org>
Date: Tue, 20 Nov 2001 11:01:12 -0800 (PST)
From: Alexander Haderer <alexander.haderer@charite.de>
To: freebsd-gnats-submit@FreeBSD.org
Subject: NFS V3/tcp server: low read performance
X-Send-Pr-Version: www-1.0

>Number:         32141
>Category:       misc
>Synopsis:       NFS V3/tcp server: low read performance
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Nov 20 11:10:00 PST 2001
>Closed-Date:    Fri Nov 30 13:43:20 PST 2001
>Last-Modified:  Fri Nov 30 13:46:51 PST 2001
>Originator:     Alexander Haderer
>Release:        4.2 Release, 4.3 Release 4.4 stable
>Organization:
Charite Berlin - Germany
>Environment:
Server:
FreeBSD xxx.charite.de 4.3-RELEASE FreeBSD 4.3-RELEASE #0: Tue Nov 20 13:11:00 CET 2001     root@marvinci.str.charite.de:/usr/src/sys/compile/GENERIC_MARVIN_NO_SCSI  i386
FreeBSD yyy.charite.de 4.4-STABLE FreeBSD 4.4-STABLE #0: Tue Nov 20 11:39:34 CET 2001     root@greenaway.str.charite.de:/opt/src/sys/compile/GENERIC  i386

Client:
FreeBSD zzz.charite.de 4.2-RELEASE FreeBSD 4.2-RELEASE #0: Mon Nov 20 13:02:55 GMT 2000     jkh@bento.FreeBSD.org:/usr/src/sys/compile/GENERIC  i386
...
>Description:
Mounting a NFS volume with NFS V3/TCP from an FreeBSD 4.3+ server
gives low read transferrates (80 kbyte/s) when mounting with
FreeBSD 4.2/4.3 clients.

The server:

(Same effect with 4.3 Release / 4.4 Release / 4.4 stable)

A FreeBSD 4.3+ machine (x86) exports a filesystem.
The installation is out of the box with no tuning done. One
ethernet card is installed and works properly: tcpblast shows
the throughput one expects for an unloaded network (1 Mbyte/s or
11 Mbyte/s for 10/100 Mbit/s network in both directions).
The ethernet config (10/100 hdx/fdx) is done manually. Connection
to the clients is a) via 10Mbit Ethernet hub or b) via switched VLAN
100Mbit/fdx set manually. Neither the server nor the clients have CPU 
or network load.
At client-side: "wire-test server" shows that the server talks 
v2/v3 and udp/tcp in all 4 variants.


(A) FreeBSD 4.2/4.3 Release Client, mount without options:

Mounting the exported filesystem with

    "mount server:/export/scratch /mnt"

and copying a large file (e.g. 3MB generic kernel) from remote
to local filesystem shows the performace for reading 
(cp /mnt/file /usr/tmp) and writing is as expected: 
about 70% .. 90% of tcpdumps bandwitdh; writing slower
than reading; copying files that were not just copyied before
because of cache effects; no tcpdump running. 
Using ftp instead of cp to get the file gives similar results.


(B) FreeBSD 4.2/4.3 Release Client, mount with options:

Mounting the exported filesystem with

    "mount_nfs -3 -T server:/export/scratch /mnt"

forces NFS v3 over TCP. Now copying a large file shows that
performace for writing is just as before. But performance for
reading breaks down to 80 kbyte/s, regardless if we use 10Mbit
or 100Mbit network. Switching down to nfs v2 tcp brings no change. 
Using ftp instead of cp to get the file gives max. transferrate.
BUT changing the server OS to 4.2 Release gives us the performance 
expected, too.
 
(C) Solaris 2.6 client, mount without options:

mounting the filesystem without options gives performance as
expected (r/w). 

(D) Solaris 2.6 client, mount with options:

Forcing NFS v3 TCP gives no change: good performance. Setting
rsize also gives:
32768 = good performance
smaller = goes downto 20%

Things tried out / checked / not appropriate here:
- odd network routing config: no, our's fine, other protocols
  work 
- wrong fdx/hdx settings: ok here: everything set manually and tested
  with tcpdump and ftp'ing files around
- high cpu / network load, lot of clients: no clients, no load here
- /kernel reports some buffer trouble in /var/log/messages:
  no messages here.
- multiplatform with odd NFS implemenentations: no, FreeBSD only
- black box network equipment: nope, only a switch or stupid
  10 mbit HUB between client/server
- unreliable network (wlan) or dialin network: no, ethernet only
- old HW: no, P500+ with PCI Ethernetcards 3com 3c905ctx
- other HW conflicts: no, VGA, Ethernet, Onboard IDE, ser+par only
- broken ipfw settings: no ipfw here
- rc.conf: tcp_extensions YES/NO plus server reboot: no change
- Changing the Server from 4.4 Release to 4.4 Stable: no change
- make all traffic go via stupid 10mbit hub to avoid vlan/switch: 
  no change
- mismatch because of incomplete cvsup: no, Release setup from CDROM

Please feel free to contact me for detailed debug output, session logs,
tcpdumps, etc...
>How-To-Repeat:
A FreeBSD 4.3 Release / 4.4 Release NFS server exports a filesystem, 
a 4.{2,3} Release client mounts the filesystem with 

    'mount_nfs -3 -T server:/export/filesys /mnt'

At the client: copying big files from the server to the client 
gives 80kbyte/s tranfserrate while writing to the server goes fast.

This problem also appears with amd using autonegotiation or forcing
amd to use v3/tcp (this way we discovered the 80kbyte/s reading).

>Fix:
When mounting the remote FS via NFS v3 TCP also set the rsize 
within the mount command:

    'mount_nfs -3 -T -r 4096 server:/export/filesys /mnt'


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->closed 
State-Changed-By: dillon 
State-Changed-When: Fri Nov 30 13:43:20 PST 2001 
State-Changed-Why:  
This plus the conversation on freebsd-hackers has led to the discovery 
of a bug in the transmit burst code for newreno.  I have just comitted 
(to both -current and -stable) a fix to disable this code. 

In my tests, NFS TCP mounts on a 100BaseT full duplex network went  
from 80K/sec to 7.9MB/sec with this fix.  If you also turn on nfsiod on 
the client (nfsiod -n 4 on the client) performance increases from 7.9MB/sec 
to 12MB/sec (which is a fully saturated 100BaseT full duplex link). 

http://www.FreeBSD.org/cgi/query-pr.cgi?pr=32141 
>Unformatted:
