From Andre.Albsmeier@siemens.com  Fri Oct 21 16:32:52 2005
Return-Path: <Andre.Albsmeier@siemens.com>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 9AAC916A41F
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 21 Oct 2005 16:32:52 +0000 (GMT)
	(envelope-from Andre.Albsmeier@siemens.com)
Received: from thoth.sbs.de (thoth.sbs.de [192.35.17.2])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 1C69343D5F
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 21 Oct 2005 16:32:46 +0000 (GMT)
	(envelope-from Andre.Albsmeier@siemens.com)
Received: from mail3.siemens.de (localhost [127.0.0.1])
	by thoth.sbs.de (8.12.6/8.12.6) with ESMTP id j9LGWjOH028281
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 21 Oct 2005 18:32:45 +0200
Received: from ims.mchp.siemens.de (ims.mchp.siemens.de [139.25.31.39])
	by mail3.siemens.de (8.12.6/8.12.6) with ESMTP id j9LGWieL031781
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 21 Oct 2005 18:32:44 +0200
Received: from mail-ct.mchp.siemens.de (mail-ct.mchp.siemens.de [139.25.31.51])
				by ims.mchp.siemens.de  with ESMTP id j9LGWiJQ005266
				for <FreeBSD-gnats-submit@freebsd.org>; Fri, 21 Oct 2005 18:32:44 +0200 (MEST)
Received: from curry.mchp.siemens.de (curry [139.25.40.130])
	by mail-ct.mchp.siemens.de (8.12.11/8.12.11) with ESMTP id j9LGWioS012900
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 21 Oct 2005 18:32:44 +0200 (MEST)
Received: (from localhost)
	by curry.mchp.siemens.de (8.13.4/8.13.4) id j9LGWhc9063475
	for FreeBSD-gnats-submit@freebsd.org; Fri, 21 Oct 2005 18:32:43 +0200 (CEST)
Message-Id: <200510211632.j9LGWhA3013446@curry.mchp.siemens.de>
Date: Fri, 21 Oct 2005 18:32:43 +0200 (CEST)
From: Andre Albsmeier <Andre.Albsmeier@siemens.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: very bad performance of cp(1) via NFS, possibly mmap() problem
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         87792
>Category:       bin
>Synopsis:       [patch] very bad performance of cp(1) via NFS, possibly mmap() problem
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Oct 21 16:40:15 GMT 2005
>Closed-Date:    
>Last-Modified:  Mon Oct 29 08:20:00 UTC 2012
>Originator:     Andre Albsmeier
>Release:        FreeBSD 5.4-STABLE i386
>Organization:
>Environment:

Two machines, a local one which copies files to a remote
machine via NFS using cp. Both machines use FreeBSD 5.4-STABLE.

>Description:

cp uses mmap() to copy files < 8MB. When files are copied
from the local host to a remote box via NFS data is transferred
very slowly. For bigger files, cp uses read() and write()
alternately which performs very well.

>How-To-Repeat:

I have two files, one is 6MB in size and the other one is 12MB.
I copy them from the local box to an NFS mounted directory named
/scratch:

andre@bali:/tmp>time cp 12mb /scratch
cp 12mb /scratch  0.00s user 0.12s system 10% cpu 1.142 total

andre@bali:/tmp>time cp 6mb /scratch
cp 6mb /scratch  0.00s user 0.13s system 0% cpu 14.274 total

The 12MB filecopy performs quite well as I would expect it
from a 100MBit network. The 6mb filecopy I won't comment on :-).
(This is an extreme case, other client-server combinations
are faster. The "fastest" one I found was 4 seconds which is
still bad for a 6MB file).

In the mmap() case, the whole source file is mmap'ed and
one write() call is used which tries to write the whole file
all at once. I have modified copy_file() in utils.c in a way
that instead of one "big" write(), several small ones are
being made (this is no fix, just for demonstration):

--- utils.c.ORI	Thu Aug 19 07:04:27 2004
+++ utils.c	Fri Oct 21 18:19:59 2005
@@ -140,7 +140,11 @@
 			wtotal = 0;
 			for (bufp = p, wresid = fs->st_size; ;
 			    bufp += wcount, wresid -= (size_t)wcount) {
+#if 0
 				wcount = write(to_fd, bufp, wresid);
+#else
+				wcount = write(to_fd, bufp, wresid > 65536 ? 65536 : wresid );
+#endif
 				wtotal += wcount;
 				if (info) {
 					info = 0;


65536 is just a value for playing; almost everything below
1M and not ridiculous small will do it:

andre@bali:/tmp>time cp 6mb /scratch 
cp 6mb /scratch  0.00s user 0.05s system 8% cpu 0.618 total

>Fix:

I have no idea. FreeBSD-4 uses the same mmap() thingy in utils.c
and here performance is ok so I assume the problem is somewhere
else than in the cp sources.
>Release-Note:
>Audit-Trail:

From: Ted Mittelstaedt <tedm@mittelstaedt.us>
To: bug-followup@FreeBSD.org, Andre.Albsmeier@siemens.com
Cc:  
Subject: Re: bin/87792: [patch] very bad performance of cp(1) via NFS, possibly
 mmap() problem
Date: Sun, 28 Oct 2012 23:53:01 -0700

 Try the following:
 
 In bin/cp/utils.c (source) there is a check, if the file is less than
 8MB or so, it uses mmap, if the file is larger, it will use write()
 
 Modify the source and recompiled to -never- use mmap, only to use
 write()
 
 Change line 143:
 original:
 fs->st_size <= 8 * 1048576) {
 
 New:
 fs->st_size <= 8 * 8) {
 
 It will use mmap still if the file is larger than 64bytes (if it uses
 bytes there, pretty sure it does).
 
 This is from a response to the freebsd-stable mailing list back in
 2009 by Brent Jones he reported a 100 fold increase.
 

From: Andre Albsmeier <Andre.Albsmeier@siemens.com>
To: Ted Mittelstaedt <tedm@mittelstaedt.us>
Cc: "bug-followup@FreeBSD.org" <bug-followup@FreeBSD.org>,
        "Albsmeier, Andre" <andre.albsmeier@siemens.com>
Subject: Re: bin/87792: [patch] very bad performance of cp(1) via NFS,
 possibly mmap() problem
Date: Mon, 29 Oct 2012 09:01:37 +0100

 I even went a step further an simply Undef'ed
 VM_AND_BUFFER_CACHE_SYNCHRONIZED in cp's Makefile
 so the entire mmap code gets disabled...
>Unformatted:
