From dkelly@Grumpy.DynDNS.org  Wed Nov 16 04:37:37 2005
Return-Path: <dkelly@Grumpy.DynDNS.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 4BE8416A420
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 16 Nov 2005 04:37:37 +0000 (GMT)
	(envelope-from dkelly@Grumpy.DynDNS.org)
Received: from smtp.knology.net (smtp.knology.net [24.214.63.101])
	by mx1.FreeBSD.org (Postfix) with SMTP id B93BD43D55
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 16 Nov 2005 04:37:36 +0000 (GMT)
	(envelope-from dkelly@Grumpy.DynDNS.org)
Received: (qmail 4097 invoked by uid 0); 16 Nov 2005 04:36:46 -0000
Received: from user-69-73-60-132.knology.net (HELO Grumpy.DynDNS.org) (69.73.60.132)
  by smtp5.knology.net with SMTP; 16 Nov 2005 04:36:46 -0000
Received: by Grumpy.DynDNS.org (Postfix, from userid 928)
	id B848A69B3; Tue, 15 Nov 2005 22:37:34 -0600 (CST)
Message-Id: <20051116043734.B848A69B3@Grumpy.DynDNS.org>
Date: Tue, 15 Nov 2005 22:37:34 -0600 (CST)
From: David Kelly <dkelly@hiwaay.net>
Reply-To: David Kelly <dkelly@hiwaay.net>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: premature EOF with ftpd on some large files
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         89100
>Category:       bin
>Synopsis:       premature EOF with ftpd on some large files
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    yar
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Nov 16 04:40:21 GMT 2005
>Closed-Date:    Tue Jan 31 14:41:33 GMT 2006
>Last-Modified:  Mon Feb  6 11:30:05 GMT 2006
>Originator:     David Kelly
>Release:        FreeBSD 6.0-STABLE i386
>Organization:
>Environment:
System: FreeBSD Grumpy.DynDNS.org 6.0-STABLE FreeBSD 6.0-STABLE #2: Sun Nov 13 19:23:43 CST 2005 dkelly@Grumpy.DynDNS.org:/usr5/obj/usr/src/sys/OPUS i386


	
>Description:

Using ftpd to send large files often results in: "226 Transfer finished
due to premature end of file."

For several days thought the problem was related to geom_vinum as most
of my big files are on a striped vinum volume whose creation dates back
to 5.0. Largely because /bin/cp could read the files correctly, write to
the simple volume on /dev/ad0s1f, and then ftpd had no problem reading
the file.

Then I started having the same problem with large files on /dev/ad0s1f
(/usr). The common denominator is ftpd.

ftp> get bigfile /dev/null
local: /dev/null remote: bigfile
229 Entering Extended Passive Mode (|||56427|)
150 Opening BINARY mode data connection for 'bigfile' (4700241920 bytes).
226 Transfer finished due to premature end of file.
405274624 bytes received in 00:08 (46.06 MB/s)
ftp> quit

% dc
10 i 16 o
4700241920 p
118280000
405274624 p
18280000

Every time I pay attention the received file is exactly 4G short. Doesn't
matter if it is written to /dev/null, differnt filesystem, or same
filesystem.

	
>How-To-Repeat:

Once a file is a problem it is always a problem in the same place. I
have not been able to create a large file which immediately has this
problem. Have not tested every large file immediately after creation,
only find this problem the next day or two.

Seems I have this problem with every file written with 5.4 on the
geom_vinum filesystem.

	
>Fix:

No permanent cure but ftpd has been able to read every fresh copy made
with /bin/cp.

	


>Release-Note:
>Audit-Trail:

From: David Kelly <dkelly@HiWAAY.net>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/89100: premature EOF with ftpd on some large files
Date: Fri, 25 Nov 2005 12:41:50 -0600

 cvsup on November 22 changed a bunch of files but did not affect this  
 problem.
 
 Of possible consequence, ACPI no longer works on this Dell 400SC.  
 When original P/R was filed had booted with ACPI disabled and only 1G  
 of 2G physical memory was detected in spite of /boot/loader.conf  
 containing:
 
 if_ath_load=YES
 snd_ich_load=YES
 geom_vinum_load=YES
 hw.ata.atapi_dma=1
 kern.maxdsiz="2G"
 kern.dfldsiz="2G"
 hw.physmem="2G"
 kern.maxssiz="128M"
 
 New kernel still doesn't like ACPI but now recognizes full 2048M of  
 memory.
 
 Every time I copy a problem file, the new copy downloads correctly  
 thru ftpd. The original still does not. Some time after making a copy  
 the copied file no longer downloads thru ftpd correctly. Always  
 exactly the last 4G is missing.
 
 Am not skilled at gdb enough to breakpoint in a forked process but  
 have sprinkled printf()'s in ftpd that I now believe the problem lies  
 in the sendfile(2) kernel routines and therefore this P/R should be  
 moved to kernel.
 
 Have not been able to grep for "sendfile" properly to find its  
 implementation in the kernel. Find it mentioned plenty of other  
 places but haven't found where the work gets done.
 
 --
 David Kelly N4HHE, dkelly@HiWAAY.net
 ========================================================================
 Whom computers would destroy, they must first drive mad.
 

From: Deomid Ryabkov <myself@rojer.pp.ru>
To: bug-followup@FreeBSD.org, dkelly@hiwaay.net
Cc:  
Subject: Re: bin/89100: premature EOF with ftpd on some large files
Date: Mon, 05 Dec 2005 23:44:05 +0300

 i have similar problem: sendfile(2) returns prematurely,
 when exactly 2^32 bytes left to transfer.
 this is demonstrated by ftpd (which uses sendfile), but applies to
 apache too (if configured with enablesendfile yes).
 in my case, sendfile misbehaves only when asked to send file residing on
 NFS-mounted filesystem.
 this seems to happen to all files > 4G resiging on NFS-mounted
 filesystems. those same files can be read without problem with read(2),
 therefore copying with cp(1) succeeds.
 
 while tracking down the problem i set up a local ftp server on my
 notebook, rooted at /stuff/ftp.
 i was able to download a large file residing on local hard disk without
 problem:
 
 nb[/stuff/ftp]# fetch -o /dev/null
 ftp://192.168.10.4/debian-31r0a-i386-binary-1.iso
 /dev/null                                     100% of 4469 MB   14 MBps
 00m00s
 
 then i tried downloading the same file from NFS-mounted filesystem:
 
 nb[/stuff/ftp]# mount
 <snip>
 tux:/spool on /stuff/ftp/spool (nfs)
 nb[/stuff/ftp]# fetch -o /dev/null
 ftp://192.168.10.4/spool/distr/debian-31r0a-i386-binary-1.iso
 /dev/null                                       8% of 4469 MB 7310 kBps
 09m30s
 fetch: /dev/null appears to be truncated: 391739392/4686706688 bytes
 
 and another one:
 
 nb[/stuff/ftp]# fetch -o /dev/null
 ftp://192.168.10.4/spool/distr/debian-31r0a-i386-binary-2.iso
 /dev/null                                       3% of 4223 MB 7367 kBps
 09m37s
 fetch: /dev/null appears to be truncated: 133883904/4428851200 bytes
 
 in both cases, transfer ends at exactly 4294967296=2^32 bytes before
 actual end of file.
 one line of debug output added to ftpd.c shows that sendfile actually
 returns without error, with correct amount of transferred bytes:
 
 Dec  5 23:34:26 nb ftpd[55338]: sendfile returned 0, errno=2,
 cnt=391739392
 Dec  5 23:39:21 nb ftpd[55352]: sendfile returned 0, errno=2,
 cnt=133883904
 
 
 

From: David Kelly <dkelly@hiwaay.net>
To: Deomid Ryabkov <myself@rojer.pp.ru>
Cc: bug-followup@FreeBSD.org
Subject: Re: bin/89100: premature EOF with ftpd on some large files
Date: Mon, 5 Dec 2005 15:39:36 -0600

 On Mon, Dec 05, 2005 at 11:44:05PM +0300, Deomid Ryabkov wrote:
 > i have similar problem: sendfile(2) returns prematurely,
 > when exactly 2^32 bytes left to transfer.
 > this is demonstrated by ftpd (which uses sendfile), but applies to
 > apache too (if configured with enablesendfile yes).
 > in my case, sendfile misbehaves only when asked to send file residing on
 > NFS-mounted filesystem.
 > this seems to happen to all files > 4G resiging on NFS-mounted
 > filesystems. those same files can be read without problem with read(2),
 > therefore copying with cp(1) succeeds.
 
 What I find is that the files "break" somehow for sendfile() once they
 have aged a bit on the filesystem. "cp -p" the problem file to the same
 filesystem and sendfile() (via ftpd) sends the copied file correctly
 today but won't a day or two later.
 
 Guess I should try a reboot after copy to see if that breaks the file.
 Its as if sendfile() doesn't know that it has not finished its job.
 Possibly a fresh copy has metadata cached by the kernel but sendfile()
 quits rather than fetch needed metadata if its missing.
 
 Of course this worked in RELENG_5.
 
 -- 
 David Kelly N4HHE, dkelly@HiWAAY.net
 ========================================================================
 Whom computers would destroy, they must first drive mad.

From: David Kelly <dkelly@hiwaay.net>
To: Deomid Ryabkov <myself@rojer.pp.ru>
Cc: bug-followup@FreeBSD.org
Subject: Re: bin/89100: premature EOF with ftpd on some large files
Date: Mon, 5 Dec 2005 16:02:28 -0600

 Forgot to mention earlier "reget bigfile" in ftp correctly finishes the
 job on a file which the moment before stopped with "premature EOF".
 
 -- 
 David Kelly N4HHE, dkelly@HiWAAY.net
 ========================================================================
 Whom computers would destroy, they must first drive mad.

Manually adding to audit trail:

see also kern/92243.
State-Changed-From-To: open->closed 
State-Changed-By: yar 
State-Changed-When: Tue Jan 31 14:39:57 UTC 2006 
State-Changed-Why:  
The problem is being investigated in kern/92243. 


Responsible-Changed-From-To: freebsd-bugs->yar 
Responsible-Changed-By: yar 
Responsible-Changed-When: Tue Jan 31 14:39:57 UTC 2006 
Responsible-Changed-Why:  
My area. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89100 

From: Deomid Ryabkov <myself@rojer.pp.ru>
To: Yar Tikhiy <yar@freebsd.org>
Cc:  
Subject: Re: bin/89100: premature EOF with ftpd on some large files
Date: Mon, 06 Feb 2006 11:23:07 +0300

 This is a cryptographically signed message in MIME format.
 
 --------------ms010705060000090100070101
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit
 
 yes, your patch fixes the problem.
 thanks!
 
 Yar Tikhiy wrote:
 > Hi there,
 >
 >   
 >> i have similar problem: sendfile(2) returns prematurely,
 >> when exactly 2^32 bytes left to transfer.
 >> this is demonstrated by ftpd (which uses sendfile), but applies to
 >> apache too (if configured with enablesendfile yes).
 >> in my case, sendfile misbehaves only when asked to send file residing on
 >> NFS-mounted filesystem.
 >> this seems to happen to all files > 4G resiging on NFS-mounted
 >> filesystems. those same files can be read without problem with read(2),
 >> therefore copying with cp(1) succeeds.
 >>     
 >
 > Could you please test my patch from kern/92243 in your NFS
 > environment?  Note that you'll have to recompile both kernel
 > and modules after applying it.  Thank you in advance!
 >
 > The URL:
 > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern%2F92243
 >
 >   
 --
 Deomid Ryabkov aka Rojer
 myself@rojer.pp.ru
 rojer@sysadmins.ru
 ICQ: 8025844
 
 
 --------------ms010705060000090100070101
 Content-Type: application/x-pkcs7-signature; name="smime.p7s"
 Content-Transfer-Encoding: base64
 Content-Disposition: attachment; filename="smime.p7s"
 Content-Description: S/MIME Cryptographic Signature
 
 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJIzCC
 AuwwggJVoAMCAQICAw5jHTANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE
 ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv
 bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwNDAxMDkwNjQzWhcNMDYwNDAxMDkwNjQz
 WjBfMRAwDgYDVQQEEwdSeWFia292MQ8wDQYDVQQqEwZEZW9taWQxFzAVBgNVBAMTDkRlb21p
 ZCBSeWFia292MSEwHwYJKoZIhvcNAQkBFhJteXNlbGZAcm9qZXIucHAucnUwggEiMA0GCSqG
 SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDEKoweumUc1/YHtlscU5xKozcKOd3lLyAZ1SM3rZvn
 iJ9VAuj9TafODcu+SoJ6sU+Crshl2Nkq/oCs6dynEqyn/jZxGm/mEYxJ+KekBQceLejdFktQ
 rOuXmjLpipESMO7w1amFn6w3pJgWXex2mcN6hcET2cUdSHjSYxLUXKCQTtzJbcPEHZ+vgEq9
 1TA4UcFeZ3d1Ax6q2A2Fs/wvFxtLUC46fq80R7lOxsJA6mxKNOJnbZTCqf8sWF5SyEnNBBj0
 yyIHfKl+iMwsqSESg5hc0l9/m6aLV24KtKtvWIEu3RQXflc380xZanF4gvAq8/NADlfEH4Rx
 SpNOJdgxZga1AgMBAAGjLzAtMB0GA1UdEQQWMBSBEm15c2VsZkByb2plci5wcC5ydTAMBgNV
 HRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUAA4GBAEaynMcbL7KaxmVMfJWXD7X4ftDolZ2CpPPN
 yoVJAIXaIHpI0JuiCnQSZivL6BvtYUNyzNAR6ceh87yWoQEJxw1cV2IgUlQ+Z3/+7brumNdb
 YwCuf3C/LlamOP5zpHDOH1euXLJe8664lC5lIaf70yO6jN7LXHwBNs73qyB7tvY/MIIC7DCC
 AlWgAwIBAgIDDmMdMA0GCSqGSIb3DQEBBAUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxU
 aGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwg
 RnJlZW1haWwgSXNzdWluZyBDQTAeFw0wNTA0MDEwOTA2NDNaFw0wNjA0MDEwOTA2NDNaMF8x
 EDAOBgNVBAQTB1J5YWJrb3YxDzANBgNVBCoTBkRlb21pZDEXMBUGA1UEAxMORGVvbWlkIFJ5
 YWJrb3YxITAfBgkqhkiG9w0BCQEWEm15c2VsZkByb2plci5wcC5ydTCCASIwDQYJKoZIhvcN
 AQEBBQADggEPADCCAQoCggEBAMQqjB66ZRzX9ge2WxxTnEqjNwo53eUvIBnVIzetm+eIn1UC
 6P1Np84Ny75KgnqxT4KuyGXY2Sr+gKzp3KcSrKf+NnEab+YRjEn4p6QFBx4t6N0WS1Cs65ea
 MumKkRIw7vDVqYWfrDekmBZd7HaZw3qFwRPZxR1IeNJjEtRcoJBO3Mltw8Qdn6+ASr3VMDhR
 wV5nd3UDHqrYDYWz/C8XG0tQLjp+rzRHuU7GwkDqbEo04mdtlMKp/yxYXlLISc0EGPTLIgd8
 qX6IzCypIRKDmFzSX3+bpotXbgq0q29YgS7dFBd+VzfzTFlqcXiC8Crz80AOV8QfhHFKk04l
 2DFmBrUCAwEAAaMvMC0wHQYDVR0RBBYwFIESbXlzZWxmQHJvamVyLnBwLnJ1MAwGA1UdEwEB
 /wQCMAAwDQYJKoZIhvcNAQEEBQADgYEARrKcxxsvsprGZUx8lZcPtfh+0OiVnYKk883KhUkA
 hdogekjQm6IKdBJmK8voG+1hQ3LM0BHpx6HzvJahAQnHDVxXYiBSVD5nf/7tuu6Y11tjAK5/
 cL8uVqY4/nOkcM4fV65csl7zrriULmUhp/vTI7qM3stcfAE2zverIHu29j8wggM/MIICqKAD
 AgECAgENMA0GCSqGSIb3DQEBBQUAMIHRMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVy
 biBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xGjAYBgNVBAoTEVRoYXd0ZSBDb25zdWx0aW5n
 MSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2VzIERpdmlzaW9uMSQwIgYDVQQDExtU
 aGF3dGUgUGVyc29uYWwgRnJlZW1haWwgQ0ExKzApBgkqhkiG9w0BCQEWHHBlcnNvbmFsLWZy
 ZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMwNzE3MDAwMDAwWhcNMTMwNzE2MjM1OTU5WjBiMQsw
 CQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoG
 A1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwgZ8wDQYJKoZIhvcN
 AQEBBQADgY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUEcJ3f6f+jHuy9zfVb8hp2vX8MOmHy
 v1HOAdTlUAow1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH5/EfkTYkKhPPK9Xzgnc9A74r/rsY
 Pge/QIACZNenprufZdHFKlSFD0gEf6e20TxhBEAeZBlyYLf7AgMBAAGjgZQwgZEwEgYDVR0T
 AQH/BAgwBgEB/wIBADBDBgNVHR8EPDA6MDigNqA0hjJodHRwOi8vY3JsLnRoYXd0ZS5jb20v
 VGhhd3RlUGVyc29uYWxGcmVlbWFpbENBLmNybDALBgNVHQ8EBAMCAQYwKQYDVR0RBCIwIKQe
 MBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDItMTM4MA0GCSqGSIb3DQEBBQUAA4GBAEiM0VCD
 6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0whuPg2H6otnzYvwPQcUCCTcDz9reFhYsPZOhl+hLGZ
 GwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBPZV+V2vf3h9bGCE6u9uo05RAaWzVNd+NWIXiC
 3CEZNd4ksdMdRv9dX2VPMYIDOzCCAzcCAQEwaTBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMc
 VGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFs
 IEZyZWVtYWlsIElzc3VpbmcgQ0ECAw5jHTAJBgUrDgMCGgUAoIIBpzAYBgkqhkiG9w0BCQMx
 CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0wNjAyMDYwODIzMDdaMCMGCSqGSIb3DQEJ
 BDEWBBRC+4zK7CIFLYYXX3LKfTFTfxlh7DBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMH
 MA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIB
 KDB4BgkrBgEEAYI3EAQxazBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29u
 c3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwg
 SXNzdWluZyBDQQIDDmMdMHoGCyqGSIb3DQEJEAILMWugaTBiMQswCQYDVQQGEwJaQTElMCMG
 A1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBl
 cnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAw5jHTANBgkqhkiG9w0BAQEFAASCAQA/SjRb
 31kQdpi0chpuRiXCb7M1XlSNnmibyS2jsV2J4OYUB6Tp08DLl8Z3o93OZ+e56aKTJRW5tALt
 mBbp9dHguyRzkckB5wAJnr03fpPT4M1QcocLiE8f3aByYMhNJO0qEPoB7F3lrat1MrFpQuEU
 rSyHDwqwtCpqNNCSbL2HC5HVNddzeEflrRM0m7wMZEu5r5k0oJdMAt5fBa/jI95cGYVrQXxX
 5c1bdCjazIkCSx/mYEw/iYNuGQguPpuVWTYg6h1qiVu7FDoOv+nV7LaOXkOq/eOHzzeAgQOf
 7VundOC/0h1xLXXqamqMOt9FQezjJAQ6gW4BSEs554isl/MrAAAAAAAA
 --------------ms010705060000090100070101--
>Unformatted:
