From nobody@FreeBSD.org  Tue Jan 24 04:20:30 2006
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 00D9A16A420
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 24 Jan 2006 04:20:30 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 7BBE643D53
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 24 Jan 2006 04:20:29 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k0O4KTGN055779
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 24 Jan 2006 04:20:29 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id k0O4KTVU055778;
	Tue, 24 Jan 2006 04:20:29 GMT
	(envelope-from nobody)
Message-Id: <200601240420.k0O4KTVU055778@www.freebsd.org>
Date: Tue, 24 Jan 2006 04:20:29 GMT
From: David Kelly <dkelly@hiwaay.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: sendfile(2) returns early on files > 4GB
X-Send-Pr-Version: www-2.3

>Number:         92243
>Category:       kern
>Synopsis:       sendfile(2) returns early on files > 4GB
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    yar
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jan 24 04:30:08 GMT 2006
>Closed-Date:    Mon Feb 20 00:58:20 GMT 2006
>Last-Modified:  Mon Feb 20 00:58:20 GMT 2006
>Originator:     David Kelly
>Release:        6.0-STABLE
>Organization:
self
>Environment:
FreeBSD Grumpy.DynDNS.org 6.0-STABLE FreeBSD 6.0-STABLE #9: Tue Jan 17 19:25:47 CST 2006     dkelly@Grumpy.DynDNS.org:/usr5/obj/usr/src/sys/OPUS  i386

>Description:
This is a restatement of bin/89100 as I now believe it is a kernel problem
rather than an application problem.  Am now convinced it is a problem with
either vm or the way sendfile(2) talks to the vm system.  Sendfile(2) works
correctly with a fresh large file. Files older than the last reboot
(possibly filesystem umount/mount will age sufficiently) always fail.
Files newer than the latest mount fail after an unknown amount of activity
on that filesystem.

If file size is > 4G then sendfile(2) returns to caller (wrongly) when
exactly 4G is remaining.  Have not experimented with 8G files.

>How-To-Repeat:
Get a file which is larger than 4G via ftpd, apache, or anything else
which uses sendfile(2).

The file must have aged a bit on the filesystem. A reboot is a sure way
to age the file enough.  Sendfile(2) works perfectly with a new file or
fresh copy.

ftp to localhost and get a big file to /dev/null to quickly reproduce
the problem.
>Fix:
If ftpd and ftp client work well together then a "reget" will complete
the truncated download. Not a fix.

If enough space exists on the filesystem a fresh copy of the file will
transfer correctly, yet the original still will not transfer completely.
>Release-Note:
>Audit-Trail:

From: "David G. Lawrence" <dg@dglawrence.com>
To: David Kelly <dkelly@hiwaay.net>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB
Date: Mon, 23 Jan 2006 23:09:08 -0800

 > >How-To-Repeat:
 > Get a file which is larger than 4G via ftpd, apache, or anything else which uses sendfile(2).
 
    Do you know if sendfile(2) is returning an error in the failure case?
 
 -DG
 
 David G. Lawrence
 President
 Download Technologies, Inc. - http://www.downloadtech.com - (866) 399 8500
 The FreeBSD Project - http://www.freebsd.org
 Pave the road of life with opportunities.

From: David Kelly <dkelly@hiwaay.net>
To: "David G. Lawrence" <dg@dglawrence.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB
Date: Tue, 24 Jan 2006 10:29:13 -0600

 On Mon, Jan 23, 2006 at 11:09:08PM -0800, David G. Lawrence wrote:
 > > >How-To-Repeat:
 > > Get a file which is larger than 4G via ftpd, apache, or anything else which uses sendfile(2).
 > 
 >    Do you know if sendfile(2) is returning an error in the failure case?
 
 I don't know the method for attaching gdb to a running daemon, have
 never had need of it before. Have sprinkled printf's in ftpd and edited
 the return message trying to understand what is happening. I think
 sendfile is returning once and ftpd is calling it again then the 2nd
 time the count is zero so ftp returns "premature end." This is untouched
 stock RELENG6:
 
 230 User dkelly logged in.
 Remote system type is UNIX.
 Using binary mode to transfer files.
 ftp> get test/bigfile /dev/null
 local: /dev/null remote: test/bigfile
 229 Entering Extended Passive Mode (|||50786|)
 150 Opening BINARY mode data connection for 'test/bigfile' (4556820480 bytes).
 226 Transfer finished due to premature end of file.
 261853184 bytes received in 00:05 (48.76 MB/s)
 ftp>
 
 It always leaves exactly 4GB behind:
 
 dkelly@Grumpy {1009} dc
 16 o
 4556820480 261853184 - p
 100000000
 
 If I copy bigfile:
 
 dkelly@Grumpy {1010} cp -p test/bigfile test/big2
 
 Then the ftpd/sendfile works for the next few hours or days or until
 reboot. Doesn't matter if the file is hosted on my simple one-disk
 filesystem or my 2-disk gvinum striped filesystem.
 
 dkelly@Grumpy {1011} ftp localhost
 Trying ::1...
 Connected to localhost.home.
 220 Grumpy.DynDNS.org FTP server (Version 6.00LS) ready.
 Name (localhost:dkelly):
 331 Password required for dkelly.
 Password:
 230 User dkelly logged in.
 Remote system type is UNIX.
 Using binary mode to transfer files.
 ftp> get test/big2 /dev/null
 local: /dev/null remote: test/big2
 229 Entering Extended Passive Mode (|||50787|)
 150 Opening BINARY mode data connection for 'test/big2' (4556820480 bytes).
 226 Transfer complete.
 4556820480 bytes received in 02:11 (32.94 MB/s)
 ftp>
 
 
 -- 
 David Kelly N4HHE, dkelly@HiWAAY.net
 ========================================================================
 Whom computers would destroy, they must first drive mad.

From: "David G. Lawrence" <dg@dglawrence.com>
To: David Kelly <dkelly@hiwaay.net>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB
Date: Thu, 26 Jan 2006 11:34:58 -0800

 > On Mon, Jan 23, 2006 at 11:09:08PM -0800, David G. Lawrence wrote:
 > > > >How-To-Repeat:
 > > > Get a file which is larger than 4G via ftpd, apache, or anything else which uses sendfile(2).
 > > 
 > >    Do you know if sendfile(2) is returning an error in the failure case?
 > 
 > I don't know the method for attaching gdb to a running daemon, have
 > never had need of it before. Have sprinkled printf's in ftpd and edited
 > the return message trying to understand what is happening. I think
 > sendfile is returning once and ftpd is calling it again then the 2nd
 > time the count is zero so ftp returns "premature end." This is untouched
 > stock RELENG6:
 
    I've been trying to reproduce the problem here, but I haven't been able
 to so far. It might be because I'm using a larger file or perhaps it has
 to do with the system configuration. Can you tell me how much main memory
 you have in your system?
    Whatever the problem is, it sure isn't obvious in the code. I've spent
 many hours combing through sendfile and the macros and functions that
 it calls, looking for anything that might cause a 64bit->32bit precision
 truncation, and so far everything looks fine to me.
 
 -DG
 
 David G. Lawrence
 President
 Download Technologies, Inc. - http://www.downloadtech.com - (866) 399 8500
 The FreeBSD Project - http://www.freebsd.org
 Pave the road of life with opportunities.

From: Yar Tikhiy <yar@comp.chem.msu.su>
To: bug-followup@FreeBSD.org, dkelly@hiwaay.net,
        "David G. Lawrence" <dg@dglawrence.com>
Cc:  
Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB
Date: Mon, 30 Jan 2006 16:12:45 +0300

 > From: "David G. Lawrence" <dg@dglawrence.com>
 > To: David Kelly <dkelly@hiwaay.net>
 > Cc: freebsd-gnats-submit@FreeBSD.org
 > Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB
 > Date: Mon, 23 Jan 2006 23:09:08 -0800
 > 
 >  > >How-To-Repeat:
 >  > Get a file which is larger than 4G via ftpd, apache, or anything else which uses sendfile(2).
 >  
 >     Do you know if sendfile(2) is returning an error in the failure case?
 
 It was me who added the code about "premature end of file" to ftpd(8).
 The code is to be hit only when sendfile(2) reports a EOF condition
 when not all file has been sent according to the byte count.
 That is, the bug should manifest itself in sendfile(2) storing 0 to
 *sbytes and returning 0 when ($filesize - 2^32) bytes have been sent.
 No errno involved.
 
 Yar

From: Yar Tikhiy <yar@comp.chem.msu.su>
To: bug-followup@FreeBSD.org, dkelly@hiwaay.net
Cc: "David G. Lawrence" <dg@dglawrence.com>, alc@FreeBSD.org
Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB
Date: Tue, 31 Jan 2006 15:49:48 +0300

 Just experimented with the issue.  It appears easily reproducable
 on a local filesystem.  The problem boils down to the fact that in
 the following code:
 
    sys/kern/uipc_syscalls.c:do_sendfile()
    1864:                /*
    1865:                 * Calculate the amount to transfer. Not to exceed a page,
    1866:                 * the EOF, or the passed in nbytes.
    1867:                 */
    1868:                xfsize = obj->un_pager.vnp.vnp_size - off;
 
 obj->un_pager.vnp.vnp_size is somehow truncated to 32 bits when the
 vnode comes from disk.  I added a printf after line 1868 and I saw
 that for a recently created file, 2^32+10000 bytes in size,
 obj->un_pager.vnp.vnp_size was the full size of the file.  The
 file's vnode was presumably still cached.  However, after a reboot
 obj->un_pager.vnp.vnp_size for the same file was just 10000.
 
    # truncate -s 4294977296 ~ftp/4g
    # ftp ftp://127.1/4g
    obj->un_pager.vnp.vnp_size = 4294977296
    # reboot
    ...
    # ftp ftp://127.1/4g
    obj->un_pager.vnp.vnp_size = 10000
    # rm ~ftp/4g
    # truncate -s 4294977296 ~ftp/4g
    # ftp ftp://127.1/4g
    obj->un_pager.vnp.vnp_size = 4294977296
 
 -- 
 Yar

From: Yar Tikhiy <yar@comp.chem.msu.su>
To: bug-followup@FreeBSD.org, dkelly@hiwaay.net
Cc: "David G. Lawrence" <dg@dglawrence.com>, alc@FreeBSD.org, phk@FreeBSD.org
Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB
Date: Tue, 31 Jan 2006 17:05:03 +0300

 On Tue, Jan 31, 2006 at 03:49:48PM +0300, Yar Tikhiy wrote:
 > Just experimented with the issue.  It appears easily reproducable
 > on a local filesystem.  The problem boils down to the fact that in
 > the following code:
 > 
 >    sys/kern/uipc_syscalls.c:do_sendfile()
 >    1864:                /*
 >    1865:                 * Calculate the amount to transfer. Not to exceed a page,
 >    1866:                 * the EOF, or the passed in nbytes.
 >    1867:                 */
 >    1868:                xfsize = obj->un_pager.vnp.vnp_size - off;
 > 
 > obj->un_pager.vnp.vnp_size is somehow truncated to 32 bits when the
 > vnode comes from disk.  I added a printf after line 1868 and I saw
 > that for a recently created file, 2^32+10000 bytes in size,
 > obj->un_pager.vnp.vnp_size was the full size of the file.  The
 > file's vnode was presumably still cached.  However, after a reboot
 > obj->un_pager.vnp.vnp_size for the same file was just 10000.
 
 Quite curiously, changing the type of the size argument to
 vnode_create_vobject() from size_t to off_t seems to remedy the
 problem in my case.  Let's ask Poul-Henning, who introduced the
 kernel function a year ago, what he thinks about this -- adding him
 to CC.
 
 The proposed patch is attached below.
 
 -- 
 Yar
 
 Index: sys/vnode.h
 ===================================================================
 RCS file: /home/ncvs/src/sys/sys/vnode.h,v
 retrieving revision 1.312
 diff -u -p -r1.312 vnode.h
 --- sys/vnode.h	15 Jan 2006 02:01:51 -0000	1.312
 +++ sys/vnode.h	31 Jan 2006 13:58:29 -0000
 @@ -700,7 +700,7 @@ void	vref(struct vnode *vp);
  int	vrefcnt(struct vnode *vp);
  void 	v_addpollinfo(struct vnode *vp);
  
 -int vnode_create_vobject(struct vnode *vp, size_t size, struct thread *td);
 +int vnode_create_vobject(struct vnode *vp, off_t size, struct thread *td);
  void vnode_destroy_vobject(struct vnode *vp);
  
  extern struct vop_vector fifo_specops;
 Index: vm/vnode_pager.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/vm/vnode_pager.c,v
 retrieving revision 1.224
 diff -u -p -r1.224 vnode_pager.c
 --- vm/vnode_pager.c	1 Nov 2005 23:00:24 -0000	1.224
 +++ vm/vnode_pager.c	31 Jan 2006 13:58:30 -0000
 @@ -97,7 +97,7 @@ int vnode_pbuf_freecnt;
  
  /* Create the VM system backing object for this vnode */
  int
 -vnode_create_vobject(struct vnode *vp, size_t isize, struct thread *td)
 +vnode_create_vobject(struct vnode *vp, off_t isize, struct thread *td)
  {
  	vm_object_t object;
  	vm_ooffset_t size = isize;
State-Changed-From-To: open->analyzed 
State-Changed-By: yar 
State-Changed-When: Tue Jan 31 14:58:33 UTC 2006 
State-Changed-Why:  
According to my investigation, the problem results from 
the size argument to vnode_create_vobject() using too small type. 
E.g., on i386 neither off_t nor vm_ooffset_t can fit in a size_t. 


Responsible-Changed-From-To: freebsd-bugs->yar 
Responsible-Changed-By: yar 
Responsible-Changed-When: Tue Jan 31 14:58:33 UTC 2006 
Responsible-Changed-Why:  
I've got involved in fixing this bug. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=92243 

From: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
To: Yar Tikhiy <yar@comp.chem.msu.su>
Cc: freebsd-bugs@freebsd.org
Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB 
Date: Tue, 31 Jan 2006 14:16:00 +0000

 In message <200601311410.k0VEAASn057130@freefall.freebsd.org>, Yar Tikhiy write
 s:
 
 > Quite curiously, changing the type of the size argument to
 > vnode_create_vobject() from size_t to off_t seems to remedy the
 > problem in my case.  Let's ask Poul-Henning, who introduced the
 > kernel function a year ago, what he thinks about this -- adding him
 > to CC.
 
 Oversight on my part.
 
 -- 
 Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 phk@FreeBSD.ORG         | TCP/IP since RFC 956
 FreeBSD committer       | BSD since 4.3-tahoe    
 Never attribute to malice what can adequately be explained by incompetence.

From: Yar Tikhiy <yar@comp.chem.msu.su>
To: Poul-Henning Kamp <phk@phk.freebsd.dk>
Cc: freebsd-bugs@freebsd.org, alc@freebsd.org
Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB
Date: Tue, 31 Jan 2006 17:36:48 +0300

 On Tue, Jan 31, 2006 at 02:16:00PM +0000, Poul-Henning Kamp wrote:
 > In message <200601311410.k0VEAASn057130@freefall.freebsd.org>, Yar Tikhiy write
 > s:
 > 
 > > Quite curiously, changing the type of the size argument to
 > > vnode_create_vobject() from size_t to off_t seems to remedy the
 > > problem in my case.  Let's ask Poul-Henning, who introduced the
 > > kernel function a year ago, what he thinks about this -- adding him
 > > to CC.
 > 
 > Oversight on my part.
 
 My last concern is whether off_t is correct there, or it should be
 vm_ooffset_t.  I.e., should the size argument represent a file size
 or VM object size?  As a matter of fact, the size argument is
 transformed into vm_ooffset_t inside vnode_create_vobject(), so we
 can safely decide that vnode_create_vobject() is on the vnode side
 of the vnode-VM interface.  Thanks!
 
 -- 
 Yar
State-Changed-From-To: analyzed->patched 
State-Changed-By: yar 
State-Changed-When: Sat Feb 4 11:44:41 UTC 2006 
State-Changed-Why:  
Fix has been committed to CURRENT.  MFC is due in a few days. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=92243 

From: Yar Tikhiy <yar@comp.chem.msu.su>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/92243: sendfile(2) returns early on files > 4GB
Date: Sat, 4 Feb 2006 14:44:21 +0300

  ----- Forwarded message from Iassen Anadoliev <korio@korio.org> -----
  From: Iassen Anadoliev <korio@korio.org>
  To: Yar Tikhiy <yar@comp.chem.msu.su>
  Subject: Re: Ftpd problems
  Date: Thu, 02 Feb 2006 16:56:26 +0200
  Cc: freebsd-net@freebsd.org
  
  [...]
  Thanks for the patch it seems to work:
  
  fetch  ftp://test:test@ibiza.korio.org/usr/src/test.zero
  test.zero                                     100% of 4293 MB 4255 kBps 
  00m00s
  [...]
  ----- End forwarded message -----
State-Changed-From-To: patched->closed 
State-Changed-By: yar 
State-Changed-When: Mon Feb 20 00:57:33 UTC 2006 
State-Changed-Why:  
The bug has also been fixed in RELENG_6.  Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=92243 
>Unformatted:
