From nobody@FreeBSD.ORG  Mon Aug  7 15:01:48 2000
Return-Path: <nobody@FreeBSD.ORG>
Received: by hub.freebsd.org (Postfix, from userid 32767)
	id 2275D37BD15; Mon,  7 Aug 2000 15:01:47 -0700 (PDT)
Message-Id: <20000807220147.2275D37BD15@hub.freebsd.org>
Date: Mon,  7 Aug 2000 15:01:47 -0700 (PDT)
From: malachai@iname.com
Sender: nobody@FreeBSD.ORG
To: freebsd-gnats-submit@FreeBSD.org
Subject: fetch(1) doesn't handle retrievals of unspecified length very well if FTP_PASSIVE_MODE is set
X-Send-Pr-Version: www-1.0

>Number:         20471
>Category:       bin
>Synopsis:       fetch(1) doesn't handle retrievals of unspecified length very well if FTP_PASSIVE_MODE is set
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    des
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 07 15:10:00 PDT 2000
>Closed-Date:    Wed Aug 30 02:02:55 PDT 2000
>Last-Modified:  Wed Aug 30 02:06:28 PDT 2000
>Originator:     Shawn Halpenny
>Release:        4.1-STABLE
>Organization:
>Environment:
FreeBSD nightrain 4.1-STABLE FreeBSD 4.1-STABLE #0: Sat Aug  5 18:39:47 EDT 2000     root@nightrain.humb.nt.com:/usr/src/sys/compile/NIGHTRAIN  i386

HTTP_PROXY=proxy:8080
FTP_PASSIVE_MODE=yes  # this is set by the default login.conf
>Description:
When using fetch(1) to retrieve a file for which the server does not
specify the file's length, fetch reports retrieving 2^32-1 bytes.  It
will properly retrieve the file, but on completion it thinks the file
was truncated and then deletes it.

Note that unsetting FTP_PASSIVE_MODE causes fetch to work as expected.
>How-To-Repeat:
First noticed when fetching packages from ftp.freebsd.org:

% fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/All/9e-1.0.tgz
Receiving 9e-1.0.tgz (4294967295 bytes): 0%
4343 bytes transferred in 3.4 seconds (1.25 kBps)
fetch: 9e-1.0.tgz appears to be truncated: 4343/4294967295 bytes

% fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/All/amaterus-0.32.5.tgz
Receiving amaterus-0.32.5.tgz (4294967295 bytes): 0%
30558 bytes transferred in 10.2 seconds (2.93 kBps)
fetch: amaterus-0.32.5.tgz appears to be truncated: 30558/4294967295 bytes

% unset FTP_PASSIVE_MODE
% fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/All/amaterus-0.32.5.tgz
Receiving amaterus-0.32.5.tgz (30558 bytes): 100%
30558 bytes transferred in 10.2 seconds (2.93 kBps)
>Fix:
Workaround:  make sure FTP_PASSIVE_MODE is not defined.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->des 
Responsible-Changed-By: sheldonh 
Responsible-Changed-When: Tue Aug 8 01:51:03 PDT 2000 
Responsible-Changed-Why:  
Over to the maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=20471 

From: Jeff Eaton <jeffe@ichips.intel.com>
To: freebsd-gnats-submit@FreeBSD.org, malachai@iname.com
Cc: jeffe@ichips.intel.com
Subject: Re: bin/20471: fetch(1) doesn't handle retrievals of unspecified length very well if FTP_PASSIVE_MODE is set
Date: Fri, 25 Aug 2000 17:42:15 -0700

 I have also seen this problem under 4.1-RELEASE
 
 Debugging it, I have found that there is a problem in libfetch/http.c
 
 The function _http_request declares the local variables clength, length,
 
 and size to be of type of type size_t, which is an unsigned 32 bit int.
 It then
 immediately initializes them to -1, which will set them to 2^32-1.  If
 no length or size
 is ever gotten, then these remain with this value.  Ultimately, size is
 copied into us->size (struct url_stat), which is of type off_t, which is
 
 a signed 64 bit int, and the -1 meaning is then totally lost.  The main
 fetch program does check for -1, but will never see it in this case.
 
 The fix would be to either to change those three variables to type
 ssize_t
 (signed 32 bit int), and make everything that touches them match (a bit
 of work), do the same thing, but change them to off_t (may be best), or
 to
 get rid of the '-1's  and replace them with SIZE_T_MAX,
 and explicitly check for that when assigning  us->size,  setting it
 directly
 to -1 if so.
 
 I have not checked to see if there are other places this might happen.
 
 Jeff Eaton
 
 

From: Jeff Eaton <jeffe@ichips.intel.com>
To: freebsd-gnats-submit@FreeBSD.org, malachai@iname.com
Cc:  
Subject: Re: bin/20471: fetch(1) doesn't handle retrievals of unspecified length very well if FTP_PASSIVE_MODE is set
Date: Fri, 25 Aug 2000 18:23:17 -0700 (PDT)

 OK, it wasn't that hard to fix.  I have appended a patch that
 fixes the problem.  I didn't see this problem anywhere else in
 the libfetch library.
 
 I have tested this patch on cases that failed before, and it now
 works.  I tested a few other cases, but I'm stuck behind a proxy,
 and can't test very many cases.  It doesn't seem to have broken
 anything.
 
 Note that in my case, setting or unsetting FTP_PASSIVE_MODE didn't
 help.  When it is using HTTP_PROXY, it should never even look at
 that variable.  Shawn should test this in his case, and see if
 this fixes his problem or not.
 
 Jeff Eaton
 
 
 --- /usr/src/lib/libfetch/http.c	Wed Jul 26 00:54:18 2000
 +++ /tmp/libfetch/http.c	Fri Aug 25 17:49:12 2000
 @@ -444,13 +444,13 @@
   * Parse a content-length header
   */
  static int
 -_http_parse_length(char *p, size_t *length)
 +_http_parse_length(char *p, off_t *length)
  {
 -    size_t len;
 +    off_t len;
      
      for (len = 0; *p && isdigit(*p); ++p)
  	len = len * 10 + (*p - '0');
 -    DEBUG(fprintf(stderr, "content length: [\033[1m%d\033[m]\n", len));
 +    DEBUG(fprintf(stderr, "content length: [\033[1m%lld\033[m]\n", len));
      *length = len;
      return 0;
  }
 @@ -459,9 +459,9 @@
   * Parse a content-range header
   */
  static int
 -_http_parse_range(char *p, off_t *offset, size_t *length, size_t *size)
 +_http_parse_range(char *p, off_t *offset, off_t *length, off_t *size)
  {
 -    int first, last, len;
 +    off_t first, last, len;
  
      if (strncasecmp(p, "bytes ", 6) != 0)
  	return -1;
 @@ -477,7 +477,7 @@
  	len = len * 10 + *p - '0';
      if (len < last - first + 1)
  	return -1;
 -    DEBUG(fprintf(stderr, "content range: [\033[1m%d-%d/%d\033[m]\n",
 +    DEBUG(fprintf(stderr, "content range: [\033[1m%lld-%lld/%lld\033[m]\n",
  		  first, last, len));
      *offset = first;
      *length = last - first + 1;
 @@ -771,7 +771,7 @@
      int chunked, need_auth, noredirect, proxy, verbose;
      int code, fd, i, n;
      off_t offset;
 -    size_t clength, length, size;
 +    off_t clength, length, size;
      time_t mtime;
      char *p;
      FILE *f;
 @@ -972,7 +972,7 @@
  	goto ouch;
      }
  
 -    DEBUG(fprintf(stderr, "offset: %lld, length: %d, size: %d, clength: %d\n",
 +    DEBUG(fprintf(stderr, "offset: %lld, length: %lld, size: %lld, clength: %lld\n",
  		  offset, length, size, clength));
      
      /* check for inconsistencies */
 
State-Changed-From-To: open->closed 
State-Changed-By: des 
State-Changed-When: Wed Aug 30 02:02:55 PDT 2000 
State-Changed-Why:  
This was fixed five days before the PR was sent. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=20471 
>Unformatted:
