From nobody@FreeBSD.org  Sun Oct  7 13:14:43 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 3D26C106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  7 Oct 2012 13:14:43 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 24CE48FC14
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  7 Oct 2012 13:14:43 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id q97DEg8O063427
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 7 Oct 2012 13:14:42 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id q97DEgaO063426;
	Sun, 7 Oct 2012 13:14:42 GMT
	(envelope-from nobody)
Message-Id: <201210071314.q97DEgaO063426@red.freebsd.org>
Date: Sun, 7 Oct 2012 13:14:42 GMT
From: Garrett Cooper <yaneurabeya@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [fetch] Not properly following redirects; causes unnecessary manual intervention with ports installs, etc
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         172451
>Category:       bin
>Synopsis:       [libfetch] Not properly following redirects; causes unnecessary manual intervention with ports installs, etc
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    eadler
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Oct 07 13:20:11 UTC 2012
>Closed-Date:    Tue Oct 30 03:30:48 UTC 2012
>Last-Modified:  Sun Feb 03 22:31:07 UTC 2013
>Originator:     Garrett Cooper
>Release:        9.1-STABLE
>Organization:
EMC Isilon
>Environment:
>Description:
I have a fair amount of issues with fetch/libfetch and HTTP redirects. Some servers do it properly, others don't. A busted example (one of samba.org's HTTP servers) is as follows:

$ fetch -vv -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
scheme:   [http]
user:     []
password: []
host:     [us1.samba.org]
port:     [0]
document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
---> us1.samba.org:80
looking up us1.samba.org
connecting to us1.samba.org:80
requesting http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
>How-To-Repeat:
fetch -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
>Fix:


Patch attached with submission follows:

Index: lib/libfetch/http.c
===================================================================
--- lib/libfetch/http.c	(revision 241309)
+++ lib/libfetch/http.c	(working copy)
@@ -1524,8 +1526,7 @@
 	/* try the provided URL first */
 	url = URL;
 
-	/* if the A flag is set, we only get one try */
-	n = noredirect ? 1 : MAX_REDIRECT;
+	n = MAX_REDIRECT;
 	i = 0;
 
 	e = HTTP_PROTOCOL_ERROR;
@@ -1772,6 +1774,15 @@
 			case hdr_location:
 				if (!HTTP_REDIRECT(conn->err))
 					break;
+				/* 
+				 * Don't redirect due to bad server
+				 * paranoia.
+				 */
+				if (noredirect &&
+				    conn->err == HTTP_MOVED_TEMP) {
+					n = 1;
+					break;
+				}
 				if (new)
 					free(new);
 				if (verbose)


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->eadler 
Responsible-Changed-By: eadler 
Responsible-Changed-When: Sun Oct 7 13:29:20 UTC 2012 
Responsible-Changed-Why:  
I'll take it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=172451 
State-Changed-From-To: open->analyzed 
State-Changed-By: eadler 
State-Changed-When: Thu Oct 11 14:14:09 UTC 2012 
State-Changed-Why:  
awaiting approval / review 

http://www.freebsd.org/cgi/query-pr.cgi?pr=172451 
State-Changed-From-To: analyzed->patched 
State-Changed-By: eadler 
State-Changed-When: Thu Oct 25 23:23:43 UTC 2012 
State-Changed-Why:  
committed in HEAD 

http://www.freebsd.org/cgi/query-pr.cgi?pr=172451 
State-Changed-From-To: patched->closed 
State-Changed-By: eadler 
State-Changed-When: Tue Oct 30 03:30:47 UTC 2012 
State-Changed-Why:  
Committed. Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=172451 
>Unformatted:
 >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
 >>> Host: us1.samba.org
 >>> User-Agent: fetch libfetch/2.0
 >>> Connection: close
 >>>
 <<< HTTP/1.1 301 Moved Permanently
 <<< Date: Sun, 07 Oct 2012 11:14:28 GMT
 <<< Server: Apache
 <<< Location: http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 <<< Vary: Accept-Encoding
 301 redirect to http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 scheme:   [http]
 user:     []
 password: []
 host:     [www.samba.org]
 port:     [0]
 document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
 <<< Content-Length: 331
 <<< Connection: close
 content length: [331]
 <<< Content-Type: text/html; charset=iso-8859-1
 <<<
 fetch: http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz: Moved Permanently
 
 A working example with samba.org (without the -A flag):
 
 $ fetch -vv -Fpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 scheme:   [http]
 user:     []
 password: []
 host:     [us1.samba.org]
 port:     [0]
 document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
 ---> us1.samba.org:80
 looking up us1.samba.org
 connecting to us1.samba.org:80
 requesting http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
 >>> Host: us1.samba.org
 >>> User-Agent: fetch libfetch/2.0
 >>> Connection: close
 >>>
 <<< HTTP/1.1 301 Moved Permanently
 <<< Date: Sun, 07 Oct 2012 11:25:32 GMT
 <<< Server: Apache
 <<< Location: http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 <<< Vary: Accept-Encoding
 301 redirect to http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 scheme:   [http]
 user:     []
 password: []
 host:     [www.samba.org]
 port:     [0]
 document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
 <<< Content-Length: 331
 <<< Connection: close
 content length: [331]
 <<< Content-Type: text/html; charset=iso-8859-1
 <<<
 ---> www.samba.org:80
 looking up www.samba.org
 connecting to www.samba.org:80
 requesting http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
 >>> Host: www.samba.org
 >>> User-Agent: fetch libfetch/2.0
 >>> Connection: close
 >>>
 <<< HTTP/1.1 301 Moved Permanently
 <<< Date: Sun, 07 Oct 2012 11:25:32 GMT
 <<< Server: Apache
 <<< Location: http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
 <<< Vary: Accept-Encoding
 301 redirect to http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
 scheme:   [http]
 user:     []
 password: []
 host:     [ftp.samba.org]
 port:     [0]
 document: [/pub/samba/stable/samba-3.6.8.tar.gz]
 <<< Content-Length: 327
 <<< Connection: close
 content length: [327]
 <<< Content-Type: text/html; charset=iso-8859-1
 <<<
 ---> ftp.samba.org:80
 looking up ftp.samba.org
 connecting to ftp.samba.org:80
 requesting http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
 >>> GET /pub/samba/stable/samba-3.6.8.tar.gz HTTP/1.1
 >>> Host: ftp.samba.org
 >>> User-Agent: fetch libfetch/2.0
 >>> Connection: close
 >>>
 <<< HTTP/1.1 200 OK
 <<< Date: Sun, 07 Oct 2012 11:25:32 GMT
 <<< Server: Apache
 <<< Last-Modified: Mon, 17 Sep 2012 07:13:13 GMT
 <<< ETag: "125001d-207e10e-4c9e081b7b440"
 last modified: [2012-09-17 07:13:13]
 <<< Accept-Ranges: bytes
 <<< Content-Length: 34070798
 <<< Connection: close
 content length: [34070798]
 <<< Content-Type: application/x-gzip
 <<< Content-Encoding: x-gzip
 <<<
 offset 0, length -1, size -1, clength 34070798
 remote size / mtime: 34070798 / 1347865993
 samba-3.6.8.tar.gz                              2% of   32 MB  659 kBps^C
 fetch: transfer interrupted
 
 I think that the issue is that HTTP 3XX codes are blacklisted with the -A flag too liberally -- in particular 301, 302, and 307 should be followed per RFC-2616 [1]. The attached patch does that exactly (and also adds support for HTTP error code 305 -- us e proxy -- although scrubbing that before commit is fine if it should be in another bug).
 
 FWIW that's what the documentation for fetch(1) suggests as well...
 
      -A          Do not automatically follow ``temporary'' (302) redirects.
                  Some broken Web sites will return a redirect instead of a
                  not-found error when the requested object does not exist.
 
 fetch(3) doesn't have anything to say on the matter (and in fact doesn't document all of the supported flags)...
 
 HTTP SCHEME
      The fetchXGetHTTP(), fetchGetHTTP() and fetchPutHTTP() functions imple-
      ment the HTTP/1.1 protocol.  With a little luck, there is even a chance
      that they comply with RFC2616 and RFC2617.
 
      If the `d' (direct) flag is specified, fetchXGetHTTP(), fetchGetHTTP()
      and fetchPutHTTP() will use a direct connection even if a proxy server is
      defined.
 
      If the `i' (if-modified-since) flag is specified, and the ims_time field
      is set in struct url, then fetchXGetHTTP() and fetchGetHTTP() will send a
      conditional If-Modified-Since HTTP header to only fetch the content if it
      is newer than ims_time.
 
      Since there seems to be no good way of implementing the HTTP PUT method
      in a manner consistent with the rest of the fetch library, fetchPutHTTP()
      is currently unimplemented.
 
 After the patch is applied things function as I would expect:
 
 $ fetch -vv -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 scheme:   [http]
 user:     []
 password: []
 host:     [us1.samba.org]
 port:     [0]
 document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
 ---> us1.samba.org:80
 looking up us1.samba.org
 connecting to us1.samba.org:80
 requesting http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
 >>> Host: us1.samba.org
 >>> User-Agent: fetch libfetch/2.0
 >>> Range: bytes=827392-
 >>> Connection: close
 >>>
 <<< HTTP/1.1 301 Moved Permanently
 <<< Date: Sun, 07 Oct 2012 12:36:27 GMT
 <<< Server: Apache
 <<< Location: http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 <<< Vary: Accept-Encoding
 301 redirect to http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 scheme:   [http]
 user:     []
 password: []
 host:     [www.samba.org]
 port:     [0]
 document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
 <<< Content-Length: 331
 <<< Connection: close
 content length: [331]
 <<< Content-Type: text/html; charset=iso-8859-1
 <<<
 ---> www.samba.org:80
 looking up www.samba.org
 connecting to www.samba.org:80
 requesting http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
 >>> Host: www.samba.org
 >>> User-Agent: fetch libfetch/2.0
 >>> Range: bytes=827392-
 >>> Connection: close
 >>>
 <<< HTTP/1.1 301 Moved Permanently
 <<< Date: Sun, 07 Oct 2012 12:36:28 GMT
 <<< Server: Apache
 <<< Location: http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
 <<< Vary: Accept-Encoding
 301 redirect to http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
 scheme:   [http]
 user:     []
 password: []
 host:     [ftp.samba.org]
 port:     [0]
 document: [/pub/samba/stable/samba-3.6.8.tar.gz]
 <<< Content-Length: 327
 <<< Connection: close
 content length: [327]
 <<< Content-Type: text/html; charset=iso-8859-1
 <<<
 ---> ftp.samba.org:80
 looking up ftp.samba.org
 connecting to ftp.samba.org:80
 requesting http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
 >>> GET /pub/samba/stable/samba-3.6.8.tar.gz HTTP/1.1
 >>> Host: ftp.samba.org
 >>> User-Agent: fetch libfetch/2.0
 >>> Range: bytes=827392-
 >>> Connection: close
 >>>
 <<< HTTP/1.1 206 Partial Content
 <<< Date: Sun, 07 Oct 2012 12:36:28 GMT
 <<< Server: Apache
 <<< Last-Modified: Mon, 17 Sep 2012 07:13:13 GMT
 <<< ETag: "125001d-207e10e-4c9e081b7b440"
 last modified: [2012-09-17 07:13:13]
 <<< Accept-Ranges: bytes
 <<< Content-Length: 33243406
 <<< Content-Range: bytes 827392-34070797/34070798
 content length: [33243406]
 <<< Connection: close
 content range: [827392-34070797/34070798]
 <<< Content-Type: application/x-gzip
 <<< Content-Encoding: x-gzip
 <<<
 offset 827392, length 33243406, size 34070798, clength 33243406
 local size / mtime: 827392 / 1347865993
 remote size / mtime: 34070798 / 1347865993
 samba-3.6.8.tar.gz                            100% of   32 MB 2301 kBps 00m00s
 
 And removing the HTTP Error 302 clause I get behavior I would expect from fetch(1):
 
 $ fetch -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
 fetch: http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz: Moved Permanently
 
 References:
 
 1. http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
