From nobody@FreeBSD.org  Thu Apr 21 02:01:17 2005
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id C7EF816A4CE
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 21 Apr 2005 02:01:17 +0000 (GMT)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 98DE643D41
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 21 Apr 2005 02:01:17 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id j3L21H7Z007319
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 21 Apr 2005 02:01:17 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id j3L21HTg007318;
	Thu, 21 Apr 2005 02:01:17 GMT
	(envelope-from nobody)
Message-Id: <200504210201.j3L21HTg007318@www.freebsd.org>
Date: Thu, 21 Apr 2005 02:01:17 GMT
From: michael johnson <ahze@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: fetch does not always work with https with a proxy.
X-Send-Pr-Version: www-2.3

>Number:         80176
>Category:       bin
>Synopsis:       fetch does not always work with https with a proxy.
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    des
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Apr 21 02:10:27 GMT 2005
>Closed-Date:    Tue May 28 11:56:54 UTC 2013
>Last-Modified:  Tue May 28 11:56:54 UTC 2013
>Originator:     michael johnson
>Release:        
>Organization:
>Environment:
Can reproduce on 4.x, 5.x and 6.x
>Description:
When fetch is used with a HTTP_PROXY=XXX FTP_PROXY=XXX it does not always work as it should, And wget always works. 

But this is only true for only a few sites, most noteably https://helixcommunity.org

This is what I get from a distfile from https://helixcommunity.org

# fetch https://helixcommunity.org/download.php/1145/RealPlayer-10.0.4.750-20050401.i586.rpm
RealPlayer-10.0.4.750-20050401.i586.rpm                  0  B    0  Bps
#  du -sh RealPlayer-10.0.4.750-20050401.i586.rpm
  0B    RealPlayer-10.0.4.750-20050401.i586.rpm
# rm RealPlayer-10.0.4.750-20050401.i586.rpm
# wget https://helixcommunity.org/download.php/1145/RealPlayer-10.0.4.750-20050401.i586.rpm--21:51:50--  https://helixcommunity.org/download.php/1145/RealPlayer-10.0.4.750-20050401.i586.rpm
           => `RealPlayer-10.0.4.750-20050401.i586.rpm'
Resolving helixcommunity.org... done.
Connecting to helixcommunity.org[207.188.25.135]:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6,642,470 [application/binary]

100%[================================================>] 6,642,470    458.47K/s    ETA 00:00

21:52:09 (458.47 KB/s) - `RealPlayer-10.0.4.750-20050401.i586.rpm' saved [6642470/6642470]
# du -sh RealPlayer-10.0.4.750-20050401.i586.rpm
  6.4M    RealPlayer-10.0.4.750-20050401.i586.rpm
# unset HTTP_PROXY
# rm RealPlayer-10.0.4.750-20050401.i586.rpm
# fetch https://helixcommunity.org/download.php/1145/RealPlayer-10.0.4.750-20050401.i586.rpmRealPlayer-10.0.4.750-20050401.i586.rpm       100% of 6486 kB  454 kBps 00m00s

This is not always true, sites such as https://gna.org and others work great.


>How-To-Repeat:
Try to 'fetch' something with HTTP_PROXY set with squid or similar from
https://helixcommunity.org
>Fix:
      
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->des 
Responsible-Changed-By: kris 
Responsible-Changed-When: Sat Apr 23 20:14:36 GMT 2005 
Responsible-Changed-Why:  
Assign to fetch maintainer 

http://www.freebsd.org/cgi/query-pr.cgi?pr=80176 
State-Changed-From-To: open->closed 
State-Changed-By: des 
State-Changed-When: Fri May 13 06:52:21 UTC 2011 
State-Changed-Why:  
unable to reproduce 

http://www.freebsd.org/cgi/query-pr.cgi?pr=80176 
State-Changed-From-To: closed->open 
State-Changed-By: edwin 
State-Changed-When: Tue Jan 17 23:01:46 UTC 2012 
State-Changed-Why:  
Easy to reproduce here: 

The proxyserver doesn't have anything listening on port 1234, so 
you assume directly a Connection Refused but you see it getting the 
https://www.ibm.com/ first: 

[edwin@freebsd90 ~]$ HTTP_PROXY=quartz.mavetju.org:1234 fetch -vvv https://www.ibm.com/ 
scheme:   [https] 
user:     [] 
password: [] 
host:     [www.ibm.com] 
port:     [0] 
document: [/] 
scheme:   [] 
user:     [] 
password: [] 
host:     [quartz.mavetju.org] 
port:     [1234] 
document: [/] 
---> www.ibm.com:443 
looking up www.ibm.com 
connecting to www.ibm.com:443 
SSL connection established using RC4-MD5 
Certificate subject: /serialNumber=Fhvd-30zZHPy-3GMAv-g6FpMEeWDl06I/C=US/ST=North Carolina/L=Research Triangle Park/O=IBM/CN=www.ibm.com 
Certificate issuer: /C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA 
requesting https://www.ibm.com/ 

From: Peter Jeremy <peter.jeremy@ALCATEL-LUCENT.COM>
To: bug-followup@FreeBSD.org, ahze@FreeBSD.org
Cc:  
Subject: Re: bin/80176: fetch does not always work with https with a proxy
Date: Wed, 18 Jan 2012 11:29:05 +1100

 --7JfCtLOvnd9MIVvH
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 I can verify this problem still exists on 8.2-STABLE.  Using ahze@'s
 original helixcommunity.org test URL, I can successfully download
 RealPlayer-10.0.4.750-20050401.i586.rpm from a system with direct
 Internet access. When I try it from a system that must use a proxy,
 I get:
 $ ktrace fetch https://helixcommunity.org/download.php/1145/RealPlayer-10.0=
 =2E4.750-20050401.i586.rpm
 fetch: https://helixcommunity.org/download.php/1145/RealPlayer-10.0.4.750-2=
 0050401.i586.rpm: Operation timed out
 
 The ktrace shows a DNS lookup of helixcommunity.org, followed by a
 connect() that times out:
  79538 fetch    CALL  socket(PF_INET,SOCK_STREAM,IPPROTO_TCP)
  79538 fetch    RET   socket 3
  79538 fetch    CALL  connect(0x3,0x8010280f0,0x10)
  79538 fetch    STRU  struct sockaddr { AF_INET, 207.188.25.135:443 }
  79538 fetch    RET   connect -1 errno 60 Operation timed out
  79538 fetch    CALL  close(0x3)
 
 It should be connecting to a proxy at a 139.188.x.x.  I can successfully
 download HTTP URLs.
 
 I have traced through the code and proxying is explicitly disabled for
 HTTPS URLs in src/lib/libfetch/http.c:http_connect() - which contains:
 =2E..
         if (purl && strcasecmp(URL->scheme, SCHEME_HTTPS) !=3D 0) {
                 URL =3D purl;
         } else if (strcasecmp(URL->scheme, SCHEME_FTP) =3D=3D 0) {
                 /* can't talk http to an ftp server */
                 /* XXX should set an error code */
                 return (NULL);
         }
 
         if ((conn =3D fetch_connect(URL->host, URL->port, af, verbose)) =3D=
 =3D NULL)
 =2E..
 At the start of this fragment, 'URL' is the broken down URL to fetch
 and 'purl' is the broken down URL of the proxy (or NULL if no proxy).
 For a HTTPS URL, the first strcasecmp() returns 0, skipping the
 following assignment and thus ignoring the proxy request.
 
 fetch_connect() directly connects to the specified host (it contains
 no proxy magic).
 
 This code has not changed in 10-current therefore I expect that fetch(1)
 (and other libfetch consumers) remain unable to proxy https requests.
 
 --=20
 Peter Jeremy
 
 --7JfCtLOvnd9MIVvH
 Content-Type: application/pgp-signature
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iEYEARECAAYFAk8WElEACgkQ/opHv/APuIcjKwCfaN964wVRlWffXkCWwvjr8CZ1
 CNEAoIBboqWmOSiOgY0HoaSVx758Uuwi
 =8EQm
 -----END PGP SIGNATURE-----
 
 --7JfCtLOvnd9MIVvH--

From: Peter Jeremy <peter.jeremy@alcatel-lucent.com>
To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= <des@des.no>
Cc: bug-followup@FreeBSD.org
Subject: Re: bin/80176: fetch does not always work with https with a proxy
Date: Thu, 19 Jan 2012 11:03:07 +1100

 On 2012-Jan-18 20:25:10 +1100, Dag-Erling Smrgrav <des@des.no> wrote:
 >HTTPS cannot be proxied.  It can only be tunneled.
 
 That's being pedantic.  If you look at curl, firefox or lynx, they all
 talk about "proxying HTTPS" or "proxy servers", when they mean
 "tunnelling HTTPS through a firewall gateway".
 
 >If you can provide documentation describing how this is supposed to
 >work, I can try to come up with a patch, but I have no way of testing
 >it.
 
 RFC2817 defines using CONNECT to request a tunnel through a proxy.
 
 The user documentation in curl(1) is:
        HTTPS_PROXY [protocol://]<host>[:port]
               Sets the proxy server to use for HTTPS.
 
 wget(1) just talks about "appropriate *_proxy environment variable".
 
 Currently, libfetch includes code to parse HTTP_PROXY as a HTTPS
 proxy, the resultant purl is just ignored in http_connect().  I would
 suggest that if a proxy is implied[*] for a HTTPS URL then
 http_connect() should setup a suitable tunnel by connect(2)ing to the
 proxy and issuing a CONNECT to the actual site.
 
 The code in wget is fairly easy to find (since it's all inside USE_SSL
 blocks) but is GPL3.  The code in curl (MIT licensed) can be found
 in lib/http.c:Curl_proxyCONNECT().  There also appears to be suitable
 code in contrib/netcat/socks.c:socks_connect() (when socksv == -1).
 
 I am happy to test patches and can suggest other people as well.
 With HTTPS-only sites (like github) becoming more popular, I suspect
 more people stucx behind proxies are going to want this functionality.
 
 [*] The way to imply this is non-standard.  libfetch uses HTTP_PROXY
 or http_proxy (HTTP and HTTPS requests both wind up in fetchXGetHTTP()
 which uses http_get_proxy()).  curl uses HTTPS_PROXY or ALL_PROXY.
 wget and lynx use https_proxy.
 
 -- 
 Peter Jeremy

From: =?iso-2022-jp?B?GyRCRmJGIxsoQiAbJEJNNDBsTzobKEI=?= <naito.yuichiro@gmail.com>
To: bug-followup@FreeBSD.org,
 ahze@FreeBSD.org
Cc:  
Subject: Re: bin/80176: fetch does not always work with https with a proxy.
Date: Tue, 17 Apr 2012 13:52:50 +0900

 --Apple-Mail=_368EE69C-200F-44AB-9684-62C1A35BB5A0
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=us-ascii
 
 I made a patch of libfetch.
 Could you try this patch?
 
 This patch supports https connection through http proxy.
 But Proxy Authentication is not supported.
 
 Please apply in src/lib/libfetch directory.
 
 -- 
 Yuichiro NAITO
   naito.yuichiro@gmail.com
 
 
 
 --Apple-Mail=_368EE69C-200F-44AB-9684-62C1A35BB5A0
 Content-Disposition: attachment;
 	filename=libfetch.diff
 Content-Type: application/octet-stream;
 	name="libfetch.diff"
 Content-Transfer-Encoding: 7bit
 
 Index: http.c
 ===================================================================
 --- http.c	(revision 233503)
 +++ http.c	(working copy)
 @@ -1356,6 +1356,7 @@
  http_connect(struct url *URL, struct url *purl, const char *flags)
  {
  	conn_t *conn;
 +	struct url *connect_url;
  	int verbose;
  	int af, val;
  
 @@ -1373,25 +1374,40 @@
  		af = AF_INET6;
  #endif
  
 -	if (purl && strcasecmp(URL->scheme, SCHEME_HTTPS) != 0) {
 -		URL = purl;
 -	} else if (strcasecmp(URL->scheme, SCHEME_FTP) == 0) {
 +	if (strcasecmp(URL->scheme, SCHEME_FTP) == 0) {
  		/* can't talk http to an ftp server */
  		/* XXX should set an error code */
  		return (NULL);
  	}
 +	if (purl)
 +		connect_url = purl;
 +	else
 +		connect_url = URL;
  
 -	if ((conn = fetch_connect(URL->host, URL->port, af, verbose)) == NULL)
 +	if ((conn = fetch_connect(connect_url->host,
 +				  connect_url->port, af, verbose)) == NULL) {
  		/* fetch_connect() has already set an error code */
  		return (NULL);
 -	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 &&
 -	    fetch_ssl(conn, verbose) == -1) {
 -		fetch_close(conn);
 -		/* grrr */
 -		errno = EAUTH;
 -		fetch_syserr();
 -		return (NULL);
  	}
 +	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0) {
 +		if (purl) {
 +			http_cmd(conn, "CONNECT %s:%d HTTP/1.1",
 +				 URL->host, URL->port);
 +			http_cmd(conn, "");
 +			if (http_get_reply(conn) != HTTP_OK) {
 +				fetch_close(conn);
 +				return (NULL);
 +			}
 +			http_get_reply(conn);
 +		}
 +		if (fetch_ssl(conn, verbose) == -1) {
 +			fetch_close(conn);
 +			/* grrr */
 +			errno = EAUTH;
 +			fetch_syserr();
 +			return (NULL);
 +		}
 +	}
  
  	val = 1;
  	setsockopt(conn->sd, IPPROTO_TCP, TCP_NOPUSH, &val, sizeof(val));
 
 --Apple-Mail=_368EE69C-200F-44AB-9684-62C1A35BB5A0--

From: Juergen Lock <nox@jelal.kn-bremen.de>
To: bug-followup@freebsd.org
Cc: ahze@freebsd.org, 4721@hushmail.com, peterj@freebsd.org, des@freebsd.org,
        naito.yuichiro@gmail.com
Subject: Re: bin/80176: fetch does not always work with https with a proxy.
Date: Wed, 26 Dec 2012 21:23:46 +0100

 --IJpNTDwzlM2Ie8A6
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 I was asked to submit this patch for frogs who has problems sending
 via email, he said he fixed the patch by naito.yuichiro so it doesn't
 break ftp requests through proxy.  He says he tested https and ftp
 and http requests with and without proxy and they worked.
 
 --IJpNTDwzlM2Ie8A6
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename="libfetch.txt"
 
 Index: lib/libfetch/http.c
 ===================================================================
 --- lib/libfetch/http.c	(revision 244180)
 +++ lib/libfetch/http.c	(working copy)
 @@ -1374,6 +1374,7 @@
  http_connect(struct url *URL, struct url *purl, const char *flags)
  {
  	conn_t *conn;
 +	struct url *connect_url;
  	int verbose;
  	int af, val;
  
 @@ -1391,25 +1392,35 @@
  		af = AF_INET6;
  #endif
  
 -	if (purl && strcasecmp(URL->scheme, SCHEME_HTTPS) != 0) {
 -		URL = purl;
 -	} else if (strcasecmp(URL->scheme, SCHEME_FTP) == 0) {
 -		/* can't talk http to an ftp server */
 -		/* XXX should set an error code */
 -		return (NULL);
 -	}
 +	if (purl)
 +		connect_url = purl;
 +	else
 +		connect_url = URL;
  
 -	if ((conn = fetch_connect(URL->host, URL->port, af, verbose)) == NULL)
 +	if ((conn = fetch_connect(connect_url->host,
 +				  connect_url->port, af, verbose)) == NULL) {
  		/* fetch_connect() has already set an error code */
  		return (NULL);
 -	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 &&
 -	    fetch_ssl(conn, verbose) == -1) {
 -		fetch_close(conn);
 -		/* grrr */
 -		errno = EAUTH;
 -		fetch_syserr();
 -		return (NULL);
  	}
 +	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0) {
 +		if (purl) {
 +			http_cmd(conn, "CONNECT %s:%d HTTP/1.1",
 +				 URL->host, URL->port);
 +			http_cmd(conn, "");
 +			if (http_get_reply(conn) != HTTP_OK) {
 +				fetch_close(conn);
 +				return (NULL);
 +			}
 +			http_get_reply(conn);
 +		}
 +		if (fetch_ssl(conn, verbose) == -1) {
 +			fetch_close(conn);
 +			/* grrr */
 +			errno = EAUTH;
 +			fetch_syserr();
 +			return (NULL);
 +		}
 +	}
  
  	val = 1;
  	setsockopt(conn->sd, IPPROTO_TCP, TCP_NOPUSH, &val, sizeof(val));
 
 --IJpNTDwzlM2Ie8A6--

From: =?iso-2022-jp?B?GyRCRmJGI000MGxPOhsoQg==?= <naito.yuichiro@gmail.com>
To: Juergen Lock <nox@jelal.kn-bremen.de>
Cc: bug-followup@freebsd.org,
 ahze@freebsd.org,
 4721@hushmail.com,
 peterj@freebsd.org,
 des@freebsd.org
Subject: Re: bin/80176: fetch does not always work with https with a proxy.
Date: Wed, 27 Mar 2013 19:08:30 +0900

 Thank you for sending me a patch of libfetch.
 This patch works fine for https and ftp and http throough proxy server.
 
 I tested on FreeBSD 9.1-R.
 
 On 2012/12/27, at 5:23, Juergen Lock <nox@jelal.kn-bremen.de> wrote:
 
 > I was asked to submit this patch for frogs who has problems sending
 > via email, he said he fixed the patch by naito.yuichiro so it doesn't
 > break ftp requests through proxy.  He says he tested https and ftp
 > and http requests with and without proxy and they worked.
 > <libfetch.txt>
 
 -- 
 Yuichiro NAITO
 naito.yuichiro@gmail.com
 
 
 

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: freebsd-gnats-submit@freebsd.org 
Cc:  
Subject: Re: bin/80176: fetch does not always work with https with a proxy.
Date: Wed, 27 Mar 2013 11:42:48 +0100

 The patch looks *mostly* fine, except it removes the check for
 SCHEME_FTP.  I'm not 100% certain this is a "can't happen" scenario.
 I'll clean it up and commit it later today.
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: freebsd-gnats-submit@freebsd.org
Cc:  
Subject: Re: bin/80176: fetch does not always work with https with a proxy.
Date: Wed, 27 Mar 2013 12:09:38 +0100

 I spoke too soon.  I'm really not confident that the FTP logic is sound,
 and I'm not sure it's easily fixable (not least because proxies which
 require authentication won't work).
 
 Could the author(s) of the patch explain to me what the expected
 behavior is when http_connect() gets an FTP URL?
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: r4721@tormail.org
To: bug-followup@freebsd.org
Cc:  
Subject: Re: bin/80176: fetch does not always work with https with a proxy.
Date: Thu, 28 Mar 2013 07:32:38 -0000

 we were using the patch by naito.yuichiro (which fixed https) but found that
 ftp requests through http proxy would simply make fetch return without any
 error or function which was traced to the return(null) in the scheme_ftp
 check. direct ftp requests go via a different function, so this is only
 called when you send ftp requests through a http proxy, which is valid so a
 check to reject scheme_ftp in http didn't seem appropriate. removing it did
 fix ftp through proxy (sends request to the http proxy) and did not break
 direct ftp/http/https requests in our tests.
 

From: Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?= <des@des.no>
To: freebsd-gnats-submit@freebsd.org 
Cc:  
Subject: Re: bin/80176: fetch does not always work with https with a proxy.
Date: Thu, 28 Mar 2013 11:13:59 +0100

 r4721@tormail.org writes:
 > we were using the patch by naito.yuichiro (which fixed https) but found t=
 hat
 > ftp requests through http proxy would simply make fetch return without any
 > error or function which was traced to the return(null) in the scheme_ftp
 > check. direct ftp requests go via a different function, so this is only
 > called when you send ftp requests through a http proxy, which is valid so=
  a
 > check to reject scheme_ftp in http didn't seem appropriate. removing it d=
 id
 > fix ftp through proxy (sends request to the http proxy) and did not break
 > direct ftp/http/https requests in our tests.
 
 Ah, I see.  This means the FTP-over-HTTP case has been broken for a long
 time :(  Thank you for clearing up my confusion.  I'll commit a
 cleaned-up patch later today.
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/80176: commit references a PR
Date: Fri, 12 Apr 2013 22:05:23 +0000 (UTC)

 Author: des
 Date: Fri Apr 12 22:05:15 2013
 New Revision: 249431
 URL: http://svnweb.freebsd.org/changeset/base/249431
 
 Log:
   Use the CONNECT method to proxy HTTPS connections through HTTP proxies.
   
   PR:		bin/80176
   Submitted by:	Yuichiro NAITO <naito.yuichiro@gmail.com>
 
 Modified:
   head/lib/libfetch/http.c
 
 Modified: head/lib/libfetch/http.c
 ==============================================================================
 --- head/lib/libfetch/http.c	Fri Apr 12 21:29:37 2013	(r249430)
 +++ head/lib/libfetch/http.c	Fri Apr 12 22:05:15 2013	(r249431)
 @@ -1373,6 +1373,7 @@ http_authorize(conn_t *conn, const char 
  static conn_t *
  http_connect(struct url *URL, struct url *purl, const char *flags)
  {
 +	struct url *curl;
  	conn_t *conn;
  	int verbose;
  	int af, val;
 @@ -1391,17 +1392,21 @@ http_connect(struct url *URL, struct url
  		af = AF_INET6;
  #endif
  
 -	if (purl && strcasecmp(URL->scheme, SCHEME_HTTPS) != 0) {
 -		URL = purl;
 -	} else if (strcasecmp(URL->scheme, SCHEME_FTP) == 0) {
 -		/* can't talk http to an ftp server */
 -		/* XXX should set an error code */
 -		return (NULL);
 -	}
 +	curl = (purl != NULL) ? purl : URL;
  
 -	if ((conn = fetch_connect(URL->host, URL->port, af, verbose)) == NULL)
 +	if ((conn = fetch_connect(curl->host, curl->port, af, verbose)) == NULL)
  		/* fetch_connect() has already set an error code */
  		return (NULL);
 +	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 && purl) {
 +		http_cmd(conn, "CONNECT %s:%d HTTP/1.1",
 +		    URL->host, URL->port);
 +		http_cmd(conn, "");
 +		if (http_get_reply(conn) != HTTP_OK) {
 +			fetch_close(conn);
 +			return (NULL);
 +		}
 +		http_get_reply(conn);
 +	}
  	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 &&
  	    fetch_ssl(conn, verbose) == -1) {
  		fetch_close(conn);
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: r4721@tormail.org
To: bug-followup@freebsd.org
Cc:  
Subject: Re: bin/80176: fetch does not always work with https with a proxy.
Date: Sat, 13 Apr 2013 22:54:10 -0000

 applying the commit diff to 9-stable here is working fully. ftp/http/https
 is working with and without proxy. hopefully this can be scheduled for MFC
 in a month or two.
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/80176: commit references a PR
Date: Sun, 26 May 2013 16:48:58 +0000 (UTC)

 Author: des
 Date: Sun May 26 16:48:51 2013
 New Revision: 251001
 URL: http://svnweb.freebsd.org/changeset/base/251001
 
 Log:
   MFH (r243149): fix indentation
   MFH (r249431): use CONNECT to proxy HTTPS over HTTP
   
   PR:		bin/80176
 
 Modified:
   stable/9/lib/libfetch/http.c
 Directory Properties:
   stable/9/lib/libfetch/   (props changed)
 
 Modified: stable/9/lib/libfetch/http.c
 ==============================================================================
 --- stable/9/lib/libfetch/http.c	Sun May 26 14:54:06 2013	(r251000)
 +++ stable/9/lib/libfetch/http.c	Sun May 26 16:48:51 2013	(r251001)
 @@ -1373,6 +1373,7 @@ http_authorize(conn_t *conn, const char 
  static conn_t *
  http_connect(struct url *URL, struct url *purl, const char *flags)
  {
 +	struct url *curl;
  	conn_t *conn;
  	int verbose;
  	int af, val;
 @@ -1391,17 +1392,21 @@ http_connect(struct url *URL, struct url
  		af = AF_INET6;
  #endif
  
 -	if (purl && strcasecmp(URL->scheme, SCHEME_HTTPS) != 0) {
 -		URL = purl;
 -	} else if (strcasecmp(URL->scheme, SCHEME_FTP) == 0) {
 -		/* can't talk http to an ftp server */
 -		/* XXX should set an error code */
 -		return (NULL);
 -	}
 +	curl = (purl != NULL) ? purl : URL;
  
 -	if ((conn = fetch_connect(URL->host, URL->port, af, verbose)) == NULL)
 +	if ((conn = fetch_connect(curl->host, curl->port, af, verbose)) == NULL)
  		/* fetch_connect() has already set an error code */
  		return (NULL);
 +	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 && purl) {
 +		http_cmd(conn, "CONNECT %s:%d HTTP/1.1",
 +		    URL->host, URL->port);
 +		http_cmd(conn, "");
 +		if (http_get_reply(conn) != HTTP_OK) {
 +			fetch_close(conn);
 +			return (NULL);
 +		}
 +		http_get_reply(conn);
 +	}
  	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 &&
  	    fetch_ssl(conn, verbose) == -1) {
  		fetch_close(conn);
 @@ -1752,11 +1757,11 @@ http_request(struct url *URL, const char
  
  		/* get headers. http_next_header expects one line readahead */
  		if (fetch_getln(conn) == -1) {
 -		    fetch_syserr();
 -		    goto ouch;
 +			fetch_syserr();
 +			goto ouch;
  		}
  		do {
 -		    switch ((h = http_next_header(conn, &headerbuf, &p))) {
 +			switch ((h = http_next_header(conn, &headerbuf, &p))) {
  			case hdr_syserror:
  				fetch_syserr();
  				goto ouch;
 @@ -1785,7 +1790,7 @@ http_request(struct url *URL, const char
  				    conn->err != HTTP_USE_PROXY) {
  					n = 1;
  					break;
 -                                }
 +				}
  				if (new)
  					free(new);
  				if (verbose)
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/80176: commit references a PR
Date: Sun, 26 May 2013 17:00:23 +0000 (UTC)

 Author: des
 Date: Sun May 26 17:00:15 2013
 New Revision: 251002
 URL: http://svnweb.freebsd.org/changeset/base/251002
 
 Log:
   MFH (r230478): fix a couple of nits in r230307 (r231248)
   MFH (r243149): fix indentation
   MFH (r249431): use CONNECT to proxy HTTPS over HTTP
   
   PR:		bin/80176
 
 Modified:
   stable/8/lib/libfetch/common.c
   stable/8/lib/libfetch/http.c
 Directory Properties:
   stable/8/lib/libfetch/   (props changed)
 
 Modified: stable/8/lib/libfetch/common.c
 ==============================================================================
 --- stable/8/lib/libfetch/common.c	Sun May 26 16:48:51 2013	(r251001)
 +++ stable/8/lib/libfetch/common.c	Sun May 26 17:00:15 2013	(r251002)
 @@ -418,7 +418,6 @@ fetch_cache_data(conn_t *conn, char *src
  	if (conn->cache.size < nbytes) {
  		tmp = realloc(conn->cache.buf, nbytes);
  		if (tmp == NULL) {
 -			errno = ENOMEM;
  			fetch_syserr();
  			return (-1);
  		}
 @@ -481,7 +480,7 @@ fetch_read(conn_t *conn, char *buf, size
  		conn->cache.len -= total;
  		conn->cache.pos += total;
  		len -= total;
 -		buf+= total;
 +		buf += total;
  	}
  
  	while (len > 0) {
 
 Modified: stable/8/lib/libfetch/http.c
 ==============================================================================
 --- stable/8/lib/libfetch/http.c	Sun May 26 16:48:51 2013	(r251001)
 +++ stable/8/lib/libfetch/http.c	Sun May 26 17:00:15 2013	(r251002)
 @@ -1373,6 +1373,7 @@ http_authorize(conn_t *conn, const char 
  static conn_t *
  http_connect(struct url *URL, struct url *purl, const char *flags)
  {
 +	struct url *curl;
  	conn_t *conn;
  	int verbose;
  	int af, val;
 @@ -1391,17 +1392,21 @@ http_connect(struct url *URL, struct url
  		af = AF_INET6;
  #endif
  
 -	if (purl && strcasecmp(URL->scheme, SCHEME_HTTPS) != 0) {
 -		URL = purl;
 -	} else if (strcasecmp(URL->scheme, SCHEME_FTP) == 0) {
 -		/* can't talk http to an ftp server */
 -		/* XXX should set an error code */
 -		return (NULL);
 -	}
 +	curl = (purl != NULL) ? purl : URL;
  
 -	if ((conn = fetch_connect(URL->host, URL->port, af, verbose)) == NULL)
 +	if ((conn = fetch_connect(curl->host, curl->port, af, verbose)) == NULL)
  		/* fetch_connect() has already set an error code */
  		return (NULL);
 +	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 && purl) {
 +		http_cmd(conn, "CONNECT %s:%d HTTP/1.1",
 +		    URL->host, URL->port);
 +		http_cmd(conn, "");
 +		if (http_get_reply(conn) != HTTP_OK) {
 +			fetch_close(conn);
 +			return (NULL);
 +		}
 +		http_get_reply(conn);
 +	}
  	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 &&
  	    fetch_ssl(conn, verbose) == -1) {
  		fetch_close(conn);
 @@ -1752,11 +1757,11 @@ http_request(struct url *URL, const char
  
  		/* get headers. http_next_header expects one line readahead */
  		if (fetch_getln(conn) == -1) {
 -		    fetch_syserr();
 -		    goto ouch;
 +			fetch_syserr();
 +			goto ouch;
  		}
  		do {
 -		    switch ((h = http_next_header(conn, &headerbuf, &p))) {
 +			switch ((h = http_next_header(conn, &headerbuf, &p))) {
  			case hdr_syserror:
  				fetch_syserr();
  				goto ouch;
 @@ -1785,7 +1790,7 @@ http_request(struct url *URL, const char
  				    conn->err != HTTP_USE_PROXY) {
  					n = 1;
  					break;
 -                                }
 +				}
  				if (new)
  					free(new);
  				if (verbose)
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->closed 
State-Changed-By: des 
State-Changed-When: Tue May 28 11:56:53 UTC 2013 
State-Changed-Why:  
fixed in head, stable/9 and stable/8 

http://www.freebsd.org/cgi/query-pr.cgi?pr=80176 
>Unformatted:
 >>> GET https://www.ibm.com/ HTTP/1.1 
 >>> Host: www.ibm.com 
 >>> User-Agent: fetch libfetch/2.0 
 >>> Connection: close 
 >>> 
 <<< HTTP/1.1 302 Found 
 <<< Date: Tue, 17 Jan 2012 22:59:49 GMT 
 <<< Server: IBM_HTTP_Server 
 <<< Location: http://www.ibm.com/ 
 <<< Content-Length: 203 
 302 redirect to http://www.ibm.com/ 
 scheme:   [http] 
 user:     [] 
 password: [] 
 host:     [www.ibm.com] 
 port:     [0] 
 document: [/] 
 <<< epKe-Alive: timeout=10, max=92 
 content length: [203] 
 <<< Connection: Keep-Alive 
 <<< Content-Type: text/html 
 <<< 
 ---> quartz.mavetju.org:1234 
 looking up quartz.mavetju.org 
 connecting to quartz.mavetju.org:1234 
 fetch: https://www.ibm.com/: Connection refused 
  
  
  
  
  http://www.freebsd.org/cgi/query-pr.cgi?pr=80176 
