From nobody@FreeBSD.org  Fri Sep  7 07:58:40 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id E112F106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  7 Sep 2012 07:58:39 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id B22D28FC0C
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  7 Sep 2012 07:58:39 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id q877wdRt022342
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 7 Sep 2012 07:58:39 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id q877wdwo022341;
	Fri, 7 Sep 2012 07:58:39 GMT
	(envelope-from nobody)
Message-Id: <201209070758.q877wdwo022341@red.freebsd.org>
Date: Fri, 7 Sep 2012 07:58:39 GMT
From: Oliver Hartmann <ohartman@zedat.fu-berlin.de>
To: freebsd-gnats-submit@FreeBSD.org
Subject: fetch(1): Authentication error or Segmentation fault on HTTPS:// URLs
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         171402
>Category:       bin
>Synopsis:       fetch(1): Authentication error or Segmentation fault on HTTPS:// URLs
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Sep 07 08:00:08 UTC 2012
>Closed-Date:    Sat Nov 24 17:51:33 UTC 2012
>Last-Modified:  Sat Nov 24 17:51:33 UTC 2012
>Originator:     Oliver Hartmann
>Release:        FreeBSD 10.0-CURRENT r240150M amd64
>Organization:
FU Berlin
>Environment:
FreeBSD 10.0-CURRENT #2 r240150M: Wed Sep  5 21:35:41 CEST 2012 amd64,  CLANG built
>Description:
While trying to fetch sources for a port located at a SSL secured URL, I run into a problem with fetch(1) droping a Segmentaion fault" or reporting "Authentication error", while wget(1) from port net/wget  doesn't segfaults: 


fetch:
fetch -o /dev/null https://launchpad.net/
fetch: https://launchpad.net/: Authentication error

wget:
wget https://launchpad.net/
--2012-09-07 09:45:59--  https://launchpad.net/
Resolving launchpad.net (launchpad.net)... 91.189.89.223, 91.189.89.222
Connecting to launchpad.net (launchpad.net)|91.189.89.223|:443... connected.
ERROR: cannot verify launchpad.net's certificate, issued by `/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287':
  Self-signed certificate encountered.
To connect to launchpad.net insecurely, use `--no-check-certificate'.



fetch -o /dev/null https://lists.sourceforge.net/
Segmentation fault

wget:
wget https://lists.sourceforge.net
--2012-09-07 09:47:02--  https://lists.sourceforge.net/
Resolving lists.sourceforge.net (lists.sourceforge.net)... 216.34.181.88
Connecting to lists.sourceforge.net (lists.sourceforge.net)|216.34.181.88|:443... connected.
ERROR: cannot verify lists.sourceforge.net's certificate, issued by `/C=US/O=GeoTrust, Inc./CN=GeoTrust SSL CA':
  Unable to locally verify the issuer's authority.
To connect to lists.sourceforge.net insecurely, use `--no-check-certificate'.

The "Authentication error" can be avoided by applying option `--no-check-certificate' as suggested for fetching sources located at https://launchpad.net/ with wget(1). But fetch(1) doesn't seem to have the ability to switch off the authentication the same way.

The Segmentation fault seems to be weird.

I'm using the system's OpenSSL 1.0.0c, no additional port installed.
>How-To-Repeat:
Try mentioned failing fetch(1) sequences and check with wget(1).
>Fix:


>Release-Note:
>Audit-Trail:

From: Mark Johnston <markjdb@gmail.com>
To: bug-followup@FreeBSD.org, ohartman@zedat.fu-berlin.de
Cc:  
Subject: Re: bin/171402: fetch(1): Authentication error or Segmentation fault
 on HTTPS:// URLs
Date: Fri, 7 Sep 2012 22:33:54 -0400

 This is interesting. I was confused when this issue was reported on
 freebsd-current yesterday, as I couldn't reproduce it - fetch(1)ing
 https://launchdpad.net works fine for me on -CURRENT. Moreover, no
 errors from libssl were getting printed: libfetch is supposed to print
 them when one of the calls in fetch_ssl() fails.
 
 I also don't see the segfault, but I do get an authentication error with
 fetch(1) for the sourceforge page, which I suppose is to be expected.
 So I think it would be good to add a flag to libfetch which tells
 openSSL to bypass certificate verification - it would just need to call
 
 SSL_set_verify(ctx, SSL_VERIFY_NONE, NULL);
 
 at the appropriate spot.
 
 Then I saw "CLANG built" above. I tried rebuilding fetch(1)'s libraries
 using clang and eventually narrowed it down to libmd: if it's built with
 clang, I get an authentication error for lanchpad.net, and a segfault
 with lists.sourceforge.net. Haven't looked into what's actually going on
 though.
 
 -Mark

From: Mark Johnston <markjdb@gmail.com>
To: bug-followup@FreeBSD.org, ohartman@zedat.fu-berlin.de
Cc:  
Subject: Re: bin/171402: fetch(1): Authentication error or Segmentation fault
 on HTTPS:// URLs
Date: Sun, 9 Sep 2012 03:15:48 -0400

 --VrqPEDrXMn8OVzN4
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 I managed to figure out what the problem is, took a while though. =)
 
 libfetch uses both libmd (for HTTP digest authentication) and libcrypto
 (through libssl for HTTPS). The problem is that some libcrypto's hash
 functions (e.g. SHA256) use the same prototypes as in libmd. For some
 reason, fetch(1) links with libmd, so libmd's functions end up getting
 used by libcrypto instead of libcrypto's hash functions. In princple
 I think this should be fine, but the problem is that libmd's prototypes
 are slightly different. Specifically, the _Init, _Update and _Final
 functions for the alorithms don't return anything, whereas libcrypto's
 implementations return an int.
 
 libcrypto actually checks the return values from these functions, so
 libcrypto/libssl will behave differently depending on what the libmd
 functions leave behind in %rax as a return value. Compiling libmd with
 clang just result in a different outcome (segfault instead of auth
 error); the problem is there regardless of which compiler is used.
 
 This can be 'fixed' by simply removing the libmd dependency from
 fetch(1). It doesn't need it, and the libmd symbols used by libfetch
 aren't in libcrypto. This way, libcrypto will come first in the library
 path and the expected hash function implementations will be used. It's a
 bit sketych, and I'll try to audit the rest of the base for similar
 issues, but this particular problem should be fixed quickly, as it means
 that libfetch+https is quite broken.
 
 Thanks,
 -Mark
 
 --VrqPEDrXMn8OVzN4
 Content-Type: text/x-diff; charset=us-ascii
 Content-Disposition: attachment; filename="fetch_libmd.patch"
 
 diff --git a/usr.bin/fetch/Makefile b/usr.bin/fetch/Makefile
 index 6f0db80..5686fc6 100644
 --- a/usr.bin/fetch/Makefile
 +++ b/usr.bin/fetch/Makefile
 @@ -4,8 +4,8 @@
  
  PROG=		fetch
  CSTD?=		c99
 -DPADD=		${LIBFETCH} ${LIBMD}
 -LDADD=		-lfetch -lmd
 +DPADD=		${LIBFETCH}
 +LDADD=		-lfetch
  .if ${MK_OPENSSL} != "no"
  DPADD+=		${LIBSSL} ${LIBCRYPTO}
  LDADD+=		-lssl -lcrypto
 
 --VrqPEDrXMn8OVzN4--

From: "O. Hartmann" <ohartman@zedat.fu-berlin.de>
To: bug-followup@FreeBSD.org, ohartman@zedat.fu-berlin.de
Cc:  
Subject: Re: bin/171402: fetch(1): Authentication error or Segmentation fault
 on HTTPS:// URLs
Date: Tue, 11 Sep 2012 12:13:50 +0200

 This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
 --------------enig2C704AEC7773EBBC6DAFAA20
 Content-Type: text/plain; charset=ISO-8859-15
 Content-Transfer-Encoding: quoted-printable
 
 After patching usr.bin/fetch/Makefile as suggested, it is possible to
 fetch the file in question (for me) without any problem.
 
 fetch -o /dev/null https://lists.sourceforge.net/
 /dev/null                                     100% of   29 kB   76 kBps
 
 and
 
 fetch -o /dev/null https://launchpad.net/
 /dev/null                                     100% of   22 kB  186 kBps
 
 
 It seems that the patch (and the explanation/investigation done by Mark
 Johnston) works - at least for me.
 
 I hope this patch will find its way soon into the tree.
 
 Another port, math/openblas, is suffering from a similar problem
 described since the port's sources are located at another https:// URL
 and I receive an Authetication error on updating.
 
 
 --------------enig2C704AEC7773EBBC6DAFAA20
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.19 (FreeBSD)
 
 iQEcBAEBAgAGBQJQTw7iAAoJEOgBcD7A/5N8IJ0H/iL2Eenbzd+xDrGvJcmZVyGV
 vCvQpXn4zLDXLXUdISIqhzJ9Lr7+qpP7+RlU5hs2asn2hdNPr5SatIbTH2TGf5sK
 OXf+ngaYc6KPXDaiBnghIrwY4ak/wiHkpC4TSGYQg7SqowUl3gHW3k0iQzDjnYW6
 sHdPcisWbYPRphfosST/KdIKw8Rlo25L4lsPEoSkwQ0SSU1fTwNp6HZN5BdLAJt3
 r5MBbCDnG68xIWiXaS0z5qVdQXkfjDgO2sFTLKEEHbg7NNYF/W41WTOs1lMjVUeD
 MCRUeSnPK/5rYVgHYEvNjLQ+EU4YCfOFMA3MOK7ZcPDWvWMfBWgaJo8R0q7KdWg=
 =MJMi
 -----END PGP SIGNATURE-----
 
 --------------enig2C704AEC7773EBBC6DAFAA20--

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: freebsd-gnats-submit@freebsd.org 
Cc:  
Subject: Re: bin/171402: fetch(1): Authentication error or Segmentation fault on HTTPS:// URLs
Date: Thu, 13 Sep 2012 13:35:21 +0200

 Mark's patch removes libmd without removing or modifying the code that
 requires it.  This is not immediately noticeable because fetch(1) also
 pulls in libmd, mostly canceling the effect of the patch.  What really
 happens is that the patch changes the order in which the libraries are
 loaded, so libssl gets libcrypto's SHA256_* instead of libmd's.
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/171402: commit references a PR
Date: Fri, 14 Sep 2012 13:00:57 +0000 (UTC)

 Author: des
 Date: Fri Sep 14 13:00:43 2012
 New Revision: 240496
 URL: http://svn.freebsd.org/changeset/base/240496
 
 Log:
   Use libmd if and only if OpenSSL is not available.
   
   PR:		bin/171402
   MFC after:	3 days
 
 Modified:
   head/lib/libfetch/Makefile
   head/lib/libfetch/http.c
   head/usr.bin/fetch/Makefile
 
 Modified: head/lib/libfetch/Makefile
 ==============================================================================
 --- head/lib/libfetch/Makefile	Fri Sep 14 12:15:13 2012	(r240495)
 +++ head/lib/libfetch/Makefile	Fri Sep 14 13:00:43 2012	(r240496)
 @@ -16,8 +16,8 @@ CFLAGS+=	-DINET6
  
  .if ${MK_OPENSSL} != "no"
  CFLAGS+=	-DWITH_SSL
 -DPADD=		${LIBSSL} ${LIBCRYPTO} ${LIBMD}
 -LDADD=		-lssl -lcrypto -lmd
 +DPADD=		${LIBSSL} ${LIBCRYPTO}
 +LDADD=		-lssl -lcrypto
  .else
  DPADD=		${LIBMD}
  LDADD=		-lmd
 
 Modified: head/lib/libfetch/http.c
 ==============================================================================
 --- head/lib/libfetch/http.c	Fri Sep 14 12:15:13 2012	(r240495)
 +++ head/lib/libfetch/http.c	Fri Sep 14 13:00:43 2012	(r240496)
 @@ -76,7 +76,15 @@ __FBSDID("$FreeBSD$");
  #include <string.h>
  #include <time.h>
  #include <unistd.h>
 +
 +#ifdef WITH_SSL
 +#include <openssl/md5.h>
 +#define MD5Init(c) MD5_Init(c)
 +#define MD5Update(c, data, len) MD5_Update(c, data, len)
 +#define MD5Final(md, c) MD5_Final(md, c)
 +#else
  #include <md5.h>
 +#endif
  
  #include <netinet/in.h>
  #include <netinet/tcp.h>
 
 Modified: head/usr.bin/fetch/Makefile
 ==============================================================================
 --- head/usr.bin/fetch/Makefile	Fri Sep 14 12:15:13 2012	(r240495)
 +++ head/usr.bin/fetch/Makefile	Fri Sep 14 13:00:43 2012	(r240496)
 @@ -4,11 +4,12 @@
  
  PROG=		fetch
  CSTD?=		c99
 +.if ${MK_OPENSSL} != "no"
 +DPADD=		${LIBFETCH} ${LIBSSL} ${LIBCRYPTO}
 +LDADD=		-lfetch -lssl -lcrypto
 +.else
  DPADD=		${LIBFETCH} ${LIBMD}
  LDADD=		-lfetch -lmd
 -.if ${MK_OPENSSL} != "no"
 -DPADD+=		${LIBSSL} ${LIBCRYPTO}
 -LDADD+=		-lssl -lcrypto
  .endif
  
  .include <bsd.prog.mk>
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: Mark Johnston <markjdb@gmail.com>
To: bug-followup@FreeBSD.org, ohartman@zedat.fu-berlin.de
Cc:  
Subject: Re: bin/171402: fetch(1): Authentication error or Segmentation fault
 on HTTPS:// URLs
Date: Sat, 24 Nov 2012 12:32:27 -0500

 I think this PR can be closed.
 
 Thanks,
 -Mark

From: "O. Hartmann" <ohartman@zedat.fu-berlin.de>
To: Mark Johnston <markjdb@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: bin/171402: fetch(1): Authentication error or Segmentation fault
 on HTTPS:// URLs
Date: Sat, 24 Nov 2012 18:33:21 +0100

 This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
 --------------enigC605292AEE0F122BB6D71D07
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 On 11/24/12 18:32, Mark Johnston wrote:
 > I think this PR can be closed.
 >=20
 > Thanks,
 > -Mark
 >=20
 
 
 Yes, I think so.
 
 Regards,
 O. Hartmann
 
 
 --------------enigC605292AEE0F122BB6D71D07
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.19 (FreeBSD)
 
 iQEcBAEBAgAGBQJQsQTmAAoJEOgBcD7A/5N8j0MH/j1Lbozu78Z8qtPFquJBy4Od
 ANH+K50smhnzAkFPD8QB0PDl2K4N425qvQFXHBi2PO1QB/jJDvyJfjauCeDN9i+M
 MV1eFOhSFGaA3cw/t5o6U4F4xRwRQpmOmgq0ywI5qSsxhtPZhVHdPM+LQ6MKA5qG
 thKuLnwOZ9NZd/npiY2/9KCWrKcHf94IC4Jty0YNUWfN6KaD8/3U2H7cJzgWqIWV
 xkrHyVjCqzr88GVd2U3szC/TrVNqGdW5Fc28ejCKRo1PB8Ce1m+NmP819VgcJ41C
 JklKN5055Edfki22I61uTpD3vBmKdMWUt9EeD9FpQdPq9rGH6m613QeEH4jxRgg=
 =FTu7
 -----END PGP SIGNATURE-----
 
 --------------enigC605292AEE0F122BB6D71D07--
State-Changed-From-To: open->closed 
State-Changed-By: eadler 
State-Changed-When: Sat Nov 24 17:51:32 UTC 2012 
State-Changed-Why:  
Per submitter request 

http://www.freebsd.org/cgi/query-pr.cgi?pr=171402 
>Unformatted:
