From nobody@FreeBSD.org  Tue Jan 22 17:32:58 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id 39DC6343
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 22 Jan 2013 17:32:58 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 18680819
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 22 Jan 2013 17:32:58 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r0MHWv5k097565
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 22 Jan 2013 17:32:57 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r0MHWvYQ097564;
	Tue, 22 Jan 2013 17:32:57 GMT
	(envelope-from nobody)
Message-Id: <201301221732.r0MHWvYQ097564@red.freebsd.org>
Date: Tue, 22 Jan 2013 17:32:57 GMT
From: Michael Gmelin <freebsd@grem.de>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [patch] Support PKI in libfetch for HTTP over SSL (https)
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         175514
>Category:       kern
>Synopsis:       [patch] Support PKI in libfetch for HTTP over SSL (https)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    des
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jan 22 17:40:00 UTC 2013
>Closed-Date:    Thu Oct 10 09:44:06 UTC 2013
>Last-Modified:  Thu Oct 10 09:44:06 UTC 2013
>Originator:     Michael Gmelin
>Release:        FreeBSD 9.1-RELEASE
>Organization:
Grem Equity GmbH
>Environment:
FreeBSD bsd64 9.1-RELEASE FreeBSD 9.1-RELEASE #3 r244944: Wed Jan  2 02:10:20 UTC 2013     root@bsd64:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
The original motivation for this patch was that pkgng uses libfetch as its
transport mechanism, so I took a closer look into how https is currently
implemented in libfetch. As it turns out support is minimal and handles
nothing more than basic encryption. It does not verify the certificate
against any trust store, does not validate the CN/subjectAltName or support
client certificates. It also allows SSLv2 by default and leaks resources. In
addition it can segfault in verbose mode.

I know that there resentments to the PKI model due to its various
shortcomings, since any CA can be break the trust (as we've seen in recent
history, e.g. DigiNotar). There are also new promising concepts on the
horizon (e.g. DANE, RFC6698), I still think supporting the current PKI model
makes sense for the following reasons:

1. It matches the user's expectation:
   If I "fetch https://www.google.com" I expect it to complain if the
   certificate presented is self-signed by fraudster.com.

2. It is solution that's been deployed already (and almost all fetch like tools
   and browsers have it enabled by default). Better solutions are not in
   widespread use yet.

3. It allows running your private CA for local use, including client
   certificate authentication. In this case most of the problems of a Public
   Key Infrastructure are not relevant.

Once libfetch is patched, the user either has to install trusted
certificates (e.g. by installing security/ca_root_nss) or explicitly disable
the checks. For the sake of migrating existing setups, it might make sense
to add a configuration file (e.g. /etc/fetch.conf) that disables checks by
default, so people's existing setups won't break - having such a default
configuration for libfetch in general might make sense anyway (also for
PROXY settings). I'm open to discuss if this is considered necessary and how
it should be implemented - basically I think the feature itself should be
enabled by default, otherwise it's of limited use.

That said, what this patch does is:
- Disable SSLv2 by default. Can be enabled by setting SSL_ALLOW_SSL2.
- Enable peer certificate verification (can be disabled by setting
  SSL_NO_VERIFY_PEER, CA trust store location can be specified by setting
  SSL_CA_CERTFILE and SSL_CA_CERTPATH and default to /etc/ssl/cert.pem)
- Enable hostname verification (CN, subjectAltName, DNS, wildcards, IPv4
  IPv6), based on RFC2818, RFC2459, RFC3280 and especially RFC6125. Can be
  disabled by setting SSL_NO_VERIFY_HOSTNAME
- Allow the use of certificate revocation list (can be configured by setting
  SSL_CRL_FILE)
- Allow the use of client certificate based authentication (by setting
  SSL_CLIENT_CERT_FILE and optionally SSL_CLIENT_KEY_FILE)
- Allow disabling of SSL methods by setting SSL_NO_SSL3 and SSL_NO_TLS1
- Fix a resource leak on multiple invocations of fetch_ssl
- Fix a segmentation fault when accessing a non-working host in verbose mode
- Document all these features in detail in fetch(3)
- Add new environment variables to the list in fetch(1) that refers to
  fetch(3)

The patch does not attempt to fix the thread-unsafe use of OpenSSL in libfetch
(ultimately the man page states that it is not fully reentrant). If there is
any interest I can fix this, but I would need some mentoring if there are
scenarios where the multithreading code shouldn't be run or not etc and if
it makes sense at all.

I tried following style(9) as closely as possible, but also respected the
slightly different style of libfetch. If there is something I overlooked
helpful criticism is always welcome (also if there are any other changes
that should be done to the code).

I tested the patch on FreeBSD 9.0 and 9.1 (amd64) using system gcc and clang.
Besides the patch to fetch.1, which barfs on 9.0 due to a glitch in the
manpage on 9.1, it should apply cleanly on both versions of the OS.

>How-To-Repeat:
fetch https://173.194.44.49
(which just works, even though it shouldn't since that IP is not
specified in google's certificate)

>Fix:
Apply patches supplied.

cd /usr/src
patch < /path/to/libfetch.patch.txt


Patch attached with submission follows:

Index: lib/libfetch/fetch.3
===================================================================
--- lib/libfetch/fetch.3	(revision 245791)
+++ lib/libfetch/fetch.3	(working copy)
@@ -1,5 +1,6 @@
 .\"-
 .\" Copyright (c) 1998-2011 Dag-Erling Smørgrav
+.\" Copyright (c) 2013 Michael Gmelin <freebsd@grem.de>
 .\" All rights reserved.
 .\"
 .\" Redistribution and use in source and binary forms, with or without
@@ -392,6 +393,60 @@
 library,
 .Fn fetchPutHTTP
 is currently unimplemented.
+.Sh HTTPS SCHEME
+Based on HTTP SCHEME. By default the peer is verified
+using the CA bundle located in
+.Pa /etc/ssl/cert.pem .
+The file may contain multiple CA certificates. A common source of a
+current CA bundle is
+.Pa \%security/ca_root_nss .
+.Pp
+The CA bundle used for peer verification can be changed by setting the
+environment variables
+.Ev SSL_CA_CERT_FILE
+to point to a concatenated
+bundle of trusted certificates and
+.Ev SSL_CA_CERT_PATH
+to point to a
+directory containing hashes of trusted CAs (see
+.Xr verify 1 ) .
+.Pp
+A certificate revocation list (CRL) can be used by setting the
+environment variable
+.Ev SSL_CRL_FILE
+(see
+.Xr crl 1 ) .
+.Pp
+Peer verification can be disabled by setting the environment variable
+.Ev SSL_NO_VERIFY_PEER .
+Note that this also disables CRL checking.
+.Pp
+By default the service identity is verified according to the rules
+detailed in RFC6125 (also known as hostname verification). This
+feature can be disabled by setting the environment variable
+.Ev SSL_NO_VERIFY_HOSTNAME .
+.Pp
+Client certificate based authentication is supported. The environment
+variable
+.Ev SSL_CLIENT_CERT_FILE
+should be set to point to a file
+containing key and client certificate to be used in PEM format. In
+case the key is stored in a separate file, the environment variable
+.Ev SSL_CLIENT_KEY_FILE
+can be set to point to the key in PEM format. In
+case the key uses a password, the user will be prompted on standard input (see
+.Xr PEM 3 ) .
+.Pp
+By default
+.Nm libfetch
+allows SSLv3 and TLSv1 when negotiating the connecting with the remote
+peer. You can change this behavior by setting the environment variable
+.Ev SSL_ALLOW_SSL2
+to allow SSLv2 (not recommended) and
+.Ev SSL_NO_SSL3
+or
+.Ev SSL_NO_TLS1
+to disable the respective methods.
 .Sh AUTHENTICATION
 Apart from setting the appropriate environment variables and
 specifying the user name and password in the URL or the
@@ -579,6 +634,31 @@
 Same as
 .Ev NO_PROXY ,
 for compatibility.
+.It Ev SSL_ALLOW_SSL2
+Allow SSL version 2 when negotiating the connection (not recommended).
+.It Ev SSL_CA_CERT_FILE
+CA certificate bundle containing trusted CA certificates. Default
+value:
+.Pa /etc/ssl/cert.pem .
+.It Ev SSL_CA_CERT_PATH
+Path containing trusted CA hashes.
+.It Ev SSL_CLIENT_CERT_FILE
+PEM encoded client certificate/key which will be used in
+client certificate authentication.
+.It Ev SSL_CLIENT_KEY_FILE
+PEM encoded client key in case key and client certificate
+are stored separately.
+.It Ev SSL_CRL_FILE
+File containing certificate revocation list.
+.It Ev SSL_NO_SSL3
+Don't allow SSL version 3 when negotiating the connection.
+.It Ev SSL_NO_TLS1
+Don't allow TLV version 1 when negotiating the connection.
+.It Ev SSL_NO_VERIFY_HOSTNAME
+If set, do not verify that the hostname matches the subject of the
+certificate presented by the server.
+.It Ev SSL_NO_VERIFY_PEER
+If set, do not verify the peer certificate against trusted CAs.
 .El
 .Sh EXAMPLES
 To access a proxy server on
@@ -610,6 +690,19 @@
 .Bd -literal -offset indent
 NO_PROXY=localhost,127.0.0.1
 .Ed
+.Pp
+Access HTTPS website without any certificate verification whatsoever:
+.Bd -literal -offset indent
+SSL_NO_VERIFY_PEER=1
+SSL_NO_VERIFY_HOSTNAME=1
+.Ed
+.Pp
+Access HTTPS website using client certificate based authentication
+and a private CA:
+.Bd -literal -offset indent
+SSL_CLIENT_CERT_FILE=/path/to/client.pem
+SSL_CA_CERT_FILE=/path/to/myca.pem
+.Ed
 .Sh SEE ALSO
 .Xr fetch 1 ,
 .Xr ftpio 3 ,
@@ -678,7 +771,8 @@
 .An Hajimu Umemoto Aq ume@FreeBSD.org ,
 .An Henry Whincup Aq henry@techiebod.com ,
 .An Jukka A. Ukkonen Aq jau@iki.fi ,
-.An Jean-Fran\(,cois Dockes Aq jf@dockes.org
+.An Jean-Fran\(,cois Dockes Aq jf@dockes.org ,
+.An Michael Gmelin Aq freebsd@grem.de
 and others.
 It replaces the older
 .Nm ftpio
@@ -688,7 +782,9 @@
 .An Jordan K. Hubbard Aq jkh@FreeBSD.org .
 .Pp
 This manual page was written by
-.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org .
+.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org
+and
+.An Michael Gmelin Aq freebsd@grem.de .
 .Sh BUGS
 Some parts of the library are not yet implemented.
 The most notable
@@ -717,6 +813,10 @@
 .Fn fetchStatFTP
 does not check that the result of an MDTM command is a valid date.
 .Pp
+In case password protected keys are used for client certificate based
+authentication the user is prompted for the password on each and every
+fetch operation.
+.Pp
 The man page is incomplete, poorly written and produces badly
 formatted text.
 .Pp
Index: lib/libfetch/http.c
===================================================================
--- lib/libfetch/http.c	(revision 245791)
+++ lib/libfetch/http.c	(working copy)
@@ -1392,7 +1392,7 @@
 		/* fetch_connect() has already set an error code */
 		return (NULL);
 	if (strcasecmp(URL->scheme, SCHEME_HTTPS) == 0 &&
-	    fetch_ssl(conn, verbose) == -1) {
+			fetch_ssl(conn, URL, verbose) == -1) {
 		fetch_close(conn);
 		/* grrr */
 		errno = EAUTH;
Index: lib/libfetch/common.c
===================================================================
--- lib/libfetch/common.c	(revision 245791)
+++ lib/libfetch/common.c	(working copy)
@@ -1,5 +1,6 @@
 /*-
  * Copyright (c) 1998-2011 Dag-Erling Smørgrav
+ * Copyright (c) 2013 Michael Gmelin <freebsd@grem.de>
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -47,6 +48,10 @@
 #include <string.h>
 #include <unistd.h>
 
+#ifdef WITH_SSL
+#include <openssl/x509v3.h>
+#endif
+
 #include "fetch.h"
 #include "common.h"
 
@@ -317,12 +322,470 @@
 	return (conn);
 }
 
+#ifdef WITH_SSL
+/*
+ * Callback for SSL certificate verification, this is called on server cert
+ * verification. It takes no decision, but informs the user in case
+ * verification failed.
+ */
+int
+fetch_ssl_cb_verify_crt(int verified, X509_STORE_CTX *ctx)
+{
+	char *str;
 
+	if (!verified)
+	{
+		str = X509_NAME_oneline(
+		    X509_get_subject_name(
+			X509_STORE_CTX_get_current_cert(ctx)),
+			0, 0);
+		fprintf(stderr, "Certificate verification failed %s\n",
+		    str);
+		OPENSSL_free(str);
+	}
+	return (verified);
+}
+
 /*
+ * Convert characters A-Z to lowercase (intentionally avoid any
+ * locale specific conversions)
+ */
+static char
+fetch_ssl_tolower(char in)
+{
+	if (in >= 'A' && in <= 'Z')
+		return (in + 32);
+	else
+		return (in);
+}
+
+/*
+ * Check if passed hostnames a and b are equal
+ */
+static int
+fetch_ssl_hostname_equal(const char *a, size_t alen, const char *b,
+    size_t blen)
+{
+	size_t i;
+
+	if (alen != blen)
+		return (0);
+	for (i=0; i<alen; ++i) {
+		if (fetch_ssl_tolower(a[i]) != fetch_ssl_tolower(b[i]))
+			return (0);
+	}
+	return (1);
+}
+
+/*
+ * Check if domain label is traditional, meaning that only A-Z, a-z, 0-9 and
+ * '-' (hyphen) are allowed. Hyphens have to be surrounded by alphanumeric
+ * characters. Double hyphens (like they're found in IDN a-labels 'xn--') are
+ * not allowed.
+ */
+static int
+fetch_ssl_is_traditional_domain_label(const char *label, size_t len,
+    int allow_wildcard)
+{
+	size_t i;
+
+	for (i=0; i<len; ++i) {
+		if (!((label[i] >= '0' && label[i] <= '9') ||
+			(label[i] >= 'A' && label[i] <= 'Z') ||
+			(label[i] >= 'a' && label[i] <= 'z') ||
+			(label[i] == '-' && i != 0 && i != len - 1
+			    && label[i - 1] != '-') ||
+			(label[i] == '*' && allow_wildcard)))
+			return (0);
+	}
+	return (1);
+}
+
+/*
+ * Check if host name consists only of numbers. This might indicate
+ * an IP address, which is not a good idea for CN wildcard comparison.
+ */
+static int
+fetch_ssl_hostname_is_only_numbers(const char *hostname, size_t len)
+{
+	size_t i;
+
+	for (i=0; i<len; ++i) {
+		if (!((hostname[i] >= '0' && hostname[i] <= '9') ||
+			hostname[i] == '.'))
+			return (0);
+	}
+	return (1);
+}
+
+/*
+ * Check if the host name passed matches the pattern passed in matches which
+ * is usually part of subjectAltName or CN of a certificate presented to the
+ * client. This includes wildcard matching. The algorithm is based on RFC6125,
+ * sections 6.4.3 and 7.2, which clarifies RFC2818 and RFC3280.
+ */
+static int
+fetch_ssl_hostname_match(const char *hostname, size_t hostnamelen,
+    const char *matches, size_t matcheslen)
+{
+	const char *wildcard;
+	const char *firstmatchdot;
+	const char *secondmatchdot;
+	const char *firsthostdot;
+
+	if (hostname && *hostname && matches && *matches)
+	{
+		if (!(wildcard = strnstr(matches, "*", matcheslen)))
+			return (fetch_ssl_hostname_equal(hostname,
+			    hostnamelen, matches, matcheslen));
+
+		/*
+		 * make sure hostname is not just dots and numbers
+		 */
+		if (fetch_ssl_hostname_is_only_numbers(hostname, hostnamelen))
+			return (0);
+
+		/* only one wildcard allowed in pattern */
+		if (strnstr(wildcard + 1, "*",
+			matcheslen - (wildcard - matches) - 1) != NULL)
+			return (0);
+
+		/*
+		 * make sure there are at least two more domain labels
+		 * and wildcard is in the leftmost label
+		 */
+		if ((firstmatchdot =
+			strnstr(matches, ".", matcheslen)) == NULL ||
+		    (matcheslen - (firstmatchdot - matches)) < 4 ||
+		    firstmatchdot < wildcard) /* important! */
+			return (0);
+
+		if ((secondmatchdot = strnstr(firstmatchdot + 1, ".",
+			    matcheslen - (firstmatchdot - matches) - 1)) == NULL ||
+		    (matcheslen - (secondmatchdot - matches)) < 2)
+			return (0);
+
+		/*
+		 * make sure hostname contains a dot and it's not the first
+		 * character
+		 */
+		if ((firsthostdot =
+			strnstr(hostname, ".", hostnamelen)) == NULL ||
+		    firsthostdot == hostname)
+			return (0);
+
+		/*
+		 * make sure host part of hostname is at least as long as
+		 * pattern it's supposed to match
+		 */
+		if ((firsthostdot - hostname) < (firstmatchdot - matches))
+			return (0);
+
+		/*
+		 * don't allow wildcards in non-traditional domain names
+		 * (IDN, A-label, U-label...)
+		 */
+		if (!fetch_ssl_is_traditional_domain_label(hostname,
+			firsthostdot - hostname - 1, 0) ||
+		    !fetch_ssl_is_traditional_domain_label(matches,
+			firstmatchdot - matches - 1, 1))
+			return (0);
+
+		/* match domain part (part after first dot) */
+		if (!fetch_ssl_hostname_equal(firsthostdot,
+			hostnamelen - (firsthostdot - hostname),
+			firstmatchdot, matcheslen - (firstmatchdot - matches)))
+			return (0);
+
+		/* match part left of wildcard */
+		if (!fetch_ssl_hostname_equal(hostname, wildcard - matches,
+			matches, wildcard - matches))
+			return (0);
+
+		/* match part right of wildcard */
+		if (!fetch_ssl_hostname_equal(
+			firsthostdot - (firstmatchdot - wildcard - 1),
+			    (firstmatchdot - wildcard - 1),
+			    firstmatchdot - (firstmatchdot - wildcard - 1),
+			    (firstmatchdot - wildcard - 1)))
+			return (0);
+
+		/* all tests succeded, it's a match */
+		return (1);
+	}
+	return (0);
+}
+
+/*
+ * Get numeric host address info - returns NULL if host was not an IP address.
+ * The caller is responsible for deallocation using freeaddrinfo(3)
+ */
+static struct addrinfo *
+fetch_ssl_get_numeric_addrinfo(const char *hostname, size_t len)
+{
+	struct addrinfo hints;
+	struct addrinfo *res;
+	char *host;
+
+	host = (char*) malloc(len + 1);
+	memcpy(host, hostname, len);
+	host[len] = '\0';
+	memset(&hints, 0, sizeof(hints));
+	hints.ai_family = PF_UNSPEC;
+	hints.ai_socktype = SOCK_STREAM;
+	hints.ai_protocol = 0;
+	hints.ai_flags = AI_NUMERICHOST;
+	/* port is not relevant for this purpose */
+	getaddrinfo(host, "443", &hints, &res);
+	free(host);
+	return res;
+}
+
+/*
+ * Compare ip address in addrinfo with address passes
+ */
+static int
+fetch_ssl_ipaddress_match_binary(const struct addrinfo *hostaaddr,
+    const char *hostbaddr, size_t hostbaddrlen)
+{
+	if (hostaaddr->ai_family == AF_INET && hostbaddrlen == 4 &&
+	    !memcmp((void*)&((struct sockaddr_in*)(void*)(hostaaddr->ai_addr))->
+		sin_addr.s_addr, (const void*)hostbaddr, hostbaddrlen))
+		return (1);
+#ifdef INET6
+	else if (hostaaddr->ai_family == AF_INET6 && hostbaddrlen == 16 &&
+	    !memcmp((void*)&((struct sockaddr_in6*)(void*)hostaaddr->ai_addr)->
+		sin6_addr, (const void*)hostbaddr, hostbaddrlen))
+		return (1);
+#endif
+	return (0);
+}
+
+/*
+ * Compare ip address in addrinfo with host passed. If host is not an
+ * IP address, comparison will fail
+ */
+static int
+fetch_ssl_ipaddress_match(const struct addrinfo *hostaaddr, const char *hostb,
+    size_t hostblen)
+{
+	struct addrinfo *hostbaddr;
+	int ret = 0;
+
+	if ((hostbaddr = fetch_ssl_get_numeric_addrinfo(hostb, hostblen)) != NULL) {
+		if (hostaaddr->ai_family == AF_INET)
+			ret = fetch_ssl_ipaddress_match_binary(hostaaddr,
+			    (char*)&((struct sockaddr_in*)(void*)hostbaddr->
+				ai_addr)->sin_addr.s_addr, 4);
+#ifdef INET6
+		else if (hostaaddr->ai_family == AF_INET6)
+			ret = fetch_ssl_ipaddress_match_binary(hostaaddr,
+			    (char*)&((struct sockaddr_in6*)(void*)hostbaddr->
+				ai_addr)->sin6_addr, 16);
+#endif
+		freeaddrinfo(hostbaddr);
+	}
+	return (ret);
+}
+
+/*
+ * Verify server certificate subjectAltName/CN matches hostname
+ */
+static int
+fetch_ssl_verify_hostname(conn_t *conn, const struct url *URL)
+{
+	int i;
+	STACK_OF(GENERAL_NAME) *subject_alt_names = NULL;
+	unsigned char *cn = NULL;
+	int ret = 0;
+	struct addrinfo *hostipaddr = NULL;
+
+	hostipaddr = fetch_ssl_get_numeric_addrinfo(URL->host, strlen(URL->host));
+	if ((subject_alt_names = X509_get_ext_d2i(conn->ssl_cert,
+		    NID_subject_alt_name, NULL, NULL))) {
+		for (i=0; i<sk_GENERAL_NAME_num(subject_alt_names); ++i)
+		{
+			const GENERAL_NAME *name;
+			const char *namestr;
+			size_t namestrlen;
+			    
+			/*
+			 * This is a workaround, since the following line causes
+			 * alignment issues on clang:
+			 * name = sk_GENERAL_NAME_value(subject_alt_names, i);
+			 * OpenSSL explicitly warns to use those macros
+			 * directly, but there isn't much choice (and there
+			 * shouldn't be any ill side effects)
+			 */
+			name = (GENERAL_NAME*)SKM_sk_value(void,
+			    subject_alt_names, i);
+			namestr = (const char *) ASN1_STRING_data(name->d.ia5);
+			namestrlen = (size_t) ASN1_STRING_length(name->d.ia5);
+
+			if ((name->type == GEN_DNS &&
+				!hostipaddr &&
+				fetch_ssl_hostname_match(URL->host,
+				    strlen(URL->host), namestr, namestrlen)) ||
+			    (name->type == GEN_IPADD &&
+				hostipaddr &&
+				fetch_ssl_ipaddress_match_binary(hostipaddr,
+				    namestr, namestrlen))) {
+				ret = 1;
+				goto ssl_verify_done;
+			}
+		}
+	}
+	else { /* get most specific CN (last entry in list) and compare */
+		X509_NAME *subject;
+		int lastpos;
+		int loc = -1;
+
+		subject = X509_get_subject_name(conn->ssl_cert);
+		lastpos = -1;
+		loc = -1;
+		while ((lastpos = X509_NAME_get_index_by_NID(subject,
+			    NID_commonName, lastpos)) != -1) {
+			loc = lastpos;
+		}
+		if (loc > -1) {
+			int cnlen;
+
+			cnlen = ASN1_STRING_to_UTF8(&cn,
+			    X509_NAME_ENTRY_get_data(
+				X509_NAME_get_entry(subject, loc)));
+			if ((!hostipaddr && fetch_ssl_hostname_match(
+				URL->host, strlen(URL->host), cn, cnlen)) ||
+			    (hostipaddr && fetch_ssl_ipaddress_match(
+				hostipaddr, cn, cnlen))) {
+				ret = 1;
+				goto ssl_verify_done;
+			}
+		}
+	}
+
+ssl_verify_done:
+	if (cn)
+		OPENSSL_free(cn);
+	if (hostipaddr)
+		freeaddrinfo(hostipaddr);
+	if (subject_alt_names)
+		GENERAL_NAMES_free(subject_alt_names);
+	return (ret);
+}
+
+/*
+ * Configure transport security layer based on environment
+ */
+static void
+fetch_ssl_setup_transport_layer(SSL_CTX *ctx, int verbose)
+{
+	long ssl_ctx_options;
+	
+	ssl_ctx_options = SSL_OP_ALL | SSL_OP_NO_TICKET;
+	if (!getenv("SSL_ALLOW_SSL2"))
+		ssl_ctx_options |= SSL_OP_NO_SSLv2;
+	if (getenv("SSL_NO_SSL3"))
+		ssl_ctx_options |= SSL_OP_NO_SSLv3;
+	if (getenv("SSL_NO_TLS1"))
+		ssl_ctx_options |= SSL_OP_NO_TLSv1;
+	if (verbose)
+		fetch_info("SSL options: %x", ssl_ctx_options);
+	SSL_CTX_set_options(ctx, ssl_ctx_options);
+}
+
+
+/*
+ * Configure peer verification based on environment
+ */
+static int
+fetch_ssl_setup_peer_verification(SSL_CTX *ctx, int verbose)
+{
+	X509_LOOKUP *crl_lookup;
+	const char *ca_cert_file;
+	const char *ca_cert_path;
+	const char *crl_file;
+
+	if (!getenv("SSL_NO_VERIFY_PEER")) {
+		ca_cert_file = getenv("SSL_CA_CERT_FILE")?
+		    getenv("SSL_CA_CERT_FILE"):"/etc/ssl/cert.pem";
+		ca_cert_path = getenv("SSL_CA_CERT_PATH");
+		if (verbose) {
+			fetch_info("Peer verification enabled");
+			if (ca_cert_file)
+				fetch_info("Using CA cert file: %s",
+				    ca_cert_file);
+			if (ca_cert_path)
+				fetch_info("Using CA cert path: %s",
+				    ca_cert_path);
+		}
+		SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER,
+		    fetch_ssl_cb_verify_crt);
+		SSL_CTX_load_verify_locations(ctx, ca_cert_file, ca_cert_path);
+		if ((crl_file = getenv("SSL_CRL_FILE")) != NULL) {
+			if (verbose)
+				fetch_info("Using CRL file: %s",
+				    crl_file);
+			crl_lookup = X509_STORE_add_lookup(
+			    SSL_CTX_get_cert_store(ctx), X509_LOOKUP_file());
+			if (crl_lookup == NULL ||
+			    !X509_load_crl_file(crl_lookup, crl_file,
+				X509_FILETYPE_PEM)) {
+				fprintf(stderr,
+				    "Could not load CRL file %s\n",
+				    crl_file);
+				return (0);
+			}
+			X509_STORE_set_flags(
+			    SSL_CTX_get_cert_store(ctx), X509_V_FLAG_CRL_CHECK |
+				X509_V_FLAG_CRL_CHECK_ALL);
+		}
+	}
+	return (1);
+}
+
+/*
+ * Configure client certificate based on environment
+ */
+static int
+fetch_ssl_setup_client_certificate(SSL_CTX *ctx, int verbose)
+{
+	const char *client_cert_file;
+	const char *client_key_file;
+
+	if ((client_cert_file = getenv("SSL_CLIENT_CERT_FILE")) != NULL) {
+		client_key_file = getenv("SSL_CLIENT_KEY_FILE")?
+		    getenv("SSL_CLIENT_KEY_FILE"):client_cert_file;
+		if (verbose) {
+			fetch_info("Using client cert file: %s",
+			    client_cert_file);
+			fetch_info("Using client key file: %s",
+			    client_key_file);
+		}
+		if (SSL_CTX_use_certificate_chain_file(ctx,
+			client_cert_file) != 1) {
+			fprintf(stderr,
+			    "Could not load client certificate %s\n",
+			    client_cert_file);
+			return (0);
+		}
+		if (SSL_CTX_use_PrivateKey_file(ctx, client_key_file,
+			SSL_FILETYPE_PEM) != 1) {
+			fprintf(stderr,
+			    "Could not load client key %s\n",
+			    client_key_file);
+			return (0);
+		}
+	}
+	return (1);
+}
+
+#endif
+
+/*
  * Enable SSL on a connection.
  */
 int
-fetch_ssl(conn_t *conn, int verbose)
+fetch_ssl(conn_t *conn, const struct url *URL, int verbose)
 {
 #ifdef WITH_SSL
 	int ret, ssl_err;
@@ -339,8 +802,14 @@
 	conn->ssl_ctx = SSL_CTX_new(conn->ssl_meth);
 	SSL_CTX_set_mode(conn->ssl_ctx, SSL_MODE_AUTO_RETRY);
 
+	fetch_ssl_setup_transport_layer(conn->ssl_ctx, verbose);
+	if (!fetch_ssl_setup_peer_verification(conn->ssl_ctx, verbose))
+		return (-1);
+	if (!fetch_ssl_setup_client_certificate(conn->ssl_ctx, verbose))
+		return (-1);
+
 	conn->ssl = SSL_new(conn->ssl_ctx);
-	if (conn->ssl == NULL){
+	if (conn->ssl == NULL) {
 		fprintf(stderr, "SSL context creation failed\n");
 		return (-1);
 	}
@@ -353,22 +822,38 @@
 			return (-1);
 		}
 	}
+	conn->ssl_cert = SSL_get_peer_certificate(conn->ssl);
 
+	if (!conn->ssl_cert) {
+		fprintf(stderr, "No server SSL certificate\n");
+		return (-1);
+	}
+
+	if (!getenv("SSL_NO_VERIFY_HOSTNAME")) {
+		if (verbose)
+			fetch_info("Verify hostname");
+		if (!fetch_ssl_verify_hostname(conn, URL)) {
+			fprintf(stderr,
+			    "SSL certificate subject doesn't match hostname %s\n",
+			    URL->host);
+			return (-1);
+		}
+	}
+
 	if (verbose) {
 		X509_NAME *name;
 		char *str;
 
-		fprintf(stderr, "SSL connection established using %s\n",
+		fetch_info("SSL connection established using %s",
 		    SSL_get_cipher(conn->ssl));
-		conn->ssl_cert = SSL_get_peer_certificate(conn->ssl);
 		name = X509_get_subject_name(conn->ssl_cert);
-		str = X509_NAME_oneline(name, 0, 0);
-		printf("Certificate subject: %s\n", str);
-		free(str);
-		name = X509_get_issuer_name(conn->ssl_cert);
-		str = X509_NAME_oneline(name, 0, 0);
-		printf("Certificate issuer: %s\n", str);
-		free(str);
+        	str = X509_NAME_oneline(name, 0, 0);
+        	fetch_info("Certificate subject: %s", str);
+        	OPENSSL_free(str);
+        	name = X509_get_issuer_name(conn->ssl_cert);
+        	str = X509_NAME_oneline(name, 0, 0);
+        	fetch_info("Certificate issuer: %s", str);
+        	OPENSSL_free(str);
 	}
 
 	return (0);
@@ -726,6 +1211,22 @@
 
 	if (--conn->ref > 0)
 		return (0);
+#ifdef WITH_SSL
+	if (conn->ssl) {
+		SSL_shutdown(conn->ssl);
+		SSL_set_connect_state(conn->ssl);
+		SSL_free(conn->ssl);
+		conn->ssl = NULL;
+	}
+	if (conn->ssl_ctx) {
+		SSL_CTX_free(conn->ssl_ctx);
+		conn->ssl_ctx = NULL;
+	}
+	if (conn->ssl_cert) {
+		X509_free(conn->ssl_cert);
+		conn->ssl_cert = NULL;
+	}
+#endif
 	ret = close(conn->sd);
 	free(conn->cache.buf);
 	free(conn->buf);
Index: lib/libfetch/common.h
===================================================================
--- lib/libfetch/common.h	(revision 245791)
+++ lib/libfetch/common.h	(working copy)
@@ -87,7 +87,10 @@
 conn_t		*fetch_connect(const char *, int, int, int);
 conn_t		*fetch_reopen(int);
 conn_t		*fetch_ref(conn_t *);
-int		 fetch_ssl(conn_t *, int);
+#ifdef WITH_SSL
+int		 fetch_ssl_cb_verify_crt(int, X509_STORE_CTX*);
+#endif
+int		 fetch_ssl(conn_t *, const struct url *, int);
 ssize_t		 fetch_read(conn_t *, char *, size_t);
 int		 fetch_getln(conn_t *);
 ssize_t		 fetch_write(conn_t *, const char *, size_t);
Index: usr.bin/fetch/fetch.1
===================================================================
--- usr.bin/fetch/fetch.1	(revision 245791)
+++ usr.bin/fetch/fetch.1	(working copy)
@@ -254,8 +254,18 @@
 .Ev HTTP_REFERER ,
 .Ev HTTP_USER_AGENT ,
 .Ev NETRC ,
-.Ev NO_PROXY No and
-.Ev no_proxy .
+.Ev NO_PROXY ,
+.Ev no_proxy ,
+.Ev SSL_ALLOW_SSL2 ,
+.Ev SSL_CA_CERT_FILE ,
+.Ev SSL_CA_CERT_PATH ,
+.Ev SSL_CLIENT_CERT_FILE ,
+.Ev SSL_CLIENT_KEY_FILE ,
+.Ev SSL_CRL_FILE ,
+.Ev SSL_NO_SSL3 ,
+.Ev SSL_NO_TLS1 ,
+.Ev SSL_NO_VERIFY_HOSTNAME and
+.Ev SSL_NO_VERIFY_PEER .
 .Sh EXIT STATUS
 The
 .Nm


>Release-Note:
>Audit-Trail:

From: Michael Gmelin <freebsd@grem.de>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/175514: [patch] Support PKI in libfetch for HTTP over SSL
 (https)
Date: Tue, 22 Jan 2013 18:50:05 +0100

 Please note that due to
 
 http://www.freebsd.org/cgi/query-pr.cgi?pr=172195
 
 the patch won't apply cleanly when downloaded through the PR system.
 Fortunately this only affects the Dag-Erling's last name, so this
 should be easy to fix manually before applying it.
 
 Also note that I added my name to common.c (that's where most of the
 changes happened) and fetch.3. If this seems somehow pretentious or
 inappropriate in any respect, please feel free to change this so it
 feels proportionate to the contribution.
 
 Cheers,
 Michael
 
 -- 
 Michael Gmelin
Responsible-Changed-From-To: freebsd-bugs->des 
Responsible-Changed-By: des 
Responsible-Changed-When: Thu Jan 24 09:49:33 UTC 2013 
Responsible-Changed-Why:  
fetch is mine 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175514 

From: Michael Gmelin <freebsd@grem.de>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/175514: [patch] Support PKI in libfetch for HTTP over SSL
 (https)
Date: Sat, 6 Apr 2013 02:48:50 +0200

 --MP_/yxy.owOr_5x+gU1WTNX/iPo
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline
 
 For completeness sake so that the PR has all the details, please find
 attached the final version of the patch - I didn't get much feedback
 beyond coding style complaints, which I hope addressed all in that
 version.
 
 In case there are problems downloading it from the PR system, it is
 available for download:
 
 http://blog.grem.de/libfetch_20130307.patch
 
 I also documented how the patch can be used in a fully trusted pkgng
 setup:
 
 http://blog.grem.de/sysadmin/Trusted-Package-Distribution-With-pkgng-2013-03-30.html
 
 So long,
 Michael
 
 -- 
 Michael Gmelin
 
 --MP_/yxy.owOr_5x+gU1WTNX/iPo
 Content-Type: text/x-patch; charset=utf8
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment; filename=libfetch_20130307.patch
 
 Index: usr.bin/fetch/fetch.1
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- usr.bin/fetch/fetch.1	(revision 245791)
 +++ usr.bin/fetch/fetch.1	(working copy)
 @@ -1,5 +1,5 @@
  .\"-
 -.\" Copyright (c) 2000-2012 Dag-Erling Sm=C3=B8rgrav
 +.\" Copyright (c) 2000-2013 Dag-Erling Sm=C3=B8rgrav
  .\" All rights reserved.
  .\" Portions Copyright (c) 1999 Massachusetts Institute of Technology; used
  .\" by permission.
 @@ -29,7 +29,7 @@
  .\"
  .\" $FreeBSD$
  .\"
 -.Dd February 28, 2012
 +.Dd January 25, 2013
  .Dt FETCH 1
  .Os
  .Sh NAME
 @@ -254,8 +254,18 @@
  .Ev HTTP_REFERER ,
  .Ev HTTP_USER_AGENT ,
  .Ev NETRC ,
 -.Ev NO_PROXY No and
 -.Ev no_proxy .
 +.Ev NO_PROXY ,
 +.Ev no_proxy ,
 +.Ev SSL_ALLOW_SSL2 ,
 +.Ev SSL_CA_CERT_FILE ,
 +.Ev SSL_CA_CERT_PATH ,
 +.Ev SSL_CLIENT_CERT_FILE ,
 +.Ev SSL_CLIENT_KEY_FILE ,
 +.Ev SSL_CRL_FILE ,
 +.Ev SSL_NO_SSL3 ,
 +.Ev SSL_NO_TLS1 ,
 +.Ev SSL_NO_VERIFY_HOSTNAME and
 +.Ev SSL_NO_VERIFY_PEER .
  .Sh EXIT STATUS
  The
  .Nm
 Index: lib/libfetch/fetch.3
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- lib/libfetch/fetch.3	(revision 245791)
 +++ lib/libfetch/fetch.3	(working copy)
 @@ -1,5 +1,6 @@
  .\"-
  .\" Copyright (c) 1998-2011 Dag-Erling Sm=C3=B8rgrav
 +.\" Copyright (c) 2013 Michael Gmelin <freebsd@grem.de>
  .\" All rights reserved.
  .\"
  .\" Redistribution and use in source and binary forms, with or without
 @@ -25,7 +26,7 @@
  .\"
  .\" $FreeBSD$
  .\"
 -.Dd September 27, 2011
 +.Dd January 25, 2013
  .Dt FETCH 3
  .Os
  .Sh NAME
 @@ -392,6 +393,60 @@
  library,
  .Fn fetchPutHTTP
  is currently unimplemented.
 +.Sh HTTPS SCHEME
 +Based on HTTP SCHEME.
 +By default the peer is verified using the CA bundle located in
 +.Pa /etc/ssl/cert.pem .
 +The file may contain multiple CA certificates.
 +A common source of a current CA bundle is
 +.Pa \%security/ca_root_nss .
 +.Pp
 +The CA bundle used for peer verification can be changed by setting the
 +environment variables
 +.Ev SSL_CA_CERT_FILE
 +to point to a concatenated bundle of trusted certificates and
 +.Ev SSL_CA_CERT_PATH
 +to point to a directory containing hashes of trusted CAs (see
 +.Xr verify 1 ) .
 +.Pp
 +A certificate revocation list (CRL) can be used by setting the
 +environment variable
 +.Ev SSL_CRL_FILE
 +(see
 +.Xr crl 1 ) .
 +.Pp
 +Peer verification can be disabled by setting the environment variable
 +.Ev SSL_NO_VERIFY_PEER .
 +Note that this also disables CRL checking.
 +.Pp
 +By default the service identity is verified according to the rules
 +detailed in RFC6125 (also known as hostname verification).
 +This feature can be disabled by setting the environment variable
 +.Ev SSL_NO_VERIFY_HOSTNAME .
 +.Pp
 +Client certificate based authentication is supported.
 +The environment variable
 +.Ev SSL_CLIENT_CERT_FILE
 +should be set to point to a file containing key and client certificate
 +to be used in PEM format. In case the key is stored in a separate
 +file, the environment variable
 +.Ev SSL_CLIENT_KEY_FILE
 +can be set to point to the key in PEM format.
 +In case the key uses a password, the user will be prompted on standard
 +input (see
 +.Xr PEM 3 ) .
 +.Pp
 +By default
 +.Nm libfetch
 +allows SSLv3 and TLSv1 when negotiating the connecting with the remote
 +peer.
 +You can change this behavior by setting the environment variable
 +.Ev SSL_ALLOW_SSL2
 +to allow SSLv2 (not recommended) and
 +.Ev SSL_NO_SSL3
 +or
 +.Ev SSL_NO_TLS1
 +to disable the respective methods.
  .Sh AUTHENTICATION
  Apart from setting the appropriate environment variables and
  specifying the user name and password in the URL or the
 @@ -579,6 +634,31 @@
  Same as
  .Ev NO_PROXY ,
  for compatibility.
 +.It Ev SSL_ALLOW_SSL2
 +Allow SSL version 2 when negotiating the connection (not recommended).
 +.It Ev SSL_CA_CERT_FILE
 +CA certificate bundle containing trusted CA certificates.
 +Default value:
 +.Pa /etc/ssl/cert.pem .
 +.It Ev SSL_CA_CERT_PATH
 +Path containing trusted CA hashes.
 +.It Ev SSL_CLIENT_CERT_FILE
 +PEM encoded client certificate/key which will be used in
 +client certificate authentication.
 +.It Ev SSL_CLIENT_KEY_FILE
 +PEM encoded client key in case key and client certificate
 +are stored separately.
 +.It Ev SSL_CRL_FILE
 +File containing certificate revocation list.
 +.It Ev SSL_NO_SSL3
 +Don't allow SSL version 3 when negotiating the connection.
 +.It Ev SSL_NO_TLS1
 +Don't allow TLV version 1 when negotiating the connection.
 +.It Ev SSL_NO_VERIFY_HOSTNAME
 +If set, do not verify that the hostname matches the subject of the
 +certificate presented by the server.
 +.It Ev SSL_NO_VERIFY_PEER
 +If set, do not verify the peer certificate against trusted CAs.
  .El
  .Sh EXAMPLES
  To access a proxy server on
 @@ -610,6 +690,19 @@
  .Bd -literal -offset indent
  NO_PROXY=3Dlocalhost,127.0.0.1
  .Ed
 +.Pp
 +Access HTTPS website without any certificate verification whatsoever:
 +.Bd -literal -offset indent
 +SSL_NO_VERIFY_PEER=3D1
 +SSL_NO_VERIFY_HOSTNAME=3D1
 +.Ed
 +.Pp
 +Access HTTPS website using client certificate based authentication
 +and a private CA:
 +.Bd -literal -offset indent
 +SSL_CLIENT_CERT_FILE=3D/path/to/client.pem
 +SSL_CA_CERT_FILE=3D/path/to/myca.pem
 +.Ed
  .Sh SEE ALSO
  .Xr fetch 1 ,
  .Xr ftpio 3 ,
 @@ -678,7 +771,8 @@
  .An Hajimu Umemoto Aq ume@FreeBSD.org ,
  .An Henry Whincup Aq henry@techiebod.com ,
  .An Jukka A. Ukkonen Aq jau@iki.fi ,
 -.An Jean-Fran\(,cois Dockes Aq jf@dockes.org
 +.An Jean-Fran\(,cois Dockes Aq jf@dockes.org ,
 +.An Michael Gmelin Aq freebsd@grem.de
  and others.
  It replaces the older
  .Nm ftpio
 @@ -688,7 +782,9 @@
  .An Jordan K. Hubbard Aq jkh@FreeBSD.org .
  .Pp
  This manual page was written by
 -.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org .
 +.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org
 +and
 +.An Michael Gmelin Aq freebsd@grem.de .
  .Sh BUGS
  Some parts of the library are not yet implemented.
  The most notable
 @@ -717,6 +813,10 @@
  .Fn fetchStatFTP
  does not check that the result of an MDTM command is a valid date.
  .Pp
 +In case password protected keys are used for client certificate based
 +authentication the user is prompted for the password on each and every
 +fetch operation.
 +.Pp
  The man page is incomplete, poorly written and produces badly
  formatted text.
  .Pp
 Index: lib/libfetch/http.c
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- lib/libfetch/http.c	(revision 245791)
 +++ lib/libfetch/http.c	(working copy)
 @@ -1392,7 +1392,7 @@
  		/* fetch_connect() has already set an error code */
  		return (NULL);
  	if (strcasecmp(URL->scheme, SCHEME_HTTPS) =3D=3D 0 &&
 -	    fetch_ssl(conn, verbose) =3D=3D -1) {
 +	    fetch_ssl(conn, URL, verbose) =3D=3D -1) {
  		fetch_close(conn);
  		/* grrr */
  		errno =3D EAUTH;
 Index: lib/libfetch/common.c
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- lib/libfetch/common.c	(revision 245791)
 +++ lib/libfetch/common.c	(working copy)
 @@ -1,5 +1,6 @@
  /*-
   * Copyright (c) 1998-2011 Dag-Erling Sm=C3=B8rgrav
 + * Copyright (c) 2013 Michael Gmelin <freebsd@grem.de>
   * All rights reserved.
   *
   * Redistribution and use in source and binary forms, with or without
 @@ -47,6 +48,10 @@
  #include <string.h>
  #include <unistd.h>
 =20
 +#ifdef WITH_SSL
 +#include <openssl/x509v3.h>
 +#endif
 +
  #include "fetch.h"
  #include "common.h"
 =20
 @@ -317,15 +322,484 @@
  	return (conn);
  }
 =20
 +#ifdef WITH_SSL
 +/*
 + * Convert characters A-Z to lowercase (intentionally avoid any locale
 + * specific conversions).
 + */
 +static char
 +fetch_ssl_tolower(char in)
 +{
 +	if (in >=3D 'A' && in <=3D 'Z')
 +		return (in + 32);
 +	else
 +		return (in);
 +}
 =20
  /*
 + * isalpha implementation that intentionally avoids any locale specific
 + * conversions.
 + */
 +static int
 +fetch_ssl_isalpha(char in)
 +{
 +	return ((in >=3D 'A' && in <=3D 'Z') || (in >=3D 'a' && in <=3D 'z'));
 +}
 +
 +/*
 + * Check if passed hostnames a and b are equal.
 + */
 +static int
 +fetch_ssl_hname_equal(const char *a, size_t alen, const char *b,
 +    size_t blen)
 +{
 +	size_t i;
 +
 +	if (alen !=3D blen)
 +		return (0);
 +	for (i =3D 0; i < alen; ++i) {
 +		if (fetch_ssl_tolower(a[i]) !=3D fetch_ssl_tolower(b[i]))
 +			return (0);
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Check if domain label is traditional, meaning that only A-Z, a-z, 0-9
 + * and '-' (hyphen) are allowed. Hyphens have to be surrounded by alpha-
 + * numeric characters. Double hyphens (like they're found in IDN a-labels
 + * 'xn--') are not allowed. Empty labels are invalid.
 + */
 +static int
 +fetch_ssl_is_trad_domain_label(const char *l, size_t len, int wcok)
 +{
 +	size_t i;
 +
 +	if (!len || l[0] =3D=3D '-' || l[len-1] =3D=3D '-')
 +		return (0);
 +	for (i =3D 0; i < len; ++i) {
 +		if (!isdigit(l[i]) &&
 +		    !fetch_ssl_isalpha(l[i]) &&
 +		    !(l[i] =3D=3D '*' && wcok) &&
 +		    !(l[i] =3D=3D '-' && l[i - 1] !=3D '-'))
 +			return (0);
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Check if host name consists only of numbers. This might indicate an IP
 + * address, which is not a good idea for CN wildcard comparison.
 + */
 +static int
 +fetch_ssl_hname_is_only_numbers(const char *hostname, size_t len)
 +{
 +	size_t i;
 +
 +	for (i =3D 0; i < len; ++i) {
 +		if (!((hostname[i] >=3D '0' && hostname[i] <=3D '9') ||
 +		    hostname[i] =3D=3D '.'))
 +			return (0);
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Check if the host name h passed matches the pattern passed in m which
 + * is usually part of subjectAltName or CN of a certificate presented to
 + * the client. This includes wildcard matching. The algorithm is based on
 + * RFC6125, sections 6.4.3 and 7.2, which clarifies RFC2818 and RFC3280.
 + */
 +static int
 +fetch_ssl_hname_match(const char *h, size_t hlen, const char *m,
 +    size_t mlen)
 +{
 +	int delta, hdotidx, mdot1idx, wcidx;
 +	const char *hdot, *mdot1, *mdot2;
 +	const char *wc; /* wildcard */
 +
 +	if (!(h && *h && m && *m))
 +		return (0);
 +	if ((wc =3D strnstr(m, "*", mlen)) =3D=3D NULL)
 +		return (fetch_ssl_hname_equal(h, hlen, m, mlen));
 +	wcidx =3D wc - m;
 +	/* hostname should not be just dots and numbers */
 +	if (fetch_ssl_hname_is_only_numbers(h, hlen))
 +		return (0);
 +	/* only one wildcard allowed in pattern */
 +	if (strnstr(wc + 1, "*", mlen - wcidx - 1) !=3D NULL)
 +		return (0);
 +	/*
 +	 * there must be at least two more domain labels and
 +	 * wildcard has to be in the leftmost label (RFC6125)
 +	 */
 +	mdot1 =3D strnstr(m, ".", mlen);
 +	if (mdot1 =3D=3D NULL || mdot1 < wc || (mlen - (mdot1 - m)) < 4)
 +		return (0);
 +	mdot1idx =3D mdot1 - m;
 +	mdot2 =3D strnstr(mdot1 + 1, ".", mlen - mdot1idx - 1);
 +	if (mdot2 =3D=3D NULL || (mlen - (mdot2 - m)) < 2)
 +		return (0);
 +	/* hostname must contain a dot and not be the 1st char */
 +	hdot =3D strnstr(h, ".", hlen);
 +	if (hdot =3D=3D NULL || hdot =3D=3D h)
 +		return (0);
 +	hdotidx =3D hdot - h;
 +	/*
 +	 * host part of hostname must be at least as long as
 +	 * pattern it's supposed to match
 +	 */
 +	if (hdotidx < mdot1idx)
 +		return (0);
 +	/*
 +	 * don't allow wildcards in non-traditional domain names
 +	 * (IDN, A-label, U-label...)
 +	 */
 +	if (!fetch_ssl_is_trad_domain_label(h, hdotidx, 0) ||
 +	    !fetch_ssl_is_trad_domain_label(m, mdot1idx, 1))
 +		return (0);
 +	/* match domain part (part after first dot) */
 +	if (!fetch_ssl_hname_equal(hdot, hlen - hdotidx, mdot1,
 +	    mlen - mdot1idx))
 +		return (0);
 +	/* match part left of wildcard */
 +	if (!fetch_ssl_hname_equal(h, wcidx, m, wcidx))
 +		return (0);
 +	/* match part right of wildcard */
 +	delta =3D mdot1idx - wcidx - 1;
 +	if (!fetch_ssl_hname_equal(hdot - delta, delta,
 +	    mdot1 - delta, delta))
 +		return (0);
 +	/* all tests succeded, it's a match */
 +	return (1);
 +}
 +
 +/*
 + * Get numeric host address info - returns NULL if host was not an IP
 + * address. The caller is responsible for deallocation using
 + * freeaddrinfo(3).
 + */
 +static struct addrinfo *
 +fetch_ssl_get_numeric_addrinfo(const char *hostname, size_t len)
 +{
 +	struct addrinfo hints, *res;
 +	char *host;
 +
 +	host =3D (char *)malloc(len + 1);
 +	memcpy(host, hostname, len);
 +	host[len] =3D '\0';
 +	memset(&hints, 0, sizeof(hints));
 +	hints.ai_family =3D PF_UNSPEC;
 +	hints.ai_socktype =3D SOCK_STREAM;
 +	hints.ai_protocol =3D 0;
 +	hints.ai_flags =3D AI_NUMERICHOST;
 +	/* port is not relevant for this purpose */
 +	getaddrinfo(host, "443", &hints, &res);
 +	free(host);
 +	return res;
 +}
 +
 +/*
 + * Compare ip address in addrinfo with address passes.
 + */
 +static int
 +fetch_ssl_ipaddr_match_bin(const struct addrinfo *lhost, const char *rhost,
 +    size_t rhostlen)
 +{
 +	const void *left;
 +
 +	if (lhost->ai_family =3D=3D AF_INET && rhostlen =3D=3D 4) {
 +		left =3D (void *)&((struct sockaddr_in*)(void *)
 +		    lhost->ai_addr)->sin_addr.s_addr;
 +#ifdef INET6
 +	} else if (lhost->ai_family =3D=3D AF_INET6 && rhostlen =3D=3D 16) {
 +		left =3D (void *)&((struct sockaddr_in6 *)(void *)
 +		    lhost->ai_addr)->sin6_addr;
 +#endif
 +	} else
 +		return (0);
 +	return (!memcmp(left, (const void *)rhost, rhostlen) ? 1 : 0);
 +}
 +
 +/*
 + * Compare ip address in addrinfo with host passed. If host is not an IP
 + * address, comparison will fail.
 + */
 +static int
 +fetch_ssl_ipaddr_match(const struct addrinfo *laddr, const char *r,
 +    size_t rlen)
 +{
 +	struct addrinfo *raddr;
 +	int ret;
 +	char *rip;
 +
 +	ret =3D 0;
 +	if ((raddr =3D fetch_ssl_get_numeric_addrinfo(r, rlen)) =3D=3D NULL)
 +		return 0; /* not a numeric host */
 +
 +	if (laddr->ai_family =3D=3D raddr->ai_family) {
 +		if (laddr->ai_family =3D=3D AF_INET) {
 +			rip =3D (char *)&((struct sockaddr_in *)(void *)
 +			    raddr->ai_addr)->sin_addr.s_addr;
 +			ret =3D fetch_ssl_ipaddr_match_bin(laddr, rip, 4);
 +#ifdef INET6
 +		} else if (laddr->ai_family =3D=3D AF_INET6) {
 +			rip =3D (char *)&((struct sockaddr_in6 *)(void *)
 +			    raddr->ai_addr)->sin6_addr;
 +			ret =3D fetch_ssl_ipaddr_match_bin(laddr, rip, 16);
 +#endif
 +		}
 +
 +	}
 +	freeaddrinfo(raddr);
 +	return (ret);
 +}
 +
 +/*
 + * Verify server certificate by subjectAltName.
 + */
 +static int
 +fetch_ssl_verify_altname(STACK_OF(GENERAL_NAME) *altnames,
 +    const char *host, struct addrinfo *ip)
 +{
 +	const GENERAL_NAME *name;
 +	size_t nslen;
 +	int i;
 +	const char *ns;
 +
 +	for (i =3D 0; i < sk_GENERAL_NAME_num(altnames); ++i) {
 +		/*
 +		 * This is a workaround, since the following line causes
 +		 * alignment issues in clang:
 +		 * name =3D sk_GENERAL_NAME_value(subject_alt_names, i);
 +		 * OpenSSL explicitly warns not to use those macros
 +		 * directly, but there isn't much choice (and there
 +		 * shouldn't be any ill side effects)
 +		 */
 +		name =3D (GENERAL_NAME *)SKM_sk_value(void, altnames, i);
 +		ns =3D (const char *)ASN1_STRING_data(name->d.ia5);
 +		nslen =3D (size_t)ASN1_STRING_length(name->d.ia5);
 +
 +		if (name->type =3D=3D GEN_DNS && ip =3D=3D NULL &&
 +		    fetch_ssl_hname_match(host, strlen(host), ns, nslen))
 +			return (1);
 +		else if (name->type =3D=3D GEN_IPADD && ip !=3D NULL &&
 +		    fetch_ssl_ipaddr_match_bin(ip, ns, nslen))
 +			return (1);
 +	}
 +	return (0);
 +}
 +
 +/*
 + * Verify server certificate by CN.
 + */
 +static int
 +fetch_ssl_verify_cn(X509_NAME *subject, const char *host,
 +    struct addrinfo *ip)
 +{
 +	ASN1_STRING *namedata;
 +	X509_NAME_ENTRY *nameentry;
 +	int cnlen, lastpos, loc, ret;
 +	unsigned char *cn;
 +
 +	ret =3D 0;
 +	lastpos =3D -1;
 +	loc =3D -1;
 +	cn =3D NULL;
 +	/* get most specific CN (last entry in list) and compare */
 +	while ((lastpos =3D X509_NAME_get_index_by_NID(subject,
 +	    NID_commonName, lastpos)) !=3D -1)
 +		loc =3D lastpos;
 +
 +	if (loc > -1) {
 +		nameentry =3D X509_NAME_get_entry(subject, loc);
 +		namedata =3D X509_NAME_ENTRY_get_data(nameentry);
 +		cnlen =3D ASN1_STRING_to_UTF8(&cn, namedata);
 +		if (ip =3D=3D NULL &&
 +		    fetch_ssl_hname_match(host, strlen(host), cn, cnlen))
 +			ret =3D 1;
 +		else if (ip !=3D NULL && fetch_ssl_ipaddr_match(ip, cn, cnlen))
 +			ret =3D 1;
 +		OPENSSL_free(cn);
 +	}
 +	return (ret);
 +}
 +
 +/*
 + * Verify that server certificate subjectAltName/CN matches
 + * hostname. First check, if there are alternative subject names. If yes,
 + * those have to match. Only if those don't exist it falls back to
 + * checking the subject's CN.
 + */
 +static int
 +fetch_ssl_verify_hname(X509 *cert, const char *host)
 +{
 +	struct addrinfo *ip;
 +	STACK_OF(GENERAL_NAME) *altnames;
 +	X509_NAME *subject;
 +	int ret;=09
 +
 +	ret =3D 0;
 +	ip =3D fetch_ssl_get_numeric_addrinfo(host, strlen(host));
 +	altnames =3D X509_get_ext_d2i(cert, NID_subject_alt_name,
 +	    NULL, NULL);
 +
 +	if (altnames !=3D NULL) {
 +		ret =3D fetch_ssl_verify_altname(altnames, host, ip);
 +	} else {
 +		subject =3D X509_get_subject_name(cert);
 +		if (subject !=3D NULL)
 +			ret =3D fetch_ssl_verify_cn(subject, host, ip);
 +	}
 +
 +	if (ip !=3D NULL)
 +		freeaddrinfo(ip);
 +	if (altnames !=3D NULL)
 +		GENERAL_NAMES_free(altnames);
 +	return (ret);
 +}
 +
 +/*
 + * Configure transport security layer based on environment.
 + */
 +static void
 +fetch_ssl_setup_transport_layer(SSL_CTX *ctx, int verbose)
 +{
 +	long ssl_ctx_options;
 +
 +	ssl_ctx_options =3D SSL_OP_ALL | SSL_OP_NO_TICKET;
 +	if (getenv("SSL_ALLOW_SSL2") =3D=3D NULL)
 +		ssl_ctx_options |=3D SSL_OP_NO_SSLv2;
 +	if (getenv("SSL_NO_SSL3") !=3D NULL)
 +		ssl_ctx_options |=3D SSL_OP_NO_SSLv3;
 +	if (getenv("SSL_NO_TLS1") !=3D NULL)
 +		ssl_ctx_options |=3D SSL_OP_NO_TLSv1;
 +	if (verbose)
 +		fetch_info("SSL options: %x", ssl_ctx_options);
 +	SSL_CTX_set_options(ctx, ssl_ctx_options);
 +}
 +
 +
 +/*
 + * Configure peer verification based on environment.
 + */
 +static int
 +fetch_ssl_setup_peer_verification(SSL_CTX *ctx, int verbose)
 +{
 +	X509_LOOKUP *crl_lookup;
 +	X509_STORE *crl_store;
 +	const char *ca_cert_file, *ca_cert_path, *crl_file;
 +
 +	if (getenv("SSL_NO_VERIFY_PEER") =3D=3D NULL) {
 +		ca_cert_file =3D getenv("SSL_CA_CERT_FILE") !=3D NULL ?
 +		    getenv("SSL_CA_CERT_FILE") : "/etc/ssl/cert.pem";
 +		ca_cert_path =3D getenv("SSL_CA_CERT_PATH");
 +		if (verbose) {
 +			fetch_info("Peer verification enabled");
 +			if (ca_cert_file !=3D NULL)
 +				fetch_info("Using CA cert file: %s",
 +				    ca_cert_file);
 +			if (ca_cert_path !=3D NULL)
 +				fetch_info("Using CA cert path: %s",
 +				    ca_cert_path);
 +		}
 +		SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER,
 +		    fetch_ssl_cb_verify_crt);
 +		SSL_CTX_load_verify_locations(ctx, ca_cert_file,
 +		    ca_cert_path);
 +		if ((crl_file =3D getenv("SSL_CRL_FILE")) !=3D NULL) {
 +			if (verbose)
 +				fetch_info("Using CRL file: %s", crl_file);
 +			crl_store =3D SSL_CTX_get_cert_store(ctx);
 +			crl_lookup =3D X509_STORE_add_lookup(crl_store,
 +			    X509_LOOKUP_file());
 +			if (crl_lookup =3D=3D NULL ||
 +			    !X509_load_crl_file(crl_lookup, crl_file,
 +				X509_FILETYPE_PEM)) {
 +				fprintf(stderr,
 +				    "Could not load CRL file %s\n",
 +				    crl_file);
 +				return (0);
 +			}
 +			X509_STORE_set_flags(crl_store,
 +			    X509_V_FLAG_CRL_CHECK |
 +			    X509_V_FLAG_CRL_CHECK_ALL);
 +		}
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Configure client certificate based on environment.
 + */
 +static int
 +fetch_ssl_setup_client_certificate(SSL_CTX *ctx, int verbose)
 +{
 +	const char *client_cert_file, *client_key_file;
 +
 +	if ((client_cert_file =3D getenv("SSL_CLIENT_CERT_FILE")) !=3D NULL) {
 +		client_key_file =3D getenv("SSL_CLIENT_KEY_FILE") !=3D NULL ?
 +		    getenv("SSL_CLIENT_KEY_FILE") : client_cert_file;
 +		if (verbose) {
 +			fetch_info("Using client cert file: %s",
 +			    client_cert_file);
 +			fetch_info("Using client key file: %s",
 +			    client_key_file);
 +		}
 +		if (SSL_CTX_use_certificate_chain_file(ctx,
 +			client_cert_file) !=3D 1) {
 +			fprintf(stderr,
 +			    "Could not load client certificate %s\n",
 +			    client_cert_file);
 +			return (0);
 +		}
 +		if (SSL_CTX_use_PrivateKey_file(ctx, client_key_file,
 +			SSL_FILETYPE_PEM) !=3D 1) {
 +			fprintf(stderr,
 +			    "Could not load client key %s\n",
 +			    client_key_file);
 +			return (0);
 +		}
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Callback for SSL certificate verification, this is called on server
 + * cert verification. It takes no decision, but informs the user in case
 + * verification failed.
 + */
 +int
 +fetch_ssl_cb_verify_crt(int verified, X509_STORE_CTX *ctx)
 +{
 +	X509 *crt;
 +	X509_NAME *name;
 +	char *str;
 +
 +	str =3D NULL;
 +	if (!verified) {
 +		if ((crt =3D X509_STORE_CTX_get_current_cert(ctx)) !=3D NULL &&
 +		    (name =3D X509_get_subject_name(crt)) !=3D NULL)
 +			str =3D X509_NAME_oneline(name, 0, 0);
 +		fprintf(stderr, "Certificate verification failed for %s\n",
 +		    str !=3D NULL ? str : "no relevant certificate");
 +		OPENSSL_free(str);
 +	}
 +	return (verified);
 +}
 +
 +#endif
 +
 +/*
   * Enable SSL on a connection.
   */
  int
 -fetch_ssl(conn_t *conn, int verbose)
 +fetch_ssl(conn_t *conn, const struct url *URL, int verbose)
  {
  #ifdef WITH_SSL
  	int ret, ssl_err;
 +	X509_NAME *name;
 +	char *str;
 =20
  	/* Init the SSL library and context */
  	if (!SSL_library_init()){
 @@ -339,8 +813,14 @@
  	conn->ssl_ctx =3D SSL_CTX_new(conn->ssl_meth);
  	SSL_CTX_set_mode(conn->ssl_ctx, SSL_MODE_AUTO_RETRY);
 =20
 +	fetch_ssl_setup_transport_layer(conn->ssl_ctx, verbose);
 +	if (!fetch_ssl_setup_peer_verification(conn->ssl_ctx, verbose))
 +		return (-1);
 +	if (!fetch_ssl_setup_client_certificate(conn->ssl_ctx, verbose))
 +		return (-1);
 +
  	conn->ssl =3D SSL_new(conn->ssl_ctx);
 -	if (conn->ssl =3D=3D NULL){
 +	if (conn->ssl =3D=3D NULL) {
  		fprintf(stderr, "SSL context creation failed\n");
  		return (-1);
  	}
 @@ -353,22 +833,35 @@
  			return (-1);
  		}
  	}
 +	conn->ssl_cert =3D SSL_get_peer_certificate(conn->ssl);
 =20
 +	if (conn->ssl_cert =3D=3D NULL) {
 +		fprintf(stderr, "No server SSL certificate\n");
 +		return (-1);
 +	}
 +
 +	if (getenv("SSL_NO_VERIFY_HOSTNAME") =3D=3D NULL) {
 +		if (verbose)
 +			fetch_info("Verify hostname");
 +		if (!fetch_ssl_verify_hname(conn->ssl_cert, URL->host)) {
 +			fprintf(stderr,
 +			    "SSL certificate subject doesn't match host %s\n",
 +			    URL->host);
 +			return (-1);
 +		}
 +	}
 +
  	if (verbose) {
 -		X509_NAME *name;
 -		char *str;
 -
 -		fprintf(stderr, "SSL connection established using %s\n",
 +		fetch_info("SSL connection established using %s",
  		    SSL_get_cipher(conn->ssl));
 -		conn->ssl_cert =3D SSL_get_peer_certificate(conn->ssl);
  		name =3D X509_get_subject_name(conn->ssl_cert);
  		str =3D X509_NAME_oneline(name, 0, 0);
 -		printf("Certificate subject: %s\n", str);
 -		free(str);
 +		fetch_info("Certificate subject: %s", str);
 +		OPENSSL_free(str);
  		name =3D X509_get_issuer_name(conn->ssl_cert);
  		str =3D X509_NAME_oneline(name, 0, 0);
 -		printf("Certificate issuer: %s\n", str);
 -		free(str);
 +		fetch_info("Certificate issuer: %s", str);
 +		OPENSSL_free(str);
  	}
 =20
  	return (0);
 @@ -726,6 +1219,22 @@
 =20
  	if (--conn->ref > 0)
  		return (0);
 +#ifdef WITH_SSL
 +	if (conn->ssl) {
 +		SSL_shutdown(conn->ssl);
 +		SSL_set_connect_state(conn->ssl);
 +		SSL_free(conn->ssl);
 +		conn->ssl =3D NULL;
 +	}
 +	if (conn->ssl_ctx) {
 +		SSL_CTX_free(conn->ssl_ctx);
 +		conn->ssl_ctx =3D NULL;
 +	}
 +	if (conn->ssl_cert) {
 +		X509_free(conn->ssl_cert);
 +		conn->ssl_cert =3D NULL;
 +	}
 +#endif
  	ret =3D close(conn->sd);
  	free(conn->cache.buf);
  	free(conn->buf);
 Index: lib/libfetch/common.h
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- lib/libfetch/common.h	(revision 245791)
 +++ lib/libfetch/common.h	(working copy)
 @@ -87,7 +87,10 @@
  conn_t		*fetch_connect(const char *, int, int, int);
  conn_t		*fetch_reopen(int);
  conn_t		*fetch_ref(conn_t *);
 -int		 fetch_ssl(conn_t *, int);
 +#ifdef WITH_SSL
 +int		 fetch_ssl_cb_verify_crt(int, X509_STORE_CTX*);
 +#endif
 +int		 fetch_ssl(conn_t *, const struct url *, int);
  ssize_t		 fetch_read(conn_t *, char *, size_t);
  int		 fetch_getln(conn_t *);
  ssize_t		 fetch_write(conn_t *, const char *, size_t);
 
 --MP_/yxy.owOr_5x+gU1WTNX/iPo--

From: r4721@tormail.org
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/175514: [patch] Support PKI in libfetch for HTTP over SSL
 (https)
Date: Tue, 16 Apr 2013 12:31:46 -0000

 # uname -r -m
 9.1-STABLE amd64
 
 # fetch https://www.google.com
 fetch: https://www.google.com: size of remote file is not known
 www.google.com                                          18 kB  23 kBps
 
 # fetch https://173.194.44.49
 SSL certificate subject doesn't match host 173.194.44.49
 fetch: https://173.194.44.49: Authentication error
 
 it works also using the https-over-http proxy support imported from 10.
 
 a note, the ETCSYMLINK option must be enabled in security/ca_root_nss, and
 this is not the
 default.
 
 security/ca_root_nss installed with default options:
 
 Certificate verification failed for /C=US/O=Google Inc/CN=Google Internet
 Authority
 97694:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
 verify failed:/usr
 /src/secure/lib/libssl/../../../crypto/openssl/ssl/s3_clnt.c:993:
 fetch: https://www.google.com: Authentication error
 
 at least it should warn that ca_root_nss must be installed with
 non-default options. or make
 security/ca_root_nss default to ETCSYMLINK=on.
 
 perhaps nice, if cert db installed, work with verification. if cert db not
 installed (or not
 installed with ETCSYMLINK=on), function without verification but print
 WARNING: certificate not
 being verified due to no database present.
 

From: r4721@tormail.org
To: bug-followup@freebsd.org
Cc: freebsd@grem.de
Subject: Re: kern/175514: [patch] Support PKI in libfetch for HTTP over SSL
 (https)
Date: Sat, 13 Jul 2013 06:31:03 +0000

 testing this patch more, it seems to work well. it successfully downloaded a
 file from lighttpd using server and client certificate verification using
 TLS 1.2 and ECDHE-ECDSA-AES256-GCM-SHA384.
 
 i'd like to be asked what to do about unverifiable (such as self signed)
 certificates, like subversion does:
 
 Error validating server certificate for 'https://svn0.eu.freebsd.org:443':
  - The certificate is not issued by a trusted authority. Use the
    fingerprint to validate the certificate manually!
  - The certificate hostname does not match.
 Certificate information:
  - Hostname: svnmir.bme.FreeBSD.org
  - Valid: from Jun 29 12:24:17 2013 GMT until Jun 29 12:24:17 2015 GMT
  - Issuer: clusteradm, FreeBSD.org, (null), CA, US (clusteradm@FreeBSD.org)
  - Fingerprint: 39:B0:53:35:CE:60:C7:BB:00:54:96:96:71:10:94:BB:CE:1C:07:A7
 (R)eject, accept (t)emporarily or accept (p)ermanently?
 
 curl for reference:
 > curl https://svn0.eu.freebsd.org
 curl: (60) SSL certificate problem: unable to get local issuer certificate
 More details here: http://curl.haxx.se/docs/sslcerts.html
 
 curl performs SSL certificate verification by default, using a "bundle"
  of Certificate Authority (CA) public keys (CA certs). If the default
  bundle file isn't adequate, you can specify an alternate file
  using the --cacert option.
 If this HTTPS server uses a certificate signed by a CA represented in
  the bundle, the certificate verification probably failed due to a
  problem with the certificate (it might be expired, or the name might
  not match the domain name in the URL).
 If you'd like to turn off curl's verification of the certificate, use
  the -k (or --insecure) option.
 
 for ports, SSL_NO_VERIFY_PEER could simply be set as ports already checks
 distfiles with SHA256.
 
 it would be very nice to have a freebsd.org cert in the base system that
 pkg/pkg_add/bsdinstall could verify against.
 

From: Michael Gmelin <freebsd@grem.de>
To: r4721@tormail.org
Cc: bug-followup@freebsd.org, Dag-Erling =?UTF-8?B?U23DuHJncmF2?=
 <des@des.no>
Subject: Re: kern/175514: [patch] Support PKI in libfetch for HTTP over SSL
 (https)
Date: Sat, 13 Jul 2013 11:20:29 +0200

 --MP_/t5sG0_JQ27vT/i9nIo_rTfQ
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline
 
 Hi,
 
 Thanks for testing. Please find attached the latest version of the
 patch from May this year. It makes a couple of features available
 to fetch(1) through options at the command line. It also adds long
 versions for most fetch(1) command line options. See "man 1 fetch"
 after installation for details. I originally sent this patch to
 DES directly and forgot to add a bug follow-up.
 
 Since the PR system might corrupt the patch I also made it available at:
 http://blog.grem.de/libfetch_20130518.patch
 
 The answers below assume that you're using the latest version of the
 patch.
 
 > testing this patch more, it seems to work well. it successfully
 > downloaded a file from lighttpd using server and client certificate
 > verification using TLS 1.2 and ECDHE-ECDSA-AES256-GCM-SHA384.
 > 
 > i'd like to be asked what to do about unverifiable (such as self
 > signed) certificates, like subversion does:
 > 
 > Error validating server certificate for
 > 'https://svn0.eu.freebsd.org:443':
 >  - The certificate is not issued by a trusted authority. Use the
 >    fingerprint to validate the certificate manually!
 >  - The certificate hostname does not match.
 > Certificate information:
 >  - Hostname: svnmir.bme.FreeBSD.org
 >  - Valid: from Jun 29 12:24:17 2013 GMT until Jun 29 12:24:17 2015 GMT
 >  - Issuer: clusteradm, FreeBSD.org, (null), CA, US
 > (clusteradm@FreeBSD.org)
 >  - Fingerprint:
 > 39:B0:53:35:CE:60:C7:BB:00:54:96:96:71:10:94:BB:CE:1C:07:A7 (R)eject,
 > accept (t)emporarily or accept (p)ermanently?
 
 I like the idea, but I don't see an easy way to add this to fetch the
 way subversion does short term, since the patch is under review in its
 current form for months now. So I would like to add this once the
 current version has been committed.
 
 > 
 > curl for reference:
 > > curl https://svn0.eu.freebsd.org
 > curl: (60) SSL certificate problem: unable to get local issuer
 > certificate More details here: http://curl.haxx.se/docs/sslcerts.html
 > 
 > curl performs SSL certificate verification by default, using a
 > "bundle" of Certificate Authority (CA) public keys (CA certs). If the
 > default bundle file isn't adequate, you can specify an alternate file
 >  using the --cacert option.
 
 You can do the same with fetch(1) using the --ca-cert command line
 option or by setting SSL_CA_CERT_FILE.
 
 > If this HTTPS server uses a certificate signed by a CA represented in
 >  the bundle, the certificate verification probably failed due to a
 >  problem with the certificate (it might be expired, or the name might
 >  not match the domain name in the URL).
 > If you'd like to turn off curl's verification of the certificate, use
 >  the -k (or --insecure) option.
 
 There is no such option as --insecure, but you can combine
 --no-verify-peer and --no-verify-hostname to get the same effect - most
 of the time --no-verify-peer is sufficient though.
 
 > 
 > for ports, SSL_NO_VERIFY_PEER could simply be set as ports already
 > checks distfiles with SHA256.
 
 Agreed. Alternatively --no-verify-peer could be added to
 the default FETCH_ARGS in /usr/ports/Mk/bsd.port.mk.
 
 > 
 > it would be very nice to have a freebsd.org cert in the base system
 > that pkg/pkg_add/bsdinstall could verify against.
 > 
 
 Agreed again. I'm not sure whom to ask about adding those to the
 distribution and also what the strategy of the project is in general
 though.
 
 In case of pkg: It's not making use of SSL or signed repositories by
 default (to be fair it's still kind of beta). If you run your own
 packagesite you can set options to libfetch for pkg
 in /usr/local/etc/pkg.conf, e.g.:
 
 packagesite: https://packagemaster.example.org
 pubkey: /etc/ssl/reposign.pubkey
 PKG_ENV: { SSL_CLIENT_KEY_FILE : /etc/ssl/pkgclient.key ,
            SSL_CLIENT_CERT_FILE : /etc/ssl/pkgclient.crt ,
            SSL_CA_CERT_FILE : /etc/ssl/pkgmaster.pem }
 
 Thanks again for testing.
 
 - Michael
 
 -- 
 Michael Gmelin
 
 --MP_/t5sG0_JQ27vT/i9nIo_rTfQ
 Content-Type: text/x-patch
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment; filename=libfetch_20130518.patch
 
 Index: usr.bin/fetch/fetch.1
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- usr.bin/fetch/fetch.1	(revision 249052)
 +++ usr.bin/fetch/fetch.1	(working copy)
 @@ -1,5 +1,6 @@
  .\"-
 -.\" Copyright (c) 2000-2012 Dag-Erling Sm=C3=B8rgrav
 +.\" Copyright (c) 2000-2013 Dag-Erling Sm=C3=B8rgrav
 +.\" Copyright (c) 2013 Michael Gmelin <freebsd@grem.de>
  .\" All rights reserved.
  .\" Portions Copyright (c) 1999 Massachusetts Institute of Technology; used
  .\" by permission.
 @@ -29,7 +30,7 @@
  .\"
  .\" $FreeBSD$
  .\"
 -.Dd February 28, 2012
 +.Dd May 18, 2013
  .Dt FETCH 1
  .Os
  .Sh NAME
 @@ -38,22 +39,51 @@
  .Sh SYNOPSIS
  .Nm
  .Op Fl 146AadFlMmnPpqRrsUv
 +.Op Fl -allow-sslv2
  .Op Fl B Ar bytes
 +.Op Fl -bind-address=3D Ns Ar host
 +.Op Fl -ca-cert=3D Ns Ar file
 +.Op Fl -ca-path=3D Ns Ar dir
 +.Op Fl -cert=3D Ns Ar file
 +.Op Fl -crl=3D Ns Ar file
  .Op Fl i Ar file
 +.Op Fl -key=3D Ns Ar file
  .Op Fl N Ar file
 +.Op Fl -no-passive
 +.Op Fl -no-proxy=3D Ns Ar list
 +.Op Fl -no-sslv3
 +.Op Fl -no-tlsv1
 +.Op Fl -no-verify-hostname
 +.Op Fl -no-verify-peer
  .Op Fl o Ar file
 +.Op Fl -referer=3D Ns Ar URL
  .Op Fl S Ar bytes
  .Op Fl T Ar seconds
 +.Op Fl -user-agent=3D Ns Ar agent-string
  .Op Fl w Ar seconds
  .Ar URL ...
  .Nm
  .Op Fl 146AadFlMmnPpqRrsUv
  .Op Fl B Ar bytes
 +.Op Fl -bind-address=3D Ns Ar host
 +.Op Fl -ca-cert=3D Ns Ar file
 +.Op Fl -ca-path=3D Ns Ar dir
 +.Op Fl -cert=3D Ns Ar file
 +.Op Fl -crl=3D Ns Ar file
  .Op Fl i Ar file
 +.Op Fl -key=3D Ns Ar file
  .Op Fl N Ar file
 +.Op Fl -no-passive
 +.Op Fl -no-proxy=3D Ns Ar list
 +.Op Fl -no-sslv3
 +.Op Fl -no-tlsv1
 +.Op Fl -no-verify-hostname
 +.Op Fl -no-verify-peer
  .Op Fl o Ar file
 +.Op Fl -referer=3D Ns Ar URL
  .Op Fl S Ar bytes
  .Op Fl T Ar seconds
 +.Op Fl -user-agent=3D Ns Ar agent-string
  .Op Fl w Ar seconds
  .Fl h Ar host Fl f Ar file Oo Fl c Ar dir Oc
  .Sh DESCRIPTION
 @@ -67,23 +97,26 @@
  .Pp
  The following options are available:
  .Bl -tag -width Fl
 -.It Fl 1
 +.It Fl 1 , -one-file
  Stop and return exit code 0 at the first successfully retrieved file.
 -.It Fl 4
 +.It Fl 4 , -ipv4-only
  Forces
  .Nm
  to use IPv4 addresses only.
 -.It Fl 6
 +.It Fl 6 , -ipv6-only
  Forces
  .Nm
  to use IPv6 addresses only.
 -.It Fl A
 +.It Fl A , -no-redirect
  Do not automatically follow ``temporary'' (302) redirects.
  Some broken Web sites will return a redirect instead of a not-found
  error when the requested object does not exist.
 -.It Fl a
 +.It Fl a , -retry
  Automatically retry the transfer upon soft failures.
 -.It Fl B Ar bytes
 +.It Fl -allow-sslv2
 +[SSL]
 +Allow SSL version 2 when negotiating the connection.
 +.It Fl B Ar bytes , Fl -buffer-size=3D Ns Ar bytes
  Specify the read buffer size in bytes.
  The default is 4096 bytes.
  Attempts to set a buffer size lower than this will be silently
 @@ -92,15 +125,43 @@
  two or higher (see the
  .Fl v
  flag).
 +.It Fl -bind-address=3D Ns Ar host
 +Specifies a hostname or IP address to which sockets used for outgoing
 +connections will be bound.
  .It Fl c Ar dir
  The file to retrieve is in directory
  .Ar dir
  on the remote host.
  This option is deprecated and is provided for backward compatibility
  only.
 -.It Fl d
 +.It Fl -ca-cert=3D Ns Ar file
 +[SSL]
 +Path to certificate bundle containing trusted CA certificates.=20
 +If not specified,
 +.Pa /etc/ssl/cert.pem
 +is used.
 +The file may contain multiple CA certificates. The port
 +.Pa security/ca_root_nss
 +is a common source of a current CA bundle.
 +.It Fl -ca-path=3D Ns Ar dir
 +[SSL]
 +The directory
 +.Ar dir
 +contains trusted CA hashes.
 +.It Fl -cert=3D Ns Ar file
 +[SSL]
 +.Ar file
 +is a PEM encoded client certificate/key which will be used in
 +client certificate authentication.
 +.It Fl -crl=3D Ns Ar file
 +[SSL]
 +Points to certificate revocation list
 +.Ar file ,
 +which has to be in PEM format and may contain peer certificates that have
 +been revoked.
 +.It Fl d , -direct
  Use a direct connection even if a proxy is configured.
 -.It Fl F
 +.It Fl F , -force-restart
  In combination with the
  .Fl r
  flag, forces a restart even if the local and remote files have
 @@ -118,17 +179,22 @@
  .Ar host .
  This option is deprecated and is provided for backward compatibility
  only.
 -.It Fl i Ar file
 +.It Fl i Ar file , Fl -if-modified-since=3D Ns Ar file
  If-Modified-Since mode: the remote file will only be retrieved if it
  is newer than
  .Ar file
  on the local host.
  (HTTP only)
 -.It Fl l
 +.It Fl -key=3D Ns Ar file
 +[SSL]
 +.Ar file
 +is a PEM encoded client key that will be used in client certificate
 +authentication in case key and client certificate are stored separately.
 +.It Fl l , -symlink
  If the target is a file-scheme URL, make a symbolic link to the target
  rather than trying to copy it.
  .It Fl M
 -.It Fl m
 +.It Fl m , -mirror
  Mirror mode: if the file already exists locally and has the same size
  and modification time as the remote file, it will not be fetched.
  Note that the
 @@ -136,7 +202,7 @@
  and
  .Fl r
  flags are mutually exclusive.
 -.It Fl N Ar file
 +.It Fl N Ar file , Fl -netrc=3D Ns Ar file
  Use
  .Ar file
  instead of
 @@ -146,9 +212,28 @@
  .Xr ftp 1
  for a description of the file format.
  This feature is experimental.
 -.It Fl n
 +.It Fl n , -no-mtime
  Do not preserve the modification time of the transferred file.
 -.It Fl o Ar file
 +.It Fl -no-passive
 +Forces the FTP code to use active mode.
 +.It Fl -no-proxy=3D Ns Ar list
 +Either a single asterisk, which disables the use of proxies
 +altogether, or a comma- or whitespace-separated list of hosts for
 +which proxies should not be used.
 +.It Fl -no-sslv3
 +[SSL]
 +Don't allow SSL version 3 when negotiating the connection.
 +.It Fl -no-tlsv1
 +[SSL]
 +Don't allow TLS version 1 when negotiating the connection.
 +.It Fl -no-verify-hostname
 +[SSL]
 +Do not verify that the hostname matches the subject of the
 +certificate presented by the server.
 +.It Fl -no-verify-peer
 +[SSL]
 +Do not verify the peer certificate against trusted CAs.
 +.It Fl o Ar file , Fl output=3D Ns Ar file
  Set the output file name to
  .Ar file .
  By default, a ``pathname'' is extracted from the specified URI, and
 @@ -163,36 +248,45 @@
  argument is a directory, fetched file(s) will be placed within the
  directory, with name(s) selected as in the default behaviour.
  .It Fl P
 -.It Fl p
 +.It Fl p , -passive
  Use passive FTP.
  These flags have no effect, since passive FTP is the default, but are
  provided for compatibility with earlier versions where active FTP was
  the default.
 -To force active mode, set the
 +To force active mode, use the
 +.Fl -no-passive
 +flag or set the
  .Ev FTP_PASSIVE_MODE
  environment variable to
  .Ql NO .
 -.It Fl q
 +.It Fl -referer=3D Ns Ar URL
 +Specifies the referrer URL to use for HTTP requests.
 +If
 +.Ar URL
 +is set to
 +.Dq auto ,
 +the document URL will be used as referrer URL.
 +.It Fl q , -quiet
  Quiet mode.
 -.It Fl R
 +.It Fl R , -keep-output
  The output files are precious, and should not be deleted under any
  circumstances, even if the transfer failed or was incomplete.
 -.It Fl r
 +.It Fl r , -restart
  Restart a previously interrupted transfer.
  Note that the
  .Fl m
  and
  .Fl r
  flags are mutually exclusive.
 -.It Fl S Ar bytes
 +.It Fl S Ar bytes , Fl -require-size=3D Ns Ar bytes
  Require the file size reported by the server to match the specified
  value.
  If it does not, a message is printed and the file is not fetched.
  If the server does not support reporting file sizes, this option is
  ignored and the file is fetched unconditionally.
 -.It Fl s
 +.It Fl s , -print-size
  Print the size in bytes of each requested file, without fetching it.
 -.It Fl T Ar seconds
 +.It Fl T Ar seconds , Fl -timeout=3D Ns Ar seconds
  Set timeout value to
  .Ar seconds .
  Overrides the environment variables
 @@ -200,15 +294,19 @@
  for FTP transfers or
  .Ev HTTP_TIMEOUT
  for HTTP transfers if set.
 -.It Fl U
 +.It Fl U , -passive-portrange-default
  When using passive FTP, allocate the port for the data connection from
  the low (default) port range.
  See
  .Xr ip 4
  for details on how to specify which port range this corresponds to.
 -.It Fl v
 +.It Fl -user-agent=3D Ns Ar agent-string
 +Specifies the User-Agent string to use for HTTP requests.
 +This can be useful when working with HTTP origin or proxy servers that
 +differentiate between user agents.
 +.It Fl v , -verbose
  Increase verbosity level.
 -.It Fl w Ar seconds
 +.It Fl w Ar seconds , Fl -retry-delay=3D Ns Ar seconds
  When the
  .Fl a
  flag is specified, wait this many seconds between successive retries.
 @@ -254,8 +352,18 @@
  .Ev HTTP_REFERER ,
  .Ev HTTP_USER_AGENT ,
  .Ev NETRC ,
 -.Ev NO_PROXY No and
 -.Ev no_proxy .
 +.Ev NO_PROXY ,
 +.Ev no_proxy ,
 +.Ev SSL_ALLOW_SSL2 ,
 +.Ev SSL_CA_CERT_FILE ,
 +.Ev SSL_CA_CERT_PATH ,
 +.Ev SSL_CLIENT_CERT_FILE ,
 +.Ev SSL_CLIENT_KEY_FILE ,
 +.Ev SSL_CRL_FILE ,
 +.Ev SSL_NO_SSL3 ,
 +.Ev SSL_NO_TLS1 ,
 +.Ev SSL_NO_VERIFY_HOSTNAME and
 +.Ev SSL_NO_VERIFY_PEER .
  .Sh EXIT STATUS
  The
  .Nm
 @@ -292,7 +400,9 @@
  and later completely rewritten to use the
  .Xr fetch 3
  library by
 -.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org .
 +.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org
 +and
 +.An Michael Gmelin Aq freebsd@grem.de .
  .Sh NOTES
  The
  .Fl b
 Index: usr.bin/fetch/fetch.c
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- usr.bin/fetch/fetch.c	(revision 249052)
 +++ usr.bin/fetch/fetch.c	(working copy)
 @@ -1,5 +1,6 @@
  /*-
   * Copyright (c) 2000-2011 Dag-Erling Sm=C3=B8rgrav
 + * Copyright (c) 2013 Michael Gmelin <freebsd@grem.de>
   * All rights reserved.
   *
   * Redistribution and use in source and binary forms, with or without
 @@ -37,6 +38,7 @@
  #include <ctype.h>
  #include <err.h>
  #include <errno.h>
 +#include <getopt.h>
  #include <signal.h>
  #include <stdint.h>
  #include <stdio.h>
 @@ -93,7 +95,79 @@
  long	 http_timeout =3D TIMEOUT;	/* default timeout for HTTP transfers */
  char	*buf;		/* transfer buffer */
 =20
 +enum options
 +{
 +	OPTION_BIND_ADDRESS,
 +	OPTION_NO_FTP_PASSIVE_MODE,
 +	OPTION_HTTP_REFERER,
 +	OPTION_HTTP_USER_AGENT,
 +	OPTION_NO_PROXY,
 +	OPTION_SSL_ALLOW_SSL2,
 +	OPTION_SSL_CA_CERT_FILE,
 +	OPTION_SSL_CA_CERT_PATH,
 +	OPTION_SSL_CLIENT_CERT_FILE,
 +	OPTION_SSL_CLIENT_KEY_FILE,
 +	OPTION_SSL_CRL_FILE,
 +	OPTION_SSL_NO_SSL3,
 +	OPTION_SSL_NO_TLS1,=09
 +	OPTION_SSL_NO_VERIFY_HOSTNAME,
 +	OPTION_SSL_NO_VERIFY_PEER
 +};
 =20
 +
 +static struct option longopts[] =3D
 +{
 +	/* mapping to single character argument */
 +	{ "one-file", no_argument, NULL, '1' },
 +	{ "ipv4-only", no_argument, NULL, '4' },
 +	{ "ipv6-only", no_argument, NULL, '6' },
 +	{ "no-redirect", no_argument, NULL, 'A' },
 +	{ "retry", no_argument, NULL, 'a' },
 +	{ "buffer-size", required_argument, NULL, 'B' },
 +	/* -c not mapped, since it's deprecated */
 +	{ "direct", no_argument, NULL, 'd' },
 +	{ "force-restart", no_argument, NULL, 'F' },
 +	/* -f not mapped, since it's deprecated */
 +	/* -h not mapped, since it's deprecated */
 +	{ "if-modified-since", required_argument, NULL, 'i' },
 +	{ "symlink", no_argument, NULL, 'l' },
 +	/* -M not mapped since it's the same as -m */
 +	{ "mirror", no_argument, NULL, 'm' },
 +	{ "netrc", required_argument, NULL, 'N' },
 +	{ "no-mtime", no_argument, NULL, 'n' },
 +	{ "output", required_argument, NULL, 'o' },
 +	/* -P not mapped since it's the same as -p */
 +	{ "passive", no_argument, NULL, 'p' },
 +	{ "quiet", no_argument, NULL, 'q' },
 +	{ "keep-output", no_argument, NULL, 'R' },
 +	{ "restart", no_argument, NULL, 'r' },
 +	{ "require-size", required_argument, NULL, 'S' },
 +	{ "print-size", no_argument, NULL, 's' },
 +	{ "timeout", required_argument, NULL, 'T' },
 +	{ "passive-portrange-default", no_argument, NULL, 'T' },
 +	{ "verbose", no_argument, NULL, 'v' },
 +	{ "retry-delay", required_argument, NULL, 'w' },
 +=09
 +	/* options without a single character equivalent */
 +	{ "bind-address", required_argument, NULL, OPTION_BIND_ADDRESS },
 +	{ "no-passive", no_argument, NULL, OPTION_NO_FTP_PASSIVE_MODE },
 +	{ "referer", required_argument, NULL, OPTION_HTTP_REFERER },
 +	{ "user-agent", required_argument, NULL, OPTION_HTTP_USER_AGENT },
 +	{ "no-proxy", required_argument, NULL, OPTION_NO_PROXY },
 +	{ "allow-sslv2", no_argument, NULL, OPTION_SSL_ALLOW_SSL2 },
 +	{ "ca-cert", required_argument, NULL, OPTION_SSL_CA_CERT_FILE },
 +	{ "ca-path", required_argument, NULL, OPTION_SSL_CA_CERT_PATH },
 +	{ "cert", required_argument, NULL, OPTION_SSL_CLIENT_CERT_FILE },
 +	{ "key", required_argument, NULL, OPTION_SSL_CLIENT_KEY_FILE },
 +	{ "crl", required_argument, NULL, OPTION_SSL_CRL_FILE },
 +	{ "no-sslv3", no_argument, NULL, OPTION_SSL_NO_SSL3 },
 +	{ "no-tlsv1", no_argument, NULL, OPTION_SSL_NO_TLS1 },
 +	{ "no-verify-hostname", no_argument, NULL, OPTION_SSL_NO_VERIFY_HOSTNAME =
 },
 +	{ "no-verify-peer", no_argument, NULL, OPTION_SSL_NO_VERIFY_PEER },
 +
 +	{ NULL, 0, NULL, 0 }
 +};
 +
  /*
   * Signal handler
   */
 @@ -752,11 +826,19 @@
  static void
  usage(void)
  {
 -	fprintf(stderr, "%s\n%s\n%s\n%s\n",
 -"usage: fetch [-146AadFlMmnPpqRrsUv] [-B bytes] [-N file] [-o file] [-S by=
 tes]",
 -"       [-T seconds] [-w seconds] [-i file] URL ...",
 -"       fetch [-146AadFlMmnPpqRrsUv] [-B bytes] [-N file] [-o file] [-S by=
 tes]",
 -"       [-T seconds] [-w seconds] [-i file] -h host -f file [-c dir]");
 +	fprintf(stderr, "%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n",
 +"usage: fetch [-146AadFlMmnPpqRrsUv] [--allow-sslv2] [-B bytes]",
 +"       [--bind-address=3Dhost] [--ca-cert=3Dfile] [--ca-path=3Ddir] [--ce=
 rt=3Dfile]",
 +"       [--crl=3Dfile] [-i file] [--key=3Dfile] [-N file] [--no-passive]",
 +"       [--no-proxy=3Dlist] [--no-sslv3] [--no-tlsv1] [--no-verify-hostnam=
 e]",
 +"       [--no-verify-peer] [-o file] [--referer=3DURL] [-S bytes] [-T seco=
 nds]",
 +"       [--user-agent=3Dagent-string] [-w seconds] URL ...",
 +"       fetch [-146AadFlMmnPpqRrsUv] [--allow-sslv2] [-B bytes]",
 +"       [--bind-address=3Dhost] [--ca-cert=3Dfile] [--ca-path=3Ddir] [--ce=
 rt=3Dfile]",
 +"       [--crl=3Dfile] [-i file] [--key=3Dfile] [-N file] [--no-passive]",
 +"       [--no-proxy=3Dlist] [--no-sslv3] [--no-tlsv1] [--no-verify-hostnam=
 e]",
 +"       [--no-verify-peer] [-o file] [--referer=3DURL] [-S bytes] [-T seco=
 nds]",
 +"       [--user-agent=3Dagent-string] [-w seconds] -h host -f file [-c dir=
 ]");
  }
 =20
 =20
 @@ -772,8 +854,10 @@
  	char *end, *q;
  	int c, e, r;
 =20
 -	while ((c =3D getopt(argc, argv,
 -	    "146AaB:bc:dFf:Hh:i:lMmN:nPpo:qRrS:sT:tUvw:")) !=3D -1)
 +
 +	while ((c =3D getopt_long(argc, argv,
 +	    "146AaB:bc:dFf:Hh:i:lMmN:nPpo:qRrS:sT:tUvw:",
 +	    longopts, NULL)) !=3D -1)
  		switch (c) {
  		case '1':
  			once_flag =3D 1;
 @@ -887,6 +971,51 @@
  			if (*optarg =3D=3D '\0' || *end !=3D '\0')
  				errx(1, "invalid delay (%s)", optarg);
  			break;
 +		case OPTION_BIND_ADDRESS:
 +			setenv("FETCH_BIND_ADDRESS", optarg, 1);
 +			break;
 +		case OPTION_NO_FTP_PASSIVE_MODE:
 +			setenv("FTP_PASSIVE_MODE", "no", 1);
 +			break;
 +		case OPTION_HTTP_REFERER:
 +			setenv("HTTP_REFERER", optarg, 1);
 +			break;
 +		case OPTION_HTTP_USER_AGENT:
 +			setenv("HTTP_USER_AGENT", optarg, 1);
 +			break;
 +		case OPTION_NO_PROXY:
 +			setenv("NO_PROXY", optarg, 1);
 +			break;
 +		case OPTION_SSL_ALLOW_SSL2:
 +			setenv("SSL_ALLOW_SSL2", "", 1);
 +			break;
 +		case OPTION_SSL_CA_CERT_FILE:
 +			setenv("SSL_CA_CERT_FILE", optarg, 1);
 +			break;
 +		case OPTION_SSL_CA_CERT_PATH:
 +			setenv("SSL_CA_CERT_PATH", optarg, 1);
 +			break;
 +		case OPTION_SSL_CLIENT_CERT_FILE:
 +			setenv("SSL_CLIENT_CERT_FILE", optarg, 1);
 +			break;
 +		case OPTION_SSL_CLIENT_KEY_FILE:
 +			setenv("SSL_CLIENT_KEY_FILE", optarg, 1);
 +			break;
 +		case OPTION_SSL_CRL_FILE:
 +			setenv("SSL_CLIENT_CRL_FILE", optarg, 1);
 +			break;
 +		case OPTION_SSL_NO_SSL3:
 +			setenv("SSL_NO_SSL3", "", 1);
 +			break;
 +		case OPTION_SSL_NO_TLS1:
 +			setenv("SSL_NO_TLS1", "", 1);
 +			break;
 +		case OPTION_SSL_NO_VERIFY_HOSTNAME:
 +			setenv("SSL_NO_VERIFY_HOSTNAME", "", 1);
 +			break;
 +		case OPTION_SSL_NO_VERIFY_PEER:
 +			setenv("SSL_NO_VERIFY_PEER", "", 1);
 +			break;
  		default:
  			usage();
  			exit(1);
 Index: lib/libfetch/fetch.3
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- lib/libfetch/fetch.3	(revision 249052)
 +++ lib/libfetch/fetch.3	(working copy)
 @@ -1,5 +1,6 @@
  .\"-
  .\" Copyright (c) 1998-2011 Dag-Erling Sm=C3=B8rgrav
 +.\" Copyright (c) 2013 Michael Gmelin <freebsd@grem.de>
  .\" All rights reserved.
  .\"
  .\" Redistribution and use in source and binary forms, with or without
 @@ -25,7 +26,7 @@
  .\"
  .\" $FreeBSD$
  .\"
 -.Dd September 27, 2011
 +.Dd January 25, 2013
  .Dt FETCH 3
  .Os
  .Sh NAME
 @@ -392,6 +393,60 @@
  library,
  .Fn fetchPutHTTP
  is currently unimplemented.
 +.Sh HTTPS SCHEME
 +Based on HTTP SCHEME.
 +By default the peer is verified using the CA bundle located in
 +.Pa /etc/ssl/cert.pem .
 +The file may contain multiple CA certificates.
 +A common source of a current CA bundle is
 +.Pa \%security/ca_root_nss .
 +.Pp
 +The CA bundle used for peer verification can be changed by setting the
 +environment variables
 +.Ev SSL_CA_CERT_FILE
 +to point to a concatenated bundle of trusted certificates and
 +.Ev SSL_CA_CERT_PATH
 +to point to a directory containing hashes of trusted CAs (see
 +.Xr verify 1 ) .
 +.Pp
 +A certificate revocation list (CRL) can be used by setting the
 +environment variable
 +.Ev SSL_CRL_FILE
 +(see
 +.Xr crl 1 ) .
 +.Pp
 +Peer verification can be disabled by setting the environment variable
 +.Ev SSL_NO_VERIFY_PEER .
 +Note that this also disables CRL checking.
 +.Pp
 +By default the service identity is verified according to the rules
 +detailed in RFC6125 (also known as hostname verification).
 +This feature can be disabled by setting the environment variable
 +.Ev SSL_NO_VERIFY_HOSTNAME .
 +.Pp
 +Client certificate based authentication is supported.
 +The environment variable
 +.Ev SSL_CLIENT_CERT_FILE
 +should be set to point to a file containing key and client certificate
 +to be used in PEM format. In case the key is stored in a separate
 +file, the environment variable
 +.Ev SSL_CLIENT_KEY_FILE
 +can be set to point to the key in PEM format.
 +In case the key uses a password, the user will be prompted on standard
 +input (see
 +.Xr PEM 3 ) .
 +.Pp
 +By default
 +.Nm libfetch
 +allows SSLv3 and TLSv1 when negotiating the connecting with the remote
 +peer.
 +You can change this behavior by setting the environment variable
 +.Ev SSL_ALLOW_SSL2
 +to allow SSLv2 (not recommended) and
 +.Ev SSL_NO_SSL3
 +or
 +.Ev SSL_NO_TLS1
 +to disable the respective methods.
  .Sh AUTHENTICATION
  Apart from setting the appropriate environment variables and
  specifying the user name and password in the URL or the
 @@ -579,6 +634,31 @@
  Same as
  .Ev NO_PROXY ,
  for compatibility.
 +.It Ev SSL_ALLOW_SSL2
 +Allow SSL version 2 when negotiating the connection (not recommended).
 +.It Ev SSL_CA_CERT_FILE
 +CA certificate bundle containing trusted CA certificates.
 +Default value:
 +.Pa /etc/ssl/cert.pem .
 +.It Ev SSL_CA_CERT_PATH
 +Path containing trusted CA hashes.
 +.It Ev SSL_CLIENT_CERT_FILE
 +PEM encoded client certificate/key which will be used in
 +client certificate authentication.
 +.It Ev SSL_CLIENT_KEY_FILE
 +PEM encoded client key in case key and client certificate
 +are stored separately.
 +.It Ev SSL_CRL_FILE
 +File containing certificate revocation list.
 +.It Ev SSL_NO_SSL3
 +Don't allow SSL version 3 when negotiating the connection.
 +.It Ev SSL_NO_TLS1
 +Don't allow TLV version 1 when negotiating the connection.
 +.It Ev SSL_NO_VERIFY_HOSTNAME
 +If set, do not verify that the hostname matches the subject of the
 +certificate presented by the server.
 +.It Ev SSL_NO_VERIFY_PEER
 +If set, do not verify the peer certificate against trusted CAs.
  .El
  .Sh EXAMPLES
  To access a proxy server on
 @@ -610,6 +690,19 @@
  .Bd -literal -offset indent
  NO_PROXY=3Dlocalhost,127.0.0.1
  .Ed
 +.Pp
 +Access HTTPS website without any certificate verification whatsoever:
 +.Bd -literal -offset indent
 +SSL_NO_VERIFY_PEER=3D1
 +SSL_NO_VERIFY_HOSTNAME=3D1
 +.Ed
 +.Pp
 +Access HTTPS website using client certificate based authentication
 +and a private CA:
 +.Bd -literal -offset indent
 +SSL_CLIENT_CERT_FILE=3D/path/to/client.pem
 +SSL_CA_CERT_FILE=3D/path/to/myca.pem
 +.Ed
  .Sh SEE ALSO
  .Xr fetch 1 ,
  .Xr ftpio 3 ,
 @@ -678,7 +771,8 @@
  .An Hajimu Umemoto Aq ume@FreeBSD.org ,
  .An Henry Whincup Aq henry@techiebod.com ,
  .An Jukka A. Ukkonen Aq jau@iki.fi ,
 -.An Jean-Fran\(,cois Dockes Aq jf@dockes.org
 +.An Jean-Fran\(,cois Dockes Aq jf@dockes.org ,
 +.An Michael Gmelin Aq freebsd@grem.de
  and others.
  It replaces the older
  .Nm ftpio
 @@ -688,7 +782,9 @@
  .An Jordan K. Hubbard Aq jkh@FreeBSD.org .
  .Pp
  This manual page was written by
 -.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org .
 +.An Dag-Erling Sm\(/orgrav Aq des@FreeBSD.org
 +and
 +.An Michael Gmelin Aq freebsd@grem.de .
  .Sh BUGS
  Some parts of the library are not yet implemented.
  The most notable
 @@ -717,6 +813,10 @@
  .Fn fetchStatFTP
  does not check that the result of an MDTM command is a valid date.
  .Pp
 +In case password protected keys are used for client certificate based
 +authentication the user is prompted for the password on each and every
 +fetch operation.
 +.Pp
  The man page is incomplete, poorly written and produces badly
  formatted text.
  .Pp
 Index: lib/libfetch/http.c
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- lib/libfetch/http.c	(revision 249052)
 +++ lib/libfetch/http.c	(working copy)
 @@ -1392,7 +1392,7 @@
  		/* fetch_connect() has already set an error code */
  		return (NULL);
  	if (strcasecmp(URL->scheme, SCHEME_HTTPS) =3D=3D 0 &&
 -	    fetch_ssl(conn, verbose) =3D=3D -1) {
 +	    fetch_ssl(conn, URL, verbose) =3D=3D -1) {
  		fetch_close(conn);
  		/* grrr */
  		errno =3D EAUTH;
 Index: lib/libfetch/common.c
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- lib/libfetch/common.c	(revision 249052)
 +++ lib/libfetch/common.c	(working copy)
 @@ -1,5 +1,6 @@
  /*-
   * Copyright (c) 1998-2011 Dag-Erling Sm=C3=B8rgrav
 + * Copyright (c) 2013 Michael Gmelin <freebsd@grem.de>
   * All rights reserved.
   *
   * Redistribution and use in source and binary forms, with or without
 @@ -47,6 +48,10 @@
  #include <string.h>
  #include <unistd.h>
 =20
 +#ifdef WITH_SSL
 +#include <openssl/x509v3.h>
 +#endif
 +
  #include "fetch.h"
  #include "common.h"
 =20
 @@ -317,15 +322,484 @@
  	return (conn);
  }
 =20
 +#ifdef WITH_SSL
 +/*
 + * Convert characters A-Z to lowercase (intentionally avoid any locale
 + * specific conversions).
 + */
 +static char
 +fetch_ssl_tolower(char in)
 +{
 +	if (in >=3D 'A' && in <=3D 'Z')
 +		return (in + 32);
 +	else
 +		return (in);
 +}
 =20
  /*
 + * isalpha implementation that intentionally avoids any locale specific
 + * conversions.
 + */
 +static int
 +fetch_ssl_isalpha(char in)
 +{
 +	return ((in >=3D 'A' && in <=3D 'Z') || (in >=3D 'a' && in <=3D 'z'));
 +}
 +
 +/*
 + * Check if passed hostnames a and b are equal.
 + */
 +static int
 +fetch_ssl_hname_equal(const char *a, size_t alen, const char *b,
 +    size_t blen)
 +{
 +	size_t i;
 +
 +	if (alen !=3D blen)
 +		return (0);
 +	for (i =3D 0; i < alen; ++i) {
 +		if (fetch_ssl_tolower(a[i]) !=3D fetch_ssl_tolower(b[i]))
 +			return (0);
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Check if domain label is traditional, meaning that only A-Z, a-z, 0-9
 + * and '-' (hyphen) are allowed. Hyphens have to be surrounded by alpha-
 + * numeric characters. Double hyphens (like they're found in IDN a-labels
 + * 'xn--') are not allowed. Empty labels are invalid.
 + */
 +static int
 +fetch_ssl_is_trad_domain_label(const char *l, size_t len, int wcok)
 +{
 +	size_t i;
 +
 +	if (!len || l[0] =3D=3D '-' || l[len-1] =3D=3D '-')
 +		return (0);
 +	for (i =3D 0; i < len; ++i) {
 +		if (!isdigit(l[i]) &&
 +		    !fetch_ssl_isalpha(l[i]) &&
 +		    !(l[i] =3D=3D '*' && wcok) &&
 +		    !(l[i] =3D=3D '-' && l[i - 1] !=3D '-'))
 +			return (0);
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Check if host name consists only of numbers. This might indicate an IP
 + * address, which is not a good idea for CN wildcard comparison.
 + */
 +static int
 +fetch_ssl_hname_is_only_numbers(const char *hostname, size_t len)
 +{
 +	size_t i;
 +
 +	for (i =3D 0; i < len; ++i) {
 +		if (!((hostname[i] >=3D '0' && hostname[i] <=3D '9') ||
 +		    hostname[i] =3D=3D '.'))
 +			return (0);
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Check if the host name h passed matches the pattern passed in m which
 + * is usually part of subjectAltName or CN of a certificate presented to
 + * the client. This includes wildcard matching. The algorithm is based on
 + * RFC6125, sections 6.4.3 and 7.2, which clarifies RFC2818 and RFC3280.
 + */
 +static int
 +fetch_ssl_hname_match(const char *h, size_t hlen, const char *m,
 +    size_t mlen)
 +{
 +	int delta, hdotidx, mdot1idx, wcidx;
 +	const char *hdot, *mdot1, *mdot2;
 +	const char *wc; /* wildcard */
 +
 +	if (!(h && *h && m && *m))
 +		return (0);
 +	if ((wc =3D strnstr(m, "*", mlen)) =3D=3D NULL)
 +		return (fetch_ssl_hname_equal(h, hlen, m, mlen));
 +	wcidx =3D wc - m;
 +	/* hostname should not be just dots and numbers */
 +	if (fetch_ssl_hname_is_only_numbers(h, hlen))
 +		return (0);
 +	/* only one wildcard allowed in pattern */
 +	if (strnstr(wc + 1, "*", mlen - wcidx - 1) !=3D NULL)
 +		return (0);
 +	/*
 +	 * there must be at least two more domain labels and
 +	 * wildcard has to be in the leftmost label (RFC6125)
 +	 */
 +	mdot1 =3D strnstr(m, ".", mlen);
 +	if (mdot1 =3D=3D NULL || mdot1 < wc || (mlen - (mdot1 - m)) < 4)
 +		return (0);
 +	mdot1idx =3D mdot1 - m;
 +	mdot2 =3D strnstr(mdot1 + 1, ".", mlen - mdot1idx - 1);
 +	if (mdot2 =3D=3D NULL || (mlen - (mdot2 - m)) < 2)
 +		return (0);
 +	/* hostname must contain a dot and not be the 1st char */
 +	hdot =3D strnstr(h, ".", hlen);
 +	if (hdot =3D=3D NULL || hdot =3D=3D h)
 +		return (0);
 +	hdotidx =3D hdot - h;
 +	/*
 +	 * host part of hostname must be at least as long as
 +	 * pattern it's supposed to match
 +	 */
 +	if (hdotidx < mdot1idx)
 +		return (0);
 +	/*
 +	 * don't allow wildcards in non-traditional domain names
 +	 * (IDN, A-label, U-label...)
 +	 */
 +	if (!fetch_ssl_is_trad_domain_label(h, hdotidx, 0) ||
 +	    !fetch_ssl_is_trad_domain_label(m, mdot1idx, 1))
 +		return (0);
 +	/* match domain part (part after first dot) */
 +	if (!fetch_ssl_hname_equal(hdot, hlen - hdotidx, mdot1,
 +	    mlen - mdot1idx))
 +		return (0);
 +	/* match part left of wildcard */
 +	if (!fetch_ssl_hname_equal(h, wcidx, m, wcidx))
 +		return (0);
 +	/* match part right of wildcard */
 +	delta =3D mdot1idx - wcidx - 1;
 +	if (!fetch_ssl_hname_equal(hdot - delta, delta,
 +	    mdot1 - delta, delta))
 +		return (0);
 +	/* all tests succeded, it's a match */
 +	return (1);
 +}
 +
 +/*
 + * Get numeric host address info - returns NULL if host was not an IP
 + * address. The caller is responsible for deallocation using
 + * freeaddrinfo(3).
 + */
 +static struct addrinfo *
 +fetch_ssl_get_numeric_addrinfo(const char *hostname, size_t len)
 +{
 +	struct addrinfo hints, *res;
 +	char *host;
 +
 +	host =3D (char *)malloc(len + 1);
 +	memcpy(host, hostname, len);
 +	host[len] =3D '\0';
 +	memset(&hints, 0, sizeof(hints));
 +	hints.ai_family =3D PF_UNSPEC;
 +	hints.ai_socktype =3D SOCK_STREAM;
 +	hints.ai_protocol =3D 0;
 +	hints.ai_flags =3D AI_NUMERICHOST;
 +	/* port is not relevant for this purpose */
 +	getaddrinfo(host, "443", &hints, &res);
 +	free(host);
 +	return res;
 +}
 +
 +/*
 + * Compare ip address in addrinfo with address passes.
 + */
 +static int
 +fetch_ssl_ipaddr_match_bin(const struct addrinfo *lhost, const char *rhost,
 +    size_t rhostlen)
 +{
 +	const void *left;
 +
 +	if (lhost->ai_family =3D=3D AF_INET && rhostlen =3D=3D 4) {
 +		left =3D (void *)&((struct sockaddr_in*)(void *)
 +		    lhost->ai_addr)->sin_addr.s_addr;
 +#ifdef INET6
 +	} else if (lhost->ai_family =3D=3D AF_INET6 && rhostlen =3D=3D 16) {
 +		left =3D (void *)&((struct sockaddr_in6 *)(void *)
 +		    lhost->ai_addr)->sin6_addr;
 +#endif
 +	} else
 +		return (0);
 +	return (!memcmp(left, (const void *)rhost, rhostlen) ? 1 : 0);
 +}
 +
 +/*
 + * Compare ip address in addrinfo with host passed. If host is not an IP
 + * address, comparison will fail.
 + */
 +static int
 +fetch_ssl_ipaddr_match(const struct addrinfo *laddr, const char *r,
 +    size_t rlen)
 +{
 +	struct addrinfo *raddr;
 +	int ret;
 +	char *rip;
 +
 +	ret =3D 0;
 +	if ((raddr =3D fetch_ssl_get_numeric_addrinfo(r, rlen)) =3D=3D NULL)
 +		return 0; /* not a numeric host */
 +
 +	if (laddr->ai_family =3D=3D raddr->ai_family) {
 +		if (laddr->ai_family =3D=3D AF_INET) {
 +			rip =3D (char *)&((struct sockaddr_in *)(void *)
 +			    raddr->ai_addr)->sin_addr.s_addr;
 +			ret =3D fetch_ssl_ipaddr_match_bin(laddr, rip, 4);
 +#ifdef INET6
 +		} else if (laddr->ai_family =3D=3D AF_INET6) {
 +			rip =3D (char *)&((struct sockaddr_in6 *)(void *)
 +			    raddr->ai_addr)->sin6_addr;
 +			ret =3D fetch_ssl_ipaddr_match_bin(laddr, rip, 16);
 +#endif
 +		}
 +
 +	}
 +	freeaddrinfo(raddr);
 +	return (ret);
 +}
 +
 +/*
 + * Verify server certificate by subjectAltName.
 + */
 +static int
 +fetch_ssl_verify_altname(STACK_OF(GENERAL_NAME) *altnames,
 +    const char *host, struct addrinfo *ip)
 +{
 +	const GENERAL_NAME *name;
 +	size_t nslen;
 +	int i;
 +	const char *ns;
 +
 +	for (i =3D 0; i < sk_GENERAL_NAME_num(altnames); ++i) {
 +		/*
 +		 * This is a workaround, since the following line causes
 +		 * alignment issues in clang:
 +		 * name =3D sk_GENERAL_NAME_value(subject_alt_names, i);
 +		 * OpenSSL explicitly warns not to use those macros
 +		 * directly, but there isn't much choice (and there
 +		 * shouldn't be any ill side effects)
 +		 */
 +		name =3D (GENERAL_NAME *)SKM_sk_value(void, altnames, i);
 +		ns =3D (const char *)ASN1_STRING_data(name->d.ia5);
 +		nslen =3D (size_t)ASN1_STRING_length(name->d.ia5);
 +
 +		if (name->type =3D=3D GEN_DNS && ip =3D=3D NULL &&
 +		    fetch_ssl_hname_match(host, strlen(host), ns, nslen))
 +			return (1);
 +		else if (name->type =3D=3D GEN_IPADD && ip !=3D NULL &&
 +		    fetch_ssl_ipaddr_match_bin(ip, ns, nslen))
 +			return (1);
 +	}
 +	return (0);
 +}
 +
 +/*
 + * Verify server certificate by CN.
 + */
 +static int
 +fetch_ssl_verify_cn(X509_NAME *subject, const char *host,
 +    struct addrinfo *ip)
 +{
 +	ASN1_STRING *namedata;
 +	X509_NAME_ENTRY *nameentry;
 +	int cnlen, lastpos, loc, ret;
 +	unsigned char *cn;
 +
 +	ret =3D 0;
 +	lastpos =3D -1;
 +	loc =3D -1;
 +	cn =3D NULL;
 +	/* get most specific CN (last entry in list) and compare */
 +	while ((lastpos =3D X509_NAME_get_index_by_NID(subject,
 +	    NID_commonName, lastpos)) !=3D -1)
 +		loc =3D lastpos;
 +
 +	if (loc > -1) {
 +		nameentry =3D X509_NAME_get_entry(subject, loc);
 +		namedata =3D X509_NAME_ENTRY_get_data(nameentry);
 +		cnlen =3D ASN1_STRING_to_UTF8(&cn, namedata);
 +		if (ip =3D=3D NULL &&
 +		    fetch_ssl_hname_match(host, strlen(host), cn, cnlen))
 +			ret =3D 1;
 +		else if (ip !=3D NULL && fetch_ssl_ipaddr_match(ip, cn, cnlen))
 +			ret =3D 1;
 +		OPENSSL_free(cn);
 +	}
 +	return (ret);
 +}
 +
 +/*
 + * Verify that server certificate subjectAltName/CN matches
 + * hostname. First check, if there are alternative subject names. If yes,
 + * those have to match. Only if those don't exist it falls back to
 + * checking the subject's CN.
 + */
 +static int
 +fetch_ssl_verify_hname(X509 *cert, const char *host)
 +{
 +	struct addrinfo *ip;
 +	STACK_OF(GENERAL_NAME) *altnames;
 +	X509_NAME *subject;
 +	int ret;=09
 +
 +	ret =3D 0;
 +	ip =3D fetch_ssl_get_numeric_addrinfo(host, strlen(host));
 +	altnames =3D X509_get_ext_d2i(cert, NID_subject_alt_name,
 +	    NULL, NULL);
 +
 +	if (altnames !=3D NULL) {
 +		ret =3D fetch_ssl_verify_altname(altnames, host, ip);
 +	} else {
 +		subject =3D X509_get_subject_name(cert);
 +		if (subject !=3D NULL)
 +			ret =3D fetch_ssl_verify_cn(subject, host, ip);
 +	}
 +
 +	if (ip !=3D NULL)
 +		freeaddrinfo(ip);
 +	if (altnames !=3D NULL)
 +		GENERAL_NAMES_free(altnames);
 +	return (ret);
 +}
 +
 +/*
 + * Configure transport security layer based on environment.
 + */
 +static void
 +fetch_ssl_setup_transport_layer(SSL_CTX *ctx, int verbose)
 +{
 +	long ssl_ctx_options;
 +
 +	ssl_ctx_options =3D SSL_OP_ALL | SSL_OP_NO_TICKET;
 +	if (getenv("SSL_ALLOW_SSL2") =3D=3D NULL)
 +		ssl_ctx_options |=3D SSL_OP_NO_SSLv2;
 +	if (getenv("SSL_NO_SSL3") !=3D NULL)
 +		ssl_ctx_options |=3D SSL_OP_NO_SSLv3;
 +	if (getenv("SSL_NO_TLS1") !=3D NULL)
 +		ssl_ctx_options |=3D SSL_OP_NO_TLSv1;
 +	if (verbose)
 +		fetch_info("SSL options: %x", ssl_ctx_options);
 +	SSL_CTX_set_options(ctx, ssl_ctx_options);
 +}
 +
 +
 +/*
 + * Configure peer verification based on environment.
 + */
 +static int
 +fetch_ssl_setup_peer_verification(SSL_CTX *ctx, int verbose)
 +{
 +	X509_LOOKUP *crl_lookup;
 +	X509_STORE *crl_store;
 +	const char *ca_cert_file, *ca_cert_path, *crl_file;
 +
 +	if (getenv("SSL_NO_VERIFY_PEER") =3D=3D NULL) {
 +		ca_cert_file =3D getenv("SSL_CA_CERT_FILE") !=3D NULL ?
 +		    getenv("SSL_CA_CERT_FILE") : "/etc/ssl/cert.pem";
 +		ca_cert_path =3D getenv("SSL_CA_CERT_PATH");
 +		if (verbose) {
 +			fetch_info("Peer verification enabled");
 +			if (ca_cert_file !=3D NULL)
 +				fetch_info("Using CA cert file: %s",
 +				    ca_cert_file);
 +			if (ca_cert_path !=3D NULL)
 +				fetch_info("Using CA cert path: %s",
 +				    ca_cert_path);
 +		}
 +		SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER,
 +		    fetch_ssl_cb_verify_crt);
 +		SSL_CTX_load_verify_locations(ctx, ca_cert_file,
 +		    ca_cert_path);
 +		if ((crl_file =3D getenv("SSL_CRL_FILE")) !=3D NULL) {
 +			if (verbose)
 +				fetch_info("Using CRL file: %s", crl_file);
 +			crl_store =3D SSL_CTX_get_cert_store(ctx);
 +			crl_lookup =3D X509_STORE_add_lookup(crl_store,
 +			    X509_LOOKUP_file());
 +			if (crl_lookup =3D=3D NULL ||
 +			    !X509_load_crl_file(crl_lookup, crl_file,
 +				X509_FILETYPE_PEM)) {
 +				fprintf(stderr,
 +				    "Could not load CRL file %s\n",
 +				    crl_file);
 +				return (0);
 +			}
 +			X509_STORE_set_flags(crl_store,
 +			    X509_V_FLAG_CRL_CHECK |
 +			    X509_V_FLAG_CRL_CHECK_ALL);
 +		}
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Configure client certificate based on environment.
 + */
 +static int
 +fetch_ssl_setup_client_certificate(SSL_CTX *ctx, int verbose)
 +{
 +	const char *client_cert_file, *client_key_file;
 +
 +	if ((client_cert_file =3D getenv("SSL_CLIENT_CERT_FILE")) !=3D NULL) {
 +		client_key_file =3D getenv("SSL_CLIENT_KEY_FILE") !=3D NULL ?
 +		    getenv("SSL_CLIENT_KEY_FILE") : client_cert_file;
 +		if (verbose) {
 +			fetch_info("Using client cert file: %s",
 +			    client_cert_file);
 +			fetch_info("Using client key file: %s",
 +			    client_key_file);
 +		}
 +		if (SSL_CTX_use_certificate_chain_file(ctx,
 +			client_cert_file) !=3D 1) {
 +			fprintf(stderr,
 +			    "Could not load client certificate %s\n",
 +			    client_cert_file);
 +			return (0);
 +		}
 +		if (SSL_CTX_use_PrivateKey_file(ctx, client_key_file,
 +			SSL_FILETYPE_PEM) !=3D 1) {
 +			fprintf(stderr,
 +			    "Could not load client key %s\n",
 +			    client_key_file);
 +			return (0);
 +		}
 +	}
 +	return (1);
 +}
 +
 +/*
 + * Callback for SSL certificate verification, this is called on server
 + * cert verification. It takes no decision, but informs the user in case
 + * verification failed.
 + */
 +int
 +fetch_ssl_cb_verify_crt(int verified, X509_STORE_CTX *ctx)
 +{
 +	X509 *crt;
 +	X509_NAME *name;
 +	char *str;
 +
 +	str =3D NULL;
 +	if (!verified) {
 +		if ((crt =3D X509_STORE_CTX_get_current_cert(ctx)) !=3D NULL &&
 +		    (name =3D X509_get_subject_name(crt)) !=3D NULL)
 +			str =3D X509_NAME_oneline(name, 0, 0);
 +		fprintf(stderr, "Certificate verification failed for %s\n",
 +		    str !=3D NULL ? str : "no relevant certificate");
 +		OPENSSL_free(str);
 +	}
 +	return (verified);
 +}
 +
 +#endif
 +
 +/*
   * Enable SSL on a connection.
   */
  int
 -fetch_ssl(conn_t *conn, int verbose)
 +fetch_ssl(conn_t *conn, const struct url *URL, int verbose)
  {
  #ifdef WITH_SSL
  	int ret, ssl_err;
 +	X509_NAME *name;
 +	char *str;
 =20
  	/* Init the SSL library and context */
  	if (!SSL_library_init()){
 @@ -339,8 +813,14 @@
  	conn->ssl_ctx =3D SSL_CTX_new(conn->ssl_meth);
  	SSL_CTX_set_mode(conn->ssl_ctx, SSL_MODE_AUTO_RETRY);
 =20
 +	fetch_ssl_setup_transport_layer(conn->ssl_ctx, verbose);
 +	if (!fetch_ssl_setup_peer_verification(conn->ssl_ctx, verbose))
 +		return (-1);
 +	if (!fetch_ssl_setup_client_certificate(conn->ssl_ctx, verbose))
 +		return (-1);
 +
  	conn->ssl =3D SSL_new(conn->ssl_ctx);
 -	if (conn->ssl =3D=3D NULL){
 +	if (conn->ssl =3D=3D NULL) {
  		fprintf(stderr, "SSL context creation failed\n");
  		return (-1);
  	}
 @@ -353,22 +833,35 @@
  			return (-1);
  		}
  	}
 +	conn->ssl_cert =3D SSL_get_peer_certificate(conn->ssl);
 =20
 +	if (conn->ssl_cert =3D=3D NULL) {
 +		fprintf(stderr, "No server SSL certificate\n");
 +		return (-1);
 +	}
 +
 +	if (getenv("SSL_NO_VERIFY_HOSTNAME") =3D=3D NULL) {
 +		if (verbose)
 +			fetch_info("Verify hostname");
 +		if (!fetch_ssl_verify_hname(conn->ssl_cert, URL->host)) {
 +			fprintf(stderr,
 +			    "SSL certificate subject doesn't match host %s\n",
 +			    URL->host);
 +			return (-1);
 +		}
 +	}
 +
  	if (verbose) {
 -		X509_NAME *name;
 -		char *str;
 -
 -		fprintf(stderr, "SSL connection established using %s\n",
 +		fetch_info("SSL connection established using %s",
  		    SSL_get_cipher(conn->ssl));
 -		conn->ssl_cert =3D SSL_get_peer_certificate(conn->ssl);
  		name =3D X509_get_subject_name(conn->ssl_cert);
  		str =3D X509_NAME_oneline(name, 0, 0);
 -		printf("Certificate subject: %s\n", str);
 -		free(str);
 +		fetch_info("Certificate subject: %s", str);
 +		OPENSSL_free(str);
  		name =3D X509_get_issuer_name(conn->ssl_cert);
  		str =3D X509_NAME_oneline(name, 0, 0);
 -		printf("Certificate issuer: %s\n", str);
 -		free(str);
 +		fetch_info("Certificate issuer: %s", str);
 +		OPENSSL_free(str);
  	}
 =20
  	return (0);
 @@ -726,6 +1219,22 @@
 =20
  	if (--conn->ref > 0)
  		return (0);
 +#ifdef WITH_SSL
 +	if (conn->ssl) {
 +		SSL_shutdown(conn->ssl);
 +		SSL_set_connect_state(conn->ssl);
 +		SSL_free(conn->ssl);
 +		conn->ssl =3D NULL;
 +	}
 +	if (conn->ssl_ctx) {
 +		SSL_CTX_free(conn->ssl_ctx);
 +		conn->ssl_ctx =3D NULL;
 +	}
 +	if (conn->ssl_cert) {
 +		X509_free(conn->ssl_cert);
 +		conn->ssl_cert =3D NULL;
 +	}
 +#endif
  	ret =3D close(conn->sd);
  	free(conn->cache.buf);
  	free(conn->buf);
 Index: lib/libfetch/common.h
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 --- lib/libfetch/common.h	(revision 249052)
 +++ lib/libfetch/common.h	(working copy)
 @@ -87,7 +87,10 @@
  conn_t		*fetch_connect(const char *, int, int, int);
  conn_t		*fetch_reopen(int);
  conn_t		*fetch_ref(conn_t *);
 -int		 fetch_ssl(conn_t *, int);
 +#ifdef WITH_SSL
 +int		 fetch_ssl_cb_verify_crt(int, X509_STORE_CTX*);
 +#endif
 +int		 fetch_ssl(conn_t *, const struct url *, int);
  ssize_t		 fetch_read(conn_t *, char *, size_t);
  int		 fetch_getln(conn_t *);
  ssize_t		 fetch_write(conn_t *, const char *, size_t);
 
 --MP_/t5sG0_JQ27vT/i9nIo_rTfQ--

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: Michael Gmelin <freebsd@grem.de>
Cc: r4721@tormail.org,  bug-followup@freebsd.org
Subject: Re: kern/175514: [patch] Support PKI in libfetch for HTTP over SSL (https)
Date: Sun, 21 Jul 2013 09:27:11 +0200

 Michael Gmelin <freebsd@grem.de> writes:
 > http://blog.grem.de/libfetch_20130518.patch
 
 Is this still the absolute latest version?
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: Michael Gmelin <freebsd@grem.de>
To: Dag-Erling =?UTF-8?B?U23DuHJncmF2?= <des@des.no>
Cc: r4721@tormail.org, bug-followup@freebsd.org
Subject: Re: kern/175514: [patch] Support PKI in libfetch for HTTP over SSL
 (https)
Date: Sun, 21 Jul 2013 11:00:45 +0200

 On Sun, 21 Jul 2013 09:27:11 +0200
 Dag-Erling Sm=C3=B8rgrav <des@des.no> wrote:
 
 > Michael Gmelin <freebsd@grem.de> writes:
 > > http://blog.grem.de/libfetch_20130518.patch
 >=20
 > Is this still the absolute latest version?
 >=20
 > DES
 
 Yes it is.
 
 - Michael
 
 --=20
 Michael Gmelin
State-Changed-From-To: open->closed 
State-Changed-By: glebius 
State-Changed-When: Thu Oct 10 09:43:51 UTC 2013 
State-Changed-Why:  
Merged to stable/9. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175514 
>Unformatted:
