Example for HTTPS client including certificate verification lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- Doyenne · 2023-03-17 16:19 Can anyone give me an example of how to download a web resource via HTTPS, including certificate verification? Preferably using the FCL (e.g. fcl-web), but Synapse is also OK. I saw that in FCL trunk, there is a VerifySSLCertificate property in TFPHTTPClient, but then I need to write an event handler and get stuck. Let's say that I would like to download a web page and https://badssl.com/ should work, but https://expired.badssl.com/ and https://wrong.host.badssl.com/ should be rejected. This is what I have, but I don't know how to add certificate verification: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; {$mode objfpc}{$H+} uses SysUtils, fphttpclient, openssl, opensslsockets; const URLS: array[0..2] of string = (  'https://badssl.com/',            // should work  'https://expired.badssl.com/',    // should be rejected  'https://wrong.host.badssl.com/'  // should be rejected); procedure TryURLs;var  URL: string;begin  for URL in URLS do    try      with TFPHTTPClient.Create(nil) do        try          AllowRedirect := True;          // do something to enable certificate verification??        Get(URL);        finally          Free;        end;      WriteLn(URL, ' succeeded.');    except      on E: Exception do        WriteLn(Format('%s failed! (%s)', [URL, E.Message]));    end;end; begin  TryURLs;end. Thanks for any hint in the right direction! ---------------------------------------------------------------------- PierceNg · 2023-03-18 01:51 Quote from: Doyenne on March 17, 2023, 05:19:11 pmCan anyone give me an example of how to download a web resource via HTTPS, including certificate verification? Preferably using the FCL (e.g. fcl-web), but Synapse is also OK. I saw that in FCL trunk, there is a VerifySSLCertificate property in TFPHTTPClient, but then I need to write an event handler and get stuck. The example fcl-web/examples/httpclient/httpget.pas shows how to set up the event handler. Edit: Doesn't look fully functional as this line prints empty string: Code: [Select]TEncoding.ASCII.GetAnsiString( aHandler.CertificateData.Certificate.Value) ---------------------------------------------------------------------- PierceNg · 2023-03-18 03:01 libcurl has what you want. Below is libcurl/examples/testcurl.pp with my changes marked "<====": Code: [Select]{$mode objfpc} {$H+} program testcurl; uses libcurl; Var   URL : Pchar = 'https://wrong.host.badssl.com'; // <==== Change URL   hCurl : pCurl; begin   hCurl:= curl_easy_init;   if Assigned(hCurl) then     begin     curl_easy_setopt(hCurl,CURLOPT_VERBOSE, [True]);     curl_easy_setopt(hCurl,CURLOPT_URL,[URL]);     curl_easy_setopt(hCurl, CURLOPT_SSL_VERIFYPEER, [True]); // <==== Add this line     curl_easy_perform(hCurl);     curl_easy_cleanup(hCurl);     end; end. Running it: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i % ./testcurl*   Trying 104.154.89.105:443...* TCP_NODELAY set* Connected to wrong.host.badssl.com (104.154.89.105) port 443 (#0)* ALPN, offering h2* ALPN, offering http/1.1* successfully set certificate verify locations:*   CAfile: /etc/ssl/certs/ca-certificates.crt  CApath: /etc/ssl/certs* SSL connection using TLSv1.2 / ECDHE-RSA- AES128-GCM-SHA256* ALPN, server accepted to use http/1.1* Server certificate:*  subject: CN=*.badssl.com*  start date: Jan 22 16:33:26 2023 GMT*  expire date: Apr 22 16:33:25 2023 GMT*  subjectAltName does not match wrong.host.badssl.com* SSL: no alternative certificate subject name matches target host name 'wrong.host.badssl.com'* Closing connection 0 For expired: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i % ./testcurl*   Trying 104.154.89.105:443...* TCP_NODELAY set* Connected to expired.badssl.com (104.154.89.105) port 443 (#0)* ALPN, offering h2* ALPN, offering http/1.1* successfully set certificate verify locations:*   CAfile: /etc/ssl/certs/ca-certificates.crt  CApath: /etc/ssl/certs* SSL certificate problem: certificate has expired* Closing connection 0 ---------------------------------------------------------------------- BeniBela · 2023-03-18 11:30 My internet tools handle that. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Project1; uses internetaccess, synapseinternetaccess;var  i: TInternetAccess;begin  i := TSynapseInternetAccess.create();  i.get('https://badssl.com/');//  i.get('https://expired.badssl.com/');//  i.get('https://wrong.host.badssl.com/');end.  I started with the Synapse HTTPS handling, but changed a lot ---------------------------------------------------------------------- domasz · 2023-03-18 12:02 Does any of these solutions work with HTTPS without DLLs? ---------------------------------------------------------------------- PierceNg · 2023-03-18 12:19 Quote from: domasz on March 18, 2023, 01:02:54 pmDoes any of these solutions work with HTTPS without DLLs? The DLLs provide the TLS functionality including crypto, certificate handling, wire protocols, etc. Not using those DLLs means using a pure Pascal implementation of TLS with equivalent functionality. AFAIK there is no such thing. ---------------------------------------------------------------------- Thaddy · 2023-03-18 14:45 Quote from: PierceNg on March 18, 2023, 01:19:57 pm AFAIK there is no such thing. Of course there is: our forum member Xor-el (the one with the unpronouncable name) took care of that.... (about 4 years ago). Only "problem" is you need to know what you are doing. ---------------------------------------------------------------------- Doyenne · 2023-03-18 16:24 Quote from: BeniBela on March 18, 2023, 12:30:09 pmMy internet tools handle that. I'm highly impressed! I had to install the FLRE and POCO library as well before I could compile, but then it worked perfectly. IMHO, this is how it should be in 2023: SSL verification easy, by default and out of the box. (IMHO using HTTPS without certificate checking does not make sense.) However, your library is GPL (not LGPL) so unfortunately I cannot use it in my project. ---------------------------------------------------------------------- Doyenne · 2023-03-18 16:29 If I try using Synapse trunk, same machine, same configuration, it doesn't work: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; uses blcksock, ssl_openssl3; const URL = 'badssl.com'; var  Sock: TTCPBlockSocket;begin  Sock := TTCPBlockSocket.Create;  try    Sock.Connect(URL, '443');    Sock.SSL.VerifyCert := True;    Sock.SSLDoConnect;    if Sock.LastError <>  0 then    begin      WriteLn('Error: ', Sock.LastErrorDesc);      WriteLn('GetVerifyCert: ', Sock.SSL.GetVerifyCert);    end    else      WriteLn('Success!');  finally    Sock.Free;  end;end. Output: Code: [Select]Error: error:0A000086:SSL routines::certificate verify failed GetVerifyCert: 20 The number 20 might mean that the root certificate could not be checked (I find OpenSSL messages always a bit unclear), because if I intentionally hide the root certificates to OpenSSL, I get verify error num=20: Code: [Select]$ openssl s_client -quiet -connect badssl.com:443 -CApath /tmp depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify error:num=20:unable to get local issuer certificate verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = *.badssl.com verify return:1 Adding Code: [Select]sock.SSL.CertCAFile := '/etc/ssl/certs/ISRG_Root_X1.pem'; does not solve the problem, still same result. Any hints are welcome... ---------------------------------------------------------------------- PierceNg · 2023-03-18 16:43 Quote from: Thaddy on March 18, 2023, 03:45:59 pmQuote from: PierceNg on March 18, 2023, 01:19:57 pm AFAIK there is no such thing. Of course there is: our forum member Xor-el (the one with the unpronouncable name) took care of that.... (about 4 years ago). Only "problem" is you need to know what you are doing. That's a crypto library, not a TLS implementation. ---------------------------------------------------------------------- Thaddy · 2023-03-18 17:42 I am sorry, but all the code you need is there. If you do not understand it, ask. ---------------------------------------------------------------------- domasz · 2023-03-18 17:46 Quote from: Thaddy on March 18, 2023, 06:42:17 pmI am sorry, but all the code you need is there. If you do not understand it, ask. I don't. Care to explain? ---------------------------------------------------------------------- Doyenne · 2023-10-23 14:27 Quote from: Doyenne on March 18, 2023, 05:29:33 pmIf I try using Synapse trunk, same machine, same configuration, it doesn't work: (...) Finally I figured out why this Synapse code didn't work... Badssl.com is serving different URLs from the same IP address. Therefore, the Server Name Indication (SNI) needs to be used, so the server knows which certificate to present to the client (e.g. the badssl.com certificate, the expired.badssl.com certificate, etc.). The THTTPSend component automatically adds SNI to the request, but the TCPBlockSocket does not. Therefore, I had to add this manually: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Sock.SSL.SNIHost := URL; The easier solution is of course to use THTTPSend. The following example shows how to use certificate verification with Synapse: This code uses Synapse trunk [r270] Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; uses httpsend, ssl_openssl3; const URLs: array[0..5] of string = (  'https://badssl.com',                // should work  'https://wrong.host.badssl.com',     // should be rejected  'https://expired.badssl.com',        // should be rejected  'https://self-signed.badssl.com',    // should be rejected  'https://untrusted-root.badssl.com', // should be rejected  'https://revoked.badssl.com'         // should be rejected); var  URL: string;  HTTP: THTTPSend;begin  HTTP := THTTPSend.Create;  try    HTTP.Sock.SSL.VerifyCert := True;    HTTP.Sock.SSL.CertCAFile := 'ca- bundle.crt';    for URL in URLs do    begin      if HTTP.HTTPMethod('GET', URL) then        WriteLn('Success: ', URL)    else        WriteLn('Failure: ', URL, '; verify result: ', HTTP.Sock.SSL.GetVerifyCert);    end;  finally    HTTP.Free;  end;end. As show in this example, certificate verification also needs a reference to trusted root (CA) certificates. Without any trusted CA certificates, all verifications will fail. A file containing trusted certificates in PEM format can for example be created using the mk-ca- bundle script from the Curl project (https://curl.se/docs/mk-ca- bundle.html). Warning! Using the current Synapse code (trunk [r270]), certificate verification works, but wrong host certificates are not detected. This is a security risk. I filed a bug including patches to fix this issue: https://sourceforge.net/p/synalist/bugs/75/ ---------------------------------------------------------------------- Doyenne · 2023-10-23 14:41 Quote from: Doyenne on March 17, 2023, 05:19:11 pmCan anyone give me an example of how to download a web resource via HTTPS, including certificate verification? Preferably using the FCL (e.g. fcl-web), but Synapse is also OK. (...) I also found out why my code using fcl-web did not work. To enable certificate verification using TFPHTTPClient, the following was needed: VerifySSlCertificate := True;Add a callback function and link this to the AfterSocketHandlerCreate event, e.g. AfterSocketHandlerCreate := @DoHaveSocketHandler;In the callback function, provide a reference to a directory containing trusted root (CA) certificates, or a reference to a file containing trusted root (CA) certificates (e.g. in PEM format). I filed a suggestion to ease certificate verification by eliminating the need for the callback procedure. This suggestion was accepted (thank you!), see https://gitlab.com/freepascal.org/fpc/source/-/issues/40480 So, using FCL trunk (version 2023-10-22 or later), certificate verification is as easy as: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; {$mode objfpc}{$H+} uses  fphttpclient, opensslsockets; const  URL = 'https://example.com/'; begin  with TFPHTTPClient.Create(nil) do  try    VerifySSlCertificate := True;    TrustedCertsDir := '/etc/ssl/certs/'; // or: CertCAFileName:='ca- bundle.crt';    WriteLn(Get(URL));  finally    Free;  end;end.  Warning! Using the current FCL code (trunk 2023-10-22), certificate verification works, but wrong host certificates are not detected. This is a security risk. I filed a bug including patches to fix this issue: https://gitlab.com/freepascal.org/fpc/source/-/issues/40479 ---------------------------------------------------------------------- delphius · 2023-10-24 06:19 Quote from: Thaddy on March 18, 2023, 06:42:17 pmI am sorry, but all the code you need is there. If you do not understand it, ask. I also thought it would be a simple story, but still not Quote from: PierceNg on March 18, 2023, 05:43:13 pmThat's a crypto library, not a TLS implementation. And the set of cryptoprimitives presented in it will not be enough to implement even half of the protocol Quote from: domasz on March 18, 2023, 01:02:54 pmDoes any of these solutions work with HTTPS without DLLs? I searched the whole Internet in search of such a thing on pure pascal, all I found were scattered pieces. There are two commercial libraries in pure pascal, but this is beyond my interests Quote from: PierceNg on March 18, 2023, 01:19:57 pmAFAIK there is no such thing. It is, otherwise I wouldn't have to write my own training implementation of the TLS 1.3 protocol. https://github.com/delphius/fpmtls ---------------------------------------------------------------------- PierceNg · 2023-10-30 02:09 Quote from: delphius on October 24, 2023, 08:19:02 amQuote from: PierceNg on March 18, 2023, 01:19:57 pmAFAIK there is no such thing. It is, otherwise I wouldn't have to write my own training implementation of the TLS 1.3 protocol. https://github.com/delphius/fpmtls Nice work. And there is an open source implementation of TLS in Pascal - https:// github.com/fundamentalslib/fundamentals5/tree/master/Source/TLS ---------------------------------------------------------------------- delphius · 2023-10-30 04:05 Quote from: PierceNg on October 30, 2023, 03:09:48 amAnd there is an open source implementation of TLS in Pascal - https://github.com/funda mentalslib/fundamentals5/tree/master/Source/TLS But while I was doing it, I figured out the essence of the question So there is still a benefit. But thanks a lot for the link to the library, it will greatly help in determining further movement P.S. No miracle happened) Did not find the expected implementation of Chacha20 + Poly1305 + curve25519)))) No GCM/GHASH ---------------------------------------------------------------------- krolikbest · 2023-11-03 08:19 Hi, according to this example Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit Unit1;{$mode objfpc}{$H+} interface uses  Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls  ,httpsend,   ,ssl_openssl //it works//  ,ssl_openssl3  //not positive result at all  ,openssl; type   { TForm1 }   TForm1 = class(TForm)    Button1: TButton;    Memo1: TMemo;    procedure Button1Click(Sender: TObject);  private   public   end; const URLs: array[0..5] of string = (  'https://badssl.com',              // should work  'https://wrong.host.badssl.com',     // should be rejected  'https://expired.badssl.com',        // should be rejected  'https://self-signed.badssl.com',    // should be rejected  'https://untrusted-root.badssl.com', // should be rejected  'https://revoked.badssl.com'         // should be rejected); var  URL: string;  HTTP: THTTPSend;  Form1: TForm1; implementation {$R *.lfm} { TForm1 } procedure TForm1.Button1Click(Sender: TObject);begin  HTTP := THTTPSend.Create;  try    HTTP.Sock.SSL.VerifyCert := True;    HTTP.Sock.SSL.CertCAFile := 'somecrt.crt';    for URL in URLs do    begin      if HTTP.HTTPMethod('GET', URL) then        WriteLn('Success: ', URL)      else        WriteLn('Failure: ', URL, '; verify result: ', HTTP.Sock.SSL.GetVerifyCert);    end;  finally  HTTP.Free;  end;end; end. , I wonder what difference is between ssl_openssl and ssl_openssl3? I keep in application directory two libraries: libeay32 and ssleay32 but in case of using ssl_openssl3  I get 10091 error for all urls, whereas using ssl_openssl I get for first url positive result and for others not (but not 10091 error). So the problem is in using ssl_openssl3, could someone explain this to me? I use Lazarus 2.2.6, Win10 64bit. ---------------------------------------------------------------------- paweld · 2023-11-03 11:35 To use openssl3 you need the libraries: libssl-3.dll and libcrypto-3.dll. ssleay32.dll and libeay32.dll are used by openssl1.0 ---------------------------------------------------------------------- krolikbest · 2023-11-03 13:36 Somehow I didn't search enough, thanks a lot.