TFPHTTPClient on MacOS lazarus / 33-macOS - Mac OS X ---------------------------------------------------------------------- dbannon · 2022-02-19 10:10 Folks, I use code like below to download from an https site, on linux and windows, it works fine since FPC3.2.0 and I am currently using Laz2.2.0. But I have just noticed it fails with an ESocketError on my (aging) MacOS box, (strange, I was sure I have tested it in the past). Anyone suggest why ?  And is there a practical alternative ? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var    Client: TFPHTTPClient;     .....    Client := TFPHttpClient.Create(nil);    Client.AddHeader('User- Agent','Mozilla/5.0 (compatible; fpweb)');    Client.AddHeader('Content-Type','application/json; charset=UTF-8');  Client.AllowRedirect := true;    SomeString := Client.Get(URL);      Actual code in the attached. Davo ---------------------------------------------------------------------- sstvmaster · 2022-02-19 23:21 Try insert InitSSLInterface befor creating the client, it is needed since fpc 3.2.0. You must insert openssl in uses to use InitSSLInterface. Quote from: dbannon on February 19, 2022, 11:10:44 amCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var    Client: TFPHTTPClient; begin    { SSL initialization has to be done by hand here }    InitSSLInterface;     Client := TFPHttpClient.Create(nil);...end; see -> https://wiki.lazarus.freepascal.org/fphttpclient ---------------------------------------------------------------------- rvk · 2022-02-20 02:30 Quote from: sstvmaster on February 20, 2022, 12:21:24 amTry insert InitSSLInterface befor creating the client, it is needed since fpc 3.2.0. Small correction... It's NOT needed anymore since 3.2.2. Hence: Quote from: dbannon on February 19, 2022, 11:10:44 amFolks, I use code like below to download from an https site, on linux and windows, it works fine since FPC3.2.0 and I am currently using Laz2.2.0. ---------------------------------------------------------------------- dbannon · 2022-02-20 02:48 thanks sstvmaster, rvk, I am using FPC3.2.0 on the Mac and have tried both with and without the Init statement, it fails in both cases on the Mac but works fine on Linux (FPC3.2.2) and Windows (FPC3.2.0). Is there some suggestion that upgrading to FPC3.2.2 on the Mac might make a difference ? Otherwise, is the any Mac users out there who would be willing to test the simple app attached to first post of this thread ?  I do wonder if my continued use of MacOS Sierra with some Brew installed OpenSSL is the problem. I do not have access to a more recent Mac and are very reluctant to upgrade my 2012 Powerbook . Sigh ... Davo ---------------------------------------------------------------------- rvk · 2022-02-20 03:09 Quote from: dbannon on February 20, 2022, 03:48:41 amI am using FPC3.2.0 on the Mac and have tried both with and without the Init statement, it fails in both cases on the Mac but works fine on Linux (FPC3.2.2) and Windows (FPC3.2.0). Yeah. I don't have a Mac so I can't test it. You could look at the suggestion given here: https://forum.lazarus.freepascal.org/index.php/topic,53613.msg396820.h tml#msg396820 Especially the link to patch openssl.pas here: https://forum.lazarus.freepascal.org/index.php/topic,34058.msg396680.h tml#msg396680 Not sure if that fixes it for you too. ---------------------------------------------------------------------- dbannon · 2022-02-20 04:26 Hmm, given how I want this app to work, patching FPC is not really an option.  But following that trail, I find this - "The issue has been fixed in FPC trunk (see Bug #37977) and is slated for the release of FPC 3.2.2" In the wiki page https://wiki.lazarus.freepascal.org/macOS_Programming _Tips#OpenSSL.2C_LibreSSL.2C_Secure_Transport.2C_Network_Framework So, maybe thats an indication FPC3.2.2 is indeed a good idea.  Thanks RVK, some light at the end of the tunnel. Davo ---------------------------------------------------------------------- trev · 2022-02-20 04:30 Personally I'd use the macOS NSURLSession API and avoid any dependence on OpenSSL libraries. You could even use the deprecated, but not going away any time soon, NSURLConnection API. Example code provided in my two Wiki articles above. ---------------------------------------------------------------------- dbannon · 2022-02-20 05:32 Quote from: trev on February 20, 2022, 05:30:29 amPersonally I'd use the macOS NSURLSession API and avoid any dependence on OpenSSL libraries. You could even use the deprecated, but not going away any time soon, NSURLConnection API. Example code provided in my two Wiki articles above. Both appear to require hacking into the FPC code, "To compile the above code successfully you are going to have to add the missing initWithData function...". Does not seem a very appropriate solution IMHO.  Is this 'hack' (used in its positive meaning) something that will appear in FPC eventually ? Davo ---------------------------------------------------------------------- dbannon · 2022-02-20 05:44 Quote from: dbannon on February 20, 2022, 05:26:46 am"The issue has been fixed in FPC trunk (see Bug #37977) and is slated for the release of FPC 3.2.2" In the wiki page https://wiki.lazarus.freepascal.org/macOS_Programming _Tips#OpenSSL.2C_LibreSSL.2C_Secure_Transport.2C_Network_Framework Indeed the code mentioned in the wiki page is in FPC3.2.2 but it not help. I have both OpenSSL 1.1 and SomethingSSL 0.9.8 and not even sure which one I am linking against. The wiki page indicates both are acceptable but maybe not mixed .... Davo ---------------------------------------------------------------------- rvk · 2022-02-20 15:48 QuoteCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient, opensslsockets, ssockets, fpopenssl;Do you actually need the ssocket and fpopenssl ??? Quote from: Gustavo 'Gus' Carreno on March 07, 2021, 08:32:40 pmThere's been a change in the way fphttpclient handles SSL. Pre FPC 3.2.0 you would use ssockets, sslsockets, openssl and fpopenssl, now from FPC 3.2.0 onwards you only need to use opensslsockets. So, shouldn't you just omit the ssockets and fpopenssl from the uses clause?? ---------------------------------------------------------------------- sstvmaster · 2022-02-20 16:39 Quote from: Gustavo 'Gus' Carreno on March 07, 2021, 08:32:40 pmThere's been a change in the way fphttpclient handles SSL. Pre FPC 3.2.0 you would use ssockets, sslsockets, openssl and fpopenssl, now from FPC 3.2.0 onwards you only need to use opensslsockets. The wiki page should maybe be changed too? ---------------------------------------------------------------------- trev · 2022-02-20 21:29 Quote from: dbannon on February 20, 2022, 06:32:19 amQuote from: trev on February 20, 2022, 05:30:29 amPersonally I'd use the macOS NSURLSession API and avoid any dependence on OpenSSL libraries. You could even use the deprecated, but not going away any time soon, NSURLConnection API. Example code provided in my two Wiki articles above. Both appear to require hacking into the FPC code, "To compile the above code successfully you are going to have to add the missing initWithData function...". Does not seem a very appropriate solution IMHO.  The macOS headers are a little long in the tooth, dating from Yosemite, and are missing some of the methods - hence my adding the needed missing methods from the Objective-C header files. There are more recent modern headers produced by automated methods, but they have no testing. The current converted headers have been fixed by hand on many occasions over the years and the same or similar fixes may be necessary for the new ones (according to Ryan who converted the new ones and Jonas who fixed the old ones). QuoteIs this 'hack' (used in its positive meaning) something that will appear in FPC eventually ? As it is adding a method missing from an incomplete/out of date header file, I would hope so ---------------------------------------------------------------------- dbannon · 2022-02-21 02:24 Quote from: trev on February 20, 2022, 10:29:33 pm.... As it is adding a method missing from an incomplete/out of date header file, I would hope so But is there a bug report in there with a patch ?  My guess, its not going to happen otherwise. Davo ---------------------------------------------------------------------- dbannon · 2022-02-21 02:28 Quote from: rvk on February 20, 2022, 04:48:45 pmQuoteCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient, opensslsockets, ssockets, fpopenssl;Do you actually need the ssocket and fpopenssl ??? .... No, that lump of code has had a number of hacks applied and discarded so has a bit of history. Pretty sure they do no harm ... @sstvmaster, yep, wiki page could do with a fixup. But I am reluctant to do so until I see something work. And right now, that does not look likely. If some Mac user with a current install would like to do a test ...... Davo ---------------------------------------------------------------------- PierceNg · 2022-02-21 04:38 I'm on Catalina. FPC 3.2.2 and Lazarus 2.2.0, both built from source. I don't use Brew. Here are the SSL libraries on my machine: Code: Bash  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i % ls -l /usr/lib/*ssl*-rwxr-xr-x  1 root  wheel  1489456 Sep 22  2020 /usr/lib/libboringssl.dylib*-rwxr-xr-x  1 root  wheel   212288 Sep 22  2020 /usr/lib/libssl.0.9.7.dylib*-rwxr-xr-x  1 root  wheel   335888 Sep 22  2020 /usr/lib/libssl.0.9.8.dylib*-rwxr-xr-x  1 root  wheel   330576 Feb  5 10:08 /usr/lib/libssl.35.dylib*-rwxr-xr-x  1 root  wheel   313984 Feb  5 10:08 /usr/lib/libssl.43.dylib*-rwxr-xr-x  1 root  wheel   300480 Feb  5 10:08 /usr/lib/libssl.44.dylib*-rwxr-xr-x  1 root  wheel   294016 Feb  5 10:08 /usr/lib/libssl.46.dylib*-rwxr-xr-x  1 root  wheel    32928 Feb  5 10:08 /usr/lib/libssl.dylib* I modified line 118 of openssl.pas by adjusting the size of the array and adding '.46' as the 2nd item: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   DLLVersions: array[1..20] of string = ('', '.46', ... the rest of the original stuff Built and installed the changed version. Then your program works. I also tried setting DLLSSLName to 'libboringssl' just to try but no dice. I also tested with my own server, which is configured to accept TLS 1.2 or higher with "strong" cipher suites. Without the above mod, same error. With the mod, program works. So the issue should be Gitlab's and Github's correctly rejecting known-insecure SSL/TLS versions when using outdated OpenSSL libraries. Maybe a long term solution is for FPC to have an API to specify the particular library files to load? ---------------------------------------------------------------------- trev · 2022-02-21 09:44 Quote from: dbannon on February 21, 2022, 03:24:37 amQuote from: trev on February 20, 2022, 10:29:33 pm.... As it is adding a method missing from an incomplete/out of date header file, I would hope so But is there a bug report in there with a patch ?  My guess, its not going to happen otherwise. There seemed little point as the old outdated/incomplete headers were going to be replaced by the new up-to-date headers (https://github.com/genericptr/MacOS_11_0 NSString.inc includes the missing method), but I have no visibility of the developer process or timing for doing that. ---------------------------------------------------------------------- dbannon · 2022-02-21 10:05 Thanks Pierce, well done. Quote from: PierceNg on February 21, 2022, 05:38:26 amI'm on Catalina. FPC 3.2.2 and Lazarus 2.2.0, both built from source. I don't use Brew. Here are the SSL libraries on my machine: ...,, I modified line 118 of openssl.pas by adjusting the size of the array and adding '.46' as the 2nd item: .... Built and installed the changed version. Then your program works. ..... Well, thats pretty clear. Without manually editing and rebuilding FPC components, it does not work. At least its clear now, I won't waste more time on it for what I am working on. QuoteMaybe a long term solution is for FPC to have an API to specify the particular library files to load?I wonder why we need to specify particular version numbers ?  Its more normal to look for eg libssl.dylib which will be a symlink to the OS recommended one ?  I wonder why this approach is not used with ssl ? Obviously, we need to seriously avoid the libssl.0.9* ones, they are dangerously out of date openssl ones, I have no idea why Apple still include them. Anyway, thanks Pierce, very useful. Davo ---------------------------------------------------------------------- dbannon · 2022-02-21 10:09 Quote from: trev on February 21, 2022, 10:44:32 am... There seemed little point as the old outdated/incomplete headers were going to be replaced by the new up-to-date headers (https://github.com/genericptr/MacOS_11_0 NSString.inc includes the missing method), but I have no visibility of the developer process or timing for doing that. OK, that makes sense Trev. It sounds like it could be a big job.  I will give up on what I was trying to do, maybe revisit when we are on top of some of these problems. Davo