fphttpClient - problem in downloading file from dropbox lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- kjteng · 2019-08-28 04:13 My code below can download files from goodgle drive but not from dropbox (respective url: https://www.dropbox.com/s/vieo9xlf7i24qvy/check2.png?dl=1 https://drive.google.com/uc?id=1gsHFgNeEZfhqgj_l3BUTXAguz7f8-_sB).  Any thing wrong with my code below ? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.bt2fphttpClick(Sender: TObject);var   Client: TFPHttpClient;begin  { SSL initialization }  InitSSLInterface;  Client := TFPHttpClient.Create(nil);  try    Client.AllowRedirect := true;  Client.Get(url1.Text, fname);    Showmessage('Output file: ' + fname);  finally    Client.Free;  end;end;                 BTW, I have no problem to download the files from the above url using urlmon   ( urlMon.URLDownloadToFile(nil, Pchar(url1.Text), PChar(fname), 0, nil). Howver, urlmon is available in Windows but not Linux. Appreciate your help. ---------------------------------------------------------------------- dbannon · 2019-08-28 04:36 Hi kjteng, this might help but probably not :-( https://forum.lazarus.freepascal.org/index.php/topic,39206.msg318360.h tml I have been working on a similar problem, trying to download from github.  While I can download from many other https sites, not github.  I have noted conversations about managing the redirect response code better, I suspect its more than that. In my case, and maybe yours, the failure happens at connect time, an incorrect redirect response has not yet been received. I think. Davo ---------------------------------------------------------------------- balazsszekely · 2019-08-28 04:46 It does not work with FPC 3.0.4, however it works perfectly fine with FPC 3.2.0. ---------------------------------------------------------------------- dbannon · 2019-08-28 07:06 GetMem, which problem does 3.2.0 solve ? There is the issue about downloading from any https: site, thats actually solved with a few lines of code in the thread I mentioned. And there is the much harder one of at least one site, github, not working with code that does work with other web sites ? Davo ---------------------------------------------------------------------- balazsszekely · 2019-08-28 07:27 @dbannon If I run OP's code(FPC 3.0.4), I get the following error: "Invalid protorcol". Adding a user agent or setting redirect to true does not help. However with FPC 3.2.0 everything works out of the box, even without the user agent. All you have to do is add opensslsockets to the uses clauses. Most likely a bug in fphttpclient(FPC 3.0.4) was fixed in 3.2.0. ---------------------------------------------------------------------- dbannon · 2019-08-28 07:38 GemMem, could I ask a favour ? Could you try the OP's code and your fpc320 with this file https://github.com/tomboy-notes/tomboy-ng/archive/V0.23.zip Its a bit less than one meg big .... Davo ---------------------------------------------------------------------- Thaddy · 2019-08-28 07:44 In 3.0.4 you can explicitly set the protocol to  tls 1.1 or higher, or simply use the ssl sources from 3.2.0. There's nothing there that won't compile on 3.0.4. ---------------------------------------------------------------------- balazsszekely · 2019-08-28 08:00 @dbannon QuoteCould you try the OP's code and your fpc320 with this file https://github.com/tomboy-notes/tomboy-ng/archive/V0.23.zip Sure. It works fine(see attached screenshot). ---------------------------------------------------------------------- dbannon · 2019-08-28 08:05 Thanks Thaddy. Not quite my issue however. I can build a working fphhtClient that connects to and downloads from most https: sites using the tricks mentioned in https://forum.lazarus.freepascal.org/index.php/topic,3920 6.msg318360.html But I get an exception (thats hard to trace) if I try to download a file from github. I'd like to keep my app using release versions of fpc/Lazarus 'cos of plans to get it into debian repos. I know they don't like 'trunk' type build recipes. Sigh ... Davo ---------------------------------------------------------------------- dbannon · 2019-08-28 08:07 Quote from: GetMem on August 28, 2019, 10:00:20 amSure. It works fine(see attached screenshot). Ahh.. thanks GetMem, now you have given me something to think about. Davo ---------------------------------------------------------------------- balazsszekely · 2019-08-28 09:33 @dbannon The attached project works with FPC 3.0.4  too. PS: You need the openssl libraries on windows(ssleay32.dll and  libeay32.dll) ---------------------------------------------------------------------- dbannon · 2019-08-28 12:18 Quote from: GetMem on August 28, 2019, 11:33:22 amThe attached project works with FPC 3.0.4  too. Wow, thank you muchly Getmem. I'll try it tomorrow, getting late here.  Just had a peek, I'm a bit puzzled, you don't seem to be doing anything I am not. Tomorrow ! Davo ---------------------------------------------------------------------- dbannon · 2019-08-29 01:45 Yep, that solves my problem ! Sure hope it helps kjteng too. The key difference between my version, from https://forum.lazarus.free pascal.org/index.php/topic,39206.msg318360.html, and GetMem's is that GetMem sets SSLType to stTLSv1_2, first thread used stTLSv1_1 Evolving standards ? Thanks GetMem, big help ! Davo ---------------------------------------------------------------------- kjteng · 2019-08-29 14:10 Quote from: GetMem on August 28, 2019, 11:33:22 am@dbannon The attached project works with FPC 3.0.4  too. PS: You need the openssl libraries on windows(ssleay32.dll and  libeay32.dll) I got "Access violation" error with your code when trying to download from the following url: https://www.dropbox.com/s/vieo9xlf7i24qvy/check2.png?dl=1 https://drive.google.com/uc?id=1gsHFgNeEZfhqgj_l3BUTXAguz7f8-_sB Did I miss anything? ---------------------------------------------------------------------- balazsszekely · 2019-08-29 17:51 @dbannon QuoteYep, that solves my problem !I'm glad it's working. @kjteng QuoteI got "Access violation" error with your code when trying to download from the following url: https://www.dropbox.com/s/vieo9xlf7i24qvy/check2.png?dl=1 https://drive.google.com/uc?id=1gsHFgNeEZfhqgj_l3BUTXAguz7f8-_sB Did I miss anything?Unfortunately you need FPC 3.2.0. As I mentioned in one of my previous posts, all the links you provided works with 3.2.0 ---------------------------------------------------------------------- dbannon · 2019-08-29 22:51 OK, there must be varying degrees of this problem then.  Because for me, GetMem's fix solved the "hard ones" even on fpc304.  Maybe kjteng's issue is even harder still ?  Sigh .... Davo ---------------------------------------------------------------------- balazsszekely · 2019-08-30 05:05 @kjteng I did take another look...Your second link works fine, but you need the openssl libraries. In order for the first link to work, I had to modify fphttpclient.  As far as I can tell, now all three links works fine. Please test attached project. PS: Don't forget to download the openssl libraries before testing. 32 bit: https://packages.lazarus-ide.org/openssl-1.0.2j-i386-win32.zip 64 bit: https://packages.lazarus- ide.org/openssl-1.0.2j-x64_86-win64.zip Make sure the dlls are in the same folder as your exe. ---------------------------------------------------------------------- kjteng · 2019-08-31 05:42 Quote from: GetMem on August 30, 2019, 07:05:46 am@kjteng I did take another look...Your second link works fine, but you need the openssl libraries. In order for the first link to work, I had to modify fphttpclient.  As far as I can tell, now all three links works fine. Please test attached project. PS: Don't forget to download the openssl libraries before testing. 32 bit: https://packages.lazarus-ide.org/openssl-1.0.2j-i386-win32.zip 64 bit: https://packages.lazarus- ide.org/openssl-1.0.2j-x64_86-win64.zip Make sure the dlls are in the same folder as your exe. Yes! the modified fphttpClient works perfectly.  It even works for my original code (simply  InitSSLInterface ... httpclient.Get(url, localfile),  without setting SSLType and  httpHeader...). Thank you very much. ---------------------------------------------------------------------- ALAU2007 · 2021-04-25 08:46 Hi, I have tested the Download.zip. It is fine when compile with Lazarus 1.6.4. When it compiles with Lazarus 2.0.10, it gets 'Error:  Identifier not found "stTLSv1_2"'.   Any comments. Thanks. ---------------------------------------------------------------------- trev · 2021-04-25 09:28 It compiles for me (macOS) with Lazarus 2.0.10 and FPC 3.2.0 with no problems.  "stTLSv1_2"' is in fpopenssl.pp ---------------------------------------------------------------------- ALAU2007 · 2021-04-25 11:37 Thanks, I compiles in Windows 10.  I cannot locate the stTLSv1_2 in fpopenssl.pp. ---------------------------------------------------------------------- trev · 2021-04-25 11:54 I checked  "stTLSv1_2"' is in fpopenssl.pp in FPC 3.0.4, 3.2.0 and 3.3.1. From FPC 3.2.0: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Constructor TSSLContext.Create(AType: TSSLType); Var  C : PSSL_CTX; begin  C := nil;  Case AType of    stAny:      begin        if Assigned(SslTLSMethod) then          C := SslCtxNew(SslTLSMethod)      else          C := SslCtxNew(SslMethodV23);      end;    stSSLv2: C := SslCtxNew(SslMethodV2);    stSSLv3: C := SslCtxNew(SslMethodV3);  stTLSv1: C := SslCtxNew(SslMethodTLSV1);    stTLSv1_1: C := SslCtxNew(SslMethodTLSV1_1);    stTLSv1_2: C := SslCtxNew(SslMethodTLSV1_2);  end;  if (C=Nil) then     Raise ESSL.Create(SErrCountNotGetContext);  Create(C);end; ---------------------------------------------------------------------- Thaddy · 2021-04-25 12:29 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     stSSLv2: C := SslCtxNew(SslMethodV2);    stSSLv3: C := SslCtxNew(SslMethodV3);    stTLSv1: C := SslCtxNew(SslMethodTLSV1);???? These are all deprecated and no longer included in OpenSsl binaries. You should not - can not - use them..(Unless you compile the OpenSsl binaries yourself, not recommended.) Actually, the FPC bindings should also remove these protocols. In that sense it is a bug that such code is still there. Also note the order is wrong: you need to test from high to low, not from low to high. This goes for any mainstream Ssl library, not only OpenSsl. In Essence, Ssl libraries do not contain Ssl  Only TLS 1.1 and higher. The above lines are not even supported by any modern browser, not even recognized. ---------------------------------------------------------------------- trev · 2022-02-14 05:24 I had occasion to recompile GetMem's zipped project which failed today despite compiling ok in April 2021. To successfully compile it now I had to change: 1) Uses clause second line from: fphttpclient, ssockets, sslsockets, fpopenssl;     to: fphttpclient, ssockets, sslsockets, opensslsockets, sslbase; 2) TForm1.GetSocketHandler() from: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.GetSocketHandler(Sender: TObject; const UseSSL: Boolean; out  AHandler: TSocketHandler);begin  AHandler := TSSLSocketHandler.Create;  TSSLSocketHandler(AHandler).SSLType := stTLSv1_2;end; to: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.GetSocketHandler(Sender: TObject; const UseSSL: Boolean; out  AHandler: TSocketHandler);begin  AHandler := TOpenSSLSocketHandler.Create; // change this line  TSSLSocketHandler(AHandler).SSLType := stTLSv1_2;end; Compiled successfully on macOS with: * Lazarus 2.3.0 (rev main-2_3-694-gc152361b72) FPC 3.3.1 x86_64-darwin-cocoa * Lazarus 2.2.0 (rev lazarus_2_2_0) FPC 3.2.2 x86_64-darwin-cocoa * Lazarus 2.0.12 FPC 3.2.0 x86_64-darwin-cocoa