[SOLVED] How to download HTML from secure https site
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
bobonwhidbey · 2018-10-19 22:05

I've been using the following:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
DownloadHTTP(URL, TargetFile: string): boolean;var  HTTPSender:
THTTPSend;begin  Result := False;  HTTPSender := THTTPSend.Create; 
try    HTTPSender.HTTPMethod('GET', URL);    if (HTTPSender.ResultCode
>= 100) and (HTTPSender.ResultCode <= 299) then    begin     
HTTPSender.Document.SaveToFile(TargetFile);      Result := True;   
end;  finally    HTTPSender.Free;  end;end;  
to download the HTML from HTTP web site. Is there a similar function I
can use to download from HTTPS sites?

----------------------------------------------------------------------
CCRDude · 2018-10-20 05:23

Yes, the same, but you need to make sure you add a SSL library to the
uses list... like ssl_openssl (also from synapse)

----------------------------------------------------------------------
Thaddy · 2018-10-20 12:03

Yes, I recognize that code 
It is indeed sufficient to add ssl_openssl to the uses clause. Nothing
more required.
Full program:Code: Pascal  [Select][+][-]window.onload =
function(){var x1 = document.getElementById("main_content_section");
if (x1) { var x = document.getElementsByClassName("geshi");for (var i
= 0; i {$ifdef fpc}{$mode delphi}{$H+}{$endif}uses
sysutils,classes,httpsend, ssl_openssl; function DownloadHTTP(URL,
TargetFile: string): boolean;var  HTTPSender: THTTPSend;begin  Result
:= False;  HTTPSender := THTTPSend.Create;  try   
HTTPSender.HTTPMethod('GET', URL);    if (HTTPSender.ResultCode >=
100) and (HTTPSender.ResultCode <= 299) then    begin     
HTTPSender.Document.SaveToFile(TargetFile);      Result := True;   
end;  finally    HTTPSender.Free;  end;end;  var  s:string
='testme.html';  l:Tstringlist;begin  if fileexists(s) then
deletefile(s);   if
Downloadhttp('https://www.freepascal.org/index.html',s) then  try   
l:=Tstringlist.create;    l.LoadfromFile(s);    writeln(l.text); 
finally    l.free;  end;end.
Or much simpler, like this:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i {$ifdef
fpc}{$mode delphi}{$H+}{$endif}uses classes, httpsend,
ssl_openssl;var  l:Tstrings;begin  l:=Tstringlist.create;  try    //
this is a standard utility function from synapse    if
HttpGetText('https://www.freepascal.org/index.html',L) then
writeln(l.text); // or l.savetofile  finally    l.free;  end;end.
That's really all...

----------------------------------------------------------------------
bobonwhidbey · 2018-10-20 16:13

Thank you - works perfectly.

----------------------------------------------------------------------
bobonwhidbey · 2018-10-22 15:43

While MyApp seems to work perfectly, I am getting a compiler warning
that I would like to know how to eliminate.

"Warning: other unit files search path (aka unit paty) of "MyApp"
contains "C:\Lazarus\components\custom\Synapse\source\lib", which
belongs to package "laz_synapse".

I have added the laz_synapse package to the Project Inspector.

----------------------------------------------------------------------
Thaddy · 2018-10-22 16:37

The synapse component, not the synapse library, is utterly useless and
you should uninstall it.
It is also the cause of this warning. It will go away when you remove
the component.

Synapse (the original, real Synapse) is purposefully written to NOT be
a component but a set of classes and routines to be used from source.

Synapse itself is world-class
The  component is a piece of cow dung. The writer -  not the writer of
synapse - could have known it would cause your problem and never took
the trouble to test it let alone fix it..

Again: >Synapse is designed to be used without components and the
component (laz_synapse) is ...

----------------------------------------------------------------------
bobonwhidbey · 2018-10-22 16:46

Thank you Thaddy. I removed the component and thus got rid of the
warnings

----------------------------------------------------------------------
xos · 2025-01-27 11:15

Hello folks,
I have adopted the first example from Thaddy but unfortunately it does
not work.
The HTTPSender.HTTPMethod(‘GET’, URL); call returns the value 500.
-> Uses is changed to ssl_openssl3.


Laz 3.0
FPC 3.2.2

----------------------------------------------------------------------
TRon · 2025-01-27 12:36

Quote from: xos on January 27, 2025, 12:15:14 pmcall returns the value
500.
That is a server error, not an error in the code.

----------------------------------------------------------------------
xos · 2025-01-27 22:19

Ok, then: why is there a server error?

The URL in the example is ‘https://www.freepascal.org/index.html’.
This page can be called up in a browser.

In the example, the HTTPMethod() should also be able to call up the
page.
-> HTTPSender.HTTPMethod(‘GET’, URL);

Why then a server error message?

----------------------------------------------------------------------
dsiders · 2025-01-28 00:35

Quote from: xos on January 27, 2025, 11:19:29 pmOk, then: why is there
a server error?

The URL in the example is ‘https://www.freepascal.org/index.html’.
This page can be called up in a browser.

In the example, the HTTPMethod() should also be able to call up the
page.
-> HTTPSender.HTTPMethod(‘GET’, URL);

Why then a server error message?

What does HttpSender.ResultString tell you?

----------------------------------------------------------------------
xos · 2025-01-28 12:23

Thank you for your help!

ResultString = ''  -> is empty
ResultCode = 500