GetURLText
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
VTwin · 2023-03-05 23:04

For a user to check for updates I have been using a very simple
method, where a text file containing the current version (e.g.,
'3.21.0') is checked against the user's version.

I have been using a function GetURLText posted by trev, for quite a
while, but it seems to no longer work. A search for GetURLText does
not come up with a hit.

What is the best way to download a simple text file from a url?

----------------------------------------------------------------------
KodeZwerg · 2023-03-05 23:10

For what OS with what needs (plain old http or newer https) (username
password needed?)

----------------------------------------------------------------------
VTwin · 2023-03-05 23:58

Thanks KodeZwerg,

I need something for macOS, Windows, and Linux, as in my profile.

I just want to download a text file, plain old http, no password.

Edit: Actually the site is https.
Something seems to have changed in the past few months.

----------------------------------------------------------------------
KodeZwerg · 2023-03-06 00:26

I am unsure about target platforms, please try for yourself.
(OPM -> Synapse -> Install)
Start a new project and try that:
(add to project requirement lazSynapse)
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i unit
Unit1; {$mode objfpc}{$H+} interface uses  Classes , SysUtils , Forms
, Controls , Graphics , Dialogs , StdCtrls ,  HTTPSend; type   {
TForm1 }   TForm1 = class(TForm)    Button1: TButton;    Memo1:
TMemo;    procedure Button1Click(Sender: TObject);  strict private 
private   public   end; var  Form1: TForm1; implementation {$R
*.lfm} { TForm1 } function GetURLText(const AURL: string; out AText:
string): Boolean;begin  AText := '';  Result := False;  with
THTTPSend.Create do  begin    if HTTPMethod('GET', AURL) then    try 
   if Document.Size > 0 then      begin        Document.Position :=
0;        SetLength(AText, Document.Size);       
Document.ReadBuffer(Pointer(AText)^, Document.Size);        Result :=
AText <> '';      end;    except     
Application.MessageBox('Unknown','Error', 0);    end;    Free; 
end;end; procedure TForm1.Button1Click(Sender: TObject);var  s:
string;begin  if GetURLText('http://your url with filename', s) then 
 Memo1.Lines.Add(s)    else    Memo1.Lines.Add('Error');end; end.

----------------------------------------------------------------------
dsiders · 2023-03-06 01:12

Quote from: VTwin on March 06, 2023, 12:58:18 amThanks KodeZwerg,

I need something for macOS, Windows, and Linux, as in my profile.

I just want to download a text file, plain old http, no password.

Edit: Actually the site is https.
Something seems to have changed in the past few months.

You can use the TFPHttpClient class in the fphttpclient unit. It's
cross-platform. Supports SSL when you use the appropriate optional
units. Call the Get method, or SimpleGet to ignore protocol
errors/exceptions.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i AString :=
TFPHTTPClient.SimpleGet(AUrl);
See: https://wiki.freepascal.org/fphttpclient

----------------------------------------------------------------------
Thaddy · 2023-03-06 08:05

If simpleget won't work, do it like this (also simple, but slightly
more involved):Code: Pascal  [Select][+][-]window.onload =
function(){var x1 = document.getElementById("main_content_section");
if (x1) { var x = document.getElementsByClassName("geshi");for (var i
= 0; i {$mode objfpc}uses  
classes,sysutils,fphttpclient,opensslsockets,fpJson,jsonparser; var 
Client:TFpHttpClient;  List:TStringList;  URL:String =
'https://www.example.com';begin  Client := TfpHttpClient.Create(nil); 
try    // this is important    Client.AllowRedirect := true; // I am
not sure SimpleGet handles redirects.    // optional browser
impersonation. This is sometimes necessary, although there is a very
old default.    Client.RequestHeaders.Add('User-Agent: Mozilla/5.0
(X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0');    List :=
TStringlist.Create;    try      List.Text := Client.Get(URL);     
writeln(List.Text);    finally      List.Free;    end;  finally   
Client.Free;  end;end.

I recommend using fcl-web like above, instead of Synapse. Better
maintained and standard.

----------------------------------------------------------------------
Warfley · 2023-03-06 09:49

Just use TFPHTTPClient with the openssl handler:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses 
opensslsockets, // Include per uses to enable OpenSSL handling  
fphttpclient; var  WebsiteText: String;begin  WebsiteText :=
TFPHTTPClient.SimpleGet('https://my.https.example.com'); 
WriteLn(WebsiteText);end.OpenSSL is pre installed on windows since a
few years, and should also be available on mac and pretty much all of
the most common linux distros.

----------------------------------------------------------------------
Thaddy · 2023-03-06 13:24

@Warfley
SimpleGet does not always work, hence my example.

----------------------------------------------------------------------
VTwin · 2023-03-06 15:52

Thanks all. Trying Warfley and Thaddy's examples, I get (on Windows)
exception  'EInOutError' "Could not initialize OpenSSL library" in
both cases.

I did not try KodeZwerg's example as HTTPSend was not found.

In the distant past I was using Synapse, but switched to fphttpclient
to minimize dependencies. It was working fine for a long time, but
recently I got a compiler message to include opensslsockets. Now I am
getting exceptions.

----------------------------------------------------------------------
Thaddy · 2023-03-06 15:55

64 bit or 32 bit?
The openssl libraries must match exactly.
For now I think there is your problem.
If that is the case it is easy to solve.
(assuming windows64)
Wait for it:
- if it is 32 bit, download the 32 bit version and copy the dll's to
\windows\syswow64 (yes 32 bit goes there!)(Many wtf's here)
- if it is 64 bit on 64 bit windows, download the 64 bit version and
put the dll's in \windows\system32 (yes 64 bit goes there)(Many wtf's
here)

I know this is counter-intuitive..... Let's call it "Windows Logic".

"Surprised?  You won't be, until the next episode of Soap"
https://www.youtube.com/watch?v=0BHQT3Omqtw

btw: WoW stands for Windows32 on Windows64. On Linux it would be
multi-arch.

Microsoft's answer (ad verbatim):
"The Windows/SysWOW64 directory contains 32-bit libraries and
executables while the Windows/System32 directory contains 64-bit
libraries and executables. The SysWOW64 directory was created to
facilitate the running of 32-bit legacy applications on 64-bit Windows
systems."

Many people, even on this forum, get it wrong all the time...... 

But that is obvious, isn't it? Everyone understands that?

----------------------------------------------------------------------
Warfley · 2023-03-06 17:21

Quote from: Thaddy on March 06, 2023, 04:55:50 pm(assuming windows64)
Wait for it:
- if it is 32 bit, download the 32 bit version and copy the dll's to
\windows\syswow64 (yes 32 bit goes there!)(Many wtf's here)
- if it is 64 bit on 64 bit windows, download the 64 bit version and
put the dll's in \windows\system32 (yes 64 bit goes there)(Many wtf's
here)

I know this is counter-intuitive..... Let's call it "Windows Logic".
There is actually reasoning behind it. System32 is named the same on
32bit and 64 bit, because of backwards compatibility, this is not
pretty, but this backwards compatibility even to the lowest elevel is
one of the reasons why windows is so dominant in the market.
The syswow64 is called that way because its the system folder for the
Windows on Windows 64 (WoW64) subsystem, which basically is a 32 bit
windows environment emulated within a windows 64 environment,
therefore the name "windows" on "windows 64".

So it might not be intuitive, but it makes at least a bit sense when
thinking about it. But I would not rule out that the naming of
SysWoW64 was an april fools joke by some microsoft employee that got
through QA and got rolled out to production

----------------------------------------------------------------------
VTwin · 2023-03-06 17:23

Thanks. Wow, or is it SysWoW64?

I use 64 bit on Windows, Mac, and Linux.

Maybe that is the problem, but why doesn't fpc link in the required
version? On Mac I go to

usr/local/lib/fpc/3.2.2/units/

where I find aarch64-darwin, i386-darwin, x86_64-darwin, each with an
openssl directory.

----------------------------------------------------------------------
VTwin · 2023-03-06 17:53

I also tried KodeZwerg's solution using Synapse. No luck yet.

----------------------------------------------------------------------
balazsszekely · 2023-03-06 18:35

@VTwin

On windows you should check if the openssl libraries are available, if
not download them, something like this:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses
zipper, fphttpclient, opensslsockets; function GetURL(URL: String; var
URLRes: String): Boolean;var  SS: TStringStream;  FPHTTPClient:
TFPHTTPClient;begin  Result := False;  FPHTTPClient :=
TFPHTTPClient.Create(nil);  try    FPHTTPClient.AllowRedirect :=
True;    SS := TStringStream.Create('');    try      Screen.Cursor :=
crHourglass;      try        FPHTTPClient.Get(URL, SS);      finally 
     Screen.Cursor := crDefault;      end;      if
FPHTTPClient.ResponseStatusCode = 200 then      begin        URLRes :=
SS.DataString;        Result := True;      end;    finally     
SS.Free;    end;  finally    FPHTTPClient.Free;  end;end; function
IsOpenSSLAvailable: Boolean;const  {$IFDEF WIN64}    cOpenSSLURL =
'http://packages.lazarus-ide.org/openssl-1.1.1o-x64_86-win64.zip'; 
{$ENDIF}  {$IFDEF WIN32}    cOpenSSLURL = 'http://packages.lazarus-
ide.org/openssl-1.1.1o-i386-win32.zip';  {$ENDIF}var  {$IFDEF
MSWINDOWS}  UnZipper: TUnZipper;  FHTTPClient: TFPHTTPClient; 
ParamPath, LibCrypto, LibSSL, ZipFile: String;  {$EndIf}begin  {$IFDEF
MSWINDOWS}  ParamPath := ExtractFilePath(ParamStr(0));   {$IFDEF
WIN64}  LibCrypto := ParamPath + 'libcrypto-1_1-x64.dll';  LibSSL :=
ParamPath + 'libssl-1_1-x64.dll';  {$ENDIF}  {$IFDEF WIN32}  LibCrypto
:= ParamPath + 'libcrypto-1_1.dll';  LibSSL := ParamPath +
'libssl-1_1.dll';  {$ENDIF}   if not Result then  begin    ZipFile :=
ParamPath + ExtractFileName(cOpenSSLURL);    FHTTPClient :=
TFPHTTPClient.Create(nil);    try      try       
FHTTPClient.Get(cOpenSSLURL, ZipFile);       except       end;   
finally      FHTTPClient.Free;    end;    if FileExists(ZipFile) then 
  begin      UnZipper := TUnZipper.Create;      try        try       
 UnZipper.FileName := ZipFile;          UnZipper.Examine;         
UnZipper.UnZipAllFiles;        except        end;      finally       
UnZipper.Free;      end;      DeleteFile(ZipFile);      Result :=
FileExists(LibCrypto) and FileExists(LibSSL);    end;  end;  {$ELSE} 
FOpenSSLAvailable := True;  {$ENDIF}end; 
See attached demo for more details.


Correction: The 64 bit library name was wrong. Thanks @KodeZwerg!

----------------------------------------------------------------------
VTwin · 2023-03-06 18:54

Thanks GetMem.

I tried your example on Windows, it tells me that OpenSSL is not
available. It does not compile on Mac though, and I need something
cross-platform.

Maybe I should just give up. It is a very minor function of my
software, just a nice convenience for my users. It is weird though, if
I run my software from 2020, it works fine.

----------------------------------------------------------------------
KodeZwerg · 2023-03-06 18:57

Quote from: VTwin on March 06, 2023, 06:53:03 pmI also tried
KodeZwerg's solution using Synapse. No luck yet.Just because you
edited your post, i tested my source on http without having any
problems.

----------------------------------------------------------------------
KodeZwerg · 2023-03-06 19:04

@GetMem, on Windows 64bit you need to have different filenames!
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   {$IFDEF
WIN64}  LibCrypto := ParamPath + 'libcrypto-1_1-x64.dll';  LibSSL :=
ParamPath + 'libssl-1_1-x64.dll';  {$ENDIF}  {$IFDEF WIN32}  LibCrypto
:= ParamPath + 'libcrypto-1_1.dll';  LibSSL := ParamPath +
'libssl-1_1.dll';  {$ENDIF}That works for Windows 64bit and 32bit.

----------------------------------------------------------------------
balazsszekely · 2023-03-06 19:06

QuoteI tried your example on Windows, it tells me that OpenSSL is not
available.Then copy libcrypto-1_1.dll and libssl-1_1.dll in the same
folder as your executable(only needed on windows). Download the
libraries from here:
32 bit: http://packages.lazarus-ide.org/openssl-1.1.1o-i386-win32.zip
64 bit: 'http://packages.lazarus-
ide.org/openssl-1.1.1o-x64_86-win64.zip

Then the following code should cross platform:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses
fphttpclient, opensslsockets; function GetURL(URL: String; var URLRes:
String): Boolean;var  SS: TStringStream;  FPHTTPClient:
TFPHTTPClient;begin  Result := False;  FPHTTPClient :=
TFPHTTPClient.Create(nil);  try    FPHTTPClient.AllowRedirect :=
True;    SS := TStringStream.Create('');    try      Screen.Cursor :=
crHourglass;      try        FPHTTPClient.Get(URL, SS);      finally 
     Screen.Cursor := crDefault;      end;      if
FPHTTPClient.ResponseStatusCode = 200 then      begin        URLRes :=
SS.DataString;        Result := True;      end;    finally     
SS.Free;    end;  finally    FPHTTPClient.Free;  end;end;
QuoteMaybe I should just give up.No! Don't give up.

----------------------------------------------------------------------
balazsszekely · 2023-03-06 19:17

@KodeZwerg
Quote@GetMem, on Windows 64bit you need to have different
filenames!Yes, you're right, the name of the 64 bit version is wrong.
Thanks for catching that! I will modify my previous post.

----------------------------------------------------------------------
VTwin · 2023-03-06 19:39

Quote from: GetMem on March 06, 2023, 08:06:21 pmNo! Don't give up. 

  Thanks! I'll keep at it, but already spent half a day. Still no
luck.

Downloading a DLL with my software is probably a non-starter. More and
more end users are on locked down computers, with IT freaking out
every time someone sneezes on the keyboard. I could probably include
it in the installation zip file, but that adds an additional
complexity.

----------------------------------------------------------------------
BobDog · 2023-03-06 20:02

Windows method, takes a couple of seconds to find site.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   {$APPTYPE
CONSOLE}usessysutils,process;  function download(site:ansistring):ansi
string;varc:ansistring='';newfile:ansistring;Fin:file of
byte;len:int32;begin     newfile:=GetCurrentDir
+'/site.txt'; runcommand('powershell',['Invoke-
WebRequest',site,'-OutFile',newfile],c);   Assign (Fin,newfile); 
 Reset (Fin);   len:=filesize(Fin);   setlength(c,len);   Reset
(Fin,len);   BlockRead (Fin,c[1],1);   close(Fin); 
 deletefile(newfile);   exit(c);end;    var 
site:ansistring='https://www.example.com';  c:ansistring;  begin 
c:=download(site);  writeln(c);  writeln('PRESS RETURN TO END ...'); 
readln;  end.

----------------------------------------------------------------------
dsiders · 2023-03-06 21:27

If you're using 64-bit, the file names are:

 libcrypto-1_1-x64.dll
libssl-1_1-x64.dll

If you're using 32-bit, the file names are:

 libcrypto-1_1.dll
libssl-1_1.dll

If your URL is valid and your uses clause has fphtttpclient and
opensslsockets, that is all you need.

----------------------------------------------------------------------
KodeZwerg · 2023-03-06 23:20

Quote from: dsiders on March 06, 2023, 10:27:52 pmthe file names
arefor completeness:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   {$IFDEF
WINDOWS}  DLLSSLName: string = 'ssleay32.dll';  DLLSSLName2: string =
'libssl32.dll';  DLLSSLName3: string = {$IFDEF
WIN64}'libssl-1_1-x64.dll'{$ELSE}'libssl-1_1.dll'{$ENDIF}; 
DLLUtilName: string = 'libeay32.dll';  DLLUtilName2: string = {$IFDEF
WIN64}'libcrypto-1_1-x64.dll'{$ELSE}'libcrypto-1_1.dll'{$ENDIF}; 
{$ELSE}   {$IFDEF OS2}    {$IFDEF OS2GCC}  DLLSSLName: string =
'kssl10.dll';  DLLUtilName: string = 'kcrypt10.dll';  DLLSSLName2:
string = 'kssl.dll';  DLLUtilName2: string = 'kcrypto.dll';    {$ELSE
OS2GCC}  DLLSSLName: string = 'emssl10.dll';  DLLUtilName: string =
'emcrpt10.dll';  DLLSSLName2: string = 'ssl.dll';  DLLUtilName2:
string = 'crypto.dll';    {$ENDIF OS2GCC}   {$ELSE OS2}  DLLSSLName:
string = 'libssl';  DLLUtilName: string = 'libcrypto';   { ADD NEW
ONES WHEN THEY APPEAR!    Always make .so/dylib first, then versions,
in descending order!    Add "." .before the version, first is always
just "" }  DLLVersions: array[1..19] of string = ('', '.1.1', '.11',
'.10', '.1.0.6', '.1.0.5', '.1.0.4', '.1.0.3',                       
               '.1.0.2', '.1.0.1','.1.0.0','.0.9.8',                 
                     '.0.9.7', '.0.9.6', '.0.9.5', '.0.9.4',         
                             '.0.9.3', '.0.9.2', '.0.9.1');   {$ENDIF
OS2}  {$ENDIF WINDOWS}opensslsockets -> openssl