Download File Using Synapse Compiling Error
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
Mattze96 · 2013-08-03 10:51

Hi,

i want to create a programm launcher, that updates my programm when an
update is avaible, but i get errors, when compiling my project.
I have just a simple form with a button which should start the
download.
In line 61 at HTTPGetResult.Document.SaveToFile(TargetFile); i get an
"Ilegall qualifier" Error. Im wondering because ive copied this
function from Synapse lazarus wiki (advanced http): http://wiki.freepa
scal.org/Synapse#Downloading_files_from_an_HTTP_server
Here is my Code:
Code: [Select]unit Unit1;

{$mode objfpc}{$H+}

interface

uses
  Classes, SysUtils, FileUtil, Forms, Controls, Graphics, Dialogs,
StdCtrls, httpsend;

type

  { TForm1 }

  TForm1 = class(TForm)
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
  private
    { private declarations }
  public
    { public declarations }
    function DownloadHTTP(URL: string; FileName: string): boolean;
  end;

var
  Form1: TForm1;

implementation

{$R *.lfm}

{ TForm1 }
function DownloadHTTP(URL, TargetFile: string): Boolean;
// Download file; retry if necessary.
// Could use Synapse HttpGetBinary, but that doesn't deal
// with result codes (i.e. it happily downloads a 404 error document)
const
  MaxRetries = 3;
var
  HTTPGetResult: Boolean;
  HTTPSender: THTTPSend;
  RetryAttempt: Integer;
begin
  Result := False;
  RetryAttempt := 1;
  HTTPSender := THTTPSend.Create;
  try
    try
      // Try to get the file
      HTTPGetResult := HTTPSender.HTTPMethod('GET', URL);
      while (HTTPGetResult = False) and (RetryAttempt < MaxRetries) do
      begin
        Sleep(500 * RetryAttempt);
        HTTPGetResult := HTTPSender.HTTPMethod('GET', URL);
        RetryAttempt := RetryAttempt + 1;
      end;
      // If we have an answer from the server, check if the file
      // was sent to us.
      case HTTPSender.Resultcode of
        100..299:
          begin
            HTTPGetResult.Document.SaveToFile(TargetFile);
            Result := True;
          end; //informational, success
        300..399: Result := False; // redirection. Not implemented,
but could be.
        400..499: Result := False; // client error; 404 not found etc
        500..599: Result := False; // internal server error
        else Result := False; // unknown code
      end;
    except
      // We don't care for the reason for this error; the download
failed.
      Result := False;
    end;
  finally
    HTTPSender.Free;
  end;
end;

procedure TForm1.Button1Click(Sender: TObject);
begin

end;
end.

best regards
matthias

----------------------------------------------------------------------
kamischi · 2013-08-03 11:21

I haven't tested, but since HTTPGetResult is a boolean,
HTTPGetResult.Document.SaveToFile is an error.

I haven't tested it, but from checking the class declaration of
HTTPSend, i think that HTTPSender should be used, in other words,
replace

  HTTPGetResult.Document.SaveToFile(TargetFile);

by

  HTTPSender.Document.SaveToFile(TargetFile);

If this works, please update the wiki pages.

----------------------------------------------------------------------
Leledumbo · 2013-08-03 11:23

QuoteIn line 61 at HTTPGetResult.Document.SaveToFile(TargetFile); i
get an "Ilegall qualifier" Error
Quotevar
  HTTPGetResult: Boolean;
Clear enough? The example is wrong then, I think it should be
HTTPSender instead of HTTPGetResult.

----------------------------------------------------------------------
Mattze96 · 2013-08-04 11:33

Hi,

thanks, the change HTTPSender.Document.... works fine and it downloads
files.

But I've got a sourceforge.net page and i want, that my launcher
downloads the newest version.
Example:(https://sourceforge.net/projects/drteach/files/latest/downloa
d)

http://wiki.freepascal.org/Download_from_SourceForge

I need a working function that gets the real URL of a special
sourceforgelink so that my DownloadHTTP function can download the
file.

But I get again some Errors and i don't know how to fix them. I copied
the whole Synapse-library into my project folder.

I hope my question is clear and here is my code:

Errorlog:
Code: [Select]Hint: Start of reading config file /etc/fpc.cfg
Hint: End of reading config file /etc/fpc.cfg
Free Pascal Compiler version 2.6.2 [2013/03/17] for x86_64
Copyright (c) 1993-2012 by Florian Klaempfl and others
Target OS: Linux for x86-64
Compiling project1.lpr
Compiling unit1.pas
unit1.pas(20,28) Hint: Parameter "Sender" not used
unit1.pas(129,15) Error: Identifier not found "PosEx"
unit1.pas(133,30) Error: Identifier not found "PosEx"
unit1.pas(140,16) Error: Identifier not found "RPos"
unit1.pas(143,28) Error: Identifier not found "RPosEx"
unit1.pas(167,15) Error: Identifier not found "infoln"
unit1.pas(168,15) Error: Identifier not found "infoln"
unit1.pas(204,13) Error: Identifier not found "infoln"
unit1.pas(222,39) Error: Duplicate identifier "Refresh"
unit1.pas(222,39) Hint: Identifier already defined in unit CONTROLS at
line 1505
unit1.pas(245,18) Error: Identifier not found "PosEx"
unit1.pas(246,17) Error: Identifier not found "infoln"
unit1.pas(247,17) Error: Identifier not found "infoln"
unit1.pas(259) Fatal: There were 11 errors compiling module, stopping
Code: [Select]unit Unit1;

{$mode objfpc}{$H+}

interface

uses
  Classes, SysUtils, FileUtil, Forms, Controls, Graphics, Dialogs,
StdCtrls,
  ComCtrls, Menus, httpsend;

type

  { TForm1 }

  TForm1 = class(TForm)
    Button1: TButton;
    CheckBox1: TCheckBox;
    ProgressBar1: TProgressBar;
    StatusBar1: TStatusBar;
    procedure Button1Click(Sender: TObject);
  private
    { private declarations }
  public
    { public declarations }
    function DownloadHTTP(URL: string; TargetFile: string): boolean;
    function SourceForgeURL(URL: string): string;
    function SFDirectLinkURL(URL: string; Document: TMemoryStream):
string;
  end;

var
  Form1: TForm1;

implementation

{$R *.lfm}

{ TForm1 }
//function DownloadHTTP(URL, TargetFile: string): Boolean;
//end;

procedure TForm1.Button1Click(Sender: TObject);
var sucess: Boolean;
var url: String;
begin
  url:=SourceForgeURL('https://sourceforge.net/projects/drteach/files/
latest/download');
  sucess:=DownloadHTTP(url, 'testfile');
  if sucess then ShowMessage('Sucess');
end;

function TForm1.DownloadHTTP(URL: string; TargetFile: string):
boolean;
 // Download file; retry if necessary.
// Could use Synapse HttpGetBinary, but that doesn't deal
// with result codes (i.e. it happily downloads a 404 error document)
const
  MaxRetries = 3;
var
  HTTPGetResult: Boolean;
  HTTPSender: THTTPSend;
  RetryAttempt: Integer;
begin
  Result := False;
  //URL:=SourceForgeURL(URL);
  RetryAttempt := 1;
  HTTPSender := THTTPSend.Create;
  try
    try
      // Try to get the file
      HTTPGetResult := HTTPSender.HTTPMethod('GET', URL);
      while (HTTPGetResult = False) and (RetryAttempt < MaxRetries) do
      begin
        Sleep(500 * RetryAttempt);
        HTTPGetResult := HTTPSender.HTTPMethod('GET', URL);
        RetryAttempt := RetryAttempt + 1;
      end;
      // If we have an answer from the server, check if the file
      // was sent to us.
      case HTTPSender.Resultcode of
        100..299:
          begin
            HTTPSender.Document.SaveToFile(TargetFile);
            Result := True;
          end; //informational, success
        300..399: Result := False; // redirection. Not implemented,
but could be.
        400..499: Result := False; // client error; 404 not found etc
        500..599: Result := False; // internal server error
        else Result := False; // unknown code
      end;
    except
      // We don't care for the reason for this error; the download
failed.
      Result := False;
    end;
  finally
    HTTPSender.Free;
  end;

end;

function TForm1.SourceForgeURL(URL: string): string;
// Detects sourceforge download and tries to deal with
// redirection, and extracting direct download link.
// Thanks to
// Ocye: http://lazarus.freepascal.org/index.php/topic,13425.msg70575.
html#msg70575
const
  SFProjectPart = '//sourceforge.net/projects/';
  SFFilesPart = '/files/';
  SFDownloadPart ='/download';
var
  HTTPSender: THTTPSend;
  i, j: integer;
  FoundCorrectURL: boolean;
  SFDirectory: string; //Sourceforge directory
  SFDirectoryBegin: integer;
  SFFileBegin: integer;
  SFFilename: string; //Sourceforge name of file
  SFProject: string;
  SFProjectBegin: integer;
begin
  // Detect SourceForge download; e.g. from URL
  //          1         2         3         4         5         6    
   7         8         9
  // 12345578901234567890123455789012345678901234557890123456789012345
57890123456789012345578901234567890
  // http://sourceforge.net/projects/base64decoder/files/base64decoder
/version%202.0/b64util.zip/download
  //                                 ^^^project^^^     
 ^^^directory............^^^ ^^^file^^^
  FoundCorrectURL:=true; //Assume not a SF download
  i:=Pos(SFProjectPart, URL);
  if i>0 then
  begin
    // Possibly found project; now extract project, directory and
filename parts.
    SFProjectBegin:=i+Length(SFProjectPart);
    j := PosEx(SFFilesPart, URL, SFProjectBegin);
    if (j>0) then
    begin
      SFProject:=Copy(URL, SFProjectBegin, j-SFProjectBegin);
      SFDirectoryBegin:=PosEx(SFFilesPart, URL,
SFProjectBegin)+Length(SFFilesPart);
      if SFDirectoryBegin>0 then
      begin
        // Find file
        // URL might have trailing arguments... so: search for first
        // /download coming up from the right, but it should be after
        // /files/
        i:=RPos(SFDownloadPart, URL);
        // Now look for previous / so we can make out the file
        // This might perhaps be the trailing / in /files/
        SFFileBegin:=RPosEx('/',URL,i-1)+1;

        if SFFileBegin>0 then
        begin
          SFFilename:=Copy(URL,SFFileBegin, i-SFFileBegin);
          //Include trailing /
          SFDirectory:=Copy(URL, SFDirectoryBegin, SFFileBegin-
SFDirectoryBegin);
          FoundCorrectURL:=false;
        end;
      end;
    end;
  end;

  if not FoundCorrectURL then
  begin
    try
      // Rewrite URL if needed for Sourceforge download redirection
      // Detect direct link in HTML body and get URL from that
      HTTPSender := THTTPSend.Create;
      //Who knows, this might help:
      HTTPSender.UserAgent:='curl/7.21.0 (i686-pc-linux-gnu)
libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18';
      while not FoundCorrectURL do
      begin
        HTTPSender.HTTPMethod('GET', URL);
        infoln('debug: headers:');
        infoln(HTTPSender.Headers.Text);
        case HTTPSender.Resultcode of
          301, 302, 307:
            begin
              for i := 0 to HTTPSender.Headers.Count - 1 do
                if (Pos('Location: ', HTTPSender.Headers.Strings[i]) >
0) or
                  (Pos('location: ', HTTPSender.Headers.Strings[i]) >
0) then
                begin
                  j := Pos('use_mirror=',
HTTPSender.Headers.Strings[i]);
                  if j > 0 then
                    URL :=
                      'http://' +
RightStr(HTTPSender.Headers.Strings[i],
                      length(HTTPSender.Headers.Strings[i]) - j - 10)
+
                      '.dl.sourceforge.net/project/' +
                      SFProject + '/' + SFDirectory + SFFilename
                  else
                    URL:=StringReplace(
                      HTTPSender.Headers.Strings[i], 'Location: ', '',
[]);
                  HTTPSender.Clear;//httpsend
                  FoundCorrectURL:=true;
                  break; //out of rewriting loop
              end;
            end;
          100..200:
            begin
              //Assume a sourceforge timer/direct link page
              URL:=SFDirectLinkURL(URL, HTTPSender.Document); //Find
out
              FoundCorrectURL:=true; //We're done by now
            end;
          500: raise Exception.Create('No internet connection
available');
            //Internal Server Error ('+aURL+')');
          else
            raise Exception.Create('Download failed with error code '
+
              IntToStr(HTTPSender.ResultCode) + ' (' +
HTTPSender.ResultString + ')');
        end;//case
      end;//while
      infoln('debug: resulting url after sf redir: *' + URL + '*');
    finally
      HTTPSender.Free;
    end;
  end;
  result:=URL;
end;

function TForm1.SFDirectLinkURL(URL: string; Document: TMemoryStream):
string;
  {
Transform this part of the body:
<noscript>
<meta http-equiv="refresh" content="5; url=http://downloads.sourceforg
e.net/project/base64decoder/base64decoder/version%202.0/b64util.zip?r=
&amp;ts=1329648745&amp;use_mirror=kent">
</noscript>
into a valid URL:
http://downloads.sourceforge.net/project/base64decoder/base64decoder/v
ersion%202.0/b64util.zip?r=&amp;ts=1329648745&amp;use_mirror=kent
}
const
  Refresh='<meta http-equiv="refresh"';
  URLMarker='url=';
var
  Counter: integer;
  HTMLBody: TStringList;
  RefreshStart: integer;
  URLStart: integer;
begin
  HTMLBody:=TStringList.Create;
  try
    HTMLBody.LoadFromStream(Document);
    for Counter:=0 to HTMLBody.Count-1 do
    begin
      // This line should be between noscript tags and give the direct
download locations:
      RefreshStart:=Ansipos(Refresh, HTMLBody[Counter]);
      if RefreshStart>0 then
      begin
        URLStart:=AnsiPos(URLMarker,
HTMLBody[Counter])+Length(URLMarker);
        if URLStart>RefreshStart then
        begin
          // Look for closing "
          URL:=Copy(HTMLBody[Counter],
            URLStart,
            PosEx('"',HTMLBody[Counter],URLStart+1)-URLStart);
          infoln('debug: new url after sf noscript:');
          infoln(URL);
          break;
        end;
      end;
    end;
  finally
    HTMLBody.Free;
  end;
  result:=URL;
end;
end.

----------------------------------------------------------------------
Mattze96 · 2013-08-04 12:50

Update:

The solution is not very important. I've just setup my own server

----------------------------------------------------------------------
kamischi · 2013-08-04 13:52

PosEx is from the unit StrUtils. Add "StrUtils" to your Uses Part of
the program should fix the compile error. no idea, where Infoln comes
from.

----------------------------------------------------------------------
BigChimp · 2013-08-04 14:08

The infoln call is in my fpcup tool to output diagnostic info. Seems
somebody copied some code from fpcup instead of the wiki article (fine
by me though).

----------------------------------------------------------------------
Mattze96 · 2013-08-04 14:08

Well thank you. StrUtils are working.
These Errors are left:
Code: [Select]Hint: Start of reading config file /etc/fpc.cfg
Hint: End of reading config file /etc/fpc.cfg
Free Pascal Compiler version 2.6.2 [2013/03/17] for x86_64
Copyright (c) 1993-2012 by Florian Klaempfl and others
Target OS: Linux for x86-64
Compiling project1.lpr
Compiling unit1.pas
unit1.pas(20,28) Hint: Parameter "Sender" not used
unit1.pas(167,15) Error: Identifier not found "infoln"
unit1.pas(168,15) Error: Identifier not found "infoln"
unit1.pas(204,13) Error: Identifier not found "infoln"
unit1.pas(222,39) Error: Duplicate identifier "Refresh"
unit1.pas(222,39) Hint: Identifier already defined in unit CONTROLS at
line 1505
unit1.pas(246,17) Error: Identifier not found "infoln"
unit1.pas(247,17) Error: Identifier not found "infoln"
unit1.pas(259) Fatal: There were 6 errors compiling module, stopping

----------------------------------------------------------------------
BigChimp · 2013-08-04 14:12

Yes, you ()@#*$@*#$. If you rip out somebody's code from somewhere you
have to take care to also add any called functions.

If you don't understand that - what about adding this (off the top of
my head, might not compile):
Code: [Select]function infoln(Message: string): boolean;
begin
  result:=true;
end;

----------------------------------------------------------------------
Mattze96 · 2013-08-04 14:29

Sorry and thank you very much. It's working now.

Sorry the infoln() function was not mention in your post: http://lazar
us.freepascal.org/index.php/topic,13425.msg70575.html#msg70575 or i
did not found it.

http://wiki.freepascal.org/Download_from_SourceForge

maybe you can add it?

Sorry again and thanks.

matthias

----------------------------------------------------------------------
BigChimp · 2013-08-04 14:42

Quote from: Mattze96 on August 04, 2013, 04:29:16 pmSorry the infoln()
function was not mention in your post: http://lazarus.freepascal.org/i
ndex.php/topic,13425.msg70575.html#msg70575 or i did not found it.

I think you didn't find it - it was mentioned:
Quote from: BigChimp on February 19, 2012, 09:33:30 amDemo project
attached; I currently have this code (using Synapse, infoln is
basically writeln):

Quotehttp://wiki.freepascal.org/Download_from_SourceForge
No infoln there that I could find, so no changes needed. If any are
needed, see below...

Quote from: Mattze96 on August 04, 2013, 04:29:16 pmmaybe you can add
it?
So after posting the initial code (with instructions) on the forum,
writing the code snippet on the wiki you are also asking to perform
further modifications?
Sorry, but that is your task, not mine. I'm very happy to give free
support, help others as I have been helped, post articles etc but
there is a limit somewhere.

Quote from: Mattze96 on August 04, 2013, 04:29:16 pmSorry again and
thanks.
No problem - it wasn't personal, just that a lot of posts/people just
take, take, take without giving back (or even help you to help them by
providing the right information).

Glad you got your code working.

----------------------------------------------------------------------
Miyamoto · 2013-08-10 10:10

This example uses the same instruction but i get another error
Code: [Select]uses httpsend,
...
function DownloadHTTP(URL, TargetFile: string): Boolean;
var
  HTTPGetResult: Boolean;
  HTTPSender: THTTPSend;
begin
  Result := False;
  HTTPSender := THTTPSend.Create;
  try
    HTTPGetResult := HTTPSender.HTTPMethod('GET', URL);
    if HTTPSender.ResultCode = 100..299 then begin
      HTTPGetResult.Document.SaveToFile(TargetFile);
      Result := True;
    end;
  finally
    HTTPSender.Free;
  end;
end;

When compiling it says:

then expected but .. found on this line     if HTTPSender.ResultCode =
100..299 then begin


I don't understand why the 'Advanced version' uses:

Code: [Select]        300..399: Result := False; // redirection. Not
implemented, but could be.
        400..499: Result := False; // client error; 404 not found etc
        500..599: Result := False; // internal server error
        else Result := False; // unknown code
instead of:
else  Result := False;   

http://wiki.freepascal.org/Synapse#From_an_HTTP_server

----------------------------------------------------------------------
Jurassic Pork · 2013-08-10 10:42

hello,
Miyamoto look at    this message  or this one

----------------------------------------------------------------------
eric · 2013-08-10 11:47

Code: [Select]if HTTPSender.ResultCode = 100..299 then begin
should surely be:

Code: [Select]if (HTTPSender.ResultCode in [100..299]) then begin