[RESOLVED] Synapse HttpSend, zero-length content
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
jamhitz · 2011-08-01 22:23

I have been using Indy for quite a while, but now I have started
learning Synapse. In the process, I am documenting my findings and
will post them online to act as a user guide for new users in the
future.

Using HttpSend, I have struggled to understand why some pages return
no content at all when requested using THttpSend, but are perfectly
alright when requested using a browser; even after sniffing out the
headers and trying to make them as compatible as possible.

For example I have tried loading https://dspace.library.uvic.ca:8443/
using the following code:

Code: [Select]interface

uses
  Classes, SysUtils, FileUtil, Forms, Controls, Graphics, Dialogs,
StdCtrls,
  blcksock, httpsend, ssl_openssl;

type

  { Tfrm }

  Tfrm = class(TForm)
      btnSend: TButton;
      edtURL: TEdit;
      memHeader: TMemo;
      memBody: TMemo;
      procedure btnSendClick(Sender: TObject);
  end;

var
  frm: Tfrm;

implementation

{$R *.lfm}

{ Tfrm }

procedure Tfrm.btnSendClick(Sender: TObject);
var ht:THttpSend;
begin
    ht := THttpSend.Create;
    ht.UserAgent := 'Mozilla/5.0 (X11; Linux i686; rv:5.0)
Gecko/20100101 Firefox/5.0';
    ht.Headers.Add('Accept: text/html');
    //ht.KeepAlive := true;
    ht.Protocol := '1.1';
    try
        if ht.HTTPMethod('GET',edtUrl.Text) then
        begin
            if(ht.Document.Size > 0) then
            begin
                ht.Document.Position := 0;
                memBody.Lines.LoadFromStream(ht.Document);
            end
            else
                memBody.Lines.Text := 'Error';
        end;
        memHeader.Lines.Add(IntToStr(ht.ResultCode) + '  ' +
ht.ResultString);
    finally
        memHeader.Lines.Text := ht.Headers.Text;
        ht.Free;
    end;
end;
Any ideas?

Thanks.

----------------------------------------------------------------------
Leledumbo · 2011-08-01 22:52

QuoteFor example I have tried loading
https://dspace.library.uvic.ca:8443/Get it?
Please take a look at this as well.

----------------------------------------------------------------------
Silvio Clécio · 2011-08-01 23:28

And:

http://code.google.com/p/lazsolutions/wiki/Tip?wl=en

----------------------------------------------------------------------
jamhitz · 2011-08-02 09:30

@Leledumbo and @silvioprog:

My checklist:

I have OpenSSL installedI have the latest Synapse from trunkI can
access *many* secure sites such (eg https://mail.google.com/mail/ and
https://www.admissions.uci.edu:8443/cgi-bin/maa_uci.cgi) but not all
(eg https://dspace.library.uvic.ca:8443/)
In Synapse, it just fails silently, in Lnet it returns an SSL error,
loops perpetually (CallAction) and then segfaults when I break the
code.

I don't know how relevant this is, but I noticed (using Firebug) that
most of the sites that are failing in Lazarus (but succeeding in
Firefox) are running off of Apache-Coyote/1.1 which is part of Tomcat,
I think.  Are there any known issues? Here is my LNet code:

Code: [Select]unit uMain;

{$mode objfpc}{$H+}

interface

uses
  Classes, SysUtils, FileUtil, Forms, Controls, Graphics, Dialogs,
StdCtrls,
  lnet, lHttp, lHttpUtil, lNetSSL, UriParser;

type

  { Tfrm }

  Tfrm = class(TForm)
      btnSend: TButton;
      edtURL: TEdit;
      memHeader: TMemo;
      memBody: TMemo;
      procedure btnSendClick(Sender: TObject);
      procedure edtURLChange(Sender: TObject);
      procedure FormCreate(Sender: TObject);
  private
    { private declarations }
       isDone: Boolean;
       errMsg: String;
       aBuff: TMemoryStream;
       procedure HtDisconnect(ASocket: TlSocket);
       procedure HtDoneInput(ASocket: TlHttpClientSocket);
       procedure HtError(const Msg: string; aSocket: TLSocket);
       procedure HtProcessHeaders(ASocket: TLHttpClientSocket);
       function HtInput(ASocket:TLHttpClientSocket; ABuffer:pChar;
ASize:Integer): Integer;
       procedure HtSSLAccept(aSocket: TLSocket);
  public
    { public declarations }
  end;

var
  frm: Tfrm;

implementation

{$R *.lfm}

{ Tfrm }

procedure Tfrm.HtDisconnect(ASocket: TLSocket);
begin
        isDone := true;
end;

procedure Tfrm.HtDoneInput(ASocket: TLHTTPClientSocket);
begin
        ASocket.Disconnect();
end;

procedure Tfrm.HtError(const Msg: string; aSocket: TLSocket);
begin
    isDone := true;
    errMsg := Msg;
    memHeader.Lines.Add('Error: ' + errMsg);
    //aSocket.SocketState;
end;

procedure Tfrm.HtProcessHeaders(ASocket: TLHTTPClientSocket);
begin
   
memHeader.Lines.Add(IntToStr(HttpStatusCodes[aSocket.ResponseStatus])
+ ' ' + HTTPTexts[aSocket.ResponseStatus] + ' ' +
ASocket.ResponseReason);
end;

function Tfrm.HtInput(ASocket: TLHTTPClientSocket; ABuffer: pChar;
ASize: Integer): Integer;
var     newText:String;
begin
        aBuff.Write(ABuffer^,ASize);
end;

procedure Tfrm.HtSSLAccept(aSocket: TLSocket);
begin
        memHeader.Lines.Add('TLS Handshake successful');
end;

procedure Tfrm.btnSendClick(Sender: TObject);
var   ht:TlHttpClient;
        ssl: TlSslSession;
        url, uri, host: String;
        port: Word; isSSL:Boolean;
begin
        ht := TlHttpClient.Create(nil);
        ssl := TlSslSession.Create(ht);
        aBuff := TMemoryStream.Create;
        url := edtUrl.Text;
        isSSl := DecomposeURL(url,host,uri,port);

        {ht.AddExtraHeader('UserAgent: Mozilla/5.0 (X11; Linux i686;
rv:5.0) Gecko/20100101 Firefox/5.0');
        ht.AddExtraHeader('Accept: text/html');
        ht.AddExtraHeader('Accept-Language: en-us,en;q=0.5');
        ht.AddExtraHeader('Accept-Encoding: gzip, deflate');
        ht.AddExtraHeader('Accept-Charset:
ISO-8859-1,utf-8;q=0.7,*;q=0.7');
        ht.AddExtraHeader('Connection: keep-alive');}

    memHeader.Clear;
    memHeader.Lines.Add('Host: ' + host + ', URI: ' + uri + ', Port' +
IntToStr(port));

    ssl.SSLActive := isSSL;
    if isSSL then memHeader.Lines.Add('SSL Active');

    try
        ht.Session := ssl;
        ht.Host := host;
        ht.Method := hmGet;
        ht.Port := port;
        ht.URI := uri;
        ht.Timeout := 1;
        ssl.OnSSLAccept := @HtSSLAccept;
        ht.OnDisconnect := @HtDisconnect;
        ht.OnDoneInput := @HtDoneInput;
        ht.OnError := @HtError;
        ht.OnInput := @HtInput;
        ht.OnProcessHeaders := @HtProcessHeaders;
        ht.SendRequest;
        isDone := false;

        while not isDone do
            ht.CallAction;

        aBuff.Position := 0;
        memBody.Lines.LoadFromStream(aBuff);
    finally
        aBuff.Free;
        ht.Free;
    end;
end;
end.

----------------------------------------------------------------------
Silvio Clécio · 2011-08-02 11:05

Hi @jamhitz,

1. Download the LazSolutions via SVN from link:
http://lazsolutions.googlecode.com/svn

2. Open the demo ">lazsolutions>trunk>Demos>LSHTTPSend" and change the
code of GetHTMLButton to:
Code: [Select]procedure TMainForm.GetHTMLButtonClick(Sender: TObject);
 var
{$IFDEF UNIX}
  VStrTemp: TStringList;
{$ENDIF}
  VHTTPSend: THTTPSend;
begin
  try
    VHTTPSend := THTTPSend.Create;
    VHTTPSend.Sock.HTTPTunnelPort := '8443';
{$IFDEF UNIX}
    VStrTemp := TStringList.Create;
{$ENDIF}
    if not LSHTTPGetTextEx(VHTTPSend, HostEdit.Text,
{$IFDEF UNIX}
      VStrTemp
{$ENDIF}
{$IFDEF MSWINDOWS}
      HTMLMemo.Lines
{$ENDIF}) then
      ShowMessage(CUnableHTMLMsg);
{$IFDEF UNIX}
    HTMLMemo.Text := AnsiToUtf8(VStrTemp.Text);
{$ENDIF}
  finally
    VHTTPSend.Free;
{$IFDEF UNIX}
    VStrTemp.Free;
{$ENDIF}
  end;
end;
3. Copy "libeay32.dll" and "ssleay32.dll" to
">lazsolutions>trunk>Demos>LSHTTPSend" (download at: http://code.googl
e.com/p/lazsolutions/source/browse/#svn%2Fmedia%2Flib%2Fwin32).

4. Put the link "dspace.library.uvic.ca" in HostEdit.

5. Enjoy! ^^

----------------------------------------------------------------------
jamhitz · 2011-08-03 02:40

A bit of progress. I have tried LazSolutions. This time at least I get
a response. But the response is a redirect to the very same secure
server, not the actual content:

Code: [Select]<html>
<head>
<title>Redirect to UvicDspace</title>
<meta http-equiv="refresh" content="2;
URL=https://dspace.library.uvic.ca:8443/">
<meta name="keywords" content="automatic redirection">
</head>
<body>
If your browser doesn't automatically go there within a few seconds,
you may want to click
<a href="https://dspace.library.uvic.ca:8443/"> here UvicDspace</a>
</body>
</html>
I must be missing something, because I believe FPC/Lazarus should be
able to do this.  I am not prepared to give up... just not yet.

----------------------------------------------------------------------
Silvio Clécio · 2011-08-03 02:49

Hmm... I received this same page. I'll try to solve it. (tomorrow,
here in Brazil is already a little late - 23:47h ... ).

----------------------------------------------------------------------
jamhitz · 2011-08-03 15:38

I have managed to resolved the problem, with the assistance of the
nice people of the Synapse forum. Apparently something changed
regarding how OpenSSL version 1  supports auto-detection of the SSL
version (as opposed to the earlier versions).  As such, the version
needs to be explicitly provided:

In the synapse code add:

Code: [Select]ht.Sock.SSL.SSLType := LT_SSLv3;
In LNet code, add:

Code: [Select]ssl.Method := msSSLv3;

----------------------------------------------------------------------
BigChimp · 2011-08-04 07:20

Maybe a good idea to add this to the wiki (if not already done) and
mark the thread as SOLVED?

This will help other people in future from facing the same problems...

----------------------------------------------------------------------
jamhitz · 2011-08-04 11:22

Sure.