[SOLVED] LNET http client
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
fabienwang · 2011-04-29 10:53

Hello,

I have tested the httpclienttest example from lnet svn,
but i can't get the whole source from a webpage.
Example: when i take google.com page, it randomly stops before the end
of the webpage.
It should stop at </script>.

I tried with lazarus.freepascal.org and it seems to work great, i get
the source until </html>.

I tried to get different https pages and all i have is:

Response: 302 Found, data...
0...
Finished.
Disconnected.

Any help to make the example program working?

----------------------------------------------------------------------
Almindor · 2011-04-29 11:09

Please specify lNet/Lazarus/FPC version and specific URLs as you put
them in the URL bar.

There were issues with this, they are currently fixed in trunk.

----------------------------------------------------------------------
Dibo · 2011-04-29 11:20

LHTTPClient by default send only few headers (I remember that
wireshark sniffer show me only "host" and another when - I don't
remember which). Maybe try add some other headers like:
lhttpclient.AddExtraHeader('Connection: keep-alive');
lhttpclient.AddExtraHeader('Keep-alive: 300');

----------------------------------------------------------------------
fabienwang · 2011-04-29 11:23

lnet from svn revision 2577 (latest one)

Lazarus 0.9.30-0 r FPC 2.4.2 i386-linux-gtk 2

URLS that do work: (any HTTP)
URLS that do not work (any HTTPS)

for HTTPS pages, i have:

TLS handshake successful
Response: 200 OK, data...
0...
Finished.
Disconnected.

here is the function OnInput i changed:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TMainForm.HTTPClientInput(ASocket: TLHTTPClientSocket; ABuffer:
pchar;  ASize: dword): dword;var  oldLength: dword;begin  oldLength :=
Length(HTTPBuffer);  setlength(HTTPBuffer,oldLength + ASize); 
move(ABuffer^,HTTPBuffer[oldLength + 1], ASize); 
AppendToMemo(MemoHTML, HTTPBuffer); //changed line  //MemoHTML.Text :=
HTTPBuffer;  //MemoHTML.SelStart := Length(HTTPBuffer); 
AppendToMemo(MemoStatus, IntToStr(ASize) + '...');  Result := aSize;
// tell the http buffer we read it allend;
So it seems the HTTPBuffer is empty on HTTPS pages. Why?

----------------------------------------------------------------------
Almindor · 2011-04-30 08:50

It was a problem with the contents of some pages not being utf8
compatible. I added a UTF8Encode to the HTTP demo and it works now. Oh
and google's page really does miss </html> and other end tags

----------------------------------------------------------------------
fabienwang · 2011-04-30 08:59

Thanks Almindor.
As mentioned, we have to be careful with encoding! =)
I put [SOLVED] in the topic.