Indy Http GET problem
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
xinyiman · 2018-08-05 09:32

Hi guys, I'm creating a program with lazarus and indy to get a data
transfer through the http GET. In practice I need to make a site with
authentication via json.

Basically a function checks the validity of the credentials and
returns this string to me

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i { "token" :
"ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGT
U9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl
4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ
2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkR5d01yVmJyekM
4T2tPOW9SUzRuZ3hLTkVhNkdHR3JVdklnRk9wdVFuSGdyY3BOUG0xU25FTFFFOFdtdkg5d
FdGYTE1cnJSZUZ1Uy9GZ2UzREY1cz0=" } 
But when I use this code to recover the data

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TNGITController.GetHTML(url: string): string;var   idhttp1:TIdHTTP; 
 xpto:tmemorystream;   ret: string;begin     ret:='';   
 idhttp1:=TIdHTTP.Create;     xpto:=tmemorystream.Create;   
 idhttp1.ReadTimeout:=5000;     idhttp1.Get(url,xpto);   
 xpto.Position:=0;     SetString(ret, PAnsiChar(xpto.Memory),
xpto.Size);     xpto.Free;     idhttp1.Free;     result:=ret;end; 
I get this result

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i json login:
HTTP/1.1 200 OKConnection: closeContent-Length: 396Server: NGIT
Bootstrap ApplicationSet-Cookie: IDHTTPSESSIONID=5Ltho7JkF8YkQaw;
path=/ { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESG
J4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1R
T25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdj
c2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1l 
But it does not always happen, only occasionally. It is also fair to
tell you that the token is sent via the AResponseInfo object:
TIdHTTPResponseInfo

Like this: AResponseInfo.ContentText

Where am I doing wrong?

----------------------------------------------------------------------
xinyiman · 2018-08-05 10:25

I tried replacing the GetHTML function with some synapse code. And it
does not seem to give any more problems. I would not want there to be
any bugs in indy for reading HTML. I forgot the problem manifests
itself on Mac OSX

----------------------------------------------------------------------
Thaddy · 2018-08-05 12:12

I just wrote some examples here http://forum.lazarus.freepascal.org/in
dex.php/topic,42132.msg293506/topicseen.html#new
that do just what you want. Either synapse or fcl-web. 4 examples in
total.

----------------------------------------------------------------------
xinyiman · 2018-08-06 10:18

Thank you, but with synapse I can do it. It amazes me that indy,
however, has the problem described by me.

----------------------------------------------------------------------
Thaddy · 2018-08-06 11:15

Quote from: xinyiman on August 06, 2018, 12:18:50 pmThank you, but
with synapse I can do it. It amazes me that indy, however, has the
problem described by me.
If (after)I have quite a lot of ice cold beers I will probably add
Indy to the examples too. There isn't much difference and also at max.
a 4 liner to handle the basics.
Maybe Remy beats me to it?

----------------------------------------------------------------------
engkin · 2018-08-06 13:20

Quote from: xinyiman on August 05, 2018, 11:32:24 amCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i json login:
HTTP/1.1 200 OKIs that a space between "json" and "login"?

Edit:
If that is line is the "status-line" then the server is not responding
correctly. It should be something like:
QuoteHTTP/1.1 200 OK json login
What do I know!

@Thaddy, it is getting hot over there.

----------------------------------------------------------------------
Remy Lebeau · 2018-08-07 19:34

Quote from: xinyiman on August 05, 2018, 11:32:24 amBasically a
function checks the validity of the credentials and returns this
string to me

You are not sending any credentials, at least not with any HTTP
authentication anyway.  Are they in your URL string instead?  That is
not very secure, unless HTTPS is being used.

Also, why are you using the overloaded version of TIdHTTP.Get() that
fills a TStream, rather than using the version of TIdHTTP.Get() that
returns a String?

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TNGITController.GetHTML(url: string): string;var   idhttp1:
TIdHTTP;begin  Result := '';  idhttp1 := TIdHTTP.Create;  try   
idhttp1.ReadTimeout := 5000;    Result := idhttp1.Get(url);  finally 
  idhttp1.Free;  end;end; 
Quote from: xinyiman on August 05, 2018, 11:32:24 amBut when I use
this code to recover the data
...
I get this result

I don't understand what you are describing.  You are clearly getting a
success reply from the server with a JSON payload, so what is the
actual problem you are having?  Please be more specific.

Are you suggesting that the entire HTTP message, not just its body
data, is being stored in the TMemoryStream?  I guarantee that is not
how TIdHTTP.get() works.

Quote from: xinyiman on August 05, 2018, 11:32:24 amBut it does not
always happen, only occasionally.

WHAT doesn't always happen?

Quote from: xinyiman on August 05, 2018, 11:32:24 amIt is also fair to
tell you that the token is sent via the AResponseInfo object:
TIdHTTPResponseInfo

Like this: AResponseInfo.ContentText

How the server end is implemented is not relevant to this issue,
unless the server is not populating the AResponseInfo correctly.

Quote from: xinyiman on August 05, 2018, 11:32:24 amWhere am I doing
wrong?

We can't know, because you haven't described the problem clearly
enough.

----------------------------------------------------------------------
xinyiman · 2018-08-08 06:18

Quote from: Remy Lebeau on August 07, 2018, 09:34:19 pm
Also, why are you using the overloaded version of TIdHTTP.Get() that
fills a TStream, rather than using the version of TIdHTTP.Get() that
returns a String?


I tried both versions. In any case, the following problem occurs to
me.
The function should always return this string

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i     {
"token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUt
TUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuL
zdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk1
1RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkR5d
01yVmJyekM4T2tPOW9SUzRuZ3hLTkVhNkdHR3JVdklnRk9wdVFuSGdyY3BOUG0xU25FTFF
FOFdtdkg5dFdGYTE1cnJSZUZ1Uy9GZ2UzREY1cz0=" }
But arbitrarily every now and then (let's say once on 5/6) returns

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i     json
login: HTTP/1.1 200 OK    Connection: close    Content-Length: 396   
Server: NGIT Bootstrap Application    Set-Cookie:
IDHTTPSESSIONID=5Ltho7JkF8YkQaw; path=/         { "token" : "ZVRoQWN0a
FF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjh
na0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0b
FJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQTh
xaTBrLy9MQlBhU1l    
Last result all into string. This is problem. I think the problem is
into GET function because i use synapse GetHttp the result is always
correct

----------------------------------------------------------------------
Remy Lebeau · 2018-08-11 01:05

Quote from: xinyiman on August 08, 2018, 08:18:39 amThe function
should always return this string

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i     {
"token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUt
TUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuL
zdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk1
1RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkR5d
01yVmJyekM4T2tPOW9SUzRuZ3hLTkVhNkdHR3JVdklnRk9wdVFuSGdyY3BOUG0xU25FTFF
FOFdtdkg5dFdGYTE1cnJSZUZ1Uy9GZ2UzREY1cz0=" }
But arbitrarily every now and then (let's say once on 5/6) returns

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i     json
login: HTTP/1.1 200 OK    Connection: close    Content-Length: 396   
Server: NGIT Bootstrap Application    Set-Cookie:
IDHTTPSESSIONID=5Ltho7JkF8YkQaw; path=/         { "token" : "ZVRoQWN0a
FF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjh
na0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0b
FJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQTh
xaTBrLy9MQlBhU1l    
That is simply not possible under normal conditions.  Especially the
"json login: " portion, which is not even part of the HTTP protocol. 
TIdHTTP DOES NOT store a response's headers in the output TStream,
only the response's body.  The server would have to be sending that
extra text in its response body, which would be a flaw on the server's
part, not in TIdHTTP itself.  Which might also make sense, given that
the base64 token in the erroneous response data doesn't match the
base64 token in the correct response data.

To verify this, use a packet sniffer like Wireshark, or attach one of
Indy's TIdLog... components (such as TIdLogFile) to the
TIdHTTP.Intercept property.  That will log the raw data that TIdHTTP
is receiving from the server.  Please post that data for review.

----------------------------------------------------------------------
xinyiman · 2018-10-24 07:57

Sorry for the delay, but at this link you find an example. If you
compile it to the current state use both synapse and indy. Synapse is
used only in unit_controller.pas in the GetHTML function.

http://forum.lazarus-
ide.org/index.php/topic,42973.msg300168/topicseen.html#new

To replicate the problem, just change the function from this

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TNGITController.GetHTML(url: string): string;var   idhttp1:TIdHTTP; 
 xpto:tmemorystream;   ret: string;begin     {ret:='';   
 idhttp1:=TIdHTTP.Create;     xpto:=tmemorystream.Create;   
 idhttp1.ReadTimeout:=5000;     idhttp1.Get(url,xpto);   
 xpto.Position:=0;     SetString(ret, PAnsiChar(xpto.Memory),
xpto.Size);     xpto.Free;     idhttp1.Free;     result:=ret;  }    
 //in questo punto ho optato su synapse perchè il get di indy sembra
non funzionare bene     //in alcuni casi toppa e non ho capito perchè,
lo fa in maniera del tutto arbitraria      ret:=RecuperaHtml(url);   
 result:=ret;end; 
to this
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TNGITController.GetHTML(url: string): string;var   idhttp1:TIdHTTP; 
 xpto:tmemorystream;   ret: string;begin     ret:='';   
 idhttp1:=TIdHTTP.Create;     xpto:=tmemorystream.Create;   
 idhttp1.ReadTimeout:=5000;     idhttp1.Get(url,xpto);   
 xpto.Position:=0;     SetString(ret, PAnsiChar(xpto.Memory),
xpto.Size);     xpto.Free;     idhttp1.Free;     result:=ret;end;