[SOLVED] Indy HTTP - Persistent Connections & Data Compression
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
JD · 2016-12-17 11:01

Hi there everyone,

I have an Indy 10 based REST server that provides data to clients
(browsers & desktop clients). The server has its KeepAlive property
set to true so as to persist connections.

When a desktop client talks to the server, it creates a new thread and
a new instance of IdHTTP as in the example below:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TRestThread.Get(AURL: string): string;var  AString: string;begin  // 
Result := EmptyStr;  AString := EmptyStr;  //  IdHTTP :=
TIdHTTP.Create(nil);  IdHTTP.Request.UserAgent        := 'MyApp'; 
IdHTTP.Request.Connection       := 'keep-alive'; 
IdHTTP.Request.ContentType      := 'application/json'; 
IdHTTP.Request.ContentEncoding  := 'UTF-8';  IdHTTP.Request.Accept   
      := 'application/json';  IdHttp.Request.AcceptLanguage   := 'fr,
fr-FR';  IdHttp.HTTPOptions              := [hoForceEncodeParams]; 
IdHttp.Request.AcceptEncoding   := 'gzip, deflate';   //  try    //
did the server send a gzip response?    if Pos('gzip',
IdHttp.Request.AcceptEncoding) > 0 then    begin      AString :=
IdHttp.Get(AURL, IndyTextEncoding_UTF8);      Result :=
UTF8Decode(ZDecompressStr(AString));    end    else      Result :=
UTF8Decode(IdHttp.Get(AURL, IndyTextEncoding_UTF8));  finally   
IdHTTP.Free;  end;end; 
Sometimes I have to call this GET procedure as much as 20 times to
fill many screen controls. As you can see, IdHTTP is created/freed in
the GET function.

I must confess I opted for this create-free method because of users
with mobile devices who can move from one WiFi network to another just
by walking around. It ensures that I don't have a fixed connection
tunnel like TCP/IP which fails when the user's WiFi access changes.
IdTCPClient/IdTCPServer combo will require reconnection when user's
WiFi changes or even when the server has to be restarted. The downside
of this IdHTTP / REST Server method is that it is slower than the
IdTCPClient/IdTCPServer combo.

My question is this, does this create-free method for IdHTTP make
persistent connections useless?

Thanks,

JD

----------------------------------------------------------------------
Remy Lebeau · 2016-12-20 00:46

Quote from: JD on December 17, 2016, 12:01:11 pmI have an Indy 10
based REST server that provides data to clients (browsers & desktop
clients). The server has its KeepAlive property set to true so as to
persist connections.

That does you no good in this situation since the code you showed is
not taking advantage of persistent connections.

Quote from: JD on December 17, 2016, 12:01:11 pmWhen a desktop client
talks to the server, it creates a new thread and a new instance of
IdHTTP as in the example below:

Setting the Request.Connection property to 'keep-alive' is wasted in
your example since your HTTP object is being freed, and thus
disconnecting, when the request is finished.

Also, UTF-8 is not a valid ContentEncoding.

Also, since you clearly want to support compressed responses, you
should assign a TIdZLibCompressorBase-derived component, like
TIdCompressorZLib, to the TIdHTTP.Compressor property, and let Indy
decompress the data for you.  However, if you are going to do it
manually, you should be looking at the Response.ContentEncoding
property to detect a compressed response, not the
Request.AcceptEncoding property.  And worse, you are indicating that
you support both 'gzip' and 'deflate' compression, but you are not
actually checking for 'deflate'-encoded responses, only 'gzip'.  They
are two separate compressions!  All the more reason not to assign
anything to Request.AcceptEncoding at all.  TIdHTTP handles that for
you when a Compressor is assigned.

But, in any case, definitely DO NOT process compressed data as a
String!  It is binary data, not textual data.  Charset handling will
corrupt the compressed bytes.  Use the overloaded version of
TIdHTTP.Get() that fills an output TStream instead, and then
decompress that if compressed.

Quote from: JD on December 17, 2016, 12:01:11 pmSometimes I have to
call this GET procedure as much as 20 times to fill many screen
controls. As you can see, IdHTTP is created/freed in the GET function.

I must confess I opted for this create-free method because of users
with mobile devices who can move from one WiFi network to another just
by walking around. It ensures that I don't have a fixed connection
tunnel like TCP/IP which fails when the user's WiFi access changes.

You do realize that HTTP runs on top of TCP/IP, don't you?  And your
code would still fail if the user happens to change WiFi in the
*middle* of an HTTP retrieval.

Quote from: JD on December 17, 2016, 12:01:11
pmIdTCPClient/IdTCPServer combo will require reconnection when user's
WiFi changes or even when the server has to be restarted.

So does TIdHTTP/TIdHTTPServer, since HTTP runs on top of TCP/IP.

Quote from: JD on December 17, 2016, 12:01:11 pmThe downside of this
IdHTTP / REST Server method is that it is slower than the
IdTCPClient/IdTCPServer combo.

Not really, since TIdHTTP derives from TIdTCPClientCustom, and
TIdHTTPServer derives from TIdCustomTCPServer, so you still have all
of the same TCP/IP semantics.

Quote from: JD on December 17, 2016, 12:01:11 pmMy question is this,
does this create-free method for IdHTTP make persistent connections
useless?

Yes.  To really take advantage of persistent connections, you have to
keep the TIdHTTP object alive and reuse it for subsequent requests. 
For instance, you could have a pool of TIdHTTP objects, and have your
REST thread pull an available object from the pool, use it as needed,
and then put it back in the pool when finished.

----------------------------------------------------------------------
JD · 2016-12-21 10:02

Thanks for your reply, Remy.

Quote from: Remy Lebeau on December 20, 2016, 01:46:04 amThat does you
no good in this situation since the code you showed is not taking
advantage of persistent connections.

Setting the Request.Connection property to 'keep-alive' is wasted in
your example since your HTTP object is being freed, and thus
disconnecting, when the request is finished.

That's what I thought was going on. Thanks for confirming it.

Quote from: Remy Lebeau on December 20, 2016, 01:46:04 amAlso, since
you clearly want to support compressed responses, you should assign a
TIdZLibCompressorBase-derived component, like TIdCompressorZLib, to
the TIdHTTP.Compressor property, and let Indy decompress the data for
you.  However, if you are going to do it manually, you should be
looking at the Response.ContentEncoding property to detect a
compressed response, not the Request.AcceptEncoding property.  And
worse, you are indicating that you support both 'gzip' and 'deflate'
compression, but you are not actually checking for 'deflate'-encoded
responses, only 'gzip'.  They are two separate compressions!  All the
more reason not to assign anything to Request.AcceptEncoding at all. 
TIdHTTP handles that for you when a Compressor is assigned.

But, in any case, definitely DO NOT process compressed data as a
String!  It is binary data, not textual data.  Charset handling will
corrupt the compressed bytes.  Use the overloaded version of
TIdHTTP.Get() that fills an output TStream instead, and then
decompress that if compressed.

OK. But I am not clear about how IdHTTP compression and decompression
works automagically with TIdCompressorZLib. On the server, I can use
the methods in IdZlib to handle compression/decompression. When I
tried to use IdHTTP with TIdCompressorZLib, I kept getting errors
concerning inflate (don't remember the exact term used).

For instance at the moment on the server side - CommandGet and
CommandOther, here is how I handle compression of strings (before the
transition to streams you suggested).

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   // SERVER
SIDE CommandGet and CommandOther code snippet   // Should serve the
entire procedure so move compression stuff here  if strJSON <>
EmptyStr then  begin    //    AResponseInfo.ResponseNo := 200;   
AResponseInfo.ContentType := 'application/json';   
AResponseInfo.CharSet := 'utf-8';     // Does the client request
demand a compressed response?    //     if Pos('gzip',
ARequestInfo.AcceptEncoding) > 0 then    begin     
AResponseInfo.ContentEncoding := 'gzip';      //      if Pos('MyApp',
ARequestInfo.UserAgent) > 0 then        strCompressedJSON :=
RUtils.ZCompressStr(strJSON, Zstream.TCompressionlevel.clMax)     
else  // browsers will be able to handle gzip compression       
strCompressedJSON := IdZlib.CompressString(strJSON,
IdZlib.TCompressionLevel.clMax, zsGZip);     
AResponseInfo.ContentStream :=
TIdMemoryBufferStream.Create(Pointer(strCompressedJSON),
Length(strCompressedJSON));    end    else     
AResponseInfo.ContentStream :=
TIdMemoryBufferStream.Create(Pointer(strJSON), Length(strJSON));  end 
else  begin    AResponseInfo.ResponseNo  := 404;            
AResponseInfo.ContentType := 'text/html';    AResponseInfo.CharSet   
:= 'utf-8';    AResponseInfo.ContentText := strNotFound;  end;    //
if strJSON <> EmptyStr then   // Send the string to the client  (here
it works with string compression method).  // One Write for all
decision branches  AResponseInfo.WriteContent; 
Can you please show me how I can change this to handle stream
compression using Indy only on the server as well as the client?

Quote from: Remy Lebeau on December 20, 2016, 01:46:04 amNot really,
since TIdHTTP derives from TIdTCPClientCustom, and TIdHTTPServer
derives from TIdCustomTCPServer, so you still have all of the same
TCP/IP semantics.

I know HTTP runs on top of TCP/IP. However I discovered that I did not
have to explicitly Connect to the server like I do in TCP/IP before
using GET/POST/PUT/DELETE. Maybe an implicit Connect is done, I did
not try to find out.

Quote from: Remy Lebeau on December 20, 2016, 01:46:04 amYes.  To
really take advantage of persistent connections, you have to keep the
TIdHTTP object alive and reuse it for subsequent requests.  For
instance, you could have a pool of TIdHTTP objects, and have your REST
thread pull an available object from the pool, use it as needed, and
then put it back in the pool when finished.

I have one for an older application using TCP/IP. In this HTTP
connection pool, does that mean I will explicitly have to call
Connect/Disconnect when the pool is created/destroyed like for the
TCP/IP pool? Sounds obvious I know but it helps to be clear. 

JD

----------------------------------------------------------------------
rvk · 2016-12-21 11:49

Quote from: JD on December 17, 2016, 12:01:11 pmSometimes I have to
call this GET procedure as much as 20 times to fill many screen
controls.
I hope you don't call this procedure 20 times for just one screen.
Because if you do it might be useful to redesign it and post ONE
complete json with all the information you want and get ONE json back
with that information. In that case compression will also be able to
do it's work because compressing a few bytes isn't that optimal.
Compression is more useful the bigger the information gets.

----------------------------------------------------------------------
JD · 2016-12-21 12:09

Quote from: rvk on December 21, 2016, 12:49:26 pmQuote from: JD on
December 17, 2016, 12:01:11 pmSometimes I have to call this GET
procedure as much as 20 times to fill many screen controls.
I hope you don't call this procedure 20 times for just one screen.
Because if you do it might be useful to redesign it and post ONE
complete json with all the information you want and get ONE json back
with that information. In that case compression will also be able to
do it's work because compressing a few bytes isn't that optimal.
Compression is more useful the bigger the information gets.

You read my mind   I'm actually working on that right now. That's why
I needed the compression. The 20 calls model was a prototype.

JD

----------------------------------------------------------------------
Remy Lebeau · 2016-12-21 23:52

Quote from: JD on December 21, 2016, 11:02:32 amOK. But I am not clear
about how IdHTTP compression and decompression works automagically
with TIdCompressorZLib.

What is unclear?  If you assign a Compressor to TIdHTTP, and then
TIdHTTP detects a compressed response, it will automatically
decompress the response for you.  The data returned to you (String or
TStream) will be the uncompressed data.

Quote from: JD on December 21, 2016, 11:02:32 amOn the server, I can
use the methods in IdZlib to handle compression/decompression.

Yes (if the client requests compression).

Quote from: JD on December 21, 2016, 11:02:32 amWhen I tried to use
IdHTTP with TIdCompressorZLib, I kept getting errors concerning
inflate (don't remember the exact term used).

Then you are likely not using it correctly.  But you have not shown
that code or the error, so I can't help you troubleshoot it.

Quote from: JD on December 21, 2016, 11:02:32 amFor instance at the
moment on the server side - CommandGet and CommandOther, here is how I
handle compression of strings (before the transition to streams you
suggested).

My suggestion to use a stream only applies to the client side.  And
even then, that doesn't change the data itself, only how TIdHTTP
delivers the data to you.  But, if you assign a Compressor to TIdHTTP,
then you don't even need to change your client code, as the String-
returning version of Get() will simply decompress the data internally
before it converts the decompressed data to a String for output.

Why are you using two different ZLib compressors on the server side,
though?  GZip is GZip regardless of the library used to implement it. 
There is no reason to use different compressors.

Quote from: JD on December 21, 2016, 11:02:32 amCan you please show me
how I can change this to handle stream compression using Indy only on
the server as well as the client?

The server code you showed is fine, you don't really need to change
it.  Though I would suggest you decide on one ZLib library or another
and stick with it.

Quote from: JD on December 21, 2016, 11:02:32 amI know HTTP runs on
top of TCP/IP. However I discovered that I did not have to explicitly
Connect to the server like I do in TCP/IP before using
GET/POST/PUT/DELETE. Maybe an implicit Connect is done, I did not try
to find out.

Yes, TIdHTTP calls Connect() internally for you.

Quote from: JD on December 21, 2016, 11:02:32 amI have one for an
older application using TCP/IP. In this HTTP connection pool, does
that mean I will explicitly have to call Connect/Disconnect when the
pool is created/destroyed like for the TCP/IP pool?

No.

----------------------------------------------------------------------
JD · 2016-12-22 11:31

Quote from: Remy Lebeau on December 22, 2016, 12:52:57 amWhat is
unclear?  If you assign a Compressor to TIdHTTP, and then TIdHTTP
detects a compressed response, it will automatically decompress the
response for you.  The data returned to you (String or TStream) will
be the uncompressed data.

OK. Here is what I did.

The server side CommandGet code using stream compression is as
follows. This works even in my browser 

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i     // Does
the client request demand a compressed response?    if Pos('gzip',
ARequestInfo.AcceptEncoding) > 0 then    begin     
AResponseInfo.ContentEncoding := 'gzip';      // Copy strJSON into a
stream and then compress the stream      // into
AResponseInfo.ContentStream in preparation for sending to      //
clients      AResponseInfo.ContentStream := TMemoryStream.Create;    
JSONStream := TStringStream.Create(strJSON);      try       
JSONStream.Position := 0;        IdZlib.CompressStreamEx(JSONStream,
AResponseInfo.ContentStream, IdZlib.TCompressionLevel.clMax, zsGzip); 
      AResponseInfo.ContentStream.Position := 0;      finally       
JSONStream.Free;      end;    end    else     
AResponseInfo.ContentStream :=
TIdMemoryBufferStream.Create(Pointer(strJSON), Length(strJSON)); 
The client side code is as follows:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TForm1.btnTestGzipClick(Sender: TObject);  function Get(AURL: string):
string;  var    AString: string;    ResultStream: TStringStream;   
var IdHttp: TIdHTTP;  begin    //    Result := EmptyStr;    AString :=
EmptyStr;    //    IdHTTP                          :=
TIdHTTP.Create(nil);    IdHTTP.Compressor               :=
TIdCompressorZlib.Create(IdHTTP);    IdHTTP.Request.ContentType    
 := 'application/json';    IdHTTP.Request.Accept           :=
'application/json';    IdHttp.Request.AcceptLanguage   := 'fr, fr-
FR';    IdHttp.HTTPOptions              := [hoForceEncodeParams];   
IdHttp.Request.AcceptEncoding   := 'gzip';     //    try      // PART
1      Result := IdHttp.Get(AURL, IndyTextEncoding_UTF8);           
//// PART 2       //// Get the data from the server      //
IdHttp.Get(AURL, IndyTextEncoding_UTF8);      //// did the server send
a gzip response?      //if Pos('gzip',
IdHttp.Response.ContentEncoding) > 0 then      //begin      //
 ResultStream := TStringStream.Create('');      //
 IdHttp.Response.ContentStream := TMemoryStream.Create;      //  //  
  //  try      //  
 IdHttp.Compressor.DecompressGZipStream(IdHttp.Response.ContentStream,
ResultStream);      //    ResultStream.Position := 0;      //  
 Result := UTF8Decode(ResultStream.DataString);      //  finally     
//    ResultStream.Free;      //  end;      //end      //else      //
 Result := UTF8Decode(AString);    finally      IdHTTP.Compressor :=
nil;      IdHTTP.Compressor.Free;      IdHTTP.Free;    end; 
end; begin  // 
ShowMessage(Get('http://localhost:80/countries'));end; 
You said HTTP.Compressor handle decompression (string/stream)
internally. The results of PART 1 and of PART 2 (the commented out
portion) are shown in the attached screenshots.

Where did I go wrong on the client side? The browser rendering of the
same URL is perfect so the problem can only be from the way I'm using
HTTP.Compressor.

Thanks a lot,

JD

----------------------------------------------------------------------
JD · 2016-12-22 11:53

Very, very strange. I had a hunch so I fired up Delphi XE4 to try the
same thing.

And lo and behold, it worked!    . All I had to do was to change

Result := IdHttp.Get(AURL, IndyTextEncoding_UTF8);

to

Result := IdHttp.Get(AURL);

It fails in Lazarus 1.6.2 BUT works in Delphi XE4. Reminds me of my
last text encoding problem before GetMem had the idea to try it in his
Delphi XE4.

The HTTP server was built using Lazarus 1.6.2.

What do I do now?

JD

----------------------------------------------------------------------
Remy Lebeau · 2016-12-22 21:01

Quote from: JD on December 22, 2016, 12:31:52 pmThe server side
CommandGet code using stream compression is as follows. This works
even in my browser 

Looks fine on the server side, though I would use ContentText instead
of ContentStream for the non-compresssed situation:

Code: [Select]AResponseInfo.ContentText := strJSON;

Quote from: JD on December 22, 2016, 12:31:52 pmThe client side code
is as follows:

I see three issues:

1. double-check to make sure that the TIdCompressorZLib.IsReady
property is returning true.  If it is not, then TIdCompressorZLib is
not loading the ZLib library correctly.  I did have to check in a bug
fix earlier this year regarding this.

2. Don't set the TIdHTTP.Request.AcceptEncoding at all.  I told you
that earlier.  Let TIdHTTP handle that assignment internally when a
Compressor is assigned, based on whether the Compressor is actually
ready for use or not.  By assigning the property manually, you are
potentially telling the server that you will accept a compressed
response even if the Compressor is not able to actually decompress it.

3. if all goes well with the Compressor then the data returned by
Get() will always be uncompressed:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
Get(AURL: string): string;var  IdHttp: TIdHTTP;begin  Result :=
EmptyStr;  IdHTTP := TIdHTTP.Create(nil);  try    IdHTTP.Compressor :=
TIdCompressorZlib.Create(IdHTTP);    IdHTTP.Request.Accept :=
'application/json';    IdHttp.Request.AcceptLanguage := 'fr, fr-FR'; 
 Result := IdHttp.Get(AURL, IndyTextEncoding_UTF8);       finally   
IdHttp.Free;  end;end; 
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
Get(AURL: string): string;var  ResultStream: TMemoryStream;  IdHttp:
TIdHTTP;begin  Result := EmptyStr;  ResultStream :=
TMemoryStream.Create;  try    IdHTTP := TIdHTTP.Create(nil);    try  
  IdHTTP.Compressor := TIdCompressorZlib.Create(IdHTTP);     
IdHTTP.Request.Accept := 'application/json';     
IdHttp.Request.AcceptLanguage := 'fr, fr-FR';      IdHttp.Get(AURL,
ResultStream);    finally      IdHTTP.Free;    end;   
ResultStream.Position := 0;    Result :=
ReadStringFromStream(ResultStream, -1, IndyTextEncoding_UTF8); 
finally    ResultStream.Free;  end;end; 
Quote from: JD on December 22, 2016, 12:31:52 pmYou said
HTTP.Compressor handle decompression (string/stream) internally.

Yes, when things go correctly.

----------------------------------------------------------------------
JD · 2016-12-23 06:49

It now works. Removing the line IdHttp.Request.AcceptEncoding   :=
'gzip'; made the difference.

However, for the second function using streams, it was necessary to
change
Result := ReadStringFromStream(ResultStream, -1,
IndyTextEncoding_UTF8);
to
Result := ReadStringFromStream(ResultStream, -1);
so that the strings could be properly rendered.

As usual, we start on one subject and end up examining other areas as
well . I have altered the title of this thread to better describe what
we discussed.

Thanks a lot for your valuable input and code review.

Merry Christmas to you,

JD

EDIT: at 15h23 CET

Sorry but it seems I was too hasty in my conclusions. Refactoring the
server side code and removing the following line from the server's
CommandGet procedure

if Pos('gzip', ARequestInfo.AcceptEncoding) > 0

made me realise that that was why it had worked like I described
earlier today. It was actually sending uncompressed data. Now I only
want it to return compressed streams and my Lazarus client application
fails to decompress the streams.

I updated my Indy intallation from indy.fulgan.com and tried it again.
It failed!
I then discovered that IdHTTP.Compressor.IsReady returns FALSE but
only in Lazarus. It returns TRUE in Delphi.

I apologize to all who have read the previous original post for this
oversight. I was in a hurry to start the weekend.

JD

----------------------------------------------------------------------
JD · 2016-12-23 14:35

So my search for a solution continues.

----------------------------------------------------------------------
rvk · 2016-12-23 17:28

If  IdHTTP.Compressor.IsReady returns FALSE there doesn't seem to be
any decompressor loaded. .IsReady checks for IdZLibHeaders.Loaded. I'm
not sure how it's done in Indy and Delphi or Lazarus but the
(de-)compression is done with idzlibheaders.pas.

If I look in there I see
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i {TODO:
 Wait for Emb to decide how to approach ZLib for their 64-bit
supportbefore we proceed at our end.}{$UNDEF STATICLOAD_ZLIB}This is
in Indy 10.6.2.0 (I used the one in onlinepackagemanager)

So I think the .obj may not be included in the compilation. It could
be that zlib1.dll is needed for decompression in that case.

What Indy version do you have on Delphi where .isReady is true?
If it's also 10.6.2.0 could you check line 22-26 if it also has these
comments mentioned above and the UNDEF for STATICLOAD_ZLIB?

Edit: Yep. I just confirmed that adding a zlib1.dll in your project-
directory will result in IdHTTP.Compressor.IsReady being TRUE.
The latest version on indy.fulgan.com (Indy10_5388) still has that
comment.

----------------------------------------------------------------------
JD · 2016-12-23 18:08

Quote from: rvk on December 23, 2016, 06:28:43 pmWhat Indy version do
you have on Delphi where .isReady is true?
If it's also 10.6.2.0 could you check line 22-26 if it also has these
comments mentioned above and the UNDEF for STATICLOAD_ZLIB?
The default Indy version in Delphi XE4. It has to be in the 10.6
series.

Quote from: rvk on December 23, 2016, 06:28:43 pmEdit: Yep. I just
confirmed that adding a zlib1.dll in your project-directory will
result in IdHTTP.Compressor.IsReady being TRUE.
The latest version on indy.fulgan.com (Indy10_5388) still has that
comment.

It worked!!!! Thanks a million.    FYI I'm using Indy10_5388 too.

Merry Christmas to you rvk.

JD