[SOLVED] Indy10 HTTPServer issues
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
JD · 2016-05-27 10:16

Hi there everyone,

Having previously worked exclusively with TCP/IP. I made a first
attempt at creating Indy10 HTTPServer which serves json & html files
to clients - browser & Indy10 client applications.

The IdHTTPServer parameters are shown below;
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TfrmServeur.FormCreate(Sender: TObject);begin  //  HTTPServer :=
TIdHTTPServer.Create(nil);  //  with HTTPServer do  begin   
OnCommandGet      := @HTTPServerCommandGet;    OnCommandOther    :=
@HTTPServerCommandOther;    OnConnect         := @HTTPServerConnect; 
 OnCreateSession   := @HTTPServerCreateSession;    OnDisconnect    
 := @HTTPServerDisconnect;    OnSessionEnd      :=
@HTTPServerSessionEnd;    OnSessionStart    :=
@HTTPServerSessionStart;  end;   // set the context class of the
server to our custom TClientContext  // the server will create
automatically our class instance  // when a client is connected and
free it when it disconnects  HTTPServer.ContextClass :=
TClientContext;end; 
The IdHTTPServer is later activated by the code below:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  
HTTPServer.Bindings.Clear;  //  for i := 0 to lbIPs.Items.Count - 1
do  if lbIPs.Checked[i] then  begin    Binding :=
HTTPServer.Bindings.Add;    Binding.IP := lbIPs.Items.Strings[i];   
Binding.Port := 8080;  //StrToInt(edPort.Text);  end;  // Activate the
server  HTTPServer.Active := True; 
The server-side handles GET commands with the method below:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TfrmServeur.HTTPServerCommandGet(AContext: TIdContext;  ARequestInfo:
TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);var 
ServerMethod: TServerMethods;  strUser, strPassword,  strLoginProfile:
string;  strJSON: string;begin  //  case Trim(ARequestInfo.URI) of   
// ARequestInfo.Params     // http://localhost:8080/departement   
'/pays', '/pays/'    :      begin        // ServerMethod       
ServerMethod := TServerMethods.Create;        //        try         
// send JSON list of countries from SQL query          strJSON :=
ServerMethod.GetResource(ARequestInfo.URI);          //          if
strJSON <> EmptyStr then          begin           
AResponseInfo.ResponseNo := 200;            AResponseInfo.ContentType
:= 'application/json';            AResponseInfo.CharSet := 'utf-8';  
        AResponseInfo.ContentText := strJSON;           
AResponseInfo.ContentLength := 99999;  <------ NO, NO, NO!!         
end    // if strJSON <> EmptyStr then          else          begin   
       //            AResponseInfo.ResponseNo := 200;                
   AResponseInfo.ContentType := 'text/html';           
AResponseInfo.CharSet := 'utf-8';            AResponseInfo.ContentText
:= strNotFound;                end;        finally         
ServerMethod.Free;        end;      end;    else      begin        //
ServerMethod        ServerMethod := TServerMethods.Create;        // 
     try          //          AResponseInfo.ResponseNo := 404;       
 AResponseInfo.ContentType := 'text/html';         
AResponseInfo.CharSet := 'utf-8';          AResponseInfo.ContentText
:= strNotFound;       // Not Found        finally         
ServerMethod.Free;        end;      end;  end;      // case
Trim(ARequestInfo.URI) ofend;  
The client-side IdHTTP create method is shown below:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
//constructor THttpConnectionIndy.Create;begin  FIdHttp :=
TIdHTTP.Create(nil);  FIdHttp.HandleRedirects := True; 
FIdHTTP.HTTPOptions := [hoForceEncodeParams]; 
FIdHTTP.Connect('localhost', 8080);  end; 
I have the following problems client-side
a) Browsers
    i) the text is not rendered correctly (see screenshot)
   ii) the entire JSON string is not sent unless
AResponseInfo.ContentLength := 99999. I chose a large number to be
able to send long strings

b) Indy client applications
    i) the entire JSON string is not sent unless
AResponseInfo.ContentLength := 99999. I chose a large number to be
able to send long strings

In discussions with Remy Lebeau, he said, and it makes perfect sense,
that setting AResponseInfo.ContentLength to a predefined number will
FORCE clients to keep reading from the socket until those bytes
arrive, or a timeout/error occurs. Indy has a procedure called
TIdHTTPResponseInfo.WriteHeader that computes
AResponseInfo.ContentLength automatically.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TIdHTTPResponseInfo.WriteHeader;var  ...begin  ...   if (ContentLength
= -1) and    ((TransferEncoding = '') or TextIsSame(TransferEncoding,
'identity')) then {do not localize}  begin    // Always check
ContentText first    if ContentText <> '' then begin     
ContentLength := CharsetToEncoding(CharSet).GetByteCount(ContentText);
// <-- HERE    end    else if Assigned(ContentStream) then begin     
ContentLength := ContentStream.Size;    end else begin      // TODO:
should this be calculating the size of the HTML message      // that
WriteContent() generates in this case?      ContentLength := 0;   
end;  end;   ...end; 
Remy assures me that it MUST be called before responses are sent but I
still don't understand why the JSON responses are not being rendered
properly in the browser & not being sent completely in browsers and
indy clients.

Thanks for your assistance.

JD

----------------------------------------------------------------------
Remy Lebeau · 2016-05-27 17:59

Quote from: JD on May 27, 2016, 12:16:31 pmCode:
[Select]AResponseInfo.ContentLength := 99999;  <------ NO, NO, NO!!

Absolutely DO NOT do that.  The ContentLength must be set to the
correct number of bytes, not characters, that will be sent:

Code: [Select]AResponseInfo.ContentLength :=
IndyTextEncoding_UTF8.GetByteCount(strJSON);

Or set to -1 (the default) to let TIdHTTPServer calculate the value
for you:

Code: [Select]AResponseInfo.ContentLength := -1;

Quote from: JD on May 27, 2016, 12:16:31 pmCode:
[Select]AResponseInfo.ResponseNo := 200;        
AResponseInfo.ContentType := 'text/html';
AResponseInfo.CharSet := 'utf-8';
AResponseInfo.ContentText := strNotFound;     

Why are returning a success response code (200) if the requested JSON
cannot be retrieved?  I would suggest either 404 or even 5xx instead.

Quote from: JD on May 27, 2016, 12:16:31 pmCode:
[Select]FIdHTTP.Connect('localhost', 8080); 

Do not call Connect() directly on a TIdHTTP.  Its various request
methods, like Get() and Post(), will call Connect() internally for you
when needed.  The correct way to use TIdHTTP in this situation is to
use the TIdHTTP.Get() method:

Code: [Select]strJSON := FIdHTTP.Get('http://localhost:8080/pays');

Which is also all the more reason NOT to have the server returns a 2xx
responsse code if the JSON is not available.  It should return an
error response code so Get() can raise an exception accordingly.

Quote from: JD on May 27, 2016, 12:16:31 pmI have the following
problems client-side
a) Browsers
    i) the text is not rendered correctly (see screenshot)

The screenshot actually shows the JSON text is encoded as UTF-8.  The
browser is simply interpreting the UTF-8 bytes as ISO-8859-1/Latin-1
when displaying the bytes.  For instance, the Unicode character 'é' is
encoded in UTF-8 as bytes $C3 $A9, which are the characters 'é' in
ISO-8859-1.

That is a browser issue, not a TIdHTTPServer issue.  TIdHTTPServer
should be sending a 'Content-Type: application/json; charset=utf-8'
response header.  The browser is likely just ignoring the charset
attribute, since technically the official MIME registration for JSON
does not include a charset attribute.  Per JSON's MIME registration),
the default byte encoding for JSON text is UTF-8, but the browser is
likely ignoring that as well.  It is likely seeing the media type as a
generic 'application/...' and displaying the raw bytes as-is, rather
than seeing the media type is specifically 'application/json' and
displaying it as UTF-8.

If you include a 'Content-Disposition' response header, you can force
a browser to save the JSON to a file instead of display it as text. 
Then you can verify the encoding of the file using any hex/text viewer
that supports UTF-8.

Code: [Select]AResponseInfo.ContentDisposition := 'attachment;
filename=pays.json';

Quote from: JD on May 27, 2016, 12:16:31 pm   ii) the entire JSON
string is not sent unless AResponseInfo.ContentLength := 99999. I
chose a large number to be able to send long strings

As we have discussed on the Embarcadero forum, I guarantee you that is
NOT how TIdHTTPServer operates.  Something else is going on to mess
with the transmission.  TIdHTTPServer does not utilize the
ContentLength property when sending the ContentText.  Setting the
ContentLength property in that case merely provides a value for the
'Content-Length' response header, nothing more.  See the source code
in the TIdHTTPResponseInfo.WriteHeader()  and
TIdHTTPResponseInfo.WriteContent() and methods in
IdCustomHTTPServer.pas.

Quote from: JD on May 27, 2016, 12:16:31 pmIn discussions with Remy
Lebeau, he said, and it makes perfect sense, that setting
AResponseInfo.ContentLength to a predefined number will FORCE clients
to keep reading from the socket until those bytes arrive, or a
timeout/error occurs.

That is true, per requirement of the HTTP spec (see RFC 2616 Section
4.4 Message Length).

Quote from: JD on May 27, 2016, 12:16:31 pmRemy assures me that it
MUST be called before responses are sent
Yes, and TIdHTTPResponseInfo.WriteContent() calls WriteHeader() if it
has not already been called:

Code: [Select]procedure TIdHTTPResponseInfo.WriteContent;
begin
  if not HeaderHasBeenWritten then begin
    WriteHeader;
  end;
  ...
end;

TIdCustomHTTPServer.DoExecute() automatically calls WriteHeader() and
WriteContent() as need, after relevant event handlers have been called
to prepare the response properties.

Quote from: JD on May 27, 2016, 12:16:31 pmbut I still don't
understand why the JSON responses are not being rendered properly in
the browser

Actually, it is, just not the way you are expecting.  Remember, the
'application/...' media type is for, well, application data, not text
data.  The browser is displaying a textual representation of something
that is technically binary data, not text data.  And there is no
official 'text/...' media type for JSON (see What is the correct JSON
content type?).

Quote from: JD on May 27, 2016, 12:16:31 pm& not being sent completely
in browsers and indy clients.

That, I do not know yet.  We have not uncovered the real culprit yet.

Use a packet sniffer, like Wireshark or Fiddler, to verify the
response is being sent correctly, and in the correct timing.  Maybe
something sitting in between the client and TIdHTTPServer is hindering
the transmission.

Use a debugger to step into the server code to see what it is really
doing while processing your request and make sure it is preparing the
response correctly.

----------------------------------------------------------------------
JD · 2016-05-28 18:30

Thanks for your reply Remy. I attach a test project to my reply along
with a screen capture of the results to show that ContentLength is not
being calculated automatically and the characters are not being
displayed normally. The debugger did not pass through the WriteHeader
procedure at all even after I had put breakpoints in the code!

This is exactly how my browser displays the data. I would be glad if
you or anybody else familiar with Indy on this forum could take a look
at it.

The test project is a composite of Indy HTTP Server & HTTPClient on
localhost:8080.

Thanks,

JD

----------------------------------------------------------------------
engkin · 2016-05-29 00:30

Add {$codepage UTF8} to unit1.

----------------------------------------------------------------------
JD · 2016-05-29 03:22

Quote from: engkin on May 29, 2016, 02:30:52 amAdd {$codepage UTF8} to
unit1.

Thanks for your reply engkin. I did as you suggested but it didn't
work! I got the same results as before.

Code: [Select]unit Unit1;

{$mode objfpc}{$H+}
{$codepage UTF8}

However I discovered that if I commented out AResponseInfo.CharSet :=
'utf-8' and computed content length using AResponseInfo.ContentLength
:= IndyTextEncoding_UTF8.GetByteCount(strJSON) as in the code below:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  if strJSON
<> EmptyStr then begin   AResponseInfo.ResponseNo := 200; 
 AResponseInfo.ContentType := 'application/json'; 
 //AResponseInfo.CharSet := 'utf-8';   AResponseInfo.ContentText :=
strJSON;   AResponseInfo.ContentLength :=
IndyTextEncoding_UTF8.GetByteCount(strJSON); end    // if strJSON <>
EmptyStr then 
then the client gets the entire JSON string! It is still not displayed
correctly though. 

JD

----------------------------------------------------------------------
balazsszekely · 2016-05-29 12:43

Hi JD,

I just converted your project to Delphi XE4, works fine without any
modification(see attachment 1). It must be another encoding
issue(windows + indy + lazarus), very similar to this one:
http://forum.lazarus.freepascal.org/index.php/topic,32694.0.html
Your project also works with lazarus, if you convert the string to
stream and send it as a stream(attachment 2). 

regards,
GetMem

----------------------------------------------------------------------
JD · 2016-05-29 13:52

Quote from: GetMem on May 29, 2016, 02:43:18 pmHi JD,

I just converted your project to Delphi XE4, works fine without any
modification(see attachment 1). It must be another encoding
issue(windows + indy + lazarus), very similar to this one:
http://forum.lazarus.freepascal.org/index.php/topic,32694.0.html
Your project also works with lazarus, if you convert the string to
stream and send it as a stream(attachment 2). 

regards,
GetMem

Thanks for your reply and your findings. How do I send the response as
a stream on the Server side? In addition, how do I make a browser read
the stream?

JD

----------------------------------------------------------------------
balazsszekely · 2016-05-29 14:31

Please try this:
1. Server:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TForm1.HTTPServerCommandGet(AContext: TIdContext;  ARequestInfo:
TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);var  strJSON:
string;begin   strJSON := String(PAYS);   if strJSON <> EmptyStr then 
begin        AResponseInfo.ContentType := 'application/json';   
AResponseInfo.CharSet := 'utf-8';     AResponseInfo.ContentStream :=
TMemoryStream.Create;    try     
AResponseInfo.ContentStream.Write(Pointer(strJSON)^,
Length(strJSON));      AResponseInfo.ContentStream.Position := 0;    
AResponseInfo.WriteContent;    finally     
AResponseInfo.ContentStream := nil;    end;  end     else  begin   
AResponseInfo.ResponseNo := 404;    AResponseInfo.ContentType :=
'text/html';    AResponseInfo.CharSet := 'utf-8';   
AResponseInfo.ContentText := 'Resource not found';  end;end;
2. Client:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TForm1.btnGetDataClick(Sender: TObject);var  Str: String;  MS:
TMemoryStream;begin  if Trim(memServer.Text) = EmptyStr then    exit 
else  begin    HTTPClient := TIdHTTP.Create(nil);    try      ms :=
TMemoryStream.Create;      try        HTTPClient.HandleRedirects :=
True;        HTTPClient.HTTPOptions := [hoForceEncodeParams];       
HTTPClient.Get(Trim(edtDomain.Text), MS);        MS.Position := 0;   
   SetLength(Str, Ms.Size);        MS.Read(Pointer(Str)^,
Length(Str));        memClient.Lines.Text := Str;      finally       
Ms.Free;      end;    finally      HTTPClient.Free;    end;  end;end;

----------------------------------------------------------------------
JD · 2016-05-29 16:45

It worked! Including the rendering in the Firefox browser! Thanks a
million GetMem.

JD

----------------------------------------------------------------------
JD · 2016-05-29 16:48

Does anyone else think we can put this up on the Lazarus wiki as an
example for other programmers?

JD

----------------------------------------------------------------------
balazsszekely · 2016-05-29 18:09

@JD
I'm glad it's working but still, I would like to find out why the same
exact project works with XE4 and not with Lazarus.

----------------------------------------------------------------------
JD · 2016-05-29 18:37

Quote from: GetMem on May 29, 2016, 08:09:30 pm@JD
I'm glad it's working but still, I would like to find out why the same
exact project works with XE4 and not with Lazarus.

It has to be the encoding. It was driving me nuts. FPC 3+ will take
some getting used to. I also have XE4 but I didn't test it inside
Delphi because it does not have a recent version of Indy10 installed &
I did not want to go to the trouble of updating it. Thank goodness you
had it also.

We'll need Remy's input to get to the bottom of this.

JD

----------------------------------------------------------------------
Graeme · 2016-05-30 15:42

QuoteIt has to be the encoding. It was driving me nuts. FPC 3+ will
take some getting used to.
Indeed, and as I mentioned in the FPC Mailing List - FPC 3.0 is what I
consider a "preview release" and really should not be used in actual
applications. Obviously the core developers says otherwise, but I
disagree with them. I have found quite a few things that FPC 3.0
breaks, and easily causes data loss due to encoding conversions. And
because the FPC 3.0 RTL is not fully Unicode ready (UTF-16 or UTF-8),
there is not much we can do - other than submit RTL fixes for the
missing parts. Then once FPC 3.2 is released [in one or two years
time], then maybe FPC will be usable again. For now, I recommend
staying with FPC 2.6.4 for actual applications. I strongly believe FPC
3.0 is now worse off than what Delphi's 2009 release was.

----------------------------------------------------------------------
Thaddy · 2016-05-30 16:49

@Graeme and Remy:

The issue at hand is much much simpler and has always been so:
The transport layer itself should not make any guesses about any
encoding. It transports bytes for god's sake...
That's the issue with Indy and also the issue why it is slow and
unreliable. It doesn't help, it tries to second-guess. Drop it Remy...
Painful it may be but it is better for society. Or re-design that
piece of sh*t in any meaningful, performant, standards compliant way
without any suggestion that high(er) level components are in any way
compliant with any standards except those of Indy itself.. Otherwise
you can predict the bug reports and questions and work-arounds
(favorite pass-time?) beforehand.

I haven't seen any Indy version that properly makes the distinction
between the underlying protocols and its component representation at
the proper level of abstraction. It's been a mess from the start.

Note for non-informed readers: I actually use Indy some times  ... But
not for anything serious.

----------------------------------------------------------------------
balazsszekely · 2016-05-30 17:33

Quote@Thaddy
Drop it Remy... Painful it may be but it is better for society. Or re-
design that piece of sh*t in any meaningful, performant, standards
compliant way without any suggestion that high(er) level components
are in any way compliant with any standards except those of Indy
itself.. Otherwise you can predict the bug reports and questions and
work-arounds (favorite pass-time?) beforehand.If you don't like Indy
it's perfectly fine, but please stop spreading nonsense. I have a
client/server application running for more then five years, I have
never had any issue with it, not once. Besides @Remy helped a lot of
people over the past 10 years(including me), so this kind of language
is unacceptable! Thaddy you're a knowledgeable programmer, but
sometimes you act like a grumpy old man.

----------------------------------------------------------------------
Thaddy · 2016-05-30 17:42

Quote from: GetMem on May 30, 2016, 07:33:56 pmQuote@Thaddy
Drop it Remy... Painful it may be but it is better for society. Or re-
design that piece of sh*t in any meaningful, performant, standards
compliant way without any suggestion that high(er) level components
are in any way compliant with any standards except those of Indy
itself.. Otherwise you can predict the bug reports and questions and
work-arounds (favorite pass-time?) beforehand.If you don't like Indy
it's perfectly fine, but please stop spreading nonsense. I have a
client/server application running for more then five years, I have
never had any issue with it, not once. Besides @Remy helped a lot of
people over the past 10 years(including me), so this kind of language
is unacceptable! Thaddy you're a knowledgeable programmer, but
sometimes you act like a grumpy old man.

Maybe I have different values. You can't use Indy to run a 100.000+
visitors a day website, even when properly spread across multiple
servers with all due infra structure in place because it is holding
back and blocking way too much. So we had to write our own stuff.
Which is performant.
As I said: Indy is a perfectly good solution for low to medium traffic
webservers. Yes: I wrote that myself. But don't ever think it is
scalable. That's the mistake many people make. I concur that Indy is a
well established solution in the Pascal community. It is just not
suited for professionals who really need to scale. Let's leave it at
that. The Indy people know I am right about why I hate it.

----------------------------------------------------------------------
Graeme · 2016-05-30 18:05

Quote from: Thaddy on May 30, 2016, 06:49:19 pmNote for non-informed
readers: I actually use Indy some times  ... But not for anything
serious.
@Thaddy: No idea what you are on about. I've used Indy since Delphi 5
days, and it works fantastic. Delphi & Kylix would also not have
included Indy as part of the "delphi" product if it was as bad as you
suggest. I've personally used Indy in many products - most commercial,
and the various software products have been in production use for 5-15
years. No problems here.

FPC 3.0 on the other hand does a lot of crazy stuff with the new auto
string encoding conversion - often not to what the developer expected.
At to that the multiple compile modes, encoding modes etc... A mess. I
did some simple tests before I wanted to move some commercial projects
over to FPC 3.0. I found so many issues, or unfinished parts in the
RTL that there is simply NO way I'm moving off FPC 2.6.4 for now.
Sometimes the auto conversion happen in your code, sometimes in the
RTL or Compiler itself. The whole 8+ string types in FPC (or Object
Pascal in general) is a damn mess! I don't know of any other language
that has so many string types - they don't for good reason. Free
Pascal used to be my go-to language [of choice], from small utility
apps, to large enterprise style applications. Now I'm not so sure any
more - my next large commercial project has already moved to Java
instead of Object Pascal. More projects will probably follow.

Coming back to Indy, as GetMem [G*d I hate these silly screen names]
says... If you don't like Indy, then simply don't use it. There are a
few other choices, but none as extensive and well used as Indy. The
choice is yours.

----------------------------------------------------------------------
Graeme · 2016-05-30 18:10

Quote from: Thaddy on May 30, 2016, 07:42:52 pmMaybe I have different
values. You can't use Indy to run a 100.000+ visitors a day website,
even when properly spread across multiple servers with all due infra
structure in place because it is holding back and blocking way too
much.
Funny that... Indy is used in a 3-tier environment managing a massive
portion of the Australian electricity grid. The Indy HTTP Client and
Server components are used as the transport layer. This code has been
in production since 1999. It works just fine in that HUGE environment
to this day.

----------------------------------------------------------------------
balazsszekely · 2016-05-30 18:20

Quote@Graeme
Coming back to Indy, as GetMem [G*d I hate these silly screen names]
says...Are you 45+ too?   Anyway I think we hijacked @JD's thread...

----------------------------------------------------------------------
JD · 2016-05-30 18:39

Quote from: GetMem on May 30, 2016, 08:20:37 pmQuote@Graeme
Coming back to Indy, as GetMem [G*d I hate these silly screen names]
says...Are you 45+ too?   Anyway I think we hijacked @JD's thread...

Please keep it coming.   I love these discussions. I made the move to
FPC 3+ no knowing how much the encoding issue will affect my
applications. This is the first time I'm actually seeing the ugly side
of encoding since in all my Indy apps, I use streams to move data and
they work fine so far.

By the way I also have a 3 tier app built using Indy (with a lot of
support from Remy as usual) started in 2009 and running till this day
(with the usual updates of course). Indy was the first thing I learned
since it came by default with Delphi & I never looked elsewhere since.

JD

----------------------------------------------------------------------
JD · 2016-05-30 18:42

Quote from: Graeme on May 30, 2016, 08:10:04 pmQuote from: Thaddy on
May 30, 2016, 07:42:52 pmMaybe I have different values. You can't use
Indy to run a 100.000+ visitors a day website, even when properly
spread across multiple servers with all due infra structure in place
because it is holding back and blocking way too much.
Funny that... Indy is used in a 3-tier environment managing a massive
portion of the Australian electricity grid. The Indy HTTP Client and
Server components are used as the transport layer. This code has been
in production since 1999. It works just fine in that HUGE environment
to this day.

Now that is good news. The kind I can use as ammunition for the Java
crowd naysayers.

JD

----------------------------------------------------------------------
Graeme · 2016-05-30 22:16

Quote from: GetMem on May 30, 2016, 08:20:37 pmAre you 45+ too? 
I'm almost there. ;-)  Somebody somewhere on the internet did a pole,
and 40-50 was the avg age of a Delphi programmer. So at least I'm in
with the avg crowd.

----------------------------------------------------------------------
Remy Lebeau · 2016-05-31 19:41

Quote from: JD on May 29, 2016, 05:22:26 amHowever I discovered that
if I commented out AResponseInfo.CharSet := 'utf-8' and computed
content length using AResponseInfo.ContentLength :=
IndyTextEncoding_UTF8.GetByteCount(strJSON) as in the code below:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  if strJSON
<> EmptyStr then begin   AResponseInfo.ResponseNo := 200; 
 AResponseInfo.ContentType := 'application/json'; 
 //AResponseInfo.CharSet := 'utf-8';   AResponseInfo.ContentText :=
strJSON;   AResponseInfo.ContentLength :=
IndyTextEncoding_UTF8.GetByteCount(strJSON); end    // if strJSON <>
EmptyStr then 
then the client gets the entire JSON string! It is still not displayed
correctly though. 

By removing the AResponseInfo.Charset assignment, TIdHTTPServer will
not encode the ContentText to UTF-8 during transmission.  It will
encode it to a different charset instead (in this case, Indy's built-
in 8bit encoding).  By setting the ContentLength to the byte length of
the UTF-8 version of the ContentText, you are lying to the client, as
the number of bytes generated by the actual encoding may be different
than the number of bytes that UTF-8 would generate.

Don't lie to the client!  The ContentLength MUST be accurate.  And
once again, you DO NOT need to set the ContentLength manually, you CAN
let TIdHTTPServer calculate it for you based on the number of bytes it
is ACTUALLY going to send to the client.

----------------------------------------------------------------------
Remy Lebeau · 2016-05-31 19:57

Quote from: GetMem on May 29, 2016, 04:31:59 pmCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
AResponseInfo.ContentStream := TMemoryStream.Create;    try     
AResponseInfo.ContentStream.Write(Pointer(strJSON)^,
Length(strJSON));      AResponseInfo.ContentStream.Position := 0;    
AResponseInfo.WriteContent;    finally     
AResponseInfo.ContentStream := nil;    end; 
Or, you can (and should) alternatively let TIdHTTPServer take
ownership of the TMemoryStream, sending and freeing it for you.  That
is its default behavior.  When the OnCommand... event handler exists,
whatever has been assigned to the Content(Text/Stream) is sent (and
freed) automatically if not already sent manually.  In this case,
there is no need to call WriteContent manually at all.

Also, in this case, if strJSON is an AnsiString then it needs to be in
UTF-8 encoding before written as-is to the TMemoryStream.  Or, you can
use Indy's own WriteStringToStream() function instead, eg:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TForm1.HTTPServerCommandGet(AContext: TIdContext;  ARequestInfo:
TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);var  strJSON:
string;begin   strJSON := String(PAYS);   if strJSON <> EmptyStr then 
begin        AResponseInfo.ContentType := 'application/json';   
AResponseInfo.CharSet := 'utf-8';    AResponseInfo.ContentStream :=
TMemoryStream.Create;   
WriteStringToStream(AResponseInfo.ContentStream, strJSON,
IndyTextEncoding_UTF8      {$IFNDEF FPC_UNICODESTRINGS},
IndyTextEncoding_OSDefault{or whatever encoding the string is actually
in}{$ENDIF}    );    AResponseInfo.ContentStream.Position := 0;  end 
else  begin    AResponseInfo.ResponseNo := 404;   
AResponseInfo.ContentType := 'text/html';    AResponseInfo.CharSet :=
'utf-8';    AResponseInfo.ContentText := 'Resource not found'; 
end;end; 
Quote from: GetMem on May 29, 2016, 04:31:59 pmCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
HTTPClient := TIdHTTP.Create(nil);    try      ms :=
TMemoryStream.Create;      try        HTTPClient.HandleRedirects :=
True;        HTTPClient.HTTPOptions := [hoForceEncodeParams];       
HTTPClient.Get(Trim(edtDomain.Text), MS);        MS.Position := 0;   
   SetLength(Str, Ms.Size);        MS.Read(Pointer(Str)^,
Length(Str));        memClient.Lines.Text := Str;      finally       
Ms.Free;      end;    finally      HTTPClient.Free;    end; 
In this case, I would suggest using the overloaded version of
TIdHTTP.Get() that returns a String.  It already handles charset-
decoding of the HTTP response to a String, and in the case where
String is AnsiString then you can also specify the desired charset for
the output AnsiString, eg:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
HTTPClient := TIdHTTP.Create(nil);    try     
HTTPClient.HandleRedirects := True;      HTTPClient.HTTPOptions :=
[hoForceEncodeParams];      Str :=
HTTPClient.Get(Trim(edtDomain.Text)        {$IFNDEF
FPC_UNICODESTRINGS}, IndyTextEncoding_UTF8{$ENDIF}      );     
memClient.Lines.Text := Str;    finally      HTTPClient.Free;   
end;[code]

----------------------------------------------------------------------
Remy Lebeau · 2016-05-31 20:07

Quote from: GetMem on May 29, 2016, 08:09:30 pmI would like to find
out why the same exact project works with XE4 and not with Lazarus.

Differences in default String encodings.

In Delphi 2009 and later, String is UnicodeString, but in
FreePascal/Lazarus String is still AnsiString unless you enable {$MODE
DelphiUnicode} or {$MODESWITCH UNICODESTRINGS}.  Indy has additional
handling that has to be taken into account when String is AnsiString
so it can deal with charset conversions between Ansi and Unicode when
needed.

Indy is primarily written to use Delphi UnicodeString, but has
backwards compatibility for AnsiString in FreePascal and earlier
Delphi versions.  However, Indy does not make (very many) assumptions
about the charset used for AnsiString, so you have to be explicit
about the charset used whenever user code is giving an AnsiString to
Indy, or whenever Indy is returning an AnsiString to user code.  There
are extra properties and method parameters exposed to handle that
where needed (such as the TIdIOHandler.DefAnsiEncoding property, the
ADestEncoding parameter of TIdHTTP.Get(), etc).

As an alternatively, in Indy's IdCompilerDefines.inc files, where Indy
enables {$MODE Delphi} for FreePascal, you might try also enabling
{$MODE DelphiUnicode} (not currently enabled) and see if it makes a
difference.  That should eliminate Indy's handling of AnsiString and
move everything to UnicodeString, just like in Delphi 2009+.  But,
since FreePascal does not have a Unicode-enabled RTL yet, I can't
comment on what kinds of side effects this may cause internally.  I
have never tried enabling FreePascal's DelphiUnicode mode in Indy yet.

----------------------------------------------------------------------
Remy Lebeau · 2016-05-31 20:29

Quote from: Thaddy on May 30, 2016, 06:49:19 pmThe issue at hand is
much much simpler and has always been so:
The transport layer itself should not make any guesses about any
encoding. It transports bytes for god's sake...

Indy's transport layer operates on bytes.  But in the case of strings,
those bytes are charset-encoded on output, and are charset-decoded on
input.  For instance, when TIdHTTPServer is sending a ContentText
value to a client, the ContentText should be in Pascal's native String
encoding, and then TIdHTTPResponseInfo.WriteContent() passes the
ContentText to the TIdIOHandler.Write(String) method with the desired
charset so the IOHandler will convert the String to charset-encoded
bytes before passed them to the socket layer.

Quote from: Thaddy on May 30, 2016, 06:49:19 pmThat's the issue with
Indy and also the issue why it is slow and unreliable. It doesn't
help, it tries to second-guess.

Actually, it does not second-guess.  Things are much simpler when
String is Unicode, transport-layer charsets are much easier and
reliable to work with.  But when String is AnsiString (such as the
case in FreePascal by default), there is more logic involved to ensure
minimal data loss when converting between Ansi and Unicode in both
directions, so you usually have to be explicit about the Ansi charsets
used so Indy can make the proper conversions.

Quote from: Thaddy on May 30, 2016, 06:49:19 pmDrop it Remy... Painful
it may be but it is better for society.
What is best for society is for AnsiString to go away forever.  And
Delphi already did that, and Indy works just fine in Unicode
environments.  But FreePascal hasn't (fully) done that yet, and until
it does, things are still subject to all the legacy Ansi encoding
issues that going with it.  And Indy is doing its best to account for
that.

Quote from: Thaddy on May 30, 2016, 06:49:19 pmOr re-design that piece
of sh*t

I don't appreciate you acting like a pompous jerk to me or Indy.  I
have enough things on my plate to deal with, I don't need bad-mouthers
downplaying my hard efforts.  Good grief.

Quote from: Thaddy on May 30, 2016, 06:49:19 pmin any meaningful,
performant, standards compliant way without any suggestion that
high(er) level components are in any way compliant with any standards
except those of Indy itself.. Otherwise you can predict the bug
reports and questions and work-arounds (favorite pass-time?)
beforehand. I haven't seen any Indy version that properly makes the
distinction between the underlying protocols and its component
representation at the proper level of abstraction. It's been a mess
from the start.

I didn't write the original Indy, I just maintain and expand on it as
I can.  Indy has had to strike a very delicate balance between
backwards compatibility (plenty of users still using Indy in pre-
Unicode versions of Delphi), forwards-movement (Unicode strings and
UTF encodings where available), and IDE integrations (some of
Embarcadero own technologies use Indy, and it can't break very much
between Delphi maintenance releases).  Maybe things are not the way
you would like them to be, but it is what it needed to be.  If you
don't like how Indy works, feel free to submit fixes, or even join the
dev team (God known I could use some help), otherwise shut up and go
use something else that suits your needs.

----------------------------------------------------------------------
Remy Lebeau · 2016-05-31 20:38

Quote from: Thaddy on May 30, 2016, 07:42:52 pmMaybe I have different
values. You can't use Indy to run a 100.000+ visitors a day website,
even when properly spread across multiple servers with all due infra
structure in place because it is holding back and blocking way too
much. So we had to write our own stuff. Which is performant.
As I said: Indy is a perfectly good solution for low to medium traffic
webservers. Yes: I wrote that myself. But don't ever think it is
scalable. That's the mistake many people make. I concur that Indy is a
well established solution in the Pascal community. It is just not
suited for professionals who really need to scale. Let's leave it at
that. The Indy people know I am right about why I hate it.

Indy was never really intended for such high levels of traffic.  And
yes, there have been attempts to provide better scaling (use of
fibers, etc) without much success.  High scalability usually involves
breaking up logic into smaller manageable tasks that are asynchronous
and distributed.  Indy is not designed for that, and the Indy
devs/execs are well aware of that.  I don't see that changing anything
soon (MAJOR re-write).

----------------------------------------------------------------------
JD · 2016-05-31 21:49

Quote from: Remy Lebeau on May 31, 2016, 10:07:46 pmAs an
alternatively, in Indy's IdCompilerDefines.inc files, where Indy
enables {$MODE Delphi} for FreePascal, you might try also enabling
{$MODE DelphiUnicode} (not currently enabled) and see if it makes a
difference.  That should eliminate Indy's handling of AnsiString and
move everything to UnicodeString, just like in Delphi 2009+.  But,
since FreePascal does not have a Unicode-enabled RTL yet, I can't
comment on what kinds of side effects this may cause internally.  I
have never tried enabling FreePascal's DelphiUnicode mode in Indy yet.

I already tried that a while ago but it made no difference. The
ContentLength problem persisted. As Graeme said, the encoding problem
was not there in FPC 2.6.4 but FPC 3+ has changed things. I've since
followed the suggestion of GetMem & used memory streams and as you can
see from my replies, it worked.

Thanks,

JD

----------------------------------------------------------------------
Remy Lebeau · 2016-05-31 23:14

Quote from: JD on May 31, 2016, 11:49:42 pmI already tried that but a
while ago but it made no difference.

I was not aware of that.  Did it make any difference to Indy at all in
general?

Quote from: JD on May 31, 2016, 11:49:42 pmThe ContentLength problem
persisted.

But WHY does it persist?  I need to know WHY the
AResponseInfo.ContentLength fails to be sent correctly so I can fix it
if needed.  What is AResponseInfo.ContentLength actually set to
compared to the AResponseInfo.ContentText value?  Is there a Content-
Length header missing on the socket, or is it actually being sent?
Does it match the AResponseInfo.ContentLength property value?

Everything in the code says this should be working fine, and
everything in this discussion says otherwise.  But I have no DETAILS
about WHY it fails to work.  I don't have FreePascal/Lazarus installed
to test this myself (it was a PITA the last time I tried), so I need
FreePascal users to help debug it for me.

Now that I think about it, I *suspect* that when String is AnsiString,
even with AResponseInfo.Charset set to UTF-8,
AResponseInfo.ContentText is not being converted to UTF-8 correctly by
default before AResponseInfo.ContentLength is calculated from it.

The IIdTextEncoding.GetBytes() and IIdTextEncoding.GetByteCount()
methods take a WideString/UnicodeString as input, so there would be an
Ansi->Unicode conversion performed by the compiler/RTL before the
Unicode->UTF8 conversion performed by Indy.  So maybe the Ansi charset
used for the conversion doesn't match the AnsiString encoding and so
data loss occurs.  AResponseInfo does not have a property to specify
the source Ansi charset for ContentText, but you could use the
AResponseInfo.Connection.IOHandler.DefAnsiEncoding property for that
instead.

But even with data loss, if AResponseInfo.ContentLength is calculated
from AResponseInfo.ContentText, it should still be the correct byte
count for the number of data bytes actually transmitted, since they
should be going through the same encoding conversion from String to
bytes.

Quote from: JD on May 31, 2016, 11:49:42 pmAs Graeme said, the
encoding problem was not there in FPC 2.6.4 but FPC 3+ has changed
things.

But not to the point where it should be causing these kinds of
problems, I think.  Now granted, I don't know the extent of what 3+
actually changes, but I do know that the default string encoding is
not one of them, unless you count adding codepage awareness to
AnsiString.  But that is still outside the scope of Indy itself.  Any
AnsiString<->Unicode string conversions are at the RTL level, and any
Indy operations that act on AnsiString still need to be told what the
AnsiString encoding is for conversion purposes.

Quote from: JD on May 31, 2016, 11:49:42 pmI've since followed the
suggestion of GetMem & used memory streams and as you can see from my
replies, it worked.

Yes, using ContentStream instead of ContentText is a viable
alternative, provided you ensure the encoding of the ContentStream
bytes is accurate.

Even then, TIdHTTPServer can still calculate the appropriate
ContentLength when set to -1, using ContentStream.Size.  Which still
goes back to the issue that the ContentLength should be getting sent
correctly regardless of whether ContentText or ContentStream is used.

----------------------------------------------------------------------
balazsszekely · 2016-06-01 06:22

Quote@Remy
I need to know WHY the AResponseInfo.ContentLength fails to be sent
correctly so I can fix it if needed.It does not fail. Actually there
is no issue after all. Under Lazarus you explicitelly have to specify
the default encoding. This is the only difference as far as I can see.
1. Server
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   //...  if
strJSON <> EmptyStr then  begin    AResponseInfo.ResponseNo := 200;  
AResponseInfo.ContentType := 'application/json';   
AResponseInfo.CharSet := 'utf-8';   
AContext.Connection.IOHandler.DefAnsiEncoding :=
IndyTextEncoding_UTF8;     
AContext.Connection.IOHandler.DefStringEncoding :=
IndyTextEncoding_UTF8;    AResponseInfo.ContentText := strJSON;   
AResponseInfo.ContentLength := -1;  end  //...
2. Client
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   //... 
HTTPClient := TIdHTTP.Create(nil);  HTTPClient.HandleRedirects :=
True;  HTTPClient.HTTPOptions := [hoForceEncodeParams];   
memClient.Lines.Text := HTTPClient.Get(Trim(edtDomain.Text),
IndyTextEncoding_UTF8);     //...
The question is, can you set the DefAnsiEncoding/DefStringEncoding to
IndyTextEncoding_UTF8 under Lazrus/FPC? If this can be done the
conversion between Delphi/Lazrus codebase would be seamless.

----------------------------------------------------------------------
Remy Lebeau · 2016-06-03 00:06

Quote from: GetMem on June 01, 2016, 08:22:09 amIt does not fail.
Actually there is no issue after all. Under Lazarus you explicitelly
have to specify the default encoding. This is the only difference as
far as I can see.

I would suggest doing the assignment of the DefAnsiEncoding in the
OnConnect event, since a single connection can be reused for multiple
requests.  You don't need to set DefStringEncoding since TIdHTTPServer
passes an explicit encoding to TIdIOHandler.Write(String) (overriding
DefStringEncoding) based on the AResponseInfo.CharSet property.

Code: [Select]procedure TMyForm.IdHTTPServer1Connect(AContext:
TIdContext);
begin
  AContext.Connection.IOHandler.DefAnsiEncoding :=
IndyTextEncoding_UTF8; 
end;

procedure TMyForm.IdHTTPServer1CommandGet(AContext: TIdContext;
ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
begin
  //...
  if strJSON <> EmptyStr then
  begin
    AResponseInfo.ResponseNo := 200;
    AResponseInfo.ContentType := 'application/json';
    AResponseInfo.CharSet := 'utf-8';
    AResponseInfo.ContentText := strJSON;
    AResponseInfo.ContentLength := -1;
  end
  //...
end;

If you do set DefStringEncoding manually, I would suggest doing that
in the OnConnect event as well.  Do note that setting
DefStringEncoding affects TIdHTTPServer's reading/sending of HTTP
headers (which should not be an issue since HTTP does not allow non-
ASCII characters in headers anyway).

Quote from: GetMem on June 01, 2016, 08:22:09 amThe question is, can
you set the DefAnsiEncoding/DefStringEncoding to IndyTextEncoding_UTF8
under Lazrus/FPC?

Yes, they can be set to UTF-8 on all platforms/compilers.  Just note
that if you enable FreePascal's UNICODESTRINGS mode, DefAnsiEncoding
should disappear since String will be UnicodeString instead of
AnsiString, so no more Ansi issues to deal with.

----------------------------------------------------------------------
balazsszekely · 2016-06-03 04:21

@Remy
QuoteI would suggest doing the assignment of the DefAnsiEncoding in
the OnConnect event, since a single connection can be reused for
multiple requests.  You don't need to set DefStringEncoding since
TIdHTTPServer passes an explicit encoding to
TIdIOHandler.Write(String) (overriding DefStringEncoding) based on the
AResponseInfo.CharSet property.QuoteIf you do set DefStringEncoding
manually, I would suggest doing that in the OnConnect event as well. 
Do note that setting DefStringEncoding affects TIdHTTPServer's
reading/sending of HTTP headers (which should not be an issue since
HTTP does not allow non-ASCII characters in headers anyway).Noted!
Thank you for the suggestions.

@JD
Can you please add the HTTP project to wiki, so other people can learn
from it?

----------------------------------------------------------------------
JD · 2016-06-03 06:51

Quote from: GetMem on June 03, 2016, 06:21:14 am@JD
Can you please add the HTTP project to wiki, so other people can learn
from it?

I don't have access to the Wiki. We'll have to ask someone with admin
rights AFAIK.

JD

----------------------------------------------------------------------
JD · 2016-06-03 07:44

Quote from: Remy Lebeau on June 01, 2016, 01:14:03 amI was not aware
of that.  Did it make any difference to Indy at all in general?

@Remy It did not seem to make any difference to Indy in general. All I
did was define {$MODE DelphiUnicode} in IdCompilerDefines.inc. My old
3 tier TCP Server/Client projects work normally but maybe it is also
because IndyTextEncoding_UTF8 is explicitly stated in the connection
between server & client.

By the way, I have a memory leak that I'm having trouble tracing even
though I defined FREE_ON_FINAL. There is a memory leak even if the
project is empty.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i // $DEFINE
the following if the global objects in the IdStack and IdThread//
units should be freed on finalization{$DEFINE FREE_ON_FINAL}{.$UNDEF
FREE_ON_FINAL} // Make sure the following is $DEFINE'd only for
suitable environments// as specified further below.  This works in
conjunction with the// FREE_ON_FINAL define above.{.$UNDEF
REGISTER_EXPECTED_MEMORY_LEAK} 
JD

----------------------------------------------------------------------
balazsszekely · 2016-06-03 09:39

Quote@JD
Remy It did not seem to make any difference to Indy in general. All I
did was define {$MODE DelphiUnicode} in IdCompilerDefines.inc. My old
3 tier TCP Server/Client projects work normally but maybe it is also
because IndyTextEncoding_UTF8 is explicitly stated in the connection
between server & client.
According to the wiki {$MODE DelphiUnicode} affects only the current
unit, besides if you want to target multiple platform, I don't think
DelphiUnicode is a good idea, personally I would stay with
IndyTextEncoding_UTF8, of course the choice is yours.

Quote@JD
By the way, I have a memory leak that I'm having trouble tracing even
though I defined FREE_ON_FINAL. There is a memory leak even if the
project is empty.
With {$DEFINE FREE_ON_FINAL} I don't get any memory leaks(see
attachment).

----------------------------------------------------------------------
Thaddy · 2016-06-03 09:40

Quote from: JD on June 03, 2016, 08:51:53 amQuote from: GetMem on June
03, 2016, 06:21:14 am@JD
Can you please add the HTTP project to wiki, so other people can learn
from it?

I don't have access to the Wiki. We'll have to ask someone with admin
rights AFAIK.

JD
JD, normally you should have access to the wiki, since your
credentials warrant that. The wiki is not open to everybody but is
open for esteemed members (meaning, sufficient posts on the forum, I
believe). I may be wrong, in that case ask a moderator to provide you
with sufficient rights.
The quality of your postings here warrant that, so I am all for it ...
(I am not saying I always agree with you, of course   )

----------------------------------------------------------------------
ChrisF · 2016-06-03 12:50

Quote from: JD on June 03, 2016, 08:51:53 amQuote from: GetMem on June
03, 2016, 06:21:14 am@JD
Can you please add the HTTP project to wiki, so other people can learn
from it?

I don't have access to the Wiki. We'll have to ask someone with admin
rights AFAIK.

JD

AFAIR, the wiki is "self-service" and opened to everybody. You must
just register first (i.e. create an account specifically for it) in
order to be able to modify it.

----------------------------------------------------------------------
JD · 2016-06-09 19:54

Quote from: ChrisF on June 03, 2016, 02:50:07 pmAFAIR, the wiki is
"self-service" and opened to everybody. You must just register first
(i.e. create an account specifically for it) in order to be able to
modify it.

Thanks for the info ChrisF. I'll do so as soon as I can.

JD

----------------------------------------------------------------------
Remy Lebeau · 2016-07-06 00:44

Quote from: GetMem on June 03, 2016, 11:39:17 amAccording to the wiki
{$MODE DelphiUnicode} affects only the current unit

If DelphiUnicode mode is enabled in IdCompilerDefines.inc then it will
take effect throughout Indy, as most units include that .inc file.

Since that mode only affects the calling unit, and if you enable it in
Indy, I would suggest enabling it in any units that use Indy directly
to match (or at least units that pass strings to/from Indy).

Quote from: GetMem on June 03, 2016, 11:39:17 ambesides if you want to
target multiple platform, I don't think DelphiUnicode is a good idea,
personally I would stay with IndyTextEncoding_UTF8, of course the
choice is yours.

If DelphiUnicode mode is enabled, Indy would handle Unicode strings
the same way on all platforms, and that is more portable.

If DelphiUnicode mode is not enabled, you are back to dealing with
platform-specific Ansi issues, which is not portable.

At the very least, I would suggest enabling DelphiUnicode mode in Indy
just to get Ansi out of its interfaces, if you still want to use Ansi
strings in the rest of your code.

I would hate to have to add a 3rd mode to Indy's codebase where all
strings are UTF-8 instead of UTF-16 or Ansi.  But that might be an
option to consider.  It would re-introduce some performance hits on
non-UTF8 RTL/API functions, though.  But I guess since FreePascal does
not have a Unicode RTL yet, you are probably getting some of those
hits anyway if DelphiUnicode mode is enabled.