Linux-Lazarus-Indy lazarus / 1-Linux ---------------------------------------------------------------------- scasparz · 2018-06-28 01:04 Not sure if this is the proper place to put this, but I would like to give it a try. Please Mod forgive me if I erred. Am using Linux Mint 18.3-64 XFCE, Lazarus 1.8.4 with FPC 3.0.4-3. Trying to break into the TCP world using Indy for Lazarus. Have downloaded Indy version 10.2.0.3 and installed it into Lazarus without issues. Have located a number of trivia Indy examples at https://github.com/tinydew4/indy-project-demos/tree/master/TCPIP%20Del phi%20%26%20Indy10%20Client%20Server%20Demo. Have started with the simplest one, that is the first 1_Sample simple string exchange. In this the author provides an elementary TCP client sending a string to an elementary TCP server. Understand this was written for the Indy-Delphi-Windows version. Have managed to port the client one to Linux without significant issues. There are some minimal differences, was not hard to modify. Of course remembered to add the indy path to the Lazarus Project unit path. Had significant issues trying to port the server demo though. To begin with there seems no way to get the tIdTCPServer widget from the palette and drag it on my form, as this can crash my Lazarus. Did not give up though. Declared an IdTCPServer property on my form, had it created manually at the Form OnCreate event handler, so far so good. Eg. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TStringServerForm.FormCreate(Sender: TObject);begin  IdTCPServer1 := tIdTCPServer.Create(Self);  IdTCPServer1.Name := 'IdTCPServer1';  IdTCPServer1.Active := False;  IdTCPServer1.ListenQueue := 15;  IdTCPServer1.MaxConnections := 0;  IdTCPServer1.ReuseSocket := rsOSDependent;  IdTCPServer1.TerminateWaitTime := 5000;  // there is no nagle property available for Linux   IdTCPServer1.Bindings.Add.IP := '127.0.0.1';  IdTCPServer1.Bindings.Add.Port := 6000;  IdTCPServer1.OnConnect := IdTCPServer1Connect;  IdTCPServer1.OnDisconnect := IdTCPServer1Disconnect;  IdTCPServer1.OnException := IdTCPServer1Exception;  IdTCPServer1.OnExecute := IdTCPServer1Execute;end;The above runs fine to the end. Yet there does not seem to be a way to activate server with the: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdTCPServer1.Active := Truecommand, which raises an exception and terminates the server app. Still did not give up. Transfered the server demo app to a virtual Windows 7 running Delphi XE which comes with Indy 10 installed. In this case the server demo app worked to a point. I mean I could send a string in Latin from the Mint-Lazarus demo-client-app to the Windows- Delphi demo-server-app, but I could not find a way to include international characters to the string. Believe this must be a transliteration issue. AFAIK Linux uses UTF8 while Redmond uses UTF16. Have used: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LLine := AContext.Connection.IOHandler.ReadLn(TIdTextEncoding.UTF8);to get the string on the server at the IdTCPServer1Execute OnExecute handler, while similarly I used Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdTCPClient1.IOHandler.WriteLn(Edit1.Text, enDefault);at the client to send the string. This did not work. Had some better luck using Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdTCPClient1.IOHandler.WriteLn(Edit1.Text, enUTF8);at the client side. While Latin text comes OK, still have failed with international (Greek) chars. In any case, each and every time I disconnect the client, the server fires the OnException event with message ‘Connection closed gracefully’. Is this an exception really? To summarize: 1. Is there a newer more stable Indy version for Lazarus-Linux and where to get it from? Similarly are there any specific tweaks that should be done during the Indy installation on Lazarus, other than the usual (eg compile the .lpk twice)? 2. Is there any hope to drag the TCPServer widget from the palette onto the Lazarus form without crashing Lazarus? 3. Is there a way to activate my TCPServer instance on Linux-Lazarus without crashing the app? 4. Is there a way to have the correct transliteration with international character sets between a client and a server? In any case this I believe is a secondary issue as I do not plan a Linux to Windows connection but for educational use only. and perhaps the most important of them all: 5. Do I stand a chance with Linux-Lazarus-Indy or should I have to look elsewhere? Indy is highly reputable and would be glad in case I could overcome my issues. On the other hand Windows and Delphi are not an option other than for fooling around. Thanks for your time in advance. s ---------------------------------------------------------------------- denver · 2018-06-28 05:54 In Linux Version of Indy10, you need to confirm the IPVersion ( id_IPV4 or id_IPv6 ) IdTCPServer1.Bindings.Items[ 0 ].IPVersion := Id_IPv4 ; ---------------------------------------------------------------------- balazsszekely · 2018-06-28 06:32 QuoteTo summarize: 1. Is there a newer more stable Indy version for Lazarus-Linux and where to get it from? Similarly are there any specific tweaks that should be done during the Indy installation on Lazarus, other than the usual (eg compile the .lpk twice)? 2. Is there any hope to drag the TCPServer widget from the palette onto the Lazarus form without crashing Lazarus? 3. Is there a way to activate my TCPServer instance on Linux-Lazarus without crashing the app? 4. Is there a way to have the correct transliteration with international character sets between a client and a server? In any case this I believe is a secondary issue as I do not plan a Linux to Windows connection but for educational use only. and perhaps the most important of them all: 5. Do I stand a chance with Linux-Lazarus-Indy or should I have to look elsewhere? Indy is highly reputable and would be glad in case I could overcome my issues. On the other hand Windows and Delphi are not an option other than for fooling around. Just install the latest version. Most of the issues mentioned above are fixed. You can download the latest version from here: https://svn.atozed.com:444/svn/Indy10 (use a svn client). Alternatively you can install indy with OPM(http://wiki.freepascal.org/Online_Package_Manager) ---------------------------------------------------------------------- scasparz · 2018-06-29 02:57 To begin with many thanks for your kind responses. Have installed a new vbox installation of Linux Mint 18.2, with FPC 3.0.4-3 and Lazarus 1.8.4. Have test it, got no issues. Then downloaded Indy10 from http://packages.lazarus-ide.org (version 10.6.2.5457) as suggested by GetMem and have it installed again without issues . Newer version seems to work considerably better than the previous. I can now drag the tIdTCPServer component from the palette onto my form without breaking Lazarus. Also have added the Id_IPv4 clause as suggested by Denver. But I still fail to Activate IdTCPServer1 exactly as before with run time message 'RunError(232)'. Not to mention that as I write code, I get the message: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Codetools, Errors: 1IdGlobal.pas(1889,105) Error: expected:; but deprecated found. refering to line:GOffsetFromUTC: TDateTime = 0{$IFDEF HAS_DEPRECATED}{$IFDEF USE_SEMICOLON_BEFORE_DEPRECATED};{$ENDIF} deprecated{$ENDIF}; at the message window. What am I doing wrong? s ---------------------------------------------------------------------- denver · 2018-06-29 05:32 Quote from: scasparz on June 29, 2018, 04:57:37 amTo begin with many thanks for your kind responses. But I still fail to Activate IdTCPServer1 exactly as before with run time message 'RunError(232)'. Not to mention that as I write code, I get the message: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Codetools, Errors: 1IdGlobal.pas(1889,105) Error: expected:; but deprecated found. refering to line:GOffsetFromUTC: TDateTime = 0{$IFDEF HAS_DEPRECATED}{$IFDEF USE_SEMICOLON_BEFORE_DEPRECATED};{$ENDIF} deprecated{$ENDIF}; at the message window. What am I doing wrong? s For 'RunError(232)'. , you need to add   cthreads, as the first unit  in unit section in the main  .pas or  .lpr  in linux base and only one Like this  : uses   {$DEFINE UseCThreads}   {$IFDEF UNIX}{$IFDEF UseCThreads}   cthreads,   {$ENDIF}{$ENDIF}   Interfaces, // this includes the LCL widgetset   Forms ; In the IdGlobal.pas, I just modify as follow and it work for me.   GOffsetFromUTC: TDateTime = 0 ; deprecated ; ---------------------------------------------------------------------- scasparz · 2018-06-30 00:19 Denver bingo! Was with the wrong impression the Lazarus latest versions had been using cthreads by default. As you wrote it was exactly this the cause why I failed to activate the IdTCPServer. It now seems to work fine. In regard to the statement: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i GOffsetFromUTC: TDateTime = 0{$IFDEF HAS_DEPRECATED}{$IFDEF USE_SEMICOLON_BEFORE_DEPRECATED};{$ENDIF} deprecated{$ENDIF}; at the IdGlobal.pas, your suggestion to have it replaced with Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i GOffsetFromUTC: TDateTime = 0; deprecated;did not work for me. On compiling Indy10 prior to installation Lazarus raises an error with message: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdGlobal.pas(1889,44) Fatal: Syntax error, ":" expected but ";" foundHad never used the deprecated directive before. Tried to experiment a bit, used Lazarus to open a new trivia project where I declared a variable using identical syntax: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  Form1: TForm1;  pap: Integer; deprecated;and Lazarus raised exactly the same error again. Tried this on Delphi XE and once more got exactly the same error with the same message. Could not find some additional documentation either on the Delphi XE help subsystem or on the internet over deprecated. Am left with the impression -please correct me if wrong- that this directive only works with procedures or methods but not with variables. Consequently the only way I could find to avoid that error was to modify IdGlobal.pas omitting deprecated completely, eg: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i GOffsetFromUTC: TDateTime = 0;It is my understanding this was a bit arbitrary, nevertheless now not only Indy compiles without issues, but post installation codetools seems to work fine as well, allowing me for example to type in methods or properties the lazy way. This did not seem to work before by virtue of deprecated. Looks like I now have a working Indy at least for elementary work. More important it looks like the package evolves as I have some serious expectations from it. Will keep experimenting on the TCP using Indy. Many thanks for your kind help. Without you and GetMem, I simply couldn't have done it. best regards s ---------------------------------------------------------------------- Remy Lebeau · 2018-07-01 22:22 Quote from: scasparz on June 28, 2018, 03:04:06 amHave downloaded Indy version 10.2.0.3 and installed it into Lazarus without issues. Why such an OLD version?  The CURRENT version is 10.6.2(.5462). Quote from: scasparz on June 28, 2018, 03:04:06 amCode: [Select]  IdTCPServer1.Bindings.Add.IP   := '127.0.0.1';   IdTCPServer1.Bindings.Add.Port := 6000; Typical newbie mistake, usually due to bad 3rd party examples.  You are adding *2* bindings to the collection, one for 127.0.0.1:0, and the other for 0.0.0.0:6000.  You want to add only *1* binding, for 127.0.0.1:6000: Code: [Select]Binding := IdTCPServer1.Bindings.Add; Binding.IP   := '127.0.0.1'; Binding.Port := 6000; Or: Code: [Select]with IdTCPServer1.Bindings.Add do begin   IP   := '127.0.0.1';   Port := 6000; end; Quote from: scasparz on June 28, 2018, 03:04:06 amI mean I could send a string in Latin from the Mint-Lazarus demo-client-app to the Windows-Delphi demo-server-app, but I could not find a way to include international characters to the string. Believe this must be a transliteration issue. AFAIK Linux uses UTF8 while Redmond uses UTF16. Indy's default text encoding is actually US-ASCII, unless you explicitly specify otherwise, either by: - setting the global GIdDefaultTextEncoding variable in the IdGlobal unit. - setting the TIdIOHandler.DefStringEncoding property (and TIdIOHandler.DefAnsiEncoding property in pre-Unicode compilers). - specifying an encoding parameter on a per-read/write operation basis. Quote from: scasparz on June 28, 2018, 03:04:06 amHave used: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LLine := AContext.Connection.IOHandler.ReadLn(TIdTextEncoding.UTF8);to get the string on the server at the IdTCPServer1Execute OnExecute handler, while similarly I used Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdTCPClient1.IOHandler.WriteLn(Edit1.Text, enDefault);at the client to send the string. This did not work. Because you are mixing encodings, since enDefault is US-ASCII by default.  If you replace enDefault with enUTF8, it should then work OK.  Or, you can configure enDefault to be UTF-8 by setting the IdGlobal.GIdDefaultTextEncoding variable to encUTF8. Quote from: scasparz on June 28, 2018, 03:04:06 amHad some better luck using Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdTCPClient1.IOHandler.WriteLn(Edit1.Text, enUTF8);at the client side. Yes, exactly. Quote from: scasparz on June 28, 2018, 03:04:06 amWhile Latin text comes OK, still have failed with international (Greek) chars. UTF-8 works fine for the entirety of Unicode, Greek and all. Quote from: scasparz on June 28, 2018, 03:04:06 amIn any case, each and every time I disconnect the client, the server fires the OnException event with message ‘Connection closed gracefully’. Is this an exception really? Yes, that is normal behavior.  Indy uses exceptions for error handling and notifications.  In a TIdTCPServer, when a client disconnects, subsequent read/write operations on that connection will raise an exception, which you should let TIdTCPServer catch and handle for you so that it can close the socket and clean up the worker thread that was managing that socket. Quote from: scasparz on June 28, 2018, 03:04:06 am1. Is there a newer more stable Indy version for Lazarus-Linux and where to get it from? Yes.  Links are on Indy's website: http://www.indyproject.org/Sockets/Docs/Indy10Installation.aspx http://www.indyproject.org/Sockets/Download/DevSnapshot.aspx Indy is also available in Lazarus's "Online Package Manager" (OPM). Quote from: scasparz on June 28, 2018, 03:04:06 am2. Is there any hope to drag the TCPServer widget from the palette onto the Lazarus form without crashing Lazarus? This is the first that I'm hearing that it crashes like that.  Do you have any information about the actual crash? Quote from: scasparz on June 28, 2018, 03:04:06 am3. Is there a way to activate my TCPServer instance on Linux-Lazarus without crashing the app? Fix the way you are initializing the server's Bindings. Quote from: scasparz on June 28, 2018, 03:04:06 am4. Is there a way to have the correct transliteration with international character sets between a client and a server? Make sure you are using the same text encoding on both sides. Quote from: scasparz on June 28, 2018, 03:04:06 am5. Do I stand a chance with Linux-Lazarus-Indy or should I have to look elsewhere? Plenty of FPC users are using Indy in Linux.  And more users to come in the future as the latest Delphi (and upcoming C++Builder) now supports Linux development, too. Quote from: denver on June 28, 2018, 07:54:29 amIn Linux Version of Indy10, you need to confirm the IPVersion ( id_IPV4 or id_IPv6 ) IdTCPServer1.Bindings.Items[ 0 ].IPVersion := Id_IPv4 ; Id_IPv4 is the default, unless specified otherwise, either explicitly with the IPVersion property, or implicitly by recompiling Indy with {$DEFINE IdIPv6} enabled in IdCompilerDefines.inc. Quote from: GetMem on June 28, 2018, 08:32:16 amYou can download the latest version from here: https://svn.atozed.com:444/svn/Indy10 (use a svn client). Or the nightly ZIPPED version of the latest SVN revision at https://indy.fulgan.com/ZIP/. Quote from: scasparz on June 29, 2018, 04:57:37 amBut I still fail to Activate IdTCPServer1 exactly as before with run time message 'RunError(232)'. Did you fix your incorrect Bindings setup, as described above? Quote from: scasparz on June 29, 2018, 04:57:37 amNot to mention that as I write code, I get the message: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Codetools, Errors: 1IdGlobal.pas(1889,105) Error: expected:; but deprecated found. refering to line:GOffsetFromUTC: TDateTime = 0{$IFDEF HAS_DEPRECATED}{$IFDEF USE_SEMICOLON_BEFORE_DEPRECATED};{$ENDIF} deprecated{$ENDIF}; at the message window. Refer to past discussions on this forum about that issue.  The USE_SEMICOLON_BEFORE_DEPRECATED conditional was supposed to address that very issue.  Are you saying it is broken AGAIN?  USE_SEMICOLON_BEFORE_DEPRECATED was specifically added for FPC, which handles that line (and 1 other line, in IdWship6.pas) differently than Delphi does. Quote from: denver on June 29, 2018, 07:32:35 amIn the IdGlobal.pas, I just modify as follow and it work for me.   GOffsetFromUTC: TDateTime = 0 ; deprecated ; Per past discussions, USE_SEMICOLON_BEFORE_DEPRECATED is defined only for FPC 2.4.4 - 2.6.1, as the issue of the semicolon was supposedly fixed in 2.6.2 to allow the same syntax as Delphi (see below) (especially since Indy puts FPC into Delphi mode).  IdCompilerDefines.inc has this code: Code: [Select]{$IFDEF FPC}   ...   {$IFDEF FPC_2_4_4_OR_ABOVE}     ...     {$IFNDEF FPC_2_6_2_OR_ABOVE}       {$DEFINE USE_SEMICOLON_BEFORE_DEPRECATED} // TODO: which earlier versions require a semicolon?     {$ENDIF}   {$ENDIF}   ... {$ENDIF} Quote from: scasparz on June 30, 2018, 02:19:39 amHad never used the deprecated directive before. Tried to experiment a bit, used Lazarus to open a new trivia project where I declared a variable using identical syntax: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  Form1: TForm1;  pap: Integer; deprecated;and Lazarus raised exactly the same error again. Tried this on Delphi XE and once more got exactly the same error with the same message. Delphi, and modern FPC, do not require the extra semicolon: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  pap: Integer deprecated; That is why the semicolon is IFDEF'ed for earlier versions of FPC. Quote from: scasparz on June 30, 2018, 02:19:39 amCould not find some additional documentation either on the Delphi XE help subsystem or on the internet over deprecated. http://docwiki.embarcadero.com/RADStudio/en/Declarations_and_Statement s_(Delphi)#Hinting_Directives Quote from: scasparz on June 30, 2018, 02:19:39 amAm left with the impression -please correct me if wrong- that this directive only works with procedures or methods but not with variables. Hinting directives, like deprecated, work with variables, too.  The only thing they don't work with is properties (sadly, since Indy has a quite few of those). Quote from: scasparz on June 30, 2018, 02:19:39 amMany thanks for your kind help. Without you and GetMem, I simply couldn't have done it. I'm late to this discussion, but keep in mind that I am Indy's primary developer, so if you have any issues with Indy, you can pass them along to me.  Also note that Indy has an issue tracker on GitHub: https://github.com/IndySockets/Indy/issues. ---------------------------------------------------------------------- marcov · 2018-07-02 08:19 Code: [Select]{$mode delphi} var   GOffsetFromUTC: TDateTime =0 deprecated ; begin end. compiles for me in 2.6.2,2.6.4,3.0.2,3.0.4 and trunk ---------------------------------------------------------------------- Remy Lebeau · 2018-07-02 17:18 Quote from: marcov on July 02, 2018, 10:19:10 amCode: [Select]{$mode delphi} var   GOffsetFromUTC: TDateTime =0 deprecated ; begin end. compiles for me in 2.6.2,2.6.4,3.0.2,3.0.4 and trunk Yes, that is the format that Indy uses in 2.6.2+. Unlike with Delphi (which Indy supports all the way back to D5), Indy has little support for past FPC versions.  The goal is to prefer to support only recent versions, since FPC is open-source and relatively easy for users to update at will. ---------------------------------------------------------------------- scasparz · 2018-07-06 02:34 Many thanks for your kind response Remy. As I wrote have downloaded a more recent version of Indy and indeed most of the early issues I had have now been addressed. For example I can now drag the tIdTCPServer from the component palette onto my form without crashing Lazarus, no need to further investigate topic. Again as suggested by Denver have activated cthreads and now the server app does not crash on server activation. Have omitted the Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdTCPClient1.IPVersion := Id_IPv4;clause and indeed had no issues. I understand your suggestion in regard to the newbie mistake and made the relevant correction on the server app, again this worked fine. Also have used: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdTCPClient1.IOHandler.WriteLn(Edit1.Text, IndyTextEncoding_OSDefault);to send a string from the client app and Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LLine := AContext.Connection.IOHandler.ReadLn(IndyTextEncoding_OSDefault);to get the client string on the server app and this works happily with international characters. However replacing IndyTextEncoding_OSDefault with IndyTextEncoding_UTF8 on both apps wouldn’t work. Seems unnecessary when both apps are running under the same OS. Just for completion have tested again the Linux-Lazarus-Indy client app with the Windows-Delphi XE-Indy server app, this time using: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LLine := AContext.Connection.IOHandler.ReadLn(enUTF8);at the windows server and it too worked as a breeze including international characters. But I still have issues each time I disconnect the client from the server. On Linux, both the server and the client app raise an exception. But while on the server app I can handle this with the OnException event, I have failed to find a way to do the same client side. I understand perfectly the reasons you have provided as to why this exception is raised and again I understand this exception is raised from deeply into the Indy code. Have tried a call to IdTCPClient1.CheckForGracefulDisconnect(False) before calling IdTCPClient1.Disconnect without success. Have also tried  IdTCPClient1.Disconnect(False) again with no success. On this topic, one noticeable difference between the Linux-Lazarus- Indy-Server App and the Windows-Delphi XE-Indy-Server app has been that when closing the connection from the Linux-Lazarus-Indy-Client App the same exception is raised on the server side, yet it is not raised on the Lazarus client side. Presume there must be some better way to close the connection from the client both graciously and without raising this exception but have failed to find it. How should I close the client-side connection without all this fuzz? One last question please: I understand the tIdTCPServer has to be multi-threaded as multiple connections are supposed to be made against it from a number of clients. However is the tIdTCPClient multi threaded too? Should I have to make thread-safe calls? As I wrote on my previous posts had considerable difficulties to start with Indy on Lazarus. Things are getting much better after a couple of posts on this forum. Indy is a great package and am glad to notice the more I use it the more I find it workable from Linux/Lazarus as well. Hope to get the same dynamics on Lazarus as it already has with Delphi.  As a last note, have taken a look on the deprecated directive and your syntax is absolutely correct. In that sense, my assumption that deprecated cannot be used on variables was completely wrong. regards s ---------------------------------------------------------------------- marcov · 2018-07-06 09:14 Quote from: Remy Lebeau on July 02, 2018, 07:18:58 pmYes, that is the format that Indy uses in 2.6.2+. Unlike with Delphi (which Indy supports all the way back to D5), Indy has little support for past FPC versions.  The goal is to prefer to support only recent versions, since FPC is open-source and relatively easy for users to update at will. From what I see use below 2.6.4. is rare.   While one occasionally sees old versions they are mostly because of the versions in LTS versions of Linux. Occasionally also win9x holdolds and people that get the version from their education institution. (which don't renew the course every year) ---------------------------------------------------------------------- Remy Lebeau · 2018-07-06 19:02 Quote from: scasparz on July 06, 2018, 04:34:48 amAlso have used: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdTCPClient1.IOHandler.WriteLn(Edit1.Text, IndyTextEncoding_OSDefault);to send a string from the client app and Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LLine := AContext.Connection.IOHandler.ReadLn(IndyTextEncoding_OSDefault);to get the client string on the server app and this works happily with international characters. That entirely depends on the particular charset locale that the OS uses, which is very machine-dependent and user-dependent.  It is really not a good idea to use OSDefault for network communications, especially once you start dealing with communications between multiple machines.  You really should be using UTF-8 instead.  OSDefault is lossy, UTF-8 is not. Quote from: scasparz on July 06, 2018, 04:34:48 amHowever replacing IndyTextEncoding_OSDefault with IndyTextEncoding_UTF8 on both apps wouldn’t work. Works fine for me, on Windows at least.  I don't use Linux.  Maybe there is a bug in Indy's implementation of UTF-8 on Linux? Indy uses platform-specific APIs (Win32 on Windows, libiconv on Linux) for most of its charset handling.  But there are some quirks in how libiconv works, particularly in how Indy calculates the FMaxCharSize and FIsSingleByte members of TIdTextEncodingBase.  So, all I can say is try to debug into Indy's source code and see if TIdUTF8Encoding is misbehaving on Linux. Quote from: scasparz on July 06, 2018, 04:34:48 amSeems unnecessary when both apps are running under the same OS. Even under the same OS, the OSDefault locale may be different between machines.  So you should stay away from OSDefault. Quote from: scasparz on July 06, 2018, 04:34:48 amJust for completion have tested again the Linux-Lazarus-Indy client app with the Windows- Delphi XE-Indy server app, this time using: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LLine := AContext.Connection.IOHandler.ReadLn(enUTF8);at the windows server and it too worked as a breeze including international characters. So the problem is only in reading UTF-8 on Linux, not with sending UTF-8? Quote from: scasparz on July 06, 2018, 04:34:48 amBut I still have issues each time I disconnect the client from the server. On Linux, both the server and the client app raise an exception. That is normal behavior. Quote from: scasparz on July 06, 2018, 04:34:48 amBut while on the server app I can handle this with the OnException event, I have failed to find a way to do the same client side. Because there is no event for it.  You will have to wrap your socket I/O operations inside of try/catch blocks. Quote from: scasparz on July 06, 2018, 04:34:48 amI understand perfectly the reasons you have provided as to why this exception is raised and again I understand this exception is raised from deeply into the Indy code. Have tried a call to IdTCPClient1.CheckForGracefulDisconnect(False) before calling IdTCPClient1.Disconnect without success. Calling CheckForGracefulDisconnect() like that doesn't serve any useful purpose. Quote from: scasparz on July 06, 2018, 04:34:48 amHave also tried  IdTCPClient1.Disconnect(False) again with no success. Disconnect() should not be raising an exception.  Are you saying it does?  Write() and Read...() certainly can. Quote from: scasparz on July 06, 2018, 04:34:48 amOn this topic, one noticeable difference between the Linux-Lazarus-Indy-Server App and the Windows-Delphi XE-Indy-Server app has been that when closing the connection from the Linux-Lazarus-Indy-Client App the same exception is raised on the server side, yet it is not raised on the Lazarus client side. What is the EXACT exception that you are seeing?  From which code?  Can you be more specific? Quote from: scasparz on July 06, 2018, 04:34:48 amPresume there must be some better way to close the connection from the client both graciously and without raising this exception but have failed to find it. How should I close the client-side connection without all this fuzz? Calling Disconnect() is the correct way.  If it is failing, something is wrong that needs to be fixed. Quote from: scasparz on July 06, 2018, 04:34:48 amOne last question please: I understand the tIdTCPServer has to be multi-threaded as multiple connections are supposed to be made against it from a number of clients. However is the tIdTCPClient multi threaded too? No, it is not.  Most of Indy's components are not multi-threaded, they operate in whatever thread your code uses them in.  Only TCP/UDP servers, a few clients (TIdTelnet, TIdCmdTCPClient, TIdIPMCastClient) and a couple of utility components (TIdIPWatch, TIdIPAddrMon) are multi-threaded. Quote from: scasparz on July 06, 2018, 04:34:48 amShould I have to make thread-safe calls? That entirely depends on how you use it in your code. ---------------------------------------------------------------------- scasparz · 2018-07-13 03:04 Remy sorry for being late to answer, had an awful lot of homework to do. In regard to the on disconnect exception, which I believed it was raised also client-side it was my fault. It does not. I used to execute two Lazarus instances on separate workspace screens, one for the server and one for the client app. I used to be on client workspace disconnecting the client from the server, this exception was risen on the server only, but Linux brought it to me at the client workspace, atop the client app, which made me believe erroneously it originated from the client rather than from the server. My bad, apologies for this confusion. Was exactly this the reason why I though I could not see this exception client wise, when it was disconnecting from a virtual windows server running Delphi-Indy. Apparently there are no issues there. And client-wise Disconnect() is working fine to close a connection, nothing else is actually required, everything is by the book here. Looks like enUTF8 appears to be deprecated now, Indy prefers the IndyTextEncoding_*() functions instead, at least on Linux. I understand the use of IndyTextEncoding_UTF8() should work for Linux and that it is the choice of preference, however for some reason it seems it doesn’t not work. Looks the problem arises client side, as it is the client app that freezes while the server one appears as not getting anything. Have tried a number of permutations, still the only way I managed to pass international characters on Linux has been with the use of IndyTextEncoding_OSDefault() on both the client and the server app. Yes, it may not really be a good idea, still it is the only working one I could find on Linux. Have managed to pass a Stream, had a lot of difficulties, until I read the example at https://github.com/tinydew4/indy-project-demos/tree/mas ter/TCPIP%20Delphi%20%26%20Indy10%20Client%20Server%20Demo/4_sample%20 Stream%20Exchange, and realize the use of buffered writes. One very simple question in case you have the time: It is my understanding that to pass a stream I have to start by sending server the stream size first, then the actual stream so that the server will know the size of the stream to read. In the example that saved my life, the author sends the size, then he ignites a buffered write by calling WriteBufferOpen that is before he sends the stream itself. Is this the correct way? Or should I have to make a call to WriteBufferOpen before I do anything else, such as this? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   IdTCPClient1.IOHandler.WriteBufferOpen;  IdTCPClient1.IOHandler.Write(MemoryStream.Size); // send size  IdTCPClient1.IOHandler.Write(MemoryStream, MemoryStream.Size); // send stream  IdTCPClient1.IOHandler.WriteBufferFlush;   IdTCPClient1.IOHandler.WriteBufferClose; // close buffered write I appreciate you bothered to answer multiple questions from the ignorant me. Many thanks again. kind regards s ---------------------------------------------------------------------- denver · 2018-07-13 06:02 Quote from: scasparz on July 13, 2018, 05:04:57 am In regard to the on disconnect exception, .... You have to try to handle the IdTCPServer1Disconnect(AContext: TIdContext) event to prevent disconnect except. If you want to send any kind of data between TIdTCPServer and TidTCPClient, I have a very simple component that work for you. It passes Strings, Integer, DateTime, Blob,even dataset ( Modified TBufDataSet ) between  TIdTCPServer and TidTCPClient at the following link : https://github.com/denverjen/LazarusMiddleWare Happy programming ... ---------------------------------------------------------------------- scasparz · 2018-07-13 18:52 Thanks denver. Have already downloaded it. Will definitely give it a shot! kind regards s ---------------------------------------------------------------------- Remy Lebeau · 2018-07-13 20:29 Quote from: scasparz on July 13, 2018, 05:04:57 amLooks like enUTF8 appears to be deprecated now, Indy prefers the IndyTextEncoding_*() functions instead, at least on Linux. On all platforms. Quote from: scasparz on July 13, 2018, 05:04:57 amI understand the use of IndyTextEncoding_UTF8() should work for Linux and that it is the choice of preference, however for some reason it seems it doesn’t not work. Not sure why.  Again, can you please test it?  You don't even need to make a TCP connection, simply use IndyTextEncoding_UTF8() by itself and encode/decode some strings with it.  Make sure everything works correctly on Linux. In any case, I have just now checked in some tweaks to how Indy's TIdUTF8Encoding class is initialized.  See if it makes any difference. Quote from: scasparz on July 13, 2018, 05:04:57 amLooks the problem arises client side, as it is the client app that freezes while the server one appears as not getting anything. That implies to me that you are performing your socket I/O in the main UI thread (client side), and that are not correctly reading whatever you send.  Sends and Reads needs to be matched up correctly, so the receiver doesn't block waiting for data the sender does not send. Quote from: scasparz on July 13, 2018, 05:04:57 amHave managed to pass a Stream, had a lot of difficulties, until I read the example at https://github.com/tinydew4/indy-project-demos/tree/master/TCPIP%20Del phi%20%26%20Indy10%20Client%20Server%20Demo/4_sample%20Stream%20Exchan ge, and realize the use of buffered writes. You really should not use buffered writes when sending streams, especially large streams.  That is a good way to waste a lot of memory unnecessarily. Quote from: scasparz on July 13, 2018, 05:04:57 amIt is my understanding that to pass a stream I have to start by sending server the stream size first, then the actual stream so that the server will know the size of the stream to read. You don't HAVE to do it that way, but it is the RECOMMENDED way, yes. Quote from: scasparz on July 13, 2018, 05:04:57 amIn the example that saved my life, the author sends the size, then he ignites a buffered write by calling WriteBufferOpen that is before he sends the stream itself. Is this the correct way? I would not use buffered writes, no.  But regardless of buffering, yes, the general approach of sending the stream size first is a good choice.  However, you don't need to send the stream size manually, TIdIOHandler.Write(TStream) can send it for you: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdTCPClient1.IOHandler.LargeStream := ...; // False to send the size as a 32bit integer, True for 64bit integerIdTCPClient1.IOHandler.Write(MemoryStream, 0, True); // send size + stream Unless the formatting of the stream size will not match your communication protocol, in which case you would have to send it manually as needed. ---------------------------------------------------------------------- Remy Lebeau · 2018-07-13 20:33 Quote from: denver on July 13, 2018, 08:02:06 amYou have to try to handle the IdTCPServer1Disconnect(AContext: TIdContext) event to prevent disconnect except. That is not true at all.  An exception is going to happen whether you handle the OnDisconnect event or not.  But, if an exception DOES occur, and the OnExecute event doesn't catch it, the OnDisconnect event will still be called as expected (followed by the OnException event). ---------------------------------------------------------------------- scasparz · 2018-07-14 23:07 QuoteNot sure why. Again, can you please test it? You don't even need to make a TCP connection, simply use IndyTextEncoding_UTF8() by itself and encode/decode some strings with it.  Make sure everything works correctly on Linux.Sorry Remy, have taken a look there http://packages.lazarus-ide.org, where it claims to provide the latest Indy version 10.6.2.5457, which is exactly the one I currently use. By the way both the tIdTCPClient as well as tIdTCPServer report their corresponding versions to be 10.6.2.0 instead. Anyway whatever this version may be there is not much I have not tried so far, on this topic. I presume perhaps you expect me to download the latest version from https://svn.atozed.com:444/svn/Indy10. Have no previous experience with SVN. Have taken a look on synaptic -the debian source of heaven-, have found a number of SVN packages, do not know which one is the correct one to use. Not to mention that having tried to get on this page through http, I was asked for a user name and password that I could not provide. I ‘d be more than just happy to try the latest patches had I know the way to. At the moment am afraid I simply cannot get the latest updates. Other than that have tried non-buffered writes successfully. Thanks for the LargeStream property, which I was not aware of, am using the 64-bit version, had no issues without LargeStream by using: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   IdTCPClient1.IOHandler.Write(MemoryStream.Size);   IdTCPClient1.IOHandler.Write(MemoryStream, MemoryStream.Size); at the client and Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   VLOC_MemoryStreamSize := AContext.Connection.IOHandler.ReadInt64();  AContext.Connection.IOHandler.ReadStream(MemoryStream, VLOC_MemoryStreamSize);on the server side. As for the OnDisconnect server side exception, seems reasonable to me and I can live with it, as is. s ---------------------------------------------------------------------- Remy Lebeau · 2018-07-16 19:02 Quote from: scasparz on July 15, 2018, 01:07:53 amSorry Remy, have taken a look there http://packages.lazarus-ide.org, where it claims to provide the latest Indy version 10.6.2.5457, which is exactly the one I currently use. By the way both the tIdTCPClient as well as tIdTCPServer report their corresponding versions to be 10.6.2.0 instead. FYI, the latest version in Indy's SVN is actually 10.6.2.5464, so OPM is a few revisions behind.  But either way, the version number is actually stored as 10.6.2.0 in the source code, and has to be overwritten with a command-line script before compiling.  I don't think OPM does that, though. Quote from: scasparz on July 15, 2018, 01:07:53 amAnyway whatever this version may be there is not much I have not tried so far, on this topic. But, did you try what I ASKED you to try?  Have you tested IndyTextEncoding_UTF8() BY ITSELF to make sure it encodes/decodes UTF-8 strings correctly on Linux in general? Quote from: scasparz on July 15, 2018, 01:07:53 amI presume perhaps you expect me to download the latest version from https://svn.atozed.com:444/svn/Indy10. Have no previous experience with SVN. You can download a nightly snapshot from https://indy.fulgan.com/ZIP/ instead. Quote from: scasparz on July 15, 2018, 01:07:53 amHave taken a look on synaptic -the debian source of heaven-, have found a number of SVN packages, do not know which one is the correct one to use. Not to mention that having tried to get on this page through http, I was asked for a user name and password that I could not provide. http://www.indyproject.org/Sockets/Download/DevSnapshot.aspx The username for Indy's SVN is "Indy-Public-RO".  There is no password. ---------------------------------------------------------------------- scasparz · 2018-07-17 09:30 Many thanks Remy, I managed to download the latest version (10.6.2.5465) from https://indy.fulgan.com/ZIP/ as suggested. Unistalled the package I had so far (10.6.2.5457) and attempted to install the new one. This new package has a different directory structure from the previous one. On adding package indylaz.lpk I got errors coming from unit IdSSLOpenSSL: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdSSLOpenSSL.pas(3304,7) Error: Identifier not found "M"IdSSLOpenSSL.pas(3305,13) Error: ENDIF without IF(N)DEFIdSSLOpenSSL.pas(3306,7) Fatal: Syntax error, ";" expected but "identifier SSL_DEFAULT_CIPHER_LIST" foundas a result of which installation failed. Reversed back to the old version, but replaced unit IdGlobal.pas that contains the IndyTextEncoding_*() functions with that of the new version. Reinstalled patched old Indy version sans issues. Then tested: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i at the clientIdTCPClient1.IOHandler.WriteLn(Edit1.Text, IndyTextEncoding_UTF8()); at the serverLLine := AContext.Connection.IOHandler.ReadLn(IndyTextEncoding_UTF8());and failed. Much like with the previous version the client app freezes while apparently the server one does not get anything. Tested again, this time using NIL at the server only, still the same. Tested the opposite permutation with NIL on the client and IndyTextEncoding_UTF8() on the server, again came with nothing. Can it be that the IndyTextEncoding_UTF8() function actually depends on one of the numerous include files to be found at the IdGlobal unit? If yes then please advise me which one(s) and I could give it another run. How am I supposed to test IndyTextEncoding_UTF8() by itself given that apparently this function does not bear any arguments while resulting an interface? Have never worked with interfaces before. Could you please provide me with a syntactic example of what you expect me to try? kind regards s ---------------------------------------------------------------------- Remy Lebeau · 2018-07-17 18:55 Quote from: scasparz on July 17, 2018, 11:30:53 amThis new package has a different directory structure from the previous one. Actually, what is in the nightly ZIP is the official folder structure.  It is Lazarus that doesn't use the standard structure. You can just replace the Lazarus source files with the official source files in whatever folder structure Lazarus uses. Quote from: scasparz on July 17, 2018, 11:30:53 amOn adding package indylaz.lpk I got errors coming from unit IdSSLOpenSSL: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i IdSSLOpenSSL.pas(3304,7) Error: Identifier not found "M"IdSSLOpenSSL.pas(3305,13) Error: ENDIF without IF(N)DEFIdSSLOpenSSL.pas(3306,7) Fatal: Syntax error, ";" expected but "identifier SSL_DEFAULT_CIPHER_LIST" foundas a result of which installation failed. That was my bad.  It has been fixed now. Quote from: scasparz on July 17, 2018, 11:30:53 amReversed back to the old version, but replaced unit IdGlobal.pas that contains the IndyTextEncoding_*() functions with that of the new version. Reinstalled patched old Indy version sans issues. Then tested: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i at the clientIdTCPClient1.IOHandler.WriteLn(Edit1.Text, IndyTextEncoding_UTF8()); at the serverLLine := AContext.Connection.IOHandler.ReadLn(IndyTextEncoding_UTF8());and failed. (sigh) Can you please test IndyTextEncoding_UTF8() BY ITSELF, NO TCP COMPONENTS AT ALL?  For example, something like this: Code: [Select]var   b: TIdBytes; begin   b := IndyTextEncoding_UTF8.GetBytes(Edit1.Text);   // verify bytes are properly UTF-8 encoded...   Edit2.Text := IndyTextEncoding_UTF8.GetString(b); end; Or this: Code: [Select]var   b: TIdBytes; begin   b := ToBytes(Edit1.Text, IndyTextEncoding_UTF8);   // verify bytes are properly UTF-8 encoded...   Edit2.Text := BytesToString(b, IndyTextEncoding_UTF8); end; Which is roughly what TIdIOHandler actually does internally. Quote from: scasparz on July 17, 2018, 11:30:53 amMuch like with the previous version the client app freezes while apparently the server one does not get anything. Tested again, this time using NIL at the server only, still the same. Tested the opposite permutation with NIL on the client and IndyTextEncoding_UTF8() on the server, again came with nothing. Sniff the TCP traffic with a packet sniffer, such as Wireshark.  Does the client actually transmitting the string data?  Does the server actually receiving the data? There are a lot of steps involved in string handling and TCP transmissions.  Can't you debug into Indy's source code at runtime at all?  That would eliminate so much of this guesswork if you could actually isolate with a debugger the exact code that is freezing. Quote from: scasparz on July 17, 2018, 11:30:53 amCan it be that the IndyTextEncoding_UTF8() function actually depends on one of the numerous include files to be found at the IdGlobal unit? On Linux, Indy's TextEncoding classes rely on the iconv library.  The usage of that library is all contained in IdGlobal.pas, using the library definitions from FreePascal's own 'iconvenc' unit. ---------------------------------------------------------------------- scasparz · 2018-07-17 22:53 Hi again, Downloaded version Indy10_5465.zip from https://indy.fulgan.com/ZIP/ again and had exactly the same installation issues I had on the previous attempt. You may find more information on this at my previous post. Anyway intend to give it another try as soon as I will be able to have a newer version installed. Reversed back to the old working Indy version, unpatched -that is with the original IdGlobal.pas unit coming with it- and tried both examples you provided. The only Indy unit I used has been IdGlobal.pas. To do this had to include the Indy path at the project options - paths. The first example worked fine, was able to test the entire Greek alphabet without issues. The second one did not work at all. All EditBox Edit2 could get was an empty string. Used the debugger on either procedures. On the first example variable b was populated after the execution of the statement: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i b := IndyTextEncoding_UTF8.GetBytes(Edit1.Text);On the second example variable b remained empty after a call to the statement: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i b := ToBytes(Edit1.Text, IndyTextEncoding_UTF8);The full execution path (in F7 steps) of the second example was: Unit IdGlobal procedure IndyTextEncoding_UTF8, obviously it was this resulting the empty string bfile control.inc procedure TControl.GetText: TCaption which apparently is irrelevant to the issuethen the statemennt Edit2.Text := BytesToString(b, IndyTextEncoding_UTF8); bringing the empty string on Edit2.In both examples, executing procedure IndyTextEncoding_UTF8(), Indy finds GIdUTF8Encoding to be NIL before proceeding with the statement: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Result := GIdUTF8Encoding; Hope it helps. Understand this problem keeps everyone busy, but can you tell the ignorant me if I can use the tIdContrext.Data property which is a tObject to store my local variables into the connection thread (eg the one to store the incoming stream and its size and perhaps any manipulative methods) or should I keep it off limits? At your service with any suggestions you make. ---------------------------------------------------------------------- Remy Lebeau · 2018-07-18 20:40 Quote from: scasparz on July 18, 2018, 12:53:45 amDownloaded version Indy10_5465.zip from https://indy.fulgan.com/ZIP/ again and had exactly the same installation issues I had on the previous attempt. You may find more information on this at my previous post. Anyway intend to give it another try as soon as I will be able to have a newer version installed. See http://wiki.freepascal.org/Indy_with_Lazarus Quote from: scasparz on July 18, 2018, 12:53:45 amThe first example worked fine, was able to test the entire Greek alphabet without issues. The second one did not work at all. All EditBox Edit2 could get was an empty string. Hmm... Quote from: scasparz on July 18, 2018, 12:53:45 amUsed the debugger on either procedures. On the first example variable b was populated after the execution of the statement: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i b := IndyTextEncoding_UTF8.GetBytes(Edit1.Text); And, was it populated *correctly*?  Did you verify that the bytes were actually encoded using proper UTF-8 for Greek? Quote from: scasparz on July 18, 2018, 12:53:45 amOn the second example variable b remained empty after a call to the statement: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i b := ToBytes(Edit1.Text, IndyTextEncoding_UTF8); Hmm, ideally it should have output the same bytes as the first example, since ToBytes() will use IndyTextEncoding_UTF8 internally. However, there is a small but important difference between the two examples: If you are using FreePascal in a mode where the 'string' type maps to 'AnsiString', ToBytes() will explicitly convert the input string from ANSI to UTF-16 using IndyTextEncoding_OSDefault (unless you specify otherwise via the optional ASrcEncoding parameter), and then convert the UTF-16 to UTF-8 using IndyTextEncoding_UTF8.  So, my ToBytes() example would be more on-par with this example for an 'AnsiString': Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  b: TIdBytes;  s: AnsiString;  ch: TIdWideChars;begin  //b := ToBytes(Edit1.Text, IndyTextEncoding_UTF8);  s := Edit1.Text;  b := RawToBytes(PAnsiChar(s)^, Length(s));  ch := IndyTextEncoding_OSDefault.GetChars(b);  b := IndyTextEncoding_UTF8.GetBytes(ch);  ...end;  So, data could be getting lost in either the call to IndyTextEncoding_OSDefault.GetChars() or the call to IndyTextEncoding_UTF8.GetBytes(). Whereas my first example, passing Edit1.Text directly to IndyTextEncoding.GetBytes(), relies on the compiler implicitly converting an 'AnsiString' to a '(Wide|Unicode)String' using its own native ANSI->Unicode conversion. When 'string' maps to 'UnicodeString' instead, then only IndyTextEncoding_UTF8 is involved when calling ToBytes(), eg: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  s: UnicodeString;  b: TIdBytes;  len: Integer;begin  //b := ToBytes(Edit1.Text, IndyTextEncoding_UTF8);  s := Edit1.Text;  len := IndyTextEncoding_UTF8.GetByteCount(s, 1, Length(s));  SetLength(b, len);  if len > 0 then begin    IndyTextEncoding_UTF8.GetBytes(s, 1, Length(s), b, 0);  end;  ...end;  Quote from: scasparz on July 18, 2018, 12:53:45 amThe full execution path (in F7 steps) of the second example was: Unit IdGlobal procedure IndyTextEncoding_UTF8, obviously it was this resulting the empty string b But, that doesn't tell me WHERE inside of Indy the data is getting lost exactly.  I need you to step INSIDE of Indy's source code with the debugger, find out EXACTLY what happens at each step in the conversion process. Quote from: scasparz on July 18, 2018, 12:53:45 amIn both examples, executing procedure IndyTextEncoding_UTF8(), Indy finds GIdUTF8Encoding to be NIL before proceeding with the statement: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Result := GIdUTF8Encoding; As it should be, since GIdUTF8Encoding is initially nil.  GIdUTF8Encoding is assigned to a TIdUTF8Encoding object the first time IndyTextEncoding_UTF8() is called. Quote from: scasparz on July 18, 2018, 12:53:45 amHope it helps. Not really. Quote from: scasparz on July 18, 2018, 12:53:45 amcan you tell the ignorant me if I can use the tIdContext.Data property which is a tObject to store my local variables into the connection thread (eg the one to store the incoming stream and its size and perhaps any manipulative methods) or should I keep it off limits? Yes, you can use the TIdContext.Data property like that.  The Data property is specifically intended for users to store whatever per- connection context data they want (similar to what the TComponent.Tag property for integers). Though, it is typically better to derive a new class from TIdServerContext, put whatever you want in that class, and then assign that class type to the server's ContextClass property before activating the server. ---------------------------------------------------------------------- scasparz · 2018-07-19 04:47 QuoteQuote from: scasparz on July 18, 2018, 12:53:45 am Hope it helps. Not really.Sorry Remy, am trying to do my best while am not a system developer. Quotehttp://wiki.freepascal.org/Indy_with_LazarusAgain thanks for the information. Will use it next time. Took a look for a newer nightly at https://indy.fulgan.com/ZIP/, seems it is now Forbidden, while I have been there multiple times. No other options for the moment other than keep using version 10.6.2.5457 unpatched. QuoteAnd, was it populated *correctly*?  Did you verify that the bytes were actually encoded using proper UTF-8 for Greek?Testing was done with the 'αβγδεζηθικλμνξοπρστυφχψω ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ άέήίόύώ ϊϋ ΐΰ' which is the entire Greek character set on the Edit1 editbox. All am trying to say is that what I got on Edit2 was a facsimile replica of what I had typed on Edit1. Plus Edit1.Text and Edit2.Text compare identical. No corruptions, no losses, everything came as it should. Looks example 1 works fine. If you want me to try something different with it please ask. QuoteBut, that doesn't tell me WHERE inside of Indy the data is getting lost exactly.  I need you to step INSIDE of Indy's source code with the debugger, find out EXACTLY what happens at each step in the conversion process.I can absolutely understand you here, however F7 is the most detailed granularity with the debugger that I know. If there is a way to increase this further please advise me on how to do this. Meanwhile all I could do was to set a breakpoint at statement: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i b := ToBytes(Edit1.Text, IndyTextEncoding_UTF8); //(1)then stepped forward with F7 and I was send to the first statement of the IndyTextEncoding_UTF8() on unit IdGlobal.pas. There has been no Indy code before my eyes other than that am afraid. And yes, I was expecting to see something related with the ToBytes function afterwards, but IMHO either this has not been transparent with the debugger or for some reason that I ignore this call to the ToBytes was not materialized. The value of the Result variable is the same ( = ) on both examples at least with Greek characters input. Ran example 2 with Latin chars, actually it contained the default editbox 'Edit1' text which I did not alter. Looks this time the execution path was different though: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LEncoding := TIdUTF8Encoding.Create; constructor TIdUTF8Encoding.Create;TIdMBCSEncoding.Create;constructor TIdMBCSEncoding.Create(const CharSet: String);function PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer;function TextIsSame(const A1, A2: string): Boolean; resulting Falsefunction PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer;function TextIsSame(const A1, A2: string): Boolean;function PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer;function TextIsSame(const A1, A2: string): Boolean;function PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer;function TextIsSame(const A1, A2: string): Boolean;function PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer;function TextIsSame(const A1, A2: string): Boolean;function PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer;function TextIsSame(const A1, A2: string): Boolean;function PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer;function TextIsSame(const A1, A2: string): Boolean;function PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer;function TextIsSame(const A1, A2: string): Boolean;function PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer; exited with -1back to constructor TIdMBCSEncoding.Create(const CharSet: String);function TIdUTF7Encoding.GetByteCount(const AChars: PIdWideChar; ACharCount: Integer): Integer;function TIdMBCSEncoding.GetByteCount(const AChars: PIdWideChar; ACharCount: Integer): Integer;function DoIconvCharsToBytes(const ACharset: string; AChars: PIdWideChar; ACharCount: Integer;function CreateIconvHandle(const ACharSet: String; AToUTF16: Boolean): iconv_t;from where I got a dialog box with message The file "iconv_open.c" was not found. Do you want to locate it yourself? Answered no, which brought me the assembler, then most probably I lost him while trying to maintain this log. Am with the impression Indy was inspecting the 'Edit1' string character by character, chances are I have missed an iteration. Anyway Remy this is French to me and I mean I do not speak French. You have been extremely helpful from the very beginning, would at least be unfair to not cooperate with you fully. Correcting Indy is important to either of us. Am still at your service mate with whatever suggestions you have. Run it for the second time without changing the Edit1 contents. This time the execution path was different though. Instead of all that fuzz, there has been no call to the Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LEncoding := TIdUTF8Encoding.Create; at the IndyTextEncoding_UTF8() function, however Edit2 contents remained correct. Many thanks for your reply on the TIdContext.Data property. Actually deriving a new tIdContext descendant and employ it through the use of the TIdServerContextClass was what I had in mind. PS. https://indy.fulgan.com/ZIP/ came back just before I was to send this post. Will try latest version 5468 as soon as I can find some time. ---------------------------------------------------------------------- Remy Lebeau · 2018-07-19 17:14 Quote from: scasparz on July 19, 2018, 06:47:45 amTook a look for a newer nightly at https://indy.fulgan.com/ZIP/, seems it is now Forbidden, while I have been there multiple times. Works fine for me.  You might have been accessing it during maintenance time, or while it was syncing with Indy's SVN, or something like that. Quote from: scasparz on July 19, 2018, 06:47:45 amNo other options for the moment other than keep using version 10.6.2.5457 unpatched. You can always download files from Indy's SVN directly, or one of its mirrors: http://www.indyproject.org/Sockets/Download/DevSnapshot.aspx Quote from: scasparz on July 19, 2018, 06:47:45 amTesting was done with the 'αβγδεζηθικλμνξοπρστυφχψω ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ άέήίόύώ ϊϋ ΐΰ' which is the entire Greek character set on the Edit1 editbox. The *correct* UTF-8 byte sequence for that string of Greek characters is: Code: [Select]CE B1 CE B2 CE B3 CE B4 CE B5 CE B6 CE B7 CE B8 CE B9 CE BA CE BB CE BC CE BD CE BE CE BF CF 80 CF 81 CF B3 CF 84 CF 85 CF 86 CF 87 CF 88 CF 89 20 CE 91 CE 92 CE 93 CE 94 CE 95 CE 96 CE 97 CE 98 CE 99 CE 9A CE 9B CE 9C CE 9D CE 9E CE 9F CE A0 CE A1 CE A3 CE A4 CE A5 CE A6 CE A7 CE A8 CE A9 20 CE AC CE AD CE AE CE AF CF 8C CF 8D CF 8E 20 CF 8A CF 8B 20 CE 90 CE B0 Is that what IndyTextEncoding_UTF8.GetBytes() produces?  And if those exact bytes are passed to IndyTextEncoding_UTF8.GetString(), does it return the original string ('αβγδεζηθικλμνξοπρστυφχψω ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ άέήίόύώ ϊϋ ΐΰ') as expected? Quote from: scasparz on July 19, 2018, 06:47:45 amF7 is the most detailed granularity with the debugger that I know. If there is a way to increase this further please advise me on how to do this. Sounds like you don't have Indy compiled with debug info enabled.  Try recompiling it.  Then hopefully the debugger will let you step into the source code. Quote from: scasparz on July 19, 2018, 06:47:45 amRan example 2 with Latin chars, actually it contained the default editbox 'Edit1' text which I did not alter. Looks this time the execution path was different though: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LEncoding := TIdUTF8Encoding.Create; constructor TIdUTF8Encoding.Create;TIdMBCSEncoding.Create;constructor TIdMBCSEncoding.Create(const CharSet: String);... snip 8 TextIsSame() comparisons ...function PosInStrArray(const SearchStr: string; const Contents: array of string; const CaseSensitive: Boolean = True): Integer; exited with -1  Please make sure you are using the absolute latest version of IdGlobal.pas, whatever it takes to get installed.  In the older version you are testing with, the PosInStrArray() call inside of the TIdMBCSEncoding constructor compares 8 charset names .  In the latest version, it compares 16 names (of which 'UTF-8' is now one of them).  I had made changes to how TIdMBCSEncoding and its descendants calculate the FMaxCharSize member, I need you to be testing with the latest version, or we are just going to end up going around in circles. Quote from: scasparz on July 19, 2018, 06:47:45 amCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function TIdUTF7Encoding.GetByteCount(const AChars: PIdWideChar; ACharCount: Integer): Integer;  FYI, in the latest version, TIdUTF8Encoding no longer derives from TIdUTF7Encoding. Quote from: scasparz on July 19, 2018, 06:47:45 amfrom where I got a dialog box with message The file "iconv_open.c" was not found. Do you want to locate it yourself? The debugger is trying to step into the ICONV library source code.  That is not necessary in this situation, so just ignore that. Quote from: scasparz on July 19, 2018, 06:47:45 amRun it for the second time without changing the Edit1 contents. This time the execution path was different though. Instead of all that fuzz, there has been no call to the Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LEncoding := TIdUTF8Encoding.Create; at the IndyTextEncoding_UTF8() function, however Edit2 contents remained correct. If you ran the conversion a second time in the same process, then yes, because the GIdUTF8Encoding object has already been created and doesn't need to be created again. If you really wanted to re-create a new UTF8 object on each test, you could use the (deprecated) IndyUTF8Encoding() function, setting its AOwnedByIndy parameter to False. ---------------------------------------------------------------------- scasparz · 2018-07-19 22:31 Downloaded and installed latest nightly Indy10-10.6.2.5468 sans issues. Run the transliteration app again and got exactly the same results as before. The quick answer is that example 1 passes with flying colours while example 2 fails. Tried to log execution path again for the second example, focusing on the first run only, that seems the run which matters. Noticed GDB behaves erratically. Jumps mid-subprograms omitting whatever lies before rather than starting from their beginning, or jumps over grayed code -presume disabled- etc. none of which makes any sense to me. This was not like yesterday when at least GDB seemed to behave itself. Me believes this affects Indy code only, as GDB behaviour on Lazarus code appears to be the usual. Do you think I had to compile Indy twice as with older versions? Suspect but am far from being certain somehow my Lazarus installation could be corrupted, perhaps by installing or unistalling a number of Indy’s versions and rebuilding Lazarus each and every time. Intend to purge Lazarus and attempt a fresh installation in the hope of fixing these issues. Will come back as soon as I have something. ---------------------------------------------------------------------- Remy Lebeau · 2018-07-20 00:33 Quote from: scasparz on July 20, 2018, 12:31:19 amDownloaded and installed latest nightly Indy10-10.6.2.5468 sans issues. Run the transliteration app again and got exactly the same results as before. The quick answer is that example 1 passes with flying colours while example 2 fails. I've already explained the difference between the 1st and 2nd examples.  You haven't addresses any of the issues I've described.  Which mode are using FreePascal in?  Does the 'string' type map to AnsiString or UnicodeString? If AnsiString, did you do any tests using the extended examples I have shown where IndyTextEncoding_OSDefault is used as an intermediate conversion? Quote from: scasparz on July 20, 2018, 12:31:19 amDo you think I had to compile Indy twice as with older versions? I have no idea.  I don't have FreePascal/Lazarus installed, let alone tried to ever install Indy in it. ---------------------------------------------------------------------- scasparz · 2018-07-20 18:55 To start with Lazarus maps String to AnsiString through the omnipresent default directive {$H+} at least on Linux-64. Have given it a another try on a vmachine this time. GDB seems to be working now. For some reason GDB would not take me to the ToBytes function using F7 during the runs of the previous days. This time I set an explicit breakpoint and was the first time I saw the execution of this code. Suspect maybe this is where the problem lies, for IndyTextEncoding_UTF8() seems to be working fine with the syntax of the example1. Tested with Indy 10.6.2.5468 using string 'αβ' on editbox Edit1. Below only the numbered // (x) lines were executed, have added a few comments, hope they help this time. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function ToBytes(const AValue: string; const ALength: Integer; const AIndex: Integer = 1;  ADestEncoding: IIdTextEncoding = nil  {$IFDEF STRING_IS_ANSI}; ASrcEncoding: IIdTextEncoding = nil{$ENDIF}  ): TIdBytes; overload;var  LLength: Integer;  {$IFDEF STRING_IS_ANSI}  LBytes: TIdBytes;  {$ENDIF}begin  {$IFDEF STRING_IS_ANSI}  LBytes := nil; // keep the compiler happy  // (0)  {$ENDIF}  LLength := IndyLength(AValue, ALength, AIndex); // (1), LLength becomes 4  if LLength > 0 then // (2)  begin    EnsureEncoding(ADestEncoding); // (3) ADestEncoding = IUNKNOWN, GDB won't let me F7 this    {$IFDEF STRING_IS_UNICODE}    SetLength(Result, ADestEncoding.GetByteCount(AValue, AIndex, LLength));    if Length(Result) > 0 then begin      ADestEncoding.GetBytes(AValue, AIndex, LLength, Result, 0);    end;    {$ELSE}    EnsureEncoding(ASrcEncoding, encOSDefault); // (4) ASrcEncoding = IUNKNOWN    LBytes := RawToBytes(AValue[AIndex], LLength); // (5) LBytes becomes (206,177,206,178)    CheckByteEncoding(LBytes, ASrcEncoding, ADestEncoding); // (6) LBytes becomes ()    Result := LBytes; // (7)    {$ENDIF}  end else begin    SetLength(Result, 0);  end;end; Statement of interest seems to be #6. Before the call LBytes is (206,177,206,178), while ASrcEncoding as well as ADestEncoding are valued IUNKNOWN. After this statement LBytes is given the () value. Have F7ed CheckByteEncoding as well. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure CheckByteEncoding(var VBytes: TIdBytes; ASrcEncoding, ADestEncoding: IIdTextEncoding);begin  if ASrcEncoding <> ADestEncoding then begin // according to the GDB both ASrcEncoding and ADestEncoding are valued IUNKNOWN    VBytes := ADestEncoding.GetBytes(ASrcEncoding.GetChars(VBytes)); // (8) this statement is executed, should it?  end;end; Both ASrcEncoding as well as ADestEncoding are of the IIdTextEncoding type, which I understand is an interface. Have no clues on how do interfaces compare, do not know the theory behind. Would expect statement #8 to not be executed. Anyway #8 changes the value of VBytes into (), finally resulting a blanc string. Cannot F7 statement #8 of course. ---------------------------------------------------------------------- Remy Lebeau · 2018-07-21 00:37 Quote from: scasparz on July 20, 2018, 08:55:16 pmTo start with Lazarus maps String to AnsiString through the omnipresent default directive {$H+} at least on Linux-64. Perhaps.  Just note that Indy also supports when 'string' maps to 'UnicodeString' via {$ModeSwitch UnicodeStrings} or {$Mode DelphiUnicode}.  Various properties and function signatures in Indy change depending on whether AnsiString or UnicodeString is being used. IIRC, doesn't Lazarus use UTF-8 encoded AnsiStrings, though?  Would that also apply to strings from the UI?  You need to take a string's encoding into account when calling Indy's ToBytes() and BytesToString() functions when dealing with AnsiStrings.  You need to tell them that input/output strings are encoded in UTF-8 and not in the OS's default locale, via either: - the ASrcEncoding parameter of ToBytes(), and the ADestEncoding parameter of BytesToString(). - the ASrcEncoding parameter of TIdIOHandler.WriteLn(), and the ADestEncoding parameter of TIdIOHandler.ReadLn(). - the DefAnsiEncoding property of TIdIOHandler, which WriteLn() and ReadLn() default to if no encoding is passed in explicitly. - the global GIdDefaultTextEncoding variable in the IdGlobal.pas unit. Quote from: scasparz on July 20, 2018, 08:55:16 pmFor some reason GDB would not take me to the ToBytes function using F7 during the runs of the previous days. Again, did you recompile Indy with debug info enabled? Quote from: scasparz on July 20, 2018, 08:55:16 pmThis time I set an explicit breakpoint and was the first time I saw the execution of this code. Suspect maybe this is where the problem lies, for IndyTextEncoding_UTF8() seems to be working fine with the syntax of the example1. As I've already explained earlier, I don't expect any problems with example 1, because the IIdTextEncoding.GetBytes() method takes a UnicodeString as input, not a generic String.  So, if you pass in an AnsiString as input to GetString(), the compiler/RTL - not Indy - will convert the string to Unicode before GetBytes() is even entered.  And in the context of Lazarus, with its UTF-8 encoded AnsiStrings, I would expect that conversion to take UTF-8 into account, thus no data loss.  Same with the IIdTextEncoding.GetString() method, which returns a UnicodeString, which the compiler/RTL - not Indy - will convert when assigned to an AnsiString. But example 1 is not what TIdIOHandler does internally, example 2 is, and that is the one that is having issues, likely because of the use of IndyTextEncoding_OSDefault as an intermediate conversion. Quote from: scasparz on July 20, 2018, 08:55:16 pmThis time I set an explicit breakpoint and was the first time I saw the execution of this code. OK, now we are finally getting somewhere useful... Quote from: scasparz on July 20, 2018, 08:55:16 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   {$IFDEF STRING_IS_ANSI}  LBytes := nil; // keep the compiler happy  // (0)  {$ENDIF}  We've established that 'string' is 'AnsiString'... Quote from: scasparz on July 20, 2018, 08:55:16 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   LLength := IndyLength(AValue, ALength, AIndex); // (1), LLength becomes 4  And that 'string' is UTF-8 encoded ('αβ' is 4 bytes when encoded in UTF-8)... In which case, you need to either: - set the ASrcEncoding parameter of ToBytes(), and the ADestEncoding parameter of BytesToString(), to IndyTextEncoding_UTF8 instead of their default values of nil. - set the IdGlobal.GIdDefaultTextEncoding variable to encUTF8 instead of its default value of encASCII. In the context of TIdIOHandler, you can set its DefAnsiEncoding property to IndyTextEncoding_UTF8, and leave off any encodings when calling TIdIOHandler.WriteLn() and TIdIOHandler.ReadLn(). Quote from: scasparz on July 20, 2018, 08:55:16 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     EnsureEncoding(ADestEncoding); // (3) ADestEncoding = IUNKNOWN, GDB won't let me F7 this  EnsureEncoding() is also in IdGlobal.pas, don't know why the debugger won't let you step into it. If the VEncoding parameter is nil (which it is not in this example), it gets set to an encoding specified by the ADefEncoding parameter, which is encIndyDefault by default, so the VEncoding will be set to IndyTextEncoding_Default, which returns an encoding determined by the IdGlobal.GIdDefaultTextEncoding variable, which is encASCII by default, so IndyTextEncoding_ASCII is used by default. But, in this example, ADestEncoding is being set to IndyTextEncoding_UTF8 by the caller, so EnsureEncoding() is a no-op. Quote from: scasparz on July 20, 2018, 08:55:16 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     EnsureEncoding(ASrcEncoding, encOSDefault); // (4) ASrcEncoding = IUNKNOWN  Following the above logic, ASrcEncoding is initially nil, so it gets set to IndyTextEncoding_OSDefault, whose implementation is TIdMBCSEncoding using 'char' or 'ASCII' as the charset when calling into ICONV, depending on the IdGlobal.GIdIconvUseLocaleDependantAnsiEncoding variable, which is false by default so 'ASCII' is the default charset. Quote from: scasparz on July 20, 2018, 08:55:16 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     LBytes := RawToBytes(AValue[AIndex], LLength); // (5) LBytes becomes (206,177,206,178)  Those bytes are the correct UTF-8 encoded form of 'αβ'... Quote from: scasparz on July 20, 2018, 08:55:16 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     CheckByteEncoding(LBytes, ASrcEncoding, ADestEncoding); // (6) LBytes becomes ()  And this is where data loss occurs, because ASrcEncoding is set to IndyTextEncoding_OSDefault instead of IndyTextEncoding_UTF8, so the bytes will not be interpreted as UTF-8 correctly.  There is a TODO comment inside of IndyTextEncoding_OSDefault() to have it use UTF-8 on POSIX systems (which includes Linux), but that has not been enabled yet. Internally, CheckByteEncoding() looks like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure CheckByteEncoding(var VBytes: TIdBytes; ASrcEncoding, ADestEncoding: IIdTextEncoding);begin  if ASrcEncoding <> ADestEncoding then begin  VBytes := ADestEncoding.GetBytes(ASrcEncoding.GetChars(VBytes));  end;end;  We know what the output of ADestEncoding.GetBytes() is (no bytes), but what is the output of ASrcEncoding.GetChars(VBytes)?  Is it also empty?  IOW, is the loss of data happening because TIdMCBSEncoding.GetChars() returns no chars at all when given UTF-8 encoded bytes, or is the loss because TIdUTF8Encoding.GetBytes() can't process the chars that TIdMBCSEncoding.GetChars() returned?  You should be able to put breakpoints in those method implementations. Even though the input UTF-8 bytes are not being interpreted as UTF-8, I would expect TIdMBCSEncoding.GetChars() to still be able to return SOME chars.  Incorrect chars perhaps, maybe even $FFFD chars, but not zero chars.  That would imply a logic bug inside of the IdGlobal.DoIconvBytesToChars() function. You might need to set the IdGlobal.GIdIconvUseLocaleDependantAnsiEncoding variable to true, or the IdGlobal.GIdIconvIgnoreIllegalChars variable to true, too work around that. Quote from: scasparz on July 20, 2018, 08:55:16 pmHave F7ed CheckByteEncoding as well. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure CheckByteEncoding(var VBytes: TIdBytes; ASrcEncoding, ADestEncoding: IIdTextEncoding);begin  if ASrcEncoding <> ADestEncoding then begin // according to the GDB both ASrcEncoding and ADestEncoding are valued IUNKNOWN    VBytes := ADestEncoding.GetBytes(ASrcEncoding.GetChars(VBytes)); // (8) this statement is executed, should it?  end;end;[/code] Yes, the ( 8 ) line should be executed, since ASrcEncoding and ADestEncoding are pointing at different objects, so the comparison evaluates as false. Quote from: scasparz on July 20, 2018, 08:55:16 pmBoth ASrcEncoding as well as ADestEncoding are of the IIdTextEncoding type, which I understand is an interface. Yes. Quote from: scasparz on July 20, 2018, 08:55:16 pmHave no clues on how do interfaces compare For purposes of the '=' and '<>' comparison operators, they are simply raw pointer comparisons. However, for Indy 11, I'm expanding on IIdTextEncoding comparisons to take codepages and charsets into account, so even if two IIdTextEncoding variables point at different objects in memory, byte conversions can be skipped if both objects logically represent the same character encoding. Quote from: scasparz on July 20, 2018, 08:55:16 pmWould expect statement #8 to not be executed. In the situation where you pass in a UTF-8 encoded AnsiString, and ask for it to be output as a UTF-8 byte array, then you would be correct ONLY WHEN the ASrcEncoding parameter of ToBytes() is set to IndyTextEncoding_UTF8, which it is not in this example.  Had it been, the comparison in CheckByteEncoding() would have evaluated as true instead of false, and the conversion skipped. Quote from: scasparz on July 20, 2018, 08:55:16 pmCannot F7 statement #8 of course. Which is odd, since those methods are also in IdGlobal.pas.  You should be able to put breakpoints in the implementations, though. ---------------------------------------------------------------------- scasparz · 2018-07-23 18:17 Well, this has been a wealth of information, will need time to play with not to mention the need to do some homework on topics I lack experience. At the moment have just reinstalled Lazarus and Indy, apparently they work, still cannot F7 say EnsureEncoding(), and a few others, can overcome this with explicit breakpoints. Quote- set the IdGlobal.GIdDefaultTextEncoding variable to encUTF8 instead of its default value of encASCII.Added this on app main, it did not work. QuoteAgain, did you recompile Indy with debug info enabled?Do not have a clue here. GDB would let me F7 some parts but may keep a few others out of reach. Haven’t changed anything on Indy, would expect there must be some centralized management on that, probably at some include file or unit. Have taken a look at Lazpack.lpk, have found nothing to be there. Have noticed the existence of a number of DEBUG_STRING_MISMATCH on IdGlobal.pas, but would expect them to be unrelated. Perhaps then you could enlighten me on where to look for. Anyway will come up as soon as and if I have something useful to say. ---------------------------------------------------------------------- Remy Lebeau · 2018-07-23 21:16 Quote from: scasparz on July 23, 2018, 08:17:32 pmQuote- set the IdGlobal.GIdDefaultTextEncoding variable to encUTF8 instead of its default value of encASCII.Added this on app main, it did not work. I gave you several different options to try.  Did you try them all? Passing IndyTextEncoding_UTF8 as **BOTH** the AByteEncoding **AND** ASrcEncoding when calling ToBytes() and TIdIOHandler.WriteLn()? Passing IndyTextEncoding_UTF8 as **BOTH** AByteEncoding **AND** ADestEncoding when calling BytesToString() and TIdIOHandler.ReadLn()? Assigning IndyTextEncoding_UTF8 to **BOTH** TIdIOHandler.DefStringEncoding **AND** TIdIOHandler.DefAnsiEncoding? I don't know what else to tell you to try.  And I can't help further since I don't have Linux or FreePascal/Lazzarus to debug with myself. Quote from: scasparz on July 23, 2018, 08:17:32 pmHave noticed the existence of a number of DEBUG_STRING_MISMATCH on IdGlobal.pas, but would expect them to be unrelated. Yes, they are unrelated to this issue. ---------------------------------------------------------------------- orionyx · 2018-08-05 16:55 i've been battling since june to get lazarus working reliably on ubuntu 16.04 then 18.04. years ago, i used borland builder 6 to create some beautiful software, and i thought that lazarus was an approximation to builder. seems i am wrong. first, the desktop doesn't seem to work very well. forms and files disappear and reappear somewhat randomly - annoying but manageable. i have tried all kinds of things to stop this happening - nothing works. today i updated my software, and the update process removed lazarus, no telling why. so i thought i would reinstall it. this is what happens: ***** lionel@lionel-TECRA-R940:~/Downloads$ gdebi fpc_3.0.4-3_amd64.deb Reading package lists... Done Building dependency tree        Reading state information... Done This package is uninstallable A later version is already installed lionel@lionel-TECRA-R940:~/Downloads$ gdebi fpc-src_3.0.4-2_amd64.deb Reading package lists... Done Building dependency tree        Reading state information... Done Reading state information... Done Free Pascal Sources  The Free Pascal Compiler is a Delphi and Turbo Pascal 7.0 compatible  32/64bit Object Pascal Compiler. It comes with a huge run-time library.  Shared libraries can be linked and created.  This package contains the sources for the compiler, the rtl, fcl and packages. Need to be root to install packages lionel@lionel-TECRA-R940:~/Downloads$ sudo gdebi fpc- src_3.0.4-2_amd64.deb [sudo] password for lionel: Reading package lists... Done Building dependency tree        Reading state information... Done Reading state information... Done Free Pascal Sources  The Free Pascal Compiler is a Delphi and Turbo Pascal 7.0 compatible  32/64bit Object Pascal Compiler. It comes with a huge run-time library.  Shared libraries can be linked and created.  This package contains the sources for the compiler, the rtl, fcl and packages. Do you want to install the software package? [y/N]:y Selecting previously unselected package fpc-src. dpkg: considering removing fpc-source-3.0.4 in favour of fpc-src ... dpkg: yes, will remove fpc-source-3.0.4 in favour of fpc-src (Reading database ... 313100 files and directories currently installed.) Preparing to unpack fpc-src_3.0.4-2_amd64.deb ... De-configuring fp-utils-3.0.4 (3.0.4+dfsg-18ubuntu1), to allow removal of fpc-source-3.0.4 (3.0.4+dfsg-18ubuntu1) ... Unpacking fpc-src (3.0.4) ... Setting up fpc-src (3.0.4) ... dpkg: dependency problems prevent configuration of fp-utils-3.0.4:  fp-utils-3.0.4 depends on fpc-source-3.0.4; however:   Package fpc-source-3.0.4 is not installed. dpkg: error processing package fp-utils-3.0.4 (--install):  dependency problems - leaving unconfigured Errors were encountered while processing:  fp-utils-3.0.4 *** as you can see, there is an unsatisfied dependency, fpc-source-3.0.4. but as you can see, the installation process removed this in favor of fpc-src. it's annoying enough to have to install 3 packages when you want 1. it's even more annoying when the installation trips over itself. lazarus would be a beautiful tool for ubuntu if it could be installed by installing a single package, and if then it simply worked. i am trying to get this working because i want my software shop to be able to do 80% of what visual basic does, without unnecessary error and frustration. will i ever get there? ---------------------------------------------------------------------- DonAlfredo · 2018-08-05 17:02 If nothing else works, you can always try fpcupdeluxe ... ;-) ---------------------------------------------------------------------- Remy Lebeau · 2018-08-07 19:16 Quote from: orionyx on August 05, 2018, 06:55:42 pmi've been battling since june to get lazarus working reliably on ubuntu 16.04 then 18.04. This has nothing to do with getting Indy working correctly on Linux - ie, the subject of this discussion thread.  Please don't hijack a discussion with unrelated issues.  You should have posted a new discussion thread instead. ---------------------------------------------------------------------- snorkel · 2018-08-07 19:56 "i've been battling since june to get lazarus working reliably on ubuntu 16.04 then 18.04." Remy is correct, start your own thread on this and we can help you.(Believe me Lazarus works great on Ubuntu, just don't use the versions in the repository, use the .debs).