Lazarus default codepage lazarus / 64-FPC development ---------------------------------------------------------------------- DimTim · 2022-01-31 09:05 Not sure where to create this topic, sorry. When a new project is created in Lazarus (I tested it on Windows), the source file is created in UTF8 by default.  But plain ansistrings (and string constants, when passed as string parameters), have codepage 0, as if encoded in ANSI.  Wouldn't it be more correct either create new source files in ANSI, or, better, implicitly specify CODEPAGE UTF8?  Now a programmer must explicitly specify it, but it's not evident, I had to study the Reference Guide to find it out. ---------------------------------------------------------------------- JuhaManninen · 2022-01-31 09:11 https://wiki.lazarus.freepascal.org/Unicode_Support_in_Lazarus ---------------------------------------------------------------------- DimTim · 2022-01-31 09:36 There seems to be an inconsistency. In systemh.inc it is specified: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i const  CP_ACP     = 0;     // default to ANSI code page  But the article by your link says: Codepage 0 or codepage 65001 are both correct, they mean UTF-8. Well, anyway it seems to be 'not a bug, but a feature'.   Delphi compatibility? ---------------------------------------------------------------------- Bart · 2022-01-31 13:31 Since Lazarus sets the default system codepage to CP_UTF8 by default of any program using LazUtf8 (which is any LCL application), this means that CP_ACP now also means the string is UTF8 encoded. Bart ---------------------------------------------------------------------- Zoran · 2022-01-31 13:52 Quote from: DimTim on January 31, 2022, 10:36:28 amThere seems to be an inconsistency. In systemh.inc it is specified: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i const  CP_ACP     = 0;     // default to ANSI code page  But the article by your link says: Codepage 0 or codepage 65001 are both correct, they mean UTF-8. Well, anyway it seems to be 'not a bug, but a feature'.   Delphi compatibility? LazUtf8 unit changes what is declared in System. That is what Bart says. So, when your project uses LCL (or just use LazUtils package and include LazUtf8 in uses section of main program, the default code page is utf8 (not the code page fetched from operating system, which is fpc default). Strictly speaking, this is not Delphi compatible, as Delphi uses UTF16 by default. But, except in special cases (which are described in wiki article from Juha's above link), when using String (= AnsiString) type, the program behaves just like in Delphi. When Lazarus moved to FPC 3 from earlier compiler versions, this approach was chosen. It allowed very smooth transition for LCL to then newly introduced codepage-aware strings. At the same time, we got (almost) Delphi compatibility. Please read carefully the wiki page Juha linked to. If you have read it already, read it again. And then... read it again. And once more. You can also try to take a look at initialization section of LazUtf8 unit and follow what is called from there. Might help or might make more confusion... But worth trying. ---------------------------------------------------------------------- PascalDragon · 2022-01-31 14:43 Quote from: DimTim on January 31, 2022, 10:36:28 amThere seems to be an inconsistency. In systemh.inc it is specified: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i const  CP_ACP     = 0;     // default to ANSI code page  But the article by your link says: Codepage 0 or codepage 65001 are both correct, they mean UTF-8. CP_ACP is simply a pseudo code page that tells the RTL that this string is encoded in the default encoding. This is different from saying that a string is explicitly encoded in Win1252 or UTF-8, because the default encoding can change. And it is this default encoding that is changed by Lazarus to CP_UTF8. Thus it's true that in case of Lazarus both mean the same code page (at least as long as no one else changes the default code page). ---------------------------------------------------------------------- DimTim · 2022-01-31 14:59 Quote from: Zoran on January 31, 2022, 02:52:14 pmPlease read carefully the wiki page Juha linked to. If you have read it already, read it again. And then... read it again. And once more. Shikatanai.  It seems, I have no other choice.   I would prefer more clean and intuitive solution, but I like Object Pascal, despite of all its quirks.   And {$CODEPAGE UTF8} works for me. ---------------------------------------------------------------------- DimTim · 2022-01-31 15:12 Quote from: PascalDragon on January 31, 2022, 03:43:43 pmCP_ACP is simply a pseudo code page that tells the RTL that this string is encoded in the default encoding. OK, I got it.  Better not rely on it, and be explicit and strictly typed.   BTW, is there a decent function to guess text files encoding? ---------------------------------------------------------------------- PascalDragon · 2022-01-31 15:33 Quote from: DimTim on January 31, 2022, 04:12:48 pmQuote from: PascalDragon on January 31, 2022, 03:43:43 pmCP_ACP is simply a pseudo code page that tells the RTL that this string is encoded in the default encoding. OK, I got it.  Better not rely on it, and be explicit and strictly typed. On the contrary, you should rely on it. Any String is essentially a AnsiString(CP_ACP). In general you don't need to worry about it and simply use it. ---------------------------------------------------------------------- marcov · 2022-01-31 15:59 Another way is to go via "project options" -> Application, and there enable the "ansi codepage is utf8" checkbox. This raises your minimal windows version somewhat (to Win10 1903+), but it changes the default codepage to UTF8 on the system level, so also for the Windows API ---------------------------------------------------------------------- DimTim · 2022-01-31 17:43 Quote from: PascalDragon on January 31, 2022, 04:33:35 pmOn the contrary, you should rely on it. Any String is essentially a AnsiString(CP_ACP). In general you don't need to worry about it and simply use it. Well, there are specific cases, when it's not enough to 'simply use it'.   For example when you need to decompose string char by char.  Or when reading text from files with different encodings.  Or when trying to find specific substrings in a string, or split a string into words.  BTW, I had some problems with TStringList, when I tried to store in it unique words in CP1251 encoding (Sorted:=True; Duplicates:=True).  It didn't sort correctly (but then I didn't yet find $CODEPAGE switch, maybe it would help).  Then I just used  array of string  and wrote a simplest sorting procedure. ---------------------------------------------------------------------- engkin · 2022-01-31 18:16 Nonsense. ---------------------------------------------------------------------- JuhaManninen · 2022-01-31 23:44 Quote from: DimTim on January 31, 2022, 06:43:12 pmBTW, I had some problems with TStringList, when I tried to store in it unique words in CP1251 encoding ... And you will have lots of such problems because the default encoding remains UTF-8. At some point you will realize it. ---------------------------------------------------------------------- DimTim · 2022-02-01 03:43 Quote from: JuhaManninen on February 01, 2022, 12:44:27 amQuote from: DimTim on January 31, 2022, 06:43:12 pmBTW, I had some problems with TStringList, when I tried to store in it unique words in CP1251 encoding ... And you will have lots of such problems because the default encoding remains UTF-8. At some point you will realize it. My strings to add to TStringList were set to right codepage.  Why the compiler didn't use it?  Should I set {$CODEPAGE 1251} (for example) when using TStringList with non-default encoded strings?  Or the only way is to re-encode them to default? ---------------------------------------------------------------------- DimTim · 2022-02-01 04:17 Quote from: engkin on January 31, 2022, 07:16:39 pmNonsense. Your comment?  Yes, it's useless. ---------------------------------------------------------------------- engkin · 2022-02-01 04:40 Quote from: DimTim on February 01, 2022, 04:43:47 amQuote from: JuhaManninen on February 01, 2022, 12:44:27 amQuote from: DimTim on January 31, 2022, 06:43:12 pmBTW, I had some problems with TStringList, when I tried to store in it unique words in CP1251 encoding ... And you will have lots of such problems because the default encoding remains UTF-8. At some point you will realize it. My strings to add to TStringList were set to right codepage.  Why the compiler didn't use it?  Should I set {$CODEPAGE 1251} (for example) when using TStringList with non-default encoded strings?  Or the only way is to re-encode them to default? {$CODEPAGE XXX} is used to inform the compiler about the code page of the unit. If your strings are in a text file, you can use TStringList.LoadFromFile version that takes TEncoding as a second parameter. The correct way with Lazarus is to convert everything to UTF8. For indexing you can use UTF8* functions from unit LazUtf8, for instance. To customize your sort use TStringList.CustomSort. Edit: Take a look at examples in: Lazarus\components\lazutils\examples\LazUnicode ---------------------------------------------------------------------- PascalDragon · 2022-02-01 07:58 Quote from: DimTim on February 01, 2022, 04:43:47 amQuote from: JuhaManninen on February 01, 2022, 12:44:27 amQuote from: DimTim on January 31, 2022, 06:43:12 pmBTW, I had some problems with TStringList, when I tried to store in it unique words in CP1251 encoding ... And you will have lots of such problems because the default encoding remains UTF-8. At some point you will realize it. My strings to add to TStringList were set to right codepage.  Why the compiler didn't use it?  Should I set {$CODEPAGE 1251} (for example) when using TStringList with non-default encoded strings?  Or the only way is to re-encode them to default? The default encoding used by TStringList is the system default encoding which in case of Lazarus is UTF-8, thus any string added to a string list will be converted to UTF-8. This is independent of the encoding used for a single unit as specified by the $CodePage directive. ---------------------------------------------------------------------- DimTim · 2022-02-01 09:14 Quote from: PascalDragon on February 01, 2022, 08:58:19 amThe default encoding used by TStringList is the system default encoding which in case of Lazarus is UTF-8, thus any string added to a string list will be converted to UTF-8. Can I change that default and set up a specific instance of TStringList to a specific encoding?  UTF8 is great encoding, but requires specific functions for symbol-by-symbol scan and analysis, requires strings instead of chars (or WideChar's) as elements, and it's not always convenient for those used to single-byte strings. And, it seems, that automatic conversion may be incorrect in some cases.  I added to TStringList strings encoded in CP1251, with correct code page field, and addition and sorting were done wrongly. Generally, I would prefer a different behaviour: strings with their encodings are atomic, all conversions explicit, containers accept them "as is", taking into account their encodings, and in the case of incompatibility (say, different encodings in a list, or comparison of strings with different encodings) runtime error is raised. ---------------------------------------------------------------------- DimTim · 2022-02-01 10:23 Quote from: engkin on February 01, 2022, 05:40:11 amThe correct way with Lazarus is to convert everything to UTF8. I see.  It's universal, but requires change of old "single-byte" habits, since Turbo Pascal 4.0. ---------------------------------------------------------------------- wp · 2022-02-01 11:08 Sometimes we should change our old habits... ---------------------------------------------------------------------- winni · 2022-02-01 11:21 Ignoring UTF8 since 30 years should be enough ..... ---------------------------------------------------------------------- Bart · 2022-02-01 11:42 Quote from: DimTim on February 01, 2022, 10:14:55 amUTF8 is great encoding, but requires specific functions for symbol-by-symbol scan and analysis, requires strings instead of chars (or WideChar's) as elements, and it's not always convenient for those used to single-byte strings. That is a common misconception about WideStrings/UnicodeStings: not all codepoints are encoded in a single WideChar. So basically you have the same "problem" as with UTF8. Bart ---------------------------------------------------------------------- PascalDragon · 2022-02-01 12:29 Quote from: DimTim on February 01, 2022, 10:14:55 amQuote from: PascalDragon on February 01, 2022, 08:58:19 amThe default encoding used by TStringList is the system default encoding which in case of Lazarus is UTF-8, thus any string added to a string list will be converted to UTF-8. Can I change that default and set up a specific instance of TStringList to a specific encoding? You need to create a TStringList with a suitable TEncoding parameter. Quote from: DimTim on February 01, 2022, 10:14:55 amUTF8 is great encoding, but requires specific functions for symbol-by-symbol scan and analysis, requires strings instead of chars (or WideChar's) as elements, and it's not always convenient for those used to single-byte strings. And single-byte code pages are restricted in what they can contain. So better learn how to work with UTF-8, cause that topic will hit you sooner or later. Quote from: DimTim on February 01, 2022, 10:14:55 amAnd, it seems, that automatic conversion may be incorrect in some cases.  I added to TStringList strings encoded in CP1251, with correct code page field, and addition and sorting were done wrongly. Please provide an explicit example for that. Otherwise we'll only speculate what might have happened. Quote from: DimTim on February 01, 2022, 10:14:55 amGenerally, I would prefer a different behaviour: strings with their encodings are atomic, all conversions explicit, containers accept them "as is", taking into account their encodings, and in the case of incompatibility (say, different encodings in a list, or comparison of strings with different encodings) runtime error is raised. First of the code page conversion system is Delphi compatible (independent of the point that Delphi uses UnicodeString as default instead of AnsiString). Second containers accepting strings as is would lead to problems with sorting. For example they would need to be converted to a common code page upon each sort instead of only once when they're added. And raising an error in that case is not very user friendly, cause a differently encoded string might come from anywhere. ---------------------------------------------------------------------- JuhaManninen · 2022-02-01 12:30 Quote from: winni on February 01, 2022, 12:21:43 pmIgnoring UTF8 since 30 years should be enough ..... Heh, exactly!  Quote from: DimTim on February 01, 2022, 10:14:55 amGenerally, I would prefer a different behaviour: strings with their encodings are atomic, all conversions explicit, containers accept them "as is", taking into account their encodings, and in the case of incompatibility (say, different encodings in a list, or comparison of strings with different encodings) runtime error is raised. There is a fallback solution for situations like Windows console support:  https://wiki.lazarus.freepascal.org/Lazarus_with_FPC3.0_without_UTF- 8_mode It is not recommended for general usage. Please experiment with it and see if it fits your needs. I predict it will be an eternal battle against windmills... The Lazarus UTF-8 solution is quite amazing don't you think? It supports full Unicode and all its benefits. It is almost compatible with Delphi at source code level despite using a different encoding. UTF-8 also enforces code to handle codepoints correctly because multibyte codepoints are so common. UTF-16 on the other hand is often treated as UCS-2. Code that claims to support Unicode is then buggy. ---------------------------------------------------------------------- DimTim · 2022-02-01 15:47 Quote from: PascalDragon on February 01, 2022, 01:29:51 pmYou need to create a TStringList with a suitable TEncoding parameter. Seems, online documentation at https://lazarus-ccr.sourceforge.io is outdated, no Encoding and DefaultEncoding properties there!  Offline CHM help and reference at https://www.freepascal.org is more up-to date.  Thanks for the hint. ---------------------------------------------------------------------- DimTim · 2022-02-02 07:32 Quote from: JuhaManninen on February 01, 2022, 01:30:55 pm...It is not recommended for general usage. Please experiment with it and see if it fits your needs. I predict it will be an eternal battle against windmills... I did something like that, when ported an old MS-DOS project to Win32.  And it happily works even now. QuoteThe Lazarus UTF-8 solution is quite amazing don't you think? It supports full Unicode and all its benefits. Yes, it is!  And UTF-8 itself is very elegant decision.  But you have to use different routines for Pos, Copy etc, and strings instead of chars... ---------------------------------------------------------------------- Thaddy · 2022-02-02 10:18 Quote from: DimTim on February 01, 2022, 04:47:06 pmQuote from: PascalDragon on February 01, 2022, 01:29:51 pmYou need to create a TStringList with a suitable TEncoding parameter. Seems, online documentation at https://lazarus-ccr.sourceforge.io is outdated, no Encoding and DefaultEncoding properties there!  Offline CHM help and reference at https://www.freepascal.org is more up-to date.  Thanks for the hint. That is because the latter points to the official documentation. No wonder it is more accurate. (drumming with fingers..) Always refer and read the official documentation first. And that is at freepascal.org. Anything other may confuse you. (Or confuse you even more...) ---------------------------------------------------------------------- JuhaManninen · 2022-02-02 10:47 Quote from: DimTim on February 02, 2022, 08:32:20 amYes, it is!  And UTF-8 itself is very elegant decision.  But you have to use different routines for Pos, Copy etc, and strings instead of chars... Actually Pos() and Copy() are useful also with UTF-8. Depends on the situation. For example see : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function SplitInHalf()in  https://wiki.lazarus.freepascal.org/UTF8_strings_and_characters It supports Unicode in every situation. ---------------------------------------------------------------------- DimTim · 2022-02-03 04:21 Quote from: Thaddy on February 02, 2022, 11:18:26 amThat is because the latter points to the official documentation. No wonder it is more accurate. (drumming with fingers..) Some functions/methods/properties are not commented there, just headers.  For example: TStrings.Map Declaration Source position: classesh.inc line 749 public procedure TStrings.Map(   aMap: TStringsMapMethod;   aList: TStrings ); function TStrings.Map(   aMap: TStringsMapMethod ):TStrings; Well, there are sources, as the last retreat. ---------------------------------------------------------------------- dsiders · 2022-02-03 15:43 Quote from: DimTim on February 01, 2022, 04:47:06 pmSeems, online documentation at https://lazarus-ccr.sourceforge.io is outdated, no Encoding and DefaultEncoding properties there!  It is a known problem and reported to the CCR bug page. LCL and LazUtils pages haven't been updated since 2.0.10 (IIRC). Just use the CHM files. ---------------------------------------------------------------------- dsiders · 2022-02-10 20:15 Quote from: dsiders on February 03, 2022, 04:43:49 pmQuote from: DimTim on February 01, 2022, 04:47:06 pmSeems, online documentation at https://lazarus-ccr.sourceforge.io is outdated, no Encoding and DefaultEncoding properties there!  It is a known problem and reported to the CCR bug page. LCL and LazUtils pages haven't been updated since 2.0.10 (IIRC). Just use the CHM files. Or the recently added documentation preview at: https://dsiders.gitlab.io/lazdocsnext/ ---------------------------------------------------------------------- marcov · 2022-02-10 20:55 Those are not very well linked. You can't click from some object in lazutils all the way back to system/objpas.tobject. ---------------------------------------------------------------------- dsiders · 2022-02-10 21:02 Quote from: marcov on February 10, 2022, 09:55:49 pmThose are not very well linked. You can't click from some object in lazutils all the way back to system/objpas.tobject. It is the unmodified output from fpdoc. do you have a specific example I can look at? The 3.2.x version of fpdoc... not 3.3.x. [Edit] I found this: https://dsiders.gitlab.io/lazdocsnext/lazutils/stringhashlist/tstringh ashlist.html TObject is linked in the code declaration but not in the inheritance tree. I can't explain it. And I can't just download a 3.3.x pre-built binary to test for differences. ---------------------------------------------------------------------- dsiders · 2022-02-11 06:01 Quote from: dsiders on February 10, 2022, 10:02:58 pmQuote from: marcov on February 10, 2022, 09:55:49 pmThose are not very well linked. You can't click from some object in lazutils all the way back to system/objpas.tobject. It is the unmodified output from fpdoc. do you have a specific example I can look at? The 3.2.x version of fpdoc... not 3.3.x. [Edit] I found this: https://dsiders.gitlab.io/lazdocsnext/lazutils/stringhashlist/tstringh ashlist.html TObject is linked in the code declaration but not in the inheritance tree. I can't explain it. And I can't just download a 3.3.x pre-built binary to test for differences. I downloaded and built fpdoc using FPC 3.3.1. It does resolve the linking issues. It has also exposed a problem with build_lcl_docs. A patch has been prepared but not yet committed. Output using fpdoc 3.3.1 is currently online. My apologies to everyone for hijacking this thread. I'll stop now.