UnicodeString assignment from string literals seem broken in ObjFpc mode lazarus / 62-General ---------------------------------------------------------------------- Arioch · 2022-09-26 17:29 UPD. https://gitlab.com/freepascal.org/fpc/source/-/issues/39923 Win7 / Laz 2.2.3 / FPC 3.2.3 Default LCL project,  {$mode objfpc}{$H+}      Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.FormClick(Sender: TObject);var  a: ansistring;  sa, sc: unicodestring;begin  a := 'ABCD БЮЖД ABCD';  sa := a;  sc := 'ABCD БЮЖД ABCD';   a := IntToStr(StringCodePage( a )) + ' / ' + IntToStr(StringElementSize(a)) + '  ' + a;//  ShowMessage(a);  sa := IntToStr(StringCodePage( sa )) + ' / ' + IntToStr(StringElementSize(sa)) + '  ' + sa;//  ShowMessage(sa);  sc := IntToStr(StringCodePage( sc )) + ' / ' + IntToStr(StringElementSize(sc)) + '  ' + sc;//  ShowMessage(sc);  ShowMessage(a + #13#10 + sa + #13#10 + sc);end;  We should see 3 almost identical lines, right? No! ---------------------------------------------------------------------- Arioch · 2022-09-26 17:42 and conversions, mostly the same but a small difference in the first one Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   sa := '';   a := 'ABCD БЮЖД ABCD';  SetCodePage(RawByteString(a), 1251, True);  SetCodePage(RawByteString(a), 65001, False);  a := IntToStr(StringCodePage( a )) + ' / ' + IntToStr(StringElementSize(a)) + '  ' + a;  sa := sa + a + #13#10;   a := 'ABCD БЮЖД ABCD';  SetCodePage(RawByteString(a), 1251, True);  SetCodePage(RawByteString(a), 65001, False);  SetCodePage(RawByteString(a), 1251, True);  a := IntToStr(StringCodePage( a )) + ' / ' + IntToStr(StringElementSize(a)) + '  ' + a;  sa := sa + a + #13#10;   a := 'ABCD БЮЖД ABCD';  SetCodePage(RawByteString(a), 65001, True);  SetCodePage(RawByteString(a), 1251, False);  SetCodePage(RawByteString(a), 65001, True);  a := IntToStr(StringCodePage( a )) + ' / ' + IntToStr(StringElementSize(a)) + '  ' + a;  sa := sa + a + #13#10;   ShowMessage(sa); ---------------------------------------------------------------------- ASerge · 2022-09-26 18:36 Quote from: Arioch on September 26, 2022, 07:29:59 pmWe should see 3 almost identical lines, right? No! By default, the source code is considered to be in the OS encoding. Therefore, the compiler considers a string constant in the OS encoding and stores it in the executable file in this form, although the original file contains it in UTF8 encoding. During execution, a transformation occurs. You can change the compiler's "opinion" in any of two ways: 1. Save the file in UTF with BOM (via context menu) 2. Add an explicit directive {$CODEPAGE UTF8} ---------------------------------------------------------------------- Arioch · 2022-09-26 18:49 Quote from: ASerge on September 26, 2022, 08:36:56 pmstring constant in the OS encoding and stores it in the executable file in this form 1. No. EXE file has those strings in UTF-8 not in windows-1251 2. Would it be so, then both ansistring (UTF-8 in LCL)  and unicodestrings (UTF-16) would be uniformly damaged in one mode, and uniformly fixed in another We see there the compiler either fails to properly decorate string literals or fails to call encoding-check when assigning to literals... ---------------------------------------------------------------------- Jonas Maebe · 2022-09-26 19:25 Delphi versions with string = unicodestring interpret constant strings in your source file according to the default code page of the machine you are compiling on. If you want to have this behaviour in FPC, add {$modeswitch systemcodepage} (_after_ any {$mode xxx} directives), or use {$mode delphiunicode} (this also changes the string into an alias for unicodestring). ---------------------------------------------------------------------- Arioch · 2022-09-26 19:34 Quote from: Jonas Maebe on September 26, 2022, 09:25:06 pmDelphi versions with string = unicodestring interpret constant strings in your source file according to the default code page of the machine you are compiling on. All Delphi versions do. As long as you did not set source files to be UTF-8. So does Lazarus. Otherwise the source editor would be unable to render fonts properly. Delphi 2009+ then, for what i noticed, would store both constants in EXE - ansi and unicode. It doees not merge string literalls across encoding. Perhaps, so should do FPC Quote from: Jonas Maebe on September 26, 2022, 09:25:06 pm add {$modeswitch systemcodepage} (_after_ any {$mode xxx} directives) Oh, now the strings are uniform indeed - they all are broken :-) ---------------------------------------------------------------------- Arioch · 2022-09-26 19:41 i did not change Laz defaults, so it is UTF-8 also, UTF-8 does NOT need BOM, becase it is byte-oriented format. BOM is needed for UCS2 / UTF-16 and perhaps for UCS4 and UTF-32 Sources have UTF-8, as promiced by IDE HMMM...  EXE does contain both literals in FPC too But the second one has double-conversion!!! It mistreated UTF-8 sources for ANSI codepage, and then applied wrong conversion, 1251 -> UTF16 instead of correct UTF8 -> UTF16 ---------------------------------------------------------------------- Jonas Maebe · 2022-09-26 19:49 Quote from: Arioch on September 26, 2022, 09:34:21 pmQuote from: Jonas Maebe on September 26, 2022, 09:25:06 pmDelphi versions with string = unicodestring interpret constant strings in your source file according to the default code page of the machine you are compiling on. All Delphi versions do. As long as you did not set source files to be UTF-8. The different is that previous Delphi versions did not store the code page of the ansistring constant in the binary (because ansistrings did not have a code page field). As a result, regardless of the default code page of the machine on which the source code was compiled, at run time an ansistring would always be interpreted according to the code page of the machine the executable was running on. This changed with Delphi 2009, because it now adds this code page of the compiling machine to the ansistring constants. As a result, the behaviour of an executable no longer depends on which machine it runs on, but on which machine it was compiled on (which I don't think is that great of an improvement). QuoteSo does Lazarus. Otherwise the source editor would be unable to render fonts properly. By default, FPC (the compiler used by Lazarus) stores "0" (CP_ACP) as the code page for an ansistring (as your first test shows). This results in behaviour compatible with previous FPC versions and also with Delphi < 2009. QuoteQuote from: Jonas Maebe on September 26, 2022, 09:25:06 pm add {$modeswitch systemcodepage} (_after_ any {$mode xxx} directives) Oh, now the strings are uniform indeed - they all are broken :-) I saw you figured this one out already in the mean time. See also https://wiki.freepascal.org/Unicode_Support_in_Lazarus (and https://wiki.freepascal.org/FPC_Unicode_support for the FPC side) ---------------------------------------------------------------------- Jonas Maebe · 2022-09-26 19:58 Quote from: Arioch on September 26, 2022, 09:41:05 pmi did not change Laz defaults, so it is UTF-8 also, UTF-8 does NOT need BOM, becase it is byte-oriented format. BOM is needed for UCS2 / UTF-16 and perhaps for UCS4 and UTF-32 Sources have UTF-8, as promiced by IDE HMMM...  EXE does contain both literals in FPC too But the second one has double-conversion!!! It mistreated UTF-8 sources for ANSI codepage, and then applied wrong conversion, 1251 -> UTF16 instead of correct UTF8 -> UTF16 That's because {$modeswitch systemcodepage} forces the compiler to interpret all constant strings in your source file as being encoded in the system code page (which is presumably 1251 in your case). The utf8 encoding gets specified on the command line by lazarus, but this directive overrides it (since it appears inside the source file, and hence is parsed after the command line options). So this is working as intended, and it seems something else is going on after all in your initial program (I forgot about Lazarus defaulting to utf-8 everywhere, I'm not a Lazarus user but a compiler developer). ---------------------------------------------------------------------- Jonas Maebe · 2022-09-26 20:03 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc}{$h+} {$codepage utf8} uses  {$ifdef unix}cwstring,{$endif}sysutils; procedure test;var  a: ansistring;  sa, sc: unicodestring;begin  a := 'ABCD БЮЖД ABCD';  sa := a;  sc := 'ABCD БЮЖД ABCD';   a := IntToStr(StringCodePage( a )) + ' / ' + IntToStr(StringElementSize(a)) + '  ' + a;//  ShowMessage(a);  sa := IntToStr(StringCodePage( sa )) + ' / ' + IntToStr(StringElementSize(sa)) + '  ' + sa;//  ShowMessage(sa);  sc := IntToStr(StringCodePage( sc )) + ' / ' + IntToStr(StringElementSize(sc)) + '  ' + sc;//  ShowMessage(sc);  writeln(a);  writeln(sa);  writeln(sc);end; begin  test;end. This program prints the following for me. So it seems the issue related to the interaction with the LCL. The string codepage 0 for sa is weird though. ---------------------------------------------------------------------- Arioch · 2022-09-26 20:03 Quote from: Jonas Maebe on September 26, 2022, 09:49:06 pmBy default, FPC (the compiler used by Lazarus) stores "0" (CP_ACP) as the code page for an ansistring (as your first test shows). Frankly, the default LCL setup pretending CP_ACP = CP_UTF8 breaks a number of standard coding patterns. Last time i was hit by it was debugging Windows GDI code in Castle Game Engine. The normal unicodestring := ansistring conversion was broken there, So i think this LCL choice was never correct and was not even most practical. But - what was done is done. However, for UTF-16 strings the encoding is different (and frankly, should be ignored as long as "char size = 2" is detected. It is not wrong charset tag which is stored - it is rong data stream, wrong bytes representation is put instead of UTF-16 stream. ---------------------------------------------------------------------- Arioch · 2022-09-26 20:11 Quote from: Jonas Maebe on September 26, 2022, 10:03:10 pmThis program prints the following for me. So it seems the issue related to the interaction with the LCL. The string codepage 0 for sa is weird though. is it? or is it about the back-end? i do not think `strings` would show both UTF-8 and UTF-16 streams in your ELF, but who knows... Arrrgh!!! I didn't see it, i am not interested in it. ---------------------------------------------------------------------- Arioch · 2022-09-26 20:11 kludges do not work too ---------------------------------------------------------------------- Arioch · 2022-09-26 20:15 notice the same UTF-8 just dilluted with zero-bytes. It would be correct for Latin1 to UCS2 quick hack conversion, would be more or less usually correct for West-European cp1250 to UTF-16, but NOT for UTF-8 ---------------------------------------------------------------------- Jonas Maebe · 2022-09-26 20:21 Quote from: Arioch on September 26, 2022, 10:03:19 pmQuote from: Jonas Maebe on September 26, 2022, 09:49:06 pmBy default, FPC (the compiler used by Lazarus) stores "0" (CP_ACP) as the code page for an ansistring (as your first test shows). Frankly, the default LCL setup pretending CP_ACP = CP_UTF8 breaks a number of standard coding patterns. Last time i was hit by it was debugging Windows GDI code in Castle Game Engine. The normal unicodestring := ansistring conversion was broken there, So i think this LCL choice was never correct and was not even most practical. But - what was done is done. It was done because it was the only way to keep backward compatibility with previous code (which Delphi didn't care about). QuoteHowever, for UTF-16 strings the encoding is different (and frankly, should be ignored as long as "char size = 2" is detected. The code page in an utf-16 string is ignored by the FPC rtl. However, if you assing a constant string in the sourcefile to a unicodestring, and the source file is not written in UTF-16 (I think we don't even support UTF-16 source files), then the compiler must either a) store the string constant as it appeared in the source file, and convert it at run time to UTF-16, or b) convert it to UTF-16 at compile time The compiler nowadays does b), and it's this part that's going wrong if the code page of the source file is not explicitly specified. See https://wiki.freepascal.org/FPC_Unicode_support#String_constants QuoteThe compiler has to know the code page according to which it should interpret string constants, as it may have to convert them at compile time. Normally, a string constant is interpreted according to the source file codepage. If the source file codepage is CP_ACP, a default is used instead: in that case, during conversions the constant strings are assumed to have code page 28591 (ISO 8859-1 Latin 1; Western European). To be sure: in what code page is your source file that you are compiling, both in Delphi and in Lazarus? ---------------------------------------------------------------------- Arioch · 2022-09-26 20:27 both default Delphi - 1251 Lazarus - UTF-8 ---------------------------------------------------------------------- Arioch · 2022-09-26 20:29 Delphi messages show compiler commands, Laz does not, so won't tell you flags.... /AFK for an hour or until tomorrow ---------------------------------------------------------------------- Jonas Maebe · 2022-09-26 20:32 See https://wiki.freepascal.org/Unicode_Support_in_Lazarus#String_Literals for a more detailed explanation of what is happening, and what works/doesn't work depending on whether or not you explicitly tell the compiler that the source file is encoded in utf-8. It's indeed a mess and unfortunate there is no way to have a mode where everything works, but this was impossible to create while also preserving backward compatibility with older code (or at least I didn't figure out a way how to do it). ---------------------------------------------------------------------- marcov · 2022-09-26 20:42 Easiest solution might be going to "project options ->  application" and then enabling manifests and utf8 as default encoding. But note the warning there, it only is for Windows 10 from a certain date. Most supported versions however should be that date though. ---------------------------------------------------------------------- Arioch · 2022-09-26 21:14 MARCOV this should not work you propose to change runtime variables (of Windows). I already did it, by issuing `chcp` (outside a program) and `SetConcoleoutputCP` (inside) you seem to assume that EXE file is correct and it is RTL that damages correct data. It is the opposite: RTL correctly interpretes the data - but the data in EXE was broken already during compilation. Even if you can trigger some reverse bug in RTL, that woud damage data in a complementary way - that would be fragile ---------------------------------------------------------------------- Arioch · 2022-09-26 21:34 Quote from: Jonas Maebe on September 26, 2022, 10:32:47 pmIt's indeed a mess and unfortunate there is no way to have a mode where everything works, but this was impossible to create while also preserving backward compatibility with older code (or at least I didn't figure out a way how to do it). I don't see any technical problem here, but only organisanional. FPC has one set of defaultsLazarus has different set of defaultsLazarus could explicitly spell out its defaults and demand FPC to obediently adapt to them. (more complicated but still possible), FPC could explicitly lay out defaults and demand Lazarus to abide But none of the last two seem to happen. I just did a test with Delphi, console program with two units, one is ANSI-coded another is UTF-8 coded. By default everything works like a charm. It was decided, that source file i encoded in ANSI unless there is BOM. IF there is BOM - then it is Unicode. I found no other charset marks in the project files. I stripped the BOM, rebuild - and the compiler expectedly destroyed the data. As soon as i broke communication between IDE and compiler (here: BOM) - it went wrong. So, from pure technical standpoint there are obvious solutions. 1. It can be discouraged to have Unicode without BOM, just like it is made Delphi. LAzarus could make default projects with BOM, and if old project without BOMs opened - could add it. I just changed in Lazarus encoding to UTF8bom - and FPC immediately picked up the wind. ---------------------------------------------------------------------- Arioch · 2022-09-26 22:26 2. Lazarus saves the information about chosen charsets (per unit) in project1.lpi and/or project1.lps FPC picks up those files imlicitly (or is explicitly commanded to compile one of hose files instead of project1.lpr) and learns encoding. 3. Both Lazarus and FPC extract charset-related heuristics into a  separate package, and both use it. But that is fragile. I saved LCL Unit in 1251 charset, compiled - and FPC screwed things yet worse. Notice how it still "converts" by just adding zero byte. Basically, FPC conversion de facto becomes a loop of "Read byte, Write word". Delphi EXE shows AnsiString to have letters in $C0 - $DF byte range, and UTF-16 string in $0410 - $042F word range FPC instead, when fed with ANSI (1251) encoded sources, saves MBCS stream the same, but saves UTF-16 stream in a broken $00C0 - $00DF word range. It seems to me FPC makes assumption, that "non-marked" source file is Latin1 or Window 1250 or something. Would FPC query the platform's charset - it would perhaps. I speculate (though it is not hard to check) than loading sources (without BOM) Lazarus first tries to de-code from UTF-8, and if it fails then it assumes platform's current non-Unicode charset. Just like it do TStringList.LoadFromFile and TIniFile This heuristic seems reasonable and perhaps FPC could adopt it, i think it is based in FCL which is shared by both FPC and Laz. But a better choice i believe would be for Laz to explicitly store charset info, and for FPC to use it. Unless that would lead to problems with git or svn. ------ Is it reliable, can LAzarus be tricked into mis-detecting charset? Probably it can. But in some very fringe cases, that no one cares of. Windows Extended ASCII letters start with $C0 and go until $FF In UTF-8 those letters should be mapped to something like $d0 $a0 or $d1 $40 - but $Cx can not be "continuation byte", it is decoding error. So, while telling different MBCS codepages from one another is really problematic, telling UTF-8 from non-UTF is not hard. but then what is platform-default non-unicode charset? I remember from pre-Unicode Linux, it was not all that easy. There even were special libraries back then to facilitate it :-) So, if FPC and Lazarus do not want to come together and  make explicit specification of charsets in project files,  i can suggest the folloin heuristics,for every souce file: 1.1. check BOM, if it is there - assume "authoritative" charset 1.2. try to transcode from UTF-8 or UTF-7 to UTF-16 or anything. If it happenned without errors - assume UTF-8 (or UTF-7) "authoritative" charset 1.3  try to ask platform HAL is there is non-Unicode default charset. If it is, assume "unreliable" charset The above are executed as short-eval, "first true exits" 2.0.  Parse .LPI and .LPS files for charset information. Currently there is nothing, so it is "placeholder for a future hook" 2.1.  Look for {$codepage } or {$modeswitch systemcodepage} or other related pragmas. If found, assume one more "authoritative" charset 2.2.  Check FPC command line for char-set related options. It is bad place, because it can not specify different per-file charsets, only one global switch. If found - assume "unreliable" charset. The above are evaluated all. 3.1. if all the detected charsets match - great, happy compiling 3.2. if all "authoritative" charsets (1 or more) match, but "speculative" do not - emit low-priority warning and compile the unit for "authoritative" one 3.3. if no "authoritative" found but "speculative" ones agree - emit Information or low-priority Warning and compile 3.4. if no charset found at all - emit error and stop, but make this error overridable in fpc.cfg. Or the opposite, make it high-profile warning that can be uplifted to error in cfg 3.5. if no "authoritative" found and "speculative" ones disagree - emit error and stop ---------------------------------------------------------------------- paweld · 2022-09-27 13:40 The project files are encoded in utf-8, so all strings assigned to variables are as UTF-8. On Windows, UnicodeString is UTF-16, so to assign a string to a variable, you must first convert it to UTF-16 (UTF8ToUTF16). When changing the code page (SetCodePage) for a string, you must set conversion to True, because cp1251 <> utf8.Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses   LazUTF8;   procedure TForm1.FormCreate(Sender: TObject);var  a: ansistring;  sa, sc: unicodestring;begin  a := 'ABCD БЮЖД ABCD';  sa := a;  sc := UTF8ToUTF16('ABCD БЮЖД ABCD');   a := IntToStr(StringCodePage( a )) + ' / ' + IntToStr(StringElementSize(a)) + '  ' + a;  sa := IntToStr(StringCodePage( sa )) + ' / ' + IntToStr(StringElementSize(sa)) + '  ' + sa;  sc := IntToStr(StringCodePage( sc )) + ' / ' + IntToStr(StringElementSize(sc)) + '  ' + sc;  ShowMessage(a + #13#10 + sa + #13#10 + sc);   sa := '';   a := 'ABCD БЮЖД ABCD';  SetCodePage(RawByteString(a), 1251, True);  SetCodePage(RawByteString(a), 65001, True);  a := IntToStr(StringCodePage( a )) + ' / ' + IntToStr(StringElementSize(a)) + '  ' + a;  sa := sa + a + #13#10;   a := 'ABCD БЮЖД ABCD';  SetCodePage(RawByteString(a), 1251, True);  SetCodePage(RawByteString(a), 65001, True);  SetCodePage(RawByteString(a), 1251, True);  a := IntToStr(StringCodePage( a )) + ' / ' + IntToStr(StringElementSize(a)) + '  ' + a;  sa := sa + a + #13#10;   a := 'ABCD БЮЖД ABCD';  SetCodePage(RawByteString(a), 65001, False);  SetCodePage(RawByteString(a), 1251, True);  SetCodePage(RawByteString(a), 65001, True);  a := IntToStr(StringCodePage( a )) + ' / ' + IntToStr(StringElementSize(a)) + '  ' + a;  sa := sa + a + #13#10;   ShowMessage(sa);end; ---------------------------------------------------------------------- Arioch · 2022-09-27 14:31 Quote from: paweld on September 27, 2022, 03:40:53 pmWhen changing the code page (SetCodePage) for a string, you must set conversion to True, because cp1251 <> utf8. i purposedly was setting false there, to create exactly the combinations of 'raw data' and 'codepage tag' i wanted to test this whole topic, actually, grew out of https://forum.lazarus.freepascal.org/index.php/topic,60688.0.html ---------------------------------------------------------------------- paweld · 2022-09-27 14:37 For change string code page I prefer the LConvEncoding module: https://lazarus- ccr.sourceforge.io/docs/lazutils/lconvencoding/index-5.html