Delphi unicode support, windows API not using unicode variants lazarus / 62-General ---------------------------------------------------------------------- deadbeef · 2022-02-15 08:32 Lazarus 2.2.0 (rev lazarus_2_2_0) FPC 3.2.2 x86_64-win64-win32/win64 Use Ansistrings is disabled. I'm using -MDelphiUnicode and also added {$MODE DELPHIUNICODE} to my units. PChar is a PWideChar but CreateFile is mapped to CreateFileA this is not compatible with Delphi which would map this to CreateFileW. ReadDirectoryChanges is not defined at all but ReadDirectoryChangesW is. Also I get some of these Warning: Current system codepage "932" is not available for the compiler. Switching default codepage back to "28591". Am I missing something in the options or is this intended? Also it would be nice to have some templates included in Lazarus so it's easy to create a project with delphi compatibility, this would also serve as an example on which settings are needed. ---------------------------------------------------------------------- PascalDragon · 2022-02-15 08:52 Quote from: deadbeef on February 15, 2022, 09:32:40 amUse Ansistrings is disabled. I'm using -MDelphiUnicode and also added {$MODE DELPHIUNICODE} to my units. PChar is a PWideChar but CreateFile is mapped to CreateFileA this is not compatible with Delphi which would map this to CreateFileW. ReadDirectoryChanges is not defined at all but ReadDirectoryChangesW is. Changing the mode of a unit does not affect the mode of other units. The DelphiUnicode mode currently has very limited use as the remaining RTL still uses AnsiString as base string type. I personally don't recommend that mode, instead simply use mode Delphi and the UTF-8 encoding like Lazarus encourages. Quote from: deadbeef on February 15, 2022, 09:32:40 amAlso it would be nice to have some templates included in Lazarus so it's easy to create a project with delphi compatibility, this would also serve as an example on which settings are needed. While it's possible to share projects with Delphi it's not recommended. The form files need to be duplicated for example among other things, thus it's nothing that can be solved by some template, because one needs to pay a hell lot of attention. ---------------------------------------------------------------------- marcov · 2022-02-15 10:37 With some work you can share forms with Delphi. Sometimes requires some manually cleaning (I use SVN diff tool) though. This is a horrible pity, since it precludes having shared demoes for e.g. library packages ---------------------------------------------------------------------- jast · 2022-02-19 21:24 Quote from: PascalDragon on February 15, 2022, 09:52:46 am... as the remaining RTL still uses AnsiString as base string type... When will FPC / Lazarus be fully unicode? I would like to move from Delphi to Lazarus but it is currently impossible due to the above quote. ---------------------------------------------------------------------- wp · 2022-02-19 21:37 Quote from: jast on February 19, 2022, 10:24:37 pmWhen will FPC / Lazarus be fully unicode? This is the least issue in conversion between Delphi and Lazarus. ---------------------------------------------------------------------- JuhaManninen · 2022-02-20 08:48 Quote from: jast on February 19, 2022, 10:24:37 pmWhen will FPC / Lazarus be fully unicode? I would like to move from Delphi to Lazarus but it is currently impossible due to the above quote. Lazarus and LCL are fully Unicode. It is easy to write code that is 100% compatible with Delphi Unicode-wise, and also compatible with the future FPC's UTF-16 solution. Please see details here:  https://wiki.lazarus.freepascal.org/Unicode_Support_in_Lazarus ---------------------------------------------------------------------- PascalDragon · 2022-02-20 13:02 Quote from: jast on February 19, 2022, 10:24:37 pmQuote from: PascalDragon on February 15, 2022, 09:52:46 am... as the remaining RTL still uses AnsiString as base string type... When will FPC / Lazarus be fully unicode? I would like to move from Delphi to Lazarus but it is currently impossible due to the above quote. It is fully Unicode. UTF-8 is Unicode as well and it's the default encoding used by Lazarus. The only difference is that for FPC/Lazarus you use AnsiString and for Delphi you use UnicodeString and that you might need to take special care for multi Byte characters, but that is a problem with UTF-16 as well. ---------------------------------------------------------------------- jast · 2022-02-21 08:50 But in Lazarus, the whitespace character (#160) is not whitespace. Also sizeof (char) does not equal 2 etc. So in all my units I have to add {$MODE DELPHIUNICODE}. What about RTL routines: passing string parameters to them won't get weird results? Isn't it high time to switch to full Unicode (UTF-16) support like in Delphi? ---------------------------------------------------------------------- JuhaManninen · 2022-02-21 09:59 @jast, you clearly did not read the wiki page:  https://wiki.lazarus.freepascal.org/Unicode_Support_in_Lazarus Our UTF-8 solution is amazingly compatible with Delphi at source code level. Sizeof(char) = 1. Why is it a problem? ---------------------------------------------------------------------- jast · 2022-02-21 10:37 In Delphi, Sizeof(char) = 2. Each character is 2 bytes long, so it's easy to estimate the size of the data buffer or write / read data to the stream. This is, of course, one example of a complication. By creating a new application in Lazarus you can deal with it. However, moving a large application from Delphi to Lazarus takes a lot of work. Therefore, for me (at least for now - until Lazarus / FPC is not fully Unicode) Lazarus is useless. It's a pity! ---------------------------------------------------------------------- wp · 2022-02-21 10:50 Quote from: jast on February 21, 2022, 11:37:57 amIn Delphi, Sizeof(char) = 2. Each character is 2 bytes longNo. Even in UTF-16 a "character" can be 4 bytes. The same story as with UTF-8. Quote from: jast on February 21, 2022, 11:37:57 ammoving a large application from Delphi to Lazarus takes a lot of work. Absolutely. But as I wrote above: The strings are not an issue in - I would say - 95% of the code. Most issues originate in incompatibilities between VCL and LCL and, most severely, in third- party components used in the application to be converted. Don't waste your time waiting for a unicode version of Lazarus. Begin writing small Lazarus applications and learn about the differences. Without a solid knowledge of both Lazarus and Delphi it is useless to even think of converting a large application. ---------------------------------------------------------------------- jast · 2022-02-21 12:25 Quote from: wp on February 21, 2022, 11:50:00 amNo. Even in UTF-16 a "character" can be 4 bytes. The same story as with UTF-8. I think you are not right. The UTF-16 standard allows 4-byte encoding for characters above 216, but in Delphi, Sizeof (char) is always 2. Delphi introduced UCS4Char (UCS4String for strings of these characters) for 4-byte characters. By converting a 32-bit (UTF-32) character string in Delphi to a string type, surrogate pairs (two Char) are created. ---------------------------------------------------------------------- AlexTP · 2022-02-21 12:29 Emoji chars are encoded by 2 WideChars, and only half of emoji is useless (2 bytes). Ie, emoji has 2 surrogate-pair chars. ---------------------------------------------------------------------- PascalDragon · 2022-02-21 12:31 Quote from: jast on February 21, 2022, 09:50:34 amBut in Lazarus, the whitespace character (#160) is not whitespace. That has nothing to do with whether Lazarus uses UTF-8 and Delphi uses UTF-16, it's simply that Lazarus does not treat it as whitespace. Quote from: jast on February 21, 2022, 09:50:34 am Also sizeof (char) does not equal 2 etc. And where do really need the size to be 2? Quote from: jast on February 21, 2022, 09:50:34 amIsn't it high time to switch to full Unicode (UTF-16) support like in Delphi? UTF-8 is just as full Unicode support as UTF-16. And again there are also characters in UTF-16 that need 4 Byte to be encoded instead of 2 (known as “Surrogate Pairs”). And even if you'd use UTF-32 you might still have problems, cause characters might be normalized or not and there might be multi character sequences that you better don't split (for example various emojis). So one needs to do more complex handling anyway, no matter what encoding one chooses. And for those cases where one doesn't need it UTF-8 works the same as UTF-16 (and in the cases where one needs it UTF-8 works just as well). There is simply no real need for Lazarus to switch to UTF-16, because UTF-8 can handle everything just as well. Quote from: jast on February 21, 2022, 01:25:43 pmQuote from: wp on February 21, 2022, 11:50:00 amNo. Even in UTF-16 a "character" can be 4 bytes. The same story as with UTF-8. I think you are not right. The UTF-16 standard allows 4-byte encoding for characters above 216, but in Delphi, Sizeof (char) is always 2. Delphi introduced UCS4Char (UCS4String for strings of these characters) for 4-byte characters. By converting a 32-bit (UTF-32) character string in Delphi to a string type, surrogate pairs (two Char) are created. And these surrogate pairs mean that you can't assume that Length(SomeUnicodeString) is equal to the number of characters. Plus what I had written above about more complex sequences. ---------------------------------------------------------------------- zamtmn · 2022-02-21 13:52 PascalDragon Are there plans to RTL build with DELPHIUNICODE or it will never happen? Not as a main course, just as an option ---------------------------------------------------------------------- JuhaManninen · 2022-02-21 15:01 Quote from: jast on February 21, 2022, 11:37:57 amIn Delphi, Sizeof(char) = 2. Each character is 2 bytes long, so it's easy to estimate the size of the data buffer or write / read data to the stream. No no no! A user perceived "character" is composed of one or more codepoints. One codepoint is composed of one or more codeunits, both in UTF-8 and in UTF-16 encodings. Remember, UTF-16 is not fixed width. If you assume it is fixed width and treat it as UCS-2, you will create sloppy buggy code. There is lots of such code out there already, please don't add to it. UTF-8 is kind of better encoding because it forces you to do things right. The multi-byte codepoints are so common. ---------------------------------------------------------------------- Remy Lebeau · 2022-02-21 18:16 Quote from: PascalDragon on February 21, 2022, 01:31:35 pmAnd these surrogate pairs mean that you can't assume that Length(SomeUnicodeString) is equal to the number of characters. Plus what I had written above about more complex sequences. All UTF encodings are variable-length.  In UTF-8, a codepoint may be 1, 2, 3, or 4 AnsiChars.  In UTF-16, a codepoint may be 1 or 2 WideChars.  In UTF-32, all codepoints are 1 UInt32.  But even so, graphenes (which are what actually constitute what you would consider a "character" visually) can be comprised of multiple codepoints (emojis also fall into this category, too).  So, Length(SomeUnicodeString) will never tell you how many "characters" it actually has, whether that be a UTF-8 string in Lazarus or a UTF-16 string in Delphi, etc.  You have to parse the UTFs into codepoints, and the codepoints into graphenes, in order to determine that. ---------------------------------------------------------------------- winni · 2022-02-21 18:27 UTF-8 Examples: Length('A') = 1 Length('Ö') = 2 Length('⑳') = 3 Length('🚀') = 4 Winni ---------------------------------------------------------------------- PascalDragon · 2022-02-22 08:20 Quote from: zamtmn on February 21, 2022, 02:52:27 pmPascalDragon Are there plans to RTL build with DELPHIUNICODE or it will never happen? Not as a main course, just as an option While the plan is to provide a RTL with String = UnicodeString (DelphiUnicode won't be used, because RTL code is mode ObjFPC) some time in the future there is currently no time table for that and it's not high priority either (considering that UTF-8 is working well for Lazarus so far). Quote from: Remy Lebeau on February 21, 2022, 07:16:41 pmQuote from: PascalDragon on February 21, 2022, 01:31:35 pmAnd these surrogate pairs mean that you can't assume that Length(SomeUnicodeString) is equal to the number of characters. Plus what I had written above about more complex sequences. All UTF encodings are variable-length.  In UTF-8, a codepoint may be 1, 2, 3, or 4 AnsiChars.  In UTF-16, a codepoint may be 1 or 2 WideChars.  In UTF-32, all codepoints are 1 UInt32.  But even so, graphenes (which are what actually constitute what you would consider a "character" visually) can be comprised of multiple codepoints (emojis also fall into this category, too).  So, Length(SomeUnicodeString) will never tell you how many "characters" it actually has, whether that be a UTF-8 string in Lazarus or a UTF-16 string in Delphi, etc.  You have to parse the UTFs into codepoints, and the codepoints into graphenes, in order to determine that. I was mainly concentrating on the point that surrogate pairs in UTF-16 already introduce multi-Char characters you are of course right as well (and I had hinted at that at a previous answer). ---------------------------------------------------------------------- jast · 2022-02-22 11:29 Quote from: JuhaManninen on February 21, 2022, 04:01:37 pmQuote from: jast on February 21, 2022, 11:37:57 amIn Delphi, Sizeof(char) = 2. Each character is 2 bytes long, so it's easy to estimate the size of the data buffer or write / read data to the stream. No no no! ... Yes yes yes! In Delphi Sizeof(char) = 2! Example from Marco Cantù : var  str1: string; begin  str1 := 'Surr. ' + ConvertFromUtf32($1D11E);  showmessage(inttostr(sizeof(char))+'='+inttostr(sizeof(str1[8]))); Result? 2=2! length(str1)=8: number of Chars (including one surrogate pair), not logical Unicode code points. ---------------------------------------------------------------------- Sieben · 2022-02-22 12:16 Quotegraphene Grapheme. Graphene is always just one layer. (scnr) ---------------------------------------------------------------------- PascalDragon · 2022-02-22 12:29 Quote from: jast on February 22, 2022, 12:29:46 pmQuote from: JuhaManninen on February 21, 2022, 04:01:37 pmQuote from: jast on February 21, 2022, 11:37:57 amIn Delphi, Sizeof(char) = 2. Each character is 2 bytes long, so it's easy to estimate the size of the data buffer or write / read data to the stream. No no no! ... Yes yes yes! In Delphi Sizeof(char) = 2! Example from Marco Cantù : var  str1: string; begin  str1 := 'Surr. ' + ConvertFromUtf32($1D11E);  showmessage(inttostr(sizeof(char))+'='+inttostr(sizeof(str1[8]))); Result? 2=2! length(str1)=8: number of Chars (including one surrogate pair), not logical Unicode code points. But that exactly is the point: you need to correctly handle the surrogate pair. You can't just say e.g. Delete(str1, 7, 1); and then expect the resulting string to be valid, because you'll have deleted only one part of the surrogate pair. Thus you always need to be aware of multi-Char characters (in case of UTF-16: the surrogate pairs, in case of UTF-8: any multi-Byte character). And even then, which is the point we're also trying to make: there can be sequences of codepoints (aka graphemes) that need to be handled together (e.g. an a plus a diaresis for a German ä if it isn't a LATIN SMALL A WITH DIARESIS or various emoji sequences). You can't just remove a single character there either or insert some random one. So working with Unicode always requires care no matter if UTF-8, UTF-16 or UTF-32. So, again, it does not matter that much if String = AnsiString with UTF-8 encoding or String = UnicodeString, because one needs to handle it with care anyway. ---------------------------------------------------------------------- tetrastes · 2022-02-22 12:57 Quote from: jast on February 22, 2022, 12:29:46 pmQuote from: JuhaManninen on February 21, 2022, 04:01:37 pmQuote from: jast on February 21, 2022, 11:37:57 amIn Delphi, Sizeof(char) = 2. Each character is 2 bytes long, so it's easy to estimate the size of the data buffer or write / read data to the stream. No no no! ... Yes yes yes! ... What "Yes yes yes!"? The character ConvertFromUtf32($1D11E) is 4 bytes long, not 2 bytes as you wrote. ---------------------------------------------------------------------- Remy Lebeau · 2022-02-22 19:45 Quote from: jast on February 22, 2022, 12:29:46 pmYes yes yes! In Delphi Sizeof(char) = 2! Delphi's (Unicode)String is a UTF-16 string, and thus (Wide)Char is 1 UTF-16 codeunit.  1 Unicode "character" <> 1 Char. Quote from: jast on February 22, 2022, 12:29:46 pmExample from Marco Cantù : That example is comparing the size of 1 single Char to the size of another single Char.  It does not display the fact that ConvertFromUtf32($1D11E) is actually returning a UnicodeString holding 2 Chars ($D834 $DD1E).  In UTF-8, $1D11E would take 4 AnsiChars ($F0 $9D $84 $9E). ---------------------------------------------------------------------- jast · 2022-02-23 10:50 Quote from: tetrastes on February 22, 2022, 01:57:35 pmWhat "Yes yes yes!"? The character ConvertFromUtf32($1D11E) is 4 bytes long, not 2 bytes as you wrote. Where did you read it? I just wrote that the result of ConvertFromUtf32($1D11E) is 2 Char long and Sizeof(Char) = 2. ---------------------------------------------------------------------- tetrastes · 2022-02-23 11:01 I read it here: Quote from: jast on February 21, 2022, 11:37:57 amIn Delphi, Sizeof(char) = 2. Each character is 2 bytes long, ... and I quoted it in my previous post.