Unicode won't work no matter what I do. lazarus / 31-LCL ---------------------------------------------------------------------- NonSpillable · 2019-01-14 05:03 Hi. Since upgrading my debian installation and to latest lazarus (from debians repo) none of my programs accessing files will work. Nordic chars like Å, Ä and Ö is replaced by "?" in all components when reading a file name. I created a new test application to try different things, attached below. Note that my test app adds "åäö" to a TMemo to demonstrate that it's not a font issue. FindFirst/FindFirstUTF8 and every conceivable combination of Utf8ToSys, Ansitowhatever, Utftowhatever does exactly nothing. How do I fix this? My system: Debian 9 64-bit. Lazarus: 1.6.2+dfsg-2 date 2019-01-12, FPC version 3.0.0. Output from '$ locale': LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME=en_AU.utf8 LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= ---------------------------------------------------------------------- CCRDude · 2019-01-14 06:28 In your test application, where do you take the string from? If you use it as a const, have you specified {$codepage utf8}? ---------------------------------------------------------------------- egsuh · 2019-01-14 06:45 In Windows, there are something called "locale", "Locale for non unicode-supporting applications" in full.  Basically this must be the same as the operating system language but I could set it as other language and then characters are not displayed correctly in some applications. Not sure about Linux. ---------------------------------------------------------------------- NonSpillable · 2019-01-14 08:33 Quote from: CCRDude on January 14, 2019, 07:28:53 amIn your test application, where do you take the string from? If you use it as a const, have you specified {$codepage utf8}? From either a TSearchRec.Name via FindFirst/Next or FindFirstUTF8/NextUTF8 or from a TSearchRecUTF8. And a short constant in the code containing "åäö" is added to the TMemo to demonstrate that it is not a font issue. That is the "åäö" in the first line in the TMemo. As you see, reading file names from disk doesn't work. ---------------------------------------------------------------------- JuhaManninen · 2019-01-14 10:20 Your code works well but the file and directory names in testdir are not UTF-8. It feels strange because in a Linux system everything is UTF-8 by default. You must have copied it from a Windows PC. ---------------------------------------------------------------------- NonSpillable · 2019-01-14 10:56 Quote from: JuhaManninen on January 14, 2019, 11:20:00 amYour code works well but the file and directory names in testdir are not UTF-8. It feels strange because in a Linux system everything is UTF-8 by default. You must have copied it from a Windows PC. Nope, they are created in Caja (MATEs file manager). No Windows here. If the file names are not UTF8 (how do you know?), they must be something older, and should therefore work as well. The files are created today, with Caja, on a Ext4 file system, on a up-to-date Debian installation. Edit: Let me upload the exact same files but in a 7z-archive... Edit2: And one file raw. ---------------------------------------------------------------------- JuhaManninen · 2019-01-14 13:03 Quote from: NonSpillable on January 14, 2019, 11:56:06 amNope, they are created in Caja (MATEs file manager). No Windows here. If the file names are not UTF8 (how do you know?), I know by clicking your package which then opens "Ark" in my Manjaro Linux. Ark comes with KDE. No need to even extract the files. I can see in the Ark window the encoding is wrong. Quotethey must be something older, and should therefore work as well. What do you mean? Things work with UTF-8 encoding. Old or new, doesn't matter. QuoteEdit: Let me upload the exact same files but in a 7z-archive... Edit2: And one file raw. The single file has the right encoding. It means your archiving process goes wrong. Let me guess, you run 7z under Wine. Hah! Why would you do that? BTW, you should see the wrong encoding right in the Caja window. You don't need a test app for that. ---------------------------------------------------------------------- NonSpillable · 2019-01-14 15:23 Quote from: JuhaManninen on January 14, 2019, 02:03:31 pmQuote from: NonSpillable on January 14, 2019, 11:56:06 amNope, they are created in Caja (MATEs file manager). No Windows here. If the file names are not UTF8 (how do you know?), I know by clicking your package which then opens "Ark" in my Manjaro Linux. Ark comes with KDE. No need to even extract the files. I can see in the Ark window the encoding is wrong. Quotethey must be something older, and should therefore work as well. What do you mean? Things work with UTF-8 encoding. Old or new, doesn't matter. QuoteEdit: Let me upload the exact same files but in a 7z-archive... Edit2: And one file raw. The single file has the right encoding. It means your archiving process goes wrong. Let me guess, you run 7z under Wine. Hah! Why would you do that? BTW, you should see the wrong encoding right in the Caja window. You don't need a test app for that. Wine? What is it with people and windows? Noooo, I haven't run Win, nor any win program for nearly two decades! I'm using stock zip and stock 7z, from the GUI (Caja in this case). Nothing has anything with win, OSX, BSD or other OSes to do than Debian 9. No files I have uploaded here has been anywhere near any Windows or Wine, and are created today/yesterday. What I meant with "older" was that there was a time before UTF8, when we had code pages for international tokens. No matter *what* decoding the file system use, Lazarus is the only thing not handling characters correctly. On my disk I have several decades of files, some with Swedish (åäöÅÄÖ) chars, created in different programs, different OSes (some as old as C64, Amiga, Atari, etc) and yes, even files created from DOS. I have never had any problem with any applications until now, when I did a fresh install of Debian 9 AMD64 (I usually use 32-bit) on an i7, fresh lazarus/FPC and suddenly all my programs stop working (I had to recompile them, since I migrate from 32- to 64-bits¹). But no other applications seems to have any problem, new or old, with new or old files. Edit: Could there be any problems with EXT4, with linux eller other things, than lazarus? But lazarus is the only thing not working (that is, FindFirst/FindNext). 1) Side note, but since early 2000 I always thought that the time was ripe for 64-bit, but no, every single time I try 64-bits linux it let me down, something breaks, and breaks bad. This time it was lazarus. Other times it has been CAD-software, visualization software, media players/codecs, etc, etc, etc. ---------------------------------------------------------------------- engkin · 2019-01-14 16:30 Just curious, as I do not use Linux, do you guys need to add a unit like cwstring to have a WideStringManager? @NonSpillable, what do you get for: DefaultSystemCodePage DefaultFileSystemCodePage DefaultRTLFileSystemCodePage? Are you really using Lazarus: 1.6.2 and not 1.8.2 or that's just a typo? Your testdir.zip testdir.7z do not look right on my side: Win using 7z. Edit: Some of your LC_* values are between quotation marks and some without, I don't know if that makes a difference? Doing a quick search gave me the impression that the values you see for LC_* in a console could be different than their counterpart for a GUI. ---------------------------------------------------------------------- JuhaManninen · 2019-01-14 19:16 Quote from: NonSpillable on January 14, 2019, 04:23:54 pmEdit: Could there be any problems with EXT4, with linux eller other things, than lazarus? But lazarus is the only thing not working (that is, FindFirst/FindNext). FindFirst/FindNext work perfectly well. Your directory and file names are just plain wrong. I don't understand why you don't see it in your Caja file manager. See the attached screenshot of my Dolphin file manager. Quote1) Side note, but since early 2000 I always thought that the time was ripe for 64-bit, but no, every single time I try 64-bits linux it let me down, something breaks, and breaks bad. This time it was lazarus. Other times it has been CAD-software, visualization software, media players/codecs, etc, etc, etc. Nonsense. I have used 64-bit Lazarus on Linux for about 7-8 years. Works well. ---------------------------------------------------------------------- JuhaManninen · 2019-01-14 19:25 Quote from: engkin on January 14, 2019, 05:30:55 pmJust curious, as I do not use Linux, do you guys need to add a unit like cwstring to have a WideStringManager? WideStringManager is only needed on Windows. Why do you mix it here? Quotewhat do you get for: DefaultSystemCodePage DefaultFileSystemCodePage DefaultRTLFileSystemCodePage? A user does not need to set such things, especially not on Linux. Why do you mix it here? QuoteAre you really using Lazarus: 1.6.2 and not 1.8.2 or that's just a typo? Makes no difference. Both use the new UTF-8 system. On Linux it matters even less. QuoteYour testdir.zip testdir.7z do not look right on my side: Win using 7z. Exactly, now you are on the right track. QuoteSome of your LC_* values are between quotation marks and some without, I don't know if that makes a difference? What LC_* values? ---------------------------------------------------------------------- JuhaManninen · 2019-01-14 19:43 Quote from: egsuh on January 14, 2019, 07:45:18 amIn Windows, there are something called "locale", "Locale for non unicode-supporting applications" in full.  Basically this must be the same as the operating system language but I could set it as other language and then characters are not displayed correctly in some applications. Not sure about Linux. It is specific to Windows. Linux typically uses UTF-8 everywhere although there is no standard for that. I don't know why you bring the Locales up in a Linux question. ---------------------------------------------------------------------- Bart · 2019-01-14 21:17 Quote from: JuhaManninen on January 14, 2019, 08:25:18 pmWideStringManager is only needed on Windows. Why do you mix it here? Quote IIRC then a WideStringManager is needed on Linux as well to have full unicode support. Otherwise things like WideCompare* won't work. See the comments in LazUtf8. (This also means that any LCL program on *nix already has CWString unit enabled.) Of course this makes no difference at all if the filename is encode wrong in the first place. Bart ---------------------------------------------------------------------- BeniBela · 2019-01-14 21:31 Linux filenames have no encoding.  Any byte sequence not containing #0 and '/' is allowed, like an arbitrary C-string. Btw, I have collected some unusual file names here. A proper file handling needs to support all of them ---------------------------------------------------------------------- Bart · 2019-01-14 21:44 Well, then I mean malformed byte sequences that do not represent any valid UTF8 codepoint. Do Lazarus file IO routines handle correctly your collection of "nasty files"? Bart ---------------------------------------------------------------------- lucamar · 2019-01-14 21:55 Talking of "nasty" file names, I had quite a few problems with Lazarus' serialized backups: "fmain.lfm;10", "fmain.pas;23" an so on. Some routines in the RTL/FCL/LCL treat those as two names separated by a ';'. Baffled me no end until I discovered just why what I was doing didn't work as expected ---------------------------------------------------------------------- engkin · 2019-01-14 22:52 Quote from: JuhaManninen on January 14, 2019, 08:25:18 pmQuote from: engkin on January 14, 2019, 05:30:55 pmJust curious, as I do not use Linux, do you guys need to add a unit like cwstring to have a WideStringManager? WideStringManager is only needed on Windows. Why do you mix it here? According to the docs?: QuoteIt makes no sense to use this unit on a non-POSIX system like Windows, OS/2 or DOS. Therefor it should always be enclosed with an ifdef statement: program myprogram; uses   {$ifdef unix}cwstring,{$endif}    classes, sysutils; So, cwstring *is* for Linux, and it *does* change the WideStringManager according to its source code: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Procedure SetCWideStringManager;...  SetUnicodeStringManager(CWideStringManager); Just in case: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i \\rtl\inc\ustrings.incProcedure SetUnicodeStringManager (Const New : TUnicodeStringManager);begin  widestringmanager:=New;end; Quote from: JuhaManninen on January 14, 2019, 08:25:18 pmQuotewhat do you get for: DefaultSystemCodePage DefaultFileSystemCodePage DefaultRTLFileSystemCodePage? A user does not need to set such things, especially not on Linux. Why do you mix it here? I am not asking him to set these variables, the question was what values do they hold. Quote from: JuhaManninen on January 14, 2019, 08:25:18 pmQuoteAre you really using Lazarus: 1.6.2 and not 1.8.2 or that's just a typo? Makes no difference. Both use the new UTF-8 system. On Linux it matters even less. I take your word, but with the FPC version he reported in his first post the possibility of wrong settings being used to compile RTL/LCL is there. Quote from: JuhaManninen on January 14, 2019, 08:25:18 pmQuoteYour testdir.zip testdir.7z do not look right on my side: Win using 7z. Exactly, now you are on the right track. Thank you. I had to stab in every direction. Now to the fun part: åäö is 6 bytes in UTF8. In his zip file, if you open it with Lazarus and switch the encoding to UTF8, you see: Int_filename_åäö.txt  instead of Int_filename_åäö.txt If you check, using a hex editor, how many bytes between Int_filename_ and .txt you'll find *12* bytes, not 6 bytes. In his zip file every byte of the original UTF8 åäö was considered ANSI and converted to UTF8. Are you following me here? Quote from: JuhaManninen on January 14, 2019, 08:25:18 pmQuoteSome of your LC_* values are between quotation marks and some without, I don't know if that makes a difference? What LC_* values? From his first post: QuoteLANG=en_US.UTF-8 LANGUAGE=en_US:en LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME=en_AU.utf8 LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= ---------------------------------------------------------------------- JuhaManninen · 2019-01-14 23:07 Yes, sorry, cwstring is needed for string comparison and it contains WideStringManager. I confused things. ---------------------------------------------------------------------- engkin · 2019-01-14 23:20 åäö is 6 bytes in UTF8. In his zip file, if you open it with Lazarus and switch the encoding to UTF8, you see: Int_filename_åäö.txt  instead of Int_filename_åäö.txt If you check in this same file, using a hex editor, how many bytes between Int_filename_ and .txt you'll find *12* bytes, not the expected 6 bytes. In his zip file every byte of the original UTF8 åäö was considered ANSI and converted to UTF8, again. His system is not right ---------------------------------------------------------------------- NonSpillable · 2019-01-18 11:43 Quote from: engkin on January 14, 2019, 05:30:55 pmJust curious, as I do not use Linux, do you guys need to add a unit like cwstring to have a WideStringManager? @NonSpillable, what do you get for: DefaultSystemCodePage DefaultFileSystemCodePage DefaultRTLFileSystemCodePage? Are you really using Lazarus: 1.6.2 and not 1.8.2 or that's just a typo? Your testdir.zip testdir.7z do not look right on my side: Win using 7z. Edit: Some of your LC_* values are between quotation marks and some without, I don't know if that makes a difference? Doing a quick search gave me the impression that the values you see for LC_* in a console could be different than their counterpart for a GUI. I've tried those, thanks to Internet searches for the problem, but to no avail. However, the problem seems gone now! The only thing I did was to run a "dpkg-reconfigure locales", and it solved the problem! What the problem was, what kind of locale some part of the system thought I had, is probably lost to history. Strange thing that it only affected lazarus programs, not my programs in gcc or any other installed software. No, I double checked. Version 1.6.2 is correct. I'm using Debian and one well known "issue" with Debian is old packages. A package only makes its way to Debian if it's old and stable. It's both a pain and a blessing. Quote from: JuhaManninen on January 14, 2019, 08:16:29 pmQuote from: NonSpillable on January 14, 2019, 04:23:54 pmEdit: Could there be any problems with EXT4, with linux eller other things, than lazarus? But lazarus is the only thing not working (that is, FindFirst/FindNext). FindFirst/FindNext work perfectly well. Your directory and file names are just plain wrong. I don't understand why you don't see it in your Caja file manager. See the attached screenshot of my Dolphin file manager. Quote1) Side note, but since early 2000 I always thought that the time was ripe for 64-bit, but no, every single time I try 64-bits linux it let me down, something breaks, and breaks bad. This time it was lazarus. Other times it has been CAD-software, visualization software, media players/codecs, etc, etc, etc. Nonsense. I have used 64-bit Lazarus on Linux for about 7-8 years. Works well. Well, the names are correct in Caja, in Thunar and in the console. The reason I think is that encoding isn't really a thing with file names. There seem to have been something that forces lazarus (and perhaps 7z even if I cannot spot a problem) to use a "locale" or "codepage" that is nonsense. As said, a dpkg-reconfigure solved the lazarus problem. PS! I didn't say that lazarus was broken consistently on x64, only that some shit always happen with x64. This time it happens to be lazarus, but not other times. ---------------------------------------------------------------------- NonSpillable · 2019-01-18 11:48 (Sorry for multiple posts. A lot to address here. The problem is resolved by rebuilding locale on my Debian system, as I said) Quote from: engkin on January 14, 2019, 11:52:06 pmThank you. I had to stab in every direction. Now to the fun part: åäö is 6 bytes in UTF8. In his zip file, if you open it with Lazarus and switch the encoding to UTF8, you see: Int_filename_åäö.txt  instead of Int_filename_åäö.txt If you check, using a hex editor, how many bytes between Int_filename_ and .txt you'll find *12* bytes, not 6 bytes. In his zip file every byte of the original UTF8 åäö was considered ANSI and converted to UTF8. Are you following me here? Yes! I think so. It seems like an UTF8 string converted to UTF8 (from ANSI/ASCII)! I really wonder why my system ended up with such problems and why lazarus and possible 7z were the only software to fail. Let me upload the exact same dir compressed the exactly same way again and see if the name encoding has changed. ---------------------------------------------------------------------- NonSpillable · 2019-01-18 11:59 Quote from: engkin on January 15, 2019, 12:20:31 amåäö is 6 bytes in UTF8. In his zip file, if you open it with Lazarus and switch the encoding to UTF8, you see: Int_filename_åäö.txt  instead of Int_filename_åäö.txt If you check in this same file, using a hex editor, how many bytes between Int_filename_ and .txt you'll find *12* bytes, not the expected 6 bytes. In his zip file every byte of the original UTF8 åäö was considered ANSI and converted to UTF8, again. His system is not right Yep, that is precisely my interpretation too. But why? Thats the big problem with UTF8 and all schemes to encode, there is no way to tell if a string already is encoded or just have high bits sat in and older code page. Lazarus and 7z may simply be the only program on my system (since they are the only program to create some trouble) that interpret codepoint as high values of plain ANSI/ASCII. "å", "ä" and "ö" is in upper extended ASCII (>127 that is) and a string parser would yield different things, 3, 6 or 12 bytes depending on what assumptions about the string is made. My system clearly uses UTF8 as everyone else's, but some setting forces lazarus and 7z to assume that a 6-byte UTF8-string of "åäö" is a 6 character extended ASCII string. ---------------------------------------------------------------------- Thaddy · 2019-01-18 12:05 Note: In a current Debian (stretch) the versions are Laz 1.8.4 and FPC 3.0.4 when you add backports. Both are the current releases. ---------------------------------------------------------------------- engkin · 2019-01-19 15:13 Quote from: NonSpillable on January 18, 2019, 12:48:12 pmThe problem is resolved by rebuilding locale on my Debian system Your new zip file has the same problem: 12 bytes instead of 6. The problem is simplified with this code: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  s: string;begin  s := 'Int_filename_åäö.txt'; { UTF8 }   SetCodePage(RawByteString(s),65001, false); { Mark the variable as UTF8 despite it does not contain the correct file name, but it is UTF8 }  SetCodePage(RawByteString(s),1252, true);  { Convert to some ANSI code page to get the correct UTF8 name }  //<----   SetCodePage(RawByteString(s),65001, false); { Mark the variable as UTF8, again!! to get the real name }   ShowMessage(s); //or whatever  I used the following code on my Windows system: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Project1; {$mode objfpc}{$H+}{$Codepage UTF8} uses  {$IFDEF UNIX}{$IFDEF UseCThreads}  cthreads,  {$ENDIF}{$ENDIF}  Classes, windows  { you can add units after this }; var  s: string;  u:UnicodeString;begin  s := 'Int_filename_åäö.txt'; { UTF8 }   SetCodePage(RawByteString(s),65001, false); { Mark the variable as UTF8 }  SetCodePage(RawByteString(s),1252, true);  { Convert to some ASNI code page }  SetCodePage(RawByteString(s),65001, false); { Mark the variable as UTF8, again!! }   u := s;  MessageBoxW(0,@u[1],@u[1],0);end.