UTF8 filename lazarus / 49-FPSpreadsheet ---------------------------------------------------------------------- totya · 2015-04-20 20:44 Hi! It's a great component, but It seems to me, this component support only ANSI filenames. My request is UTF8 filenames support, if possible. Thanks to developers! ---------------------------------------------------------------------- JuhaManninen · 2015-04-20 21:50 Quote from: totya on April 20, 2015, 10:44:13 pmIt's a great component, ... Which component? ---------------------------------------------------------------------- typo · 2015-04-20 21:53 Juha, simply take a look at the title of the subforum (LOL). I do not understand why you ask questions like this ...(LOL) ---------------------------------------------------------------------- totya · 2015-04-21 02:18 Quote from: JuhaManninen on April 20, 2015, 11:50:03 pmQuote from: totya on April 20, 2015, 10:44:13 pmIt's a great component, ... Which component? Hi! Yes, see subforum name: fpspreadsheet see http://wiki.freepascal.org/FPSpreadsheet for example: Code: [Select]MyWorkbook.ReadFromFile(UTF8ToSys(FileName)); UTF8toSys conversion needed, but if filename(and/or path) is UTF8, then ReadFromFile doesn't work with this code: Code: [Select]MyWorkbook.ReadFromFile(FileName); First problem is in fpspreadsheet unit: Code: [Select]procedure TsWorkbook.ReadFromFile(AFileName: string); overload; var   SheetType: TsSpreadsheetFormat;   valid: Boolean;   lException: Exception = nil; begin   if not FileExists(AFileName) then     raise Exception.CreateFmt(rsFileNotFound, [AFileName]);  See, this unit use FileExists function, and not FileExistsUTF8. I hope you understand this problem now. ---------------------------------------------------------------------- JuhaManninen · 2015-04-21 06:30 Quote from: typo on April 20, 2015, 11:53:13 pmJuha, simply take a look at the title of the subforum (LOL). I do not understand why you ask questions like this ...(LOL) Oops ... My bad. Anyway, our improved Unicode support very likely solves this problem :   http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus ---------------------------------------------------------------------- Bart · 2015-04-21 08:19 I would think that the core of fpspreadsheet does not depend on anything form Lazarus, just fpc. It would be possible to add a dependency on LazUtils and use TFileStreamUtf8, but that would make the component unusable for non- Lazarus (as in: pure fpc) developers. Bart ---------------------------------------------------------------------- wp · 2015-04-21 08:46 The main spreadsheet functionality (package laz_fpspreadsheet) depends on FCL and the LazUtils. The visual components (laz_fpspreadsheet_visual) depend also on LCL and TAChart. Quote from: totyasimply take a look at the title of the subforum (LOL). I do not understand why you ask questions like this ...(LOL)And I do not understand why you expect me to take my time to answer your question which you did not take your time to formulate (LOL) ---------------------------------------------------------------------- rvk · 2015-04-21 08:57 Quote from: wp on April 21, 2015, 10:46:18 amQuote from: totyasimply take a look at the title of the subforum (LOL). I do not understand why you ask questions like this ...(LOL)And I do not understand why you expect me to take my time to answer your question which you did not take your time to formulate (LOL) Mmmm, I think the quoting is wrong here  That was a quote from typo, not totya, and was aimed at the response of JuhaManninen. ---------------------------------------------------------------------- typo · 2015-04-21 09:08 Just a joke, sorry. ---------------------------------------------------------------------- wp · 2015-04-21 09:15 QuoteMmmm, I think the quoting is wrong here  (http://forum.lazarus.fr eepascal.org/Smileys/ExcellentSmileys1/wacko.gif) That was a quote from typo, not totya, and was aimed at the response of JuhaManninen.Sorry - did not read carefully enough (as usual...). But the message remains: if users want somebody else to take their time for answering their questions they should take at least enough of their own time to formulate their questions in an understandable way. ---------------------------------------------------------------------- felipemdc · 2015-04-21 09:43 Quote from: totya on April 20, 2015, 10:44:13 pmHi! It's a great component, but It seems to me, this component support only ANSI filenames. My request is UTF8 filenames support, if possible. Thanks to developers! Try a search and replace in the main fpspreadsheet file: Search for: TFileStream Replace with: TFileStreamUtf8 And check if it fixes your problem. ---------------------------------------------------------------------- rvk · 2015-04-21 09:54 Quote from: felipemdc on April 21, 2015, 11:43:03 amSearch for: TFileStream Replace with: TFileStreamUtf8 That wouldn't solve the "not FileExists(AFileName)" in the source of fpspreadsheet, which isn't UTF-8 compatible as long as UTF-8 isn't enabled in RTL. The suggestion from JuhaManninen seems much more promising: http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#Testing_w ith_Lazarus (i.e. "Set UTF-8 in RTL) QuoteFile functions in RTL now take care of file name encoding. All (?) file name related ...UTF8() functions can be replaced with the Delphi compatible function without UTF8 suffix. For example FileExistsUTF8 can be replaced with FileExists. But that would only work with the latest version (FPC 2.7.1). ---------------------------------------------------------------------- wp · 2015-04-21 09:55 But this would make the basic spreadsheet functionality depend on LCL. It the worth the little convenience of not calling AnsiToUTF8/TUF8ToAnsi? Beyond that, there are other TFileStreams spread throughout the package: fpolebasic.pas, fpolestorage.pas, xlsooxml.pas, fpcsvdocument.pas, fpsopendocument.pas, fpsreaderwriter.pas, fpsstreams.pas, fpszipper.pas, fpsxmlcommon.pas ---------------------------------------------------------------------- wp · 2015-04-21 10:03 Quote from: rvkThat wouldn't solve the "not FileExists(AFileName)" in the source of fpspreadsheet, which isn't UTF-8 compatible as long as UTF-8 isn't enabled in RTL.Why is this an issue? All strings passed to any fpspreadsheet filename are expected to be ansi, just like for any fpc file procedure. So, for reading a spreadsheet file from Lazarus, call "workbook.ReadFromFile(UTF8ToAnsi(filename))", and the "FileExists" is handled correctly. ---------------------------------------------------------------------- rvk · 2015-04-21 10:48 "workbook.ReadFromFile(UTF8Filename);" would work too if the option is set to "Set UTF-8 in RTL" in the project-options (I think). I don't think this is a fpspreadsheet issue at all and fpspreadsheet wouldn't even have to know what kind of encoding it's handling. It passes the strings to RTL (i.e. FileExists() ) and if the project- options are set correctly it would work correctly if the RTL can handle these UTF-8 strings (via that option). ---------------------------------------------------------------------- JuhaManninen · 2015-04-21 11:20 Quote from: wp on April 21, 2015, 11:55:32 amBut this would make the basic spreadsheet functionality depend on LCL. No no! It only depends on LazUtils. Please read the "Better Unicode Support in Lazarus" page carefully. I think this component would benefit a lot from the new UTF-8 system. The old explicit UTF-8 conversion functions are rather a hack and they make the code less compatible with Delphi. The new system is nearly compatible with Delphi. Please test and report any problems you find. So far we have found only 2 problems, listed in the "Open issues" section. Maybe it works so well that nobody has anything to complain about, or maybe nobody has tested it yet properly. I think there is disbelief among Lazarus users about how good this new UTF-8 system is. Everybody is used to the old hack and nobody really expected it to improve anytime soon. ... but then it came from the bushes without warning and solved all (?) Unicode issues ... ---------------------------------------------------------------------- taazz · 2015-04-21 11:30 Quote from: JuhaManninen on April 21, 2015, 01:20:28 pmI think there is disbelief among Lazarus users about how good this new UTF-8 system is. Everybody is used to the old hack and nobody really expected it to improve anytime soon. ... but then it came from the bushes without warning and solved all (?) Unicode issues ... and this is enabled in which version exactly? by the way your bushes need trimming. ---------------------------------------------------------------------- felipemdc · 2015-04-21 11:36 Quote from: rvk on April 21, 2015, 11:54:03 amThat wouldn't solve the "not FileExists(AFileName)" in the source of fpspreadsheet, which isn't UTF-8 compatible as long as UTF-8 isn't enabled in RTL. Then add another search&replace: Search: FileExists Replace: FileExistsUTF8 Probably there is a FileExistsUTF8 and if there isnt one, maybe we should add. QuoteThe suggestion from JuhaManninen seems much more promising: http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#Testing_w ith_Lazarus (i.e. "Set UTF-8 in RTL) Yes, but that only works with development FPC, many people use only stable releases. ---------------------------------------------------------------------- wp · 2015-04-21 13:35 Quote from: juhaQuote from: wpBut this would make the basic spreadsheet functionality depend on LCL.No no! It only depends on LazUtils. I was referring to Felipe's post with the TFileStreamUTF8 which google reported to be in lcl (http://lazarus- ccr.sourceforge.net/docs/lcl/lazutf8classes/tfilestreamutf8.html), but the source tells me that it's in LazUtils. Fine! Not sure if I understand the wiki article and its consequences on fpspreadsheet correctly. My understanding is that there is no need to call any explicit conversion functions any more, and no need to use a dedicated utf8 file stream as Felipe suggests. The only thing the user has to to is to activate the utf8-rtl in the project. Therefore, there is no need to modify anything inside fpspreadsheet. Correct? ---------------------------------------------------------------------- Bart · 2015-04-21 13:54 Quote from: wp on April 21, 2015, 03:35:45 pmQuote from: juhaQuote from: wpBut this would make the basic spreadsheet functionality depend on LCL.No no! It only depends on LazUtils. I was referring to Felipe's post with the TFileStreamUTF8 which google reported to be in lcl (http://lazarus- ccr.sourceforge.net/docs/lcl/lazutf8classes/tfilestreamutf8.html), but the source tells me that it's in LazUtils. Fine! LazUtf8Classes is part of LazUtls, not of LCL. Bart ---------------------------------------------------------------------- JuhaManninen · 2015-04-21 14:28 Quote from: wp on April 21, 2015, 03:35:45 pmNot sure if I understand the wiki article and its consequences on fpspreadsheet correctly. My understanding is that there is no need to call any explicit conversion functions any more, and no need to use a dedicated utf8 file stream as Felipe suggests. The only thing the user has to to is to activate the utf8-rtl in the project. Therefore, there is no need to modify anything inside fpspreadsheet. Correct? Yes, correct. It still requires development versions of both FPC and Lazarus, as Taazz and Felipe noted. FPC can be either trunk or the fixes_3_0 branch. Things will improve when FPC 3.0 is released sometime later this year. ---------------------------------------------------------------------- totya · 2015-04-21 15:27 Quote from: JuhaManninen on April 21, 2015, 04:28:39 pmYes, correct. It still requires development versions of both FPC and Lazarus, as Taazz and Felipe noted. FPC can be either trunk or the fixes_3_0 branch. Things will improve when FPC 3.0 is released sometime later this year. Hi! Thanks to all for answers! I have seen this page already: http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus But new (not stable) FPC version need to install. How can I install latest FPC on windows? I need the simplest way. I downloaded latest fpc svn, this is okay. Quote from: JuhaManninen on April 21, 2015, 04:28:39 pmIt still requires development versions of both FPC and Lazarus FPC is okay, but really need latest Lazarus svn version too? Thanks! ---------------------------------------------------------------------- JuhaManninen · 2015-04-21 15:41 Quote from: totya on April 21, 2015, 05:27:45 pmFPC is okay, but really need latest Lazarus svn version too? Yes but it is the easy part. Basically just get the sources from SVN and then "make all" or "make bigide".   http://wiki.freepascal.org/Getting_Lazarus#Getting_Lazarus_SVN_devel opment_version Easiest way to install a new FPC is by using "fpcup". There is plenty of info in this forum about it, and it is documented. You can also get Lazarus trunk at the same go with fpcup if you want. I had problem with fpcup on Linux but on Windows it worked well for me. ---------------------------------------------------------------------- totya · 2015-04-21 22:30 Hi! Thanks, with fpcup is easy. But don't really stable, because I get appcrash once with ppc386.exe and with fpc.exe under fpcup process (with  newest Reiniero fpcup x86 and x64 version too).  But result is working, Lazarus can started, my (about 1000 lines) project is compiled, and running perfectly (after I press set UTF8 in RTL button, before I get charcode problem ex.: ???? ??? ??). svn Lazarus and svn fpc it seems to me is in very early state, for example, I create new app, I drop button on a form, I click to a button, and the result is error: TDefaultComponentEditor: Unable to find method... lol. My OS is win7. Stable Lazarus is working for me without similar problem... ---------------------------------------------------------------------- rvk · 2015-04-21 22:43 Did you try "Tools > Rescan FPC source directory"? ---------------------------------------------------------------------- totya · 2015-04-21 23:02 Hi! No I didn't try it. After "rescan FPC source directory", button event is created under IDE, without any error... Thanks! ---------------------------------------------------------------------- totya · 2015-04-21 23:19 But I see many-many compiler warnings (in my app: Warning: Implicit string type conversion from "AnsiString" to "UnicodeString" or Warning: Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString" ---------------------------------------------------------------------- rvk · 2015-04-22 07:19 Quote from: totya on April 22, 2015, 01:19:30 amBut I see many-many compiler warnings (in my app: Warning: Implicit string type conversion from "AnsiString" to "UnicodeString" or Warning: Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString" Are these warnings in your own application or in other sources? If these are in your own sources, are you using {$H+} at the top of your units (or have AnsiStrings enabled in the options)? (which is default) (if you want an Unicode-application you should set all your options to Unicode too in your own source) Could you give some examples of lines where you get these warnings? ---------------------------------------------------------------------- JuhaManninen · 2015-04-22 08:40 Quote from: totya on April 22, 2015, 01:19:30 amBut I see many-many compiler warnings (in my app: Warning: Implicit string type conversion from "AnsiString" to "UnicodeString" or Warning: Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString" Why do you use UnicodeString? The default String is now encoded with UTF-8. UnicodeString is UTF-16 and indeed triggers a conversion at every assignment. The warning is valid. Just use String everywhere and this problem is solved. @rkv: yes "AnsiStrings enabled" option must be there but I think everybody has it. The important Unicode option in a project is -dEnableUTF8RTL. It affects all dependent packages because it is set in "Additions and Overrides" (the "Set UTF-8 in RTL" button adds it there). For const strings you also need -FcUTF8. ---------------------------------------------------------------------- totya · 2015-04-22 20:18 Hi! First, today "fpcup_update" running without  ppc386/fpc.exe crash under update process. That's a good news, I don't like, if any process  is crashed. Quote from: rvk on April 22, 2015, 09:19:12 amQuote from: totya on April 22, 2015, 01:19:30 amBut I see many-many compiler warnings (in my app: Warning: Implicit string type conversion from "AnsiString" to "UnicodeString" or Warning: Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString" Are these warnings in your own application or in other sources? As I wrote "(in my app)" (Okay I know my English is terrible, I'm sorry). Quote from: rvk on April 22, 2015, 09:19:12 amIf these are in your own sources, are you using {$H+} at the top of your units (or have AnsiStrings enabled in the options)? (which is default) Okay, I removed all {$H+} from sources, and I turned off Parsing/Ansistring options, but warning messages are come again, and I think I get more messages (ex.: shortstring). Quote from: rvk on April 22, 2015, 09:19:12 am(if you want an Unicode- application you should set all your options to Unicode too in your own source) How exactly? Quote from: rvk on April 22, 2015, 09:19:12 amCould you give some examples of lines where you get these warnings? {$H+} mode in units, and Parsing/Ansistring options is enabled. Set UTF8 in RTL is pressed, and bulid. Well, I think I get Warning: Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString" and similar  messages, if I use + (plus) string operator. and new problem in IDE: my line: Naplo_Memo.Lines.Add(''); if I press enter on it, later on source inserted this: Quote  finally   end; lol. And compile is unsuccess, certainly. ---------------------------------------------------------------------- totya · 2015-04-22 20:46 Quote from: JuhaManninen on April 22, 2015, 10:40:40 amWhy do you use UnicodeString? The default String is now encoded with UTF-8. UnicodeString is UTF-16 and indeed triggers a conversion at every assignment. The warning is valid. Just use String everywhere and this problem is solved. Thanks, but I just use string everywhere... ---------------------------------------------------------------------- JuhaManninen · 2015-04-22 21:05 Quote from: totya on April 22, 2015, 10:46:00 pmThanks, but I just use string everywhere... Are the warnings from a dependent package? ---------------------------------------------------------------------- totya · 2015-04-23 19:59 Hi! Dependent package? What do you mean? ---------------------------------------------------------------------- JuhaManninen · 2015-04-24 21:54 Quote from: totya on April 23, 2015, 09:59:29 pmDependent package? What do you mean? A project can depend on packages which are also recompiled when needed. Typical dependencies are LazUtils and LCL but you may have others. The option -FcUTF8 for const strings does not affect units in those packages now, it should be moved to Additions and Overrides. If the warnings come from units belonging to your project, then I have no idea what causes them. ---------------------------------------------------------------------- totya · 2015-04-25 09:12 Quote from: JuhaManninen on April 24, 2015, 11:54:01 pmA project can depend on packages which are also recompiled when needed. (...) Hi! Okay, thank you for this information (I thought similar of this). Environment: Fpc+lazarus is latest trunk, I comiled these about one hour ago, with fpcup. I open my actual project (about 2000 lines). "UTF8 in RTL" setting is ON. Rescan fpc directories Rebuild ide Run/Clean up and build Error messages only from Dependent package, see attached file. Few ansi/wide errors visible in this log. Greetings! ---------------------------------------------------------------------- JuhaManninen · 2015-04-25 10:00 Quote from: totya on April 25, 2015, 11:12:58 amError messages only from Dependent package, see attached file. Few ansi/wide errors visible in this log. Ok, can you please add -FcUTF8 to the "Additions and Overrides" in Compiler Options, as a new custom option right next to the -dEnableUTF8RTL. Then do a clean build again. ---------------------------------------------------------------------- totya · 2015-04-25 10:46 Quote from: JuhaManninen on April 25, 2015, 12:00:48 pmQuote from: totya on April 25, 2015, 11:12:58 amError messages only from Dependent package, see attached file. Few ansi/wide errors visible in this log. Ok, can you please add -FcUTF8 to the "Additions and Overrides" in Compiler Options, as a new custom option right next to the -dEnableUTF8RTL. Then do a clean build again. Hi! After I did this, only the compile times changed in the messages. New options, see atached file. New compiler messages, see attached file. ---------------------------------------------------------------------- JuhaManninen · 2015-04-25 13:45 Quote from: totya on April 25, 2015, 12:46:10 pmAfter I did this, only the compile times changed in the messages. The UnicodeString warnings all come from RichMemo. It explicitly uses UnicodeString as I initially assumed. For example : C:\development\components\svn\richmemo\richmemoutils.pas(133,52) Warning: (4105) Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString" Line 133 says :   fs:= TFileStream.Create( UTF8Decode(FileNameUTF8), fmShareDenyNone or fmOpenRead); You see, it calls this RTL function :   function UTF8Decode(const s : RawByteString): UnicodeString; The code works but it does 2 useless conversions between UTF-8 and UTF-16. If you check the other warnings carefully, they also turn out to be valid. You can ask Dmitry 'skalogryz' Boyarintsev if he wants to add support for the "better" UTF-8 system into RichMemo. ---------------------------------------------------------------------- totya · 2015-04-26 06:28 Hi! Thanks for answers! But as I wrote, I get many warnings from my app, and I dont't like this. I use only simple strings, and only one place where I use conversion between UTF-8 and UTF-16. I will create sample code for you, to see this problem. ---------------------------------------------------------------------- totya · 2015-05-04 20:06 Quote from: JuhaManninen on April 25, 2015, 12:00:48 pmQuote from: totya on April 25, 2015, 11:12:58 amError messages only from Dependent package, see attached file. Few ansi/wide errors visible in this log. Ok, can you please add -FcUTF8 to the "Additions and Overrides" in Compiler Options, as a new custom option right next to the -dEnableUTF8RTL. Then do a clean build again. Hi! Svn versions are terribe, so I install the latest stable version with fcpup "default" parameter. Laz: 1.4.0 /  FP: 3.0.1 Set UTF-8 in RLT button is missing, but UTF8 is working in RLT, for example Memo1.Lines.LoadfromFile('xx') xx: unicode chars (!). I didn't set any extra options for example -FcUTF8 or any... Is it normal? With old Lazarus, I get file open error with this simple project certainly.