ANSI text conversion interrogation with forthcoming LCL (-dEnableUTF8RTL) lazarus / 31-LCL ---------------------------------------------------------------------- ChrisF · 2015-06-26 16:34 Preliminaries: - by conversion, I mean conversion to the by-default encoding representation of text into the LCL, aka UTF8, - only Windows is concerned. I'm a bit puzzled by the challenge to keep an ascendant compatibility with existing source code dealing with ANSI text data. Currently, when an application wants to deal (i.e. convert) simply with external ANSI text (text file, database, ...), some tools are offered by the RTL or the LCL: AnsiToUTF8/UTF8ToAnsi, SysToUTF8/UTF8ToSys, ... Of course, they are using by default only the Windows code page of the computer running the application (or eventually, the one provided by the programmer into his/her source code when modifying the DefaultSystemCodePage value). Now, when DefaultSystemCodePage is changed to UTF8 (i.e. SetMultiByteConversionCodePage(CP_UTF8), with -dEnableUTF8RTL), the existing source code converting external ANSI text to UTF8 is no more working (unless I've missed something - which is not impossible): .  ANSIToUTF8/UTF8ToANSI, SysToUTF8/UTF8ToANSI, ... are no more working (i.e. no conversion is done), .  the "internal" FreePascal conversion mechanism during assignment neither, .  forcing variables to the ansistring type neither, ... The only solution I'm currently seeing is to add some extra instructions to modify manually the code page of the string variables containing ANSI text data to the Windows code page value (or any other one), before doing the conversion (tools or "internal" FreePascal conversion). Furthermore, the Windows code page is no more available using only FreePascal instructions, because the DefaultSystemCodePage value has been internally overridden by the LCL. So, the only way is to use directly the GetACP Windows API; or to use fixed code page values into the source code. String variables with a given static code page can also be used (i.e. type MyCodePageString = type ANSIString(MyCodePageValue)), but it means that modifications are also required into the existing source code anyway. As an illustration, attached a simple program which is loading and displaying an external ANSI text file (Windows code page 1252 is required for this sample text file; if necessary, create your own 'AnsiFile.txt' file for your own Windows code page and modify the sample source code). Please note the additional instruction for the ANSI->UTF8 conversion, required with EnableUTF8RTL: Code: [Select]  // The conversion   {$if (FPC_FULLVERSION>=20701) and defined(EnableUTF8RTL)}   SetCodePage(rawbytestring(S), 1252, false);  // 1252 always !! (or Windows.GetACP())   {$ifend}   SConv := SysToUTF8(S);   // Conversion done Have I missed something evident, or is it an "accepted" ascendant compatibility issue ? ---------------------------------------------------------------------- taazz · 2015-06-26 17:23 Quote from: ChrisF on June 26, 2015, 06:34:56 pmFurthermore, the Windows code page is no more available using only FreePascal instructions, because the DefaultSystemCodePage value has been internally overridden by the LCL. I have no idea as far as I know the latest lazarus release uses fpc 2.6.4 which does not support what you describe here but the above sentence is a no starter for me. If lcl goes that way I'll have to stop upgrading here and fork it. ---------------------------------------------------------------------- ChrisF · 2015-06-26 17:28 Quote from: taazz on June 26, 2015, 07:23:14 pmI have no idea as far as I know the latest lazarus release uses fpc 2.6.4 which does not support what you describe here   [...] Yes, I'm talking only of the trunk version of Lazarus (what I've called the "forthcoming LCL"), additionally with Free Pascal 2.7.1+ (with the rawbytestring code page support, ...). ---------------------------------------------------------------------- JuhaManninen · 2015-06-26 23:50 Quote from: ChrisF on June 26, 2015, 06:34:56 pmHave I missed something evident, or is it an "accepted" ascendant compatibility issue ? Indeed a text file with Windows (or some other) codepage requires extra code. I was planning to test it on Windows and document the alternatives in the feature's wiki page. Still haven't done that. I see 2 ways to handle it. 1. Your way of setting the right codepage for a string in advance. 2. Use RawByteString and do an explicit conversion. I made a skeleton document here :   http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#Reading _text_file_with_Windows_codepage Can you ChrisF please test more and update the page with tested examples. It is important to remember that the "better" Unicode support in LCL makes everything easier when the input data is already UTF-8. Things are easier also when data with different encodings is already in string variables. The conversions now happen dynamically and automatically. Your case of a text file with Windows encoding is an exception and requires a little bit of extra code. We also expect to find more problems in libraries bacause of old WinAPI calls. They must be replaced with the new W-function versions. So far there are only 2 issues found :   http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#Open_is sues Please everybody test more on Windows and report problems. I hope FPC 3.0 gets out ASAP, it would make testing easier. ---------------------------------------------------------------------- taazz · 2015-06-27 00:07 OK this is going to be my last post on the subject since I'm not involved in either the development nor the testing. Quote from: JuhaManninen on June 27, 2015, 01:50:58 amQuote from: ChrisF on June 26, 2015, 06:34:56 pmHave I missed something evident, or is it an "accepted" ascendant compatibility issue ? Indeed a text file with Windows (or some other) codepage requires extra code. I was planning to test it on Windows and document the alternatives in the feature's wiki page. Still haven't done that. I see 2 ways to handle it. 1. Your way of setting the right codepage for a string in advance. 2. Use RawByteString and do an explicit conversion. The fact that the rtl variable DefautlSystemCodePage is changed by the lcl is a major problem. This variable should be used to talk to the system it self and lying about it makes things harder not easier. Quote from: JuhaManninen on June 27, 2015, 01:50:58 am I made a skeleton document here :   http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#Reading _text_file_with_Windows_codepage Can you ChrisF please test more and update the page with tested examples. It is important to remember that the "better" Unicode support in LCL makes everything easier when the input data is already UTF-8. which outside the browser is never in my experience. Up until 5 years ago the most common file encoding was some form of dos code page because the systems creating those where build that far back and on the data exchange front it was the default ansi windows code page or utf16. UTf 8 I have only seen on html to be frank. Quote from: JuhaManninen on June 27, 2015, 01:50:58 amThings are easier also when data with different encodings is already in string variables. The conversions now happen dynamically and automatically. Your case of a text file with Windows encoding is an exception and requires a little bit of extra code. Not really the exception is to find anything utf8 on the windows world. Quote from: JuhaManninen on June 27, 2015, 01:50:58 amWe also expect to find more problems in libraries bacause of old WinAPI calls. They must be replaced with the new W-API versions. So far there are only 2 issues found :   http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#Open_is sues Please everybody test more on Windows and report problems. I hope FPC 3.0 gets out ASAP, it would make testing easier. Sorry but as it stands now the "unicode version" will only be installed on a VM for running tests not going to change a few thousand lines of code because you think that the default windows code page is utf8. ---------------------------------------------------------------------- ChrisF · 2015-06-27 14:06 Quote from: JuhaManninen on June 27, 2015, 01:50:58 am[...] Can you ChrisF please test more and update the page with tested examples.  [...] As far as I've tested, all these 3 kinds of solution are working (BTW, they are general ANSI <-> UTF8 conversions, not limited only to text file data): Code: [Select]{$if (FPC_FULLVERSION>=20701) and defined(EnableUTF8RTL)}   SetCodePage(rawbytestring(StrIn), 1252, false);  // 1252 always, or Windows.GetACP() {$ifend}   StrOut := SysToUTF8(StrIn); Code: [Select]{$if (FPC_FULLVERSION>=20701) and defined(EnableUTF8RTL)} type   string1252 = type ansistring(1252); {$ifend} ... {$if (FPC_FULLVERSION>=20701) and defined(EnableUTF8RTL)} var StrIn: string1252; {$else} var StrIn: string; {$ifend} ...   StrOut := SysToUTF8(StrIn); Code: [Select]var StrIn: string;    // or rawbytestring (OK in both cases apparently, but rawbytestring type needs conditional code) ...   StrOut := CP1252ToUTF8(StrIn);    // 1252 always, of course For my own, I prefer the first one, because generic source code could be used for any Windows code page (providing you know this Windows code page value). Some new generic conversion functions could even be created, like (it's only a proposal): Code: [Select]{$if (FPC_FULLVERSION>=20701) and defined(EnableUTF8RTL)} function AnsiCPChange(const s: string): string; begin   result := s;   // Only if code page for string variable has not been already set (by a former AnsiCPChange call, or by user)   if StringCodePage(s) = CP_UTF8 then     if DefaultSystemCodePage = CP_UTF8 then       SetCodePage(rawbytestring(result), Windows.GetACP, false)     else       // In case DefaultSystemCodePage has been changed by user       SetCodePage(rawbytestring(result), DefaultSystemCodePage, false); end; function SysToUTF8Ext(const s: string): string; begin   result := SysToUTF8(AnsiCPChange(s)); end; function UTF8ToSysExt(const s: string): string; begin   result := AnsiCPChange(UTF8ToSys(s)); end; {$ifend} Or eventually modify the existing SysToUTF8/UTF8ToSys LazUTF8 functions ? It would be great if possible, as it would mean that no change would be required in current user's source code (i.e. current function calls would be still OK). Concerning AnsiToUTF8 and UTF8ToAnsi, I'm afraid they are no more working anyway; not without modifying the code page for ANSI string variables directly in the user's source code. Quote from: JuhaManninen on June 27, 2015, 01:50:58 am[...] Your case of a text file with Windows encoding is an exception and requires a little bit of extra code.  [...] Well, if I'm correct, in fact EVERY existing (i.e. up to Lazarus 1.4/Free Pascal 2.6.4) following conversion instruction calls need to be carefully examined in current source code, and most probably modified (whatever is the origin of the ANSI data; not only text files, I mean): ANSIToUTF8 and UTF8ToANSI, SysToUTF8 and UTF8ToANSI. Because I can't imagine any case for which these current function calls would also work properly with Lazarus 1.5+/FreePascal 2.7.1+ (as string variables containing ANSI data are now always identified as strings with an UTF8 code page by default). Except for ASCII only text, of course. As an addition, 1/ If DefaultSystemCodePage is now always set to CP_UTF8, it would be nice to have at least a new variable containing really the Windows code page; like OSDefaultSystemCodePage or any other name (for the other OS, OSDefaultSystemCodePage = DefaultSystemCodePage). It would break the Delphi compatibility (Delphi has only one DefaultSystemCodePage value), and it would require source code changes, but at least this new variable could be used in these changes (with conditional instructions however, depending of the LCL version). 2/ In new programs (i.e. not talking of the ANSI conversion problem for existing source code), how is it possible to declare a "generic" ANSI type string with LCL 1.5+ ? I mean, a string type with the Windows code page as a static code page value: neither "ansistring" nor "type ansistring(CP_ACP)" are working, as CP_ACP means now UTF8. So, how can we declare a "realansistring" type, usable for any Windows code page (corresponding of course to the targeted computer Windows OS) ? NB. Quote from: JuhaManninen on June 27, 2015, 01:50:58 am[...] It is important to remember that the "better" Unicode support in LCL makes everything easier when the input data is already UTF-8.  [...] I'm certainly not arguing against a "better" Unicode support in the LCL/RTL. I'd just wanted to identify potential compatibility issues concerning the new LCL/FPC versions, at least as it's currently implemented in the corresponding trunk versions. ---------------------------------------------------------------------- BeniBela · 2015-06-27 15:49 It seems it would be much better, if there was an option to set `string = UTF8String = string(UTF_8)` and keep ansistring as it used to be (string(CP_ACP)).  And replace every occurrence of ansistring in the LCL with string/utf8string, if it does not want the real ansistring there. ---------------------------------------------------------------------- marcov · 2015-06-27 16:20 Or simply go for string=unicodestring. ---------------------------------------------------------------------- BeniBela · 2015-06-27 17:33 But the LCL uses UTF-8 everywhere ---------------------------------------------------------------------- taazz · 2015-06-27 17:54 Quote from: BeniBela on June 27, 2015, 07:33:36 pmBut the LCL uses UTF-8 everywhere define everywhere or better yet define where this makes a difference. I'm not against lcl using utf8 I'm against covering it like its the OS default code page if lcl needs utf8, which I doubt then use utf8string instead of string and be done with it. ---------------------------------------------------------------------- ChrisF · 2015-06-27 22:17 Quote from: JuhaManninen on June 27, 2015, 01:50:58 am[...]  Can you ChrisF please test more and update the page with tested examples.  [...] A precision concerning the "explicit conversion" solution... CPxxxxToUTF8 functions are OK, but it's more complicated for their opposite UTF8ToCPxxxx. For these last ones, the data (in the result) are OK but the code page for the result returned by these functions is always the default one, i.e. CP_UTF8. Which will most probably causes a lot of problems to anybody using them. Eventually, these functions could be modified, in order to force the corresponding code page for the result. For instance, something like: Code: [Select]function UTF8ToCP1252(const s: string): string; begin   Result:=UTF8ToSingleByte(s,@UnicodeToCP1252,1252);    // Code page parameter added to all UTF8ToSingleByte call end; function UTF8ToSingleByte(const s: string;   const UTF8CharConvFunc: TUnicodeToCharID; ResultCP: TSystemCodePage): string; [...] {$if (FPC_FULLVERSION>=20701) and defined(EnableUTF8RTL)}   SetCodePage(rawbytestring(result), ResultCP, false); {$ifend} [...] ---------------------------------------------------------------------- JuhaManninen · 2015-06-30 10:27 Sorry for the delay in my answer. Quote from: ChrisF on June 27, 2015, 04:06:44 pmWell, if I'm correct, in fact EVERY existing (i.e. up to Lazarus 1.4/Free Pascal 2.6.4) following conversion instruction calls need to be carefully examined in current source code, and most probably modified (whatever is the origin of the ANSI data; not only text files, I mean): ANSIToUTF8 and UTF8ToANSI, SysToUTF8 and UTF8ToANSI. Because I can't imagine any case for which these current function calls would also work properly with Lazarus 1.5+/FreePascal 2.7.1+ (as string variables containing ANSI data are now always identified as strings with an UTF8 code page by default). Except for ASCII only text, of course. Not really. Almost all existing conversion calls can be removed. Conversions happen automatically now. Only when reading from a file or raw stream, then you must take care of the encoding explicitly. String data returned from library calls is not a problem as the strings have dynamic encoding and are converted as needed. For example functions calling WinAPI can  use UnicodeString explicitly. Many FPC libs do that and more will do so in future. This does not even pose any conflict between the encodings. Using the 'W'-WinAPI calls and UnicodeString makes LCL work right regardless of what encoding it uses. Quote1/ If DefaultSystemCodePage is now always set to CP_UTF8, it would be nice to have at least a new variable containing really the Windows code page; like OSDefaultSystemCodePage or any other name (for the other OS, OSDefaultSystemCodePage = DefaultSystemCodePage). [...] 2/ In new programs (i.e. not talking of the ANSI conversion problem for existing source code), how is it possible to declare a "generic" ANSI type string with LCL 1.5+ ? I mean, a string type with the Windows code page as a static code page value: neither "ansistring" nor "type ansistring(CP_ACP)" are working, as CP_ACP means now UTF8. Good questions but I don't have answers now. I even asked Mattias who has best knowledge on the topic but he is having holidays I think. As some people may have noticed I have learned Unicode related stuff while testing/improving/documenting our new UTF-8 support. Autumn may be better time to solve this, then also FPC 3.0RC1 is hopefully released making testing easier for many people. How about UTF8ToWinCP and WinCPToUTF8? You already solved the problem somehow using Windows.GetACP etc. QuoteCPxxxxToUTF8 functions are OK, but it's more complicated for their opposite UTF8ToCPxxxx. For these last ones, the data (in the result) are OK but the code page for the result returned by these functions is always the default one, i.e. CP_UTF8. Which will most probably causes a lot of problems to anybody using them. Eventually, these functions could be modified, in order to force the corresponding code page for the result. Yes, that is a bug. Thanks for finding it. I will add it to the wiki page. Please everybody try to find more real detailed problems in the new system. I personally was surprised how well it works. It provides Unicode support which is easier than the current hackish system. It is almost source compatible with Delphi although the encoding is different. I mean all Ansi...() string functions work etc. Its usage is not forced to anybody, the old system can still be used. It is functional now (or very soon anyway) because the new FPC features allowed implementing it with only small changes. The future UTF-16 support will take more time and effort. So, I don't see reason for negative feelings against this new UTF-8 support. It does not take anything away from anybody but it offers improvements for people who want them. ---------------------------------------------------------------------- ChrisF · 2015-06-30 15:34 Quote from: JuhaManninen on June 30, 2015, 12:27:08 pm[...]  Almost all existing conversion calls can be removed. Conversions happen automatically now.  [...] I do agree with you. But there is a big if: if the code page of the string variables are corresponding to the text data into these string variables. As there is no ansistring type any more, this mechanism doesn't work for ANSI data; at least not without a lot of precautions and a few additional code each time. Quote from: JuhaManninen on June 30, 2015, 12:27:08 pm[...]  How about UTF8ToWinCP and WinCPToUTF8?  [...] The conversions are OK, indeed: the conversions, but not the code page value for UTF8ToWinCP. And it's not working with "typed" ansistring (i.e. with constant static code page, as for "type ansistring(1252)", for instance). Hereafter a summary I've done concerning the results for an ANSI -> UTF8 text data conversion, with various versions and configurations. Except when indicated, the results are identical for [SIn = ansistring, SOut = utf8string] or for [SIn, SOut = string]. Code: [Select]!-------------------------------------------!-------!--- ----!-------!-------! !     Lazarus version and additional        !   =   !  Sys  !  Ans  !  Win  ! !            configuration                  ! OK KO ! OK KO ! OK KO ! OK KO ! !-------------------------------------------!-------!-------!-------!- ------! !                                           !       !       !       !       ! !  LCL 1.4                                  !    KO ! OK    ! OK    ! OK    ! !                                           !       !       !       !       ! !-------------------------------------------!-------!-------!-------!- ------! !                                           !       !       !       !       ! !  LCL 1.5 without UTF8RTL                  !    KO ! OK    !    KO ! OK    ! !                                           !       !       !       !       ! !-------------------------------------------!-------!-------!-------!- ------! !                                           !       !       !       !       ! !  LCL 1.5 with UTF8RTL                     !    KO !    KO !    KO ! OK    ! !                                           !       !       !       !       ! !-------------------------------------------!-------!-------!-------!- ------! !                                           !       !       !       !       ! !  LCL 1.5 with UTF8RTL and SIn=string1252  ! OK    ! OK    ! OK    !    KO ! !                                           !       !       !       !       ! !-------------------------------------------!-------!-------!-------!- ------! SIn: string variable with ANSI data SOut: string variable equal to: 'Start!' + XXX(SIn) + '!End' Legend for XXX: "=" :   SIn "Sys":  SysToUTF8(SIn) "Ans":  AnsiToUTF8(SIn) "Win":  WinCPToUTF8(SIn) type   string1252 = type ansistring(1252); According to my all -very short- tests, I've founded it's true that there is a "Better Unicode Support in Lazarus/FPC", with more simplifications especially for new programs. But the lack of a real ansistring type seems to be quite a problem to me; not to say an issue. But that's only my opinion ... ---------------------------------------------------------------------- ChrisF · 2015-06-30 15:36 Here is a very basic summary of what I've understood concerning the implementation of the incoming code page support for string variables in Lazarus/Free Pascal. I'm intentionally omitting all the technical stuff, in order to make things as simple as possible and to focus mainly on the concepts. FREE PASCAL by default (i.e. no EnableUTF8RTL define in Lazarus):  ansistring   -->   ANSI  utf8string   -->   UTF8  string = ansistring   -->   ANSI LAZARUS (with EnableUTF8RTL)  ansistring   -->   UTF8  utf8string   -->   UTF8  string = ansistring = utf8string   -->   UTF8 In the latest case the ANSI string type has now disappeared. Apparently, there is no (to be confirmed/infirmed) such way to only have "string = utf8string", without modifying the "ansistring" type at the same time. Something like:  ansistring   -->   ANSI  utf8string   -->   UTF8  string = utf8string   -->   UTF8 Though this latest configuration looks more logical to me, and furthermore more adapted to a transition step. ---------------------------------------------------------------------- JuhaManninen · 2015-06-30 16:36 ChrisF, the word ANSI apparently confuses your head. Please try to forget it and think of "encoding" instead. The "Ansi"-prefix in string type and in string functions is confusing. Originally the string functions (no Ansi...) worked with plain ASCII (7-bit) = only English. Then Borland added support for new ANSI code pages (8-bit) and new functions with "Ansi"-prefix. At some point they named the dynamic string type as AnsiString maybe to differentiate it from ShortString. Now the Ansi...() functions work with UTF-16 text (UnicodeString) in Delphi and UTF-8 text in Lazarus (with -dEnableUTF8RTL). The "Ansi"-prefix does not make much sense, it is only a historical remain. ANSI means also "American National Standards Institute" but I don't know if it has anything to do with our Ansi-stuff. Now the main rule must be to have always the right encoding in a string variable. When reading a system file/stream then maybe an extra function call is needed but so be it. Mattias may have better ideas. The discussion must later be moved to Lazarus mailing list which is followed by more developers. I am still on the edge of understanding/not-understanding all the details. Patches are welcome to fix the wrong types in conversion functions. ---------------------------------------------------------------------- marcov · 2015-06-30 16:50 Quote from: ChrisF on June 30, 2015, 05:36:09 pmSomething like:  ansistring   -->   ANSI  utf8string   -->   UTF8  string = utf8string   -->   UTF8 The problem is that there can only be one variable encoding (as in encoding set on startup) at the moment. You have two, one for "string", which you pin to utf8, and one for ansi. "utf8string" as a type has no special handling in the compiler. If you want this, you'll have to fix your code for unicodestring. The enableutf8 is already the transition hack. ---------------------------------------------------------------------- taazz · 2015-06-30 17:01 Quote from: JuhaManninen on June 30, 2015, 06:36:01 pmChrisF, the word ANSI apparently confuses your head. Please try to forget it and think of "encoding" instead. The "Ansi"-prefix in string type and in string functions is confusing. Originally the string functions (no Ansi...) worked with plain ASCII (7-bit) = only English. Then Borland added support for new ANSI code pages (8-bit) and new functions with "Ansi"-prefix. At some point they named the dynamic string type as AnsiString maybe to differentiate it from ShortString. Hell no. If you want to go as back as the dos era then you have to know that at that time the only existing standart was ascii and some companies where trying to maintain a list of code pages that used ascii for the first 7 bits and an other language for the rest values. At those times IBM was recognized as the middle ground although most big companies do their own thing. Ansi strings introduced on the windows development only and they had two things that the short strings did not 1) huge strings instead of up to 255 chars and 2) they supported the windows multibyte encodings. How the name ANSI came to be and what is its connection with the ansi institution I have no idea but I assume that the ANSI institution has some sort of multibyte char encoding defined somewhere. Quote from: JuhaManninen on June 30, 2015, 06:36:01 pmNow the Ansi...() functions work with UTF-16 text (UnicodeString) in Delphi and UTF-8 text in Lazarus (with -dEnableUTF8RTL). The "Ansi"-prefix does not make much sense, it is only a historical remain. That was a questionable decision on borlands part and something that was made to help the transition of existing code to unicode with the minimum of changes. But, they did not steam rolled the ansi strings to something it is not just to support unicode and they are still paying for it today. I expected a lot better lazarus I was wrong. ---------------------------------------------------------------------- ChrisF · 2015-06-30 17:02 Quote from: JuhaManninen on June 30, 2015, 06:36:01 pm[...]  Now the Ansi...() functions work with UTF-16 text (UnicodeString) in Delphi  [...] True, but I was talking of Ansi string variables, not of Ansi...() functions. And when Delphi has migrated to its Unicode version, the "ansistring" type has been kept untouched, in order people can quickly and easily port their old programs just by changing their "string" variables to "ansistring" ones. OK, it was probably a bit more complicated, but at least it has helped during the transition. Currently, Lazarus doesn't offer this possibility and that's why I'm a bit worried. Quote from: JuhaManninen on June 30, 2015, 06:36:01 pm[...]  ChrisF, the word ANSI apparently confuses your head. Please try to forget it and think of "encoding" instead.  [...] I agree that the fact I'm using the same term for various things might be a bit confusing. So, here is technically exactly what I mean by an Ansi string variable type: Code: [Select]type   realansistring = type ansistring( Windows.GetACP ); which of course can't be declared this way. ---------------------------------------------------------------------- ChrisF · 2015-06-30 17:26 Quote from: marcov on June 30, 2015, 06:50:22 pm[...]  The problem is that there can only be one variable encoding (as in encoding set on startup) at the moment.   [...] That's also how as I've understood the situation. Quote from: marcov on June 30, 2015, 06:50:22 pm"utf8string" as a type has no special handling in the compiler. I'm not exactly requesting an additional "special handling". What I've got in mind is rather a way to define a string type with ANSI (= GetACP of the running computer) as a static code page value. As I can see it, the problem is that here is no difference between current encoding and Windows encoding: i.e. CP_ACP means in fact DefaultSystemCodePage, and not necessarily Windows.GetACP. While CP8_UTF8 always means UTF8. So, having an additional TSystemCodePage type would be great IMHO: Code: [Select]{ some values which are used in RTL for TSystemCodePage type } const   CP_ACP     = 0;     // default to ANSI code page   CP_OEMCP   = 1;     // default to OEM (console) code page   CP_UTF16   = 1200;  // utf-16   CP_UTF16BE = 1201;  // unicodeFFFE   CP_UTF7    = 65000; // utf-7   CP_UTF8    = 65001; // utf-8   CP_ASCII   = 20127; // us-ascii   CP_NONE    = $FFFF; // rawbytestring encoding plus Code: [Select]  CP_REALACP = xxxx;     // ALWAYS to ANSI code page = Windows.GetACP And this way, an "ansistring" type could be redefined as: Code: [Select]type   ansistring = type ansistring( CP_REALACP ); and not as "ansistring( CP_ACP )", as it's currently the case. But I admit I don't know enough this Free Pascal part, in order to know if it's possible (not even to know if it really makes sense). Though I understand it's currently not possible without some additions. ---------------------------------------------------------------------- marcov · 2015-06-30 19:02 Quote from: ChrisF on June 30, 2015, 07:26:44 pmI'm not exactly requesting an additional "special handling". What I've got in mind is rather a way to define a string type with ANSI (= GetACP of the running computer) as a static code page value. The only variable type is ansistring(ACP)=ansistring(0), all conversions are done through this type. The utf8enable hack abuses a layer to fix this on non-Windows platforms. That is already a non FPC supported hack, and you ask to build on top of that. I don't expect that is going to happen. Changing any other type to something that is not a hardcoded value (iow determine on startup) means defining another runtime type, and out of compatibility doing that on more than one system. QuoteAs I can see it, the problem is that here is no difference between current encoding and Windows encoding: i.e. CP_ACP means in fact DefaultSystemCodePage, and not necessarily Windows.GetACP. While CP8_UTF8 always means UTF8. Yes. But the enableutf8 hack sets the RTL equivalent of ACP to utf8. QuoteCode: [Select]  CP_REALACP = xxxx;     // ALWAYS to ANSI code page = Windows.GetACP And since Windows.GetACP is not known compiletime, that  is another runtime typed type. QuoteBut I admit I don't know enough this Free Pascal part, in order to know if it's possible (not even to know if it really makes sense). Though I understand it's currently not possible without some additions. Simple. Long term work should go into a Delphi compatible solution, not prolonging hacks. If you need ACP,  don't enable utf8 hacks. ---------------------------------------------------------------------- JuhaManninen · 2015-06-30 22:27 Anybody who wants to improve the new UTF-8 support, please find open issues and write them to the wiki page. I am sure we can solve them. There is no need for the endless debate about how things should be in a perfect world. That debate already continued many years in FPC mailing lists wasting time and energy from many people. My impression is that all remaining issues can be solved with WinCPToUTF8(). In any case please write to wiki the cases where changes are needed for old code. Now we have the improved UTF-8 support (which is very good BTW). Later there will be UTF-16 support but it will take few years. Nothing fancy about it. Just some work is needed to implement them. ---------------------------------------------------------------------- mattias · 2015-09-22 16:07 Quote from: ChrisF on June 28, 2015, 12:17:36 amCPxxxxToUTF8 functions are OK, but it's more complicated for their opposite UTF8ToCPxxxx. For these last ones, the data (in the result) are OK but the code page for the result returned by these functions is always the default one, i.e. CP_UTF8. Which will most probably causes a lot of problems to anybody using them. Eventually, these functions could be modified, in order to force the corresponding code page for the result. The UTF8ToCPxxxx functions under FPC 2.7.1+ now have a parameter "SetTargetCodePage: boolean = false". You can choose whether the resulting string has the new codepage or the default codepage. The default is false for compatibility. ---------------------------------------------------------------------- ChrisF · 2015-09-23 17:23 Sorry, I'm not sure to understand... Do you mean that: - UTF8ToCPxxxx(String) = UTF8ToCPxxxx(String, false)  gives CP_UTF8 (with -dEnableUTF8RTL, of course) as the code page for the result of the function, - while UTF8ToCPxxxx(String, true) gives CP_xxxx ? Or the opposite ? ** Edit ** Looking at the current source code of the trunk version, the answer is apparently yes (i.e. true sets the code page of the result to CP_xxxx). BTW, I can see in this part of the current source code -at least- 2 new conditional directives: UseSystemCPCon and UseLCPConv.  What are their functions ?