Issues with CP_UTF7 lazarus / 62-General ---------------------------------------------------------------------- Bart · 2019-05-12 08:15 Hi, I stumbled upon some issues with CP_UTF7 encoded strings: You cannot assign a literal string to a CP_UTF7 encoded string typeConversion from CP_UTF7 to any other type of AnsiString results in an empty stringConversion from CP_UTF7 to ShortString treats encoded bytes as ASCII, but does not decode the string See the following example: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program cps; {$mode objfpc}{$h+} uses  sysutils; type  AsciiString = type AnsiString(CP_ASCII);  Utf7String  = type AnsiString(CP_UTF7); var  U7: Utf7String;  U8: Utf8String;  A: AsciiString;  S,S2: String;  SS: ShortString; function StrToHex(const S: rawbytestring): string;var  sd: ShortString;  i: Integer;begin  sd := format('[%5d] ',[StringCodePage(S)]);  for i := 1 to Length(s) do    sd := sd + '$' + IntToHex(Byte(s[i]), 2) + ' ';  sd := trim(sd);  result := sd;end; function StrToHex(const S: UnicodeString): shortstring;var  sd: ShortString;  i: Integer;begin  sd := '';  for i := 1 to Length(s) do    sd := sd + '$' + IntToHex(Word(s[i]), 4) + ' ';  sd := trim(sd);  result := sd;end; begin  //U7 := 'U7';     //cps.lpr(45,9) Error: Unknown codepage "65000"  repeat    write('S: ');    readln(S);    U7 := S;    U8 := S;    A := S;    writeln('S : ',StrToHex(S),' [',S,']');    writeln('U8: ',StrToHex(U8),' [',U8,']');    writeln('U7: ',StrToHex(U7),' [',U7,']');    writeln('A : ',StrToHex(A),' [',A,']');    SS := U7;    S2 := U7;    U8 := U7;  writeln('Utf7 -> ShortString: ',StrToHex(SS),' [',SS,']');    writeln('Utf7 -> CP_ACP     : ',StrToHex(S2),' [',S2,']');    writeln('Utf7 -> CP_UTF8    : ',StrToHex(U8),' [',U8,']');  until S='';end. And this sample input Code: [Select]C:\Users\Bart\LazarusProjecten\bugs\Console\cpstring>cps S: 1 + 1 = 2 S : [ 1252] $31 $20 $2B $20 $31 $20 $3D $20 $32 [1 + 1 = 2] U8: [65001] $31 $20 $2B $20 $31 $20 $3D $20 $32 [1 + 1 = 2] U7: [65000] $31 $20 $2B $2D $20 $31 $20 $2B $41 $44 $30 $2D $20 $32 [] A : [20127] $31 $20 $2B $20 $31 $20 $3D $20 $32 [1 + 1 = 2] Utf7 -> ShortString: [ 1252] $31 $20 $2B $2D $20 $31 $20 $2B $41 $44 $30 $2D $20 $32 [1 +- 1 +AD0- 2] Utf7 -> CP_ACP     : [ 1252] [] Utf7 -> CP_UTF8    : [ 1252] [] According to https://en.wikipedia.org/wiki/UTF-7 the string '1 + 1 = 2' indeed shall be encoded as '1 +- 1 +AD0- 2' in CP_UTF7 (so, $31 $20 $2B $2D $20 $31 $20 $2B $41 $44 $30 $2D $20 $32 is correct). The decoding however should IMO result in the original string (given that the input is pure ASCII, encode/decode should be lossless). Since I have no Delphi to test this I do not know if this behaviour is Delphi compatible. IMO it's a bit inconsistent regardless of wether the Greek did it the same way. Bart ---------------------------------------------------------------------- ASerge · 2019-05-12 12:48 On windows, this function is called when converting from one code page to another (syswin.inc): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure Win32Ansi2UnicodeMove(source:pchar;cp : TSystemCodePage;var dest:UnicodeString;len:SizeInt);var  destlen: SizeInt;  dwflags: DWORD;begin  // retrieve length including trailing #0  // not anymore, because this must also be usable for single characters  if cp=CP_UTF8 then    dwFlags:=0  else    dwFlags:=MB_PRECOMPOSED;  destlen:=MultiByteToWideChar(cp, dwFlags, source, len, nil, 0);  // this will null-terminate  setlength(dest, destlen);  if destlen>0 then    begin      MultiByteToWideChar(cp, dwFlags, source, len, @dest[1], destlen);      PUnicodeRec(pointer(dest)-UnicodeFirstOff)^.CodePage:=CP_UTF16;    end;end;See line 8. For UTF8 only, the flag is zero. But according to Microsoft documentation "...dwFlags must be set to 0...  65000 (UTF-7)...   Otherwise, the function fails with ERROR_INVALID_FLAGS". I went through the debugger to the point of return. Indeed, the function returns 0 and GetLastError=1004 (ERROR_INVALID_FLAGS). The fix in the Win32Ansi2UnicodeMove function should be as follows: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure Win32Ansi2UnicodeMove(source:pchar;cp : TSystemCodePage;var dest:UnicodeString;len:SizeInt);var  destlen: SizeInt;  dwflags: DWORD;begin  // retrieve length including trailing #0  // not anymore, because this must also be usable for single characters  case cp of    // Under https://docs.microsoft.com/en- us/windows/desktop/api/stringapiset/nf-stringapiset- multibytetowidechar    CP_UTF8, CP_UTF7, 50220, 50221, 50222, 50225, 50227, 50229, 57002..57011, 42:      dwFlags:=0  else    dwFlags:=MB_PRECOMPOSED;  end;  destlen:=MultiByteToWideChar(cp, dwFlags, source, len, nil, 0);  // this will null-terminate  setlength(dest, destlen);  if destlen>0 then    begin      MultiByteToWideChar(cp, dwFlags, source, len, @dest[1], destlen);    PUnicodeRec(pointer(dest)-UnicodeFirstOff)^.CodePage:=CP_UTF16;    end;end; ---------------------------------------------------------------------- ASerge · 2019-05-12 13:05 Checked with Delphi. Everything works correctly there. By the way, Delphi calls the MultiByteToWideChar function always with the flag 0, regardless of the code page. ---------------------------------------------------------------------- marcov · 2019-05-12 14:19 Quote from: ASerge on May 12, 2019, 02:48:15 pmOn windows, this function is called when converting from one code page to another (syswin.inc):  The fix in the Win32Ansi2UnicodeMove function should be as follows: r42043 ---------------------------------------------------------------------- Bart · 2019-05-12 20:09 Not fixed Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i U7 := 'U7';     //cps.lpr(45,9) Error: Unknown codepage "65000" Also the conversion to ShortString is stil wrong IMO: Code: [Select]S: 1 + 1 = 2 ... U7: [65000] $31 $20 $2B $2D $20 $31 $20 $2B $41 $44 $30 $2D $20 $32 [1 + 1 = 2] ... Utf7 -> ShortString: [ 1252] $31 $20 $2B $2D $20 $31 $20 $2B $41 $44 $30 $2D $20 $32 [1 +- 1 +AD0- 2] Utf7 -> CP_ACP     : [ 1252] $31 $20 $2B $20 $31 $20 $3D $20 $32 [1 + 1 = 2] Utf7 -> CP_UTF8    : [65001] $31 $20 $2B $20 $31 $20 $3D $20 $32 [1 + 1 = 2] Bart ---------------------------------------------------------------------- marcov · 2019-05-12 20:24 Maybe you need to load a codepage with -Fc ? ---------------------------------------------------------------------- Bart · 2019-05-12 20:40 Why would that be necessary? The entire source code is plain ASCII. Code: [Select]C:\Users\Bart\LazarusProjecten\bugs\Console\cpstring>fpc -FcUTF7 cps.lpr Error: Unknown codepage "UTF7" Error: C:\pp\bin\i386-win32\ppc386.exe returned an error exitcode C:\Users\Bart\LazarusProjecten\bugs\Console\cpstring>fpc -Fc65000 cps.lpr Error: Unknown codepage "65000" Error: C:\pp\bin\i386-win32\ppc386.exe returned an error exitcode C:\Users\Bart\LazarusProjecten\bugs\Console\cpstring>fpc -FcUTF-7 cps.lpr Error: Unknown codepage "UTF-7" Error: C:\pp\bin\i386-win32\ppc386.exe returned an error exitcode And it gets even better: Code: [Select]C:\Users\Bart\LazarusProjecten\bugs\Console\cpstring>fpc -FcUTF8 cps.lpr Free Pascal Compiler version 3.3.1 [2019/05/12] for i386 Copyright (c) 1993-2018 by Florian Klaempfl and others Target OS: Win32 for i386 Compiling cps.lpr cps.lpr(45,9) Error: Unknown codepage "65000" cps.lpr(45,9) Error: Compilation raised exception internally Fatal: Compilation aborted An unhandled exception occurred at $00477B14: EAccessViolation: Access violation   $00477B14  GETASCII,  line 697 of C:/devel/fpc/trunk/rtl/inc/charset.pp   $004C3954  TTYPECONVNODE__SIMPLIFY,  line 2926 of ncnv.pas   $004C27F9  TTYPECONVNODE__PASS_TYPECHECK,  line 2426 of ncnv.pas   $004CC597  TYPECHECKPASS_INTERNAL,  line 81 of pass_1.pas   $004BD2E7  INSERTTYPECONV,  line 380 of ncnv.pas   $004CC597  TYPECHECKPASS_INTERNAL,  line 81 of pass_1.pas   $00552EFB  STATEMENT_BLOCK,  line 1367 of pstatmnt.pas   $00538079  BLOCK,  line 381 of psub.pas   $00439919  COMPILE,  line 395 of parser.pas   $00416674  COMPILE,  line 278 of compiler.pas Bart ---------------------------------------------------------------------- marcov · 2019-05-13 08:26 Quote from: Bart on May 12, 2019, 10:40:54 pmWhy would that be necessary? To encode literals in a certain encoding, the compiler needs tables. A few common ones are builtin, weird ones not.  I believe the tables can be made using utils/creumap or something. ---------------------------------------------------------------------- Bart · 2019-05-13 09:28 Quote from: marcov on May 13, 2019, 10:26:39 amQuote from: Bart on May 12, 2019, 10:40:54 pmWhy would that be necessary? To encode literals in a certain encoding, the compiler needs tables. A few common ones are builtin, weird ones not.  I believe the tables can be made using utils/creumap or something. If that were the case then assigning a CP_ACP string to a CP_UTF7 sting would then also have to fail? But the example shows that the contents of the CP_UTF7 string are correct after assigning. I agree that CP_UTF7 is a weird one  Bart ---------------------------------------------------------------------- marcov · 2019-05-13 09:35 Quote from: Bart on May 13, 2019, 11:28:03 amIf that were the case then assigning a CP_ACP string to a CP_UTF7 sting would then also have to fail? But the example shows that the contents of the CP_UTF7 string are correct after assigning. But that might not be handled compiletime, and the runtime conversions go over OS/iconv conversion routines. Check generated assembler to see if the compiler handles a given conversion or not. Note that this is not my strong point, my comments in this threads are intuition about things/directions to check, not absolute truths. As far as I can see utf7 is next to useless, but testing the codepage system is not useless, so I go along with it :-) ---------------------------------------------------------------------- Bart · 2019-05-13 11:26 Should I ask on fpc-devel ML? Bart ---------------------------------------------------------------------- marcov · 2019-05-13 12:07 If you can't check the asm yourself, maybe that is better. You can also make two examples (acs:=utf7 and vice versa), and I'll check. But if utf7 can't be implemented using lookup tables, it will be hard. The mechanism is simpler in the compiler ---------------------------------------------------------------------- Bart · 2019-05-13 12:30 Quote from: marcov on May 13, 2019, 02:07:57 pmIf you can't check the asm yourself, maybe that is better. You can also make two examples (acs:=utf7 and vice versa), and I'll check. Well, it does not compile, so there is no assembler output I would think. And my assemble knowlegde is zero (at best). Bart ---------------------------------------------------------------------- marcov · 2019-05-13 12:44 Quote from: Bart on May 13, 2019, 02:30:12 pmQuote from: marcov on May 13, 2019, 02:07:57 pmIf you can't check the asm yourself, maybe that is better. You can also make two examples (acs:=utf7 and vice versa), and I'll check. Well, it does not compile, so there is no assembler output I would think. And my assemble knowlegde is zero (at best). It might not just be literal conversion tables, it also can be that the compiler has an hard limit on allowed mappings somewhere (e.g. the ones that come from the unicode consortium tables). Better ask on fpc-devel. ---------------------------------------------------------------------- Bart · 2019-05-13 20:37 Quote from: marcov on May 13, 2019, 02:44:54 pmIt might not just be literal conversion tables, it also can be that the compiler has an hard limit on allowed mappings somewhere (e.g. the ones that come from the unicode consortium tables). Better ask on fpc-devel. I did. I also reported the compiler crash in the bugtracker. Bart