Bug in string concatenation? lazarus / 62-General ---------------------------------------------------------------------- Avinash · 2024-12-11 16:16 Version: FPC 3.2.2 Win32     OS: Windows 7 32-bit Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  S: String;begin   //DefaultSystemCodePage := CP_OEMCP;   S := ShortString(UTF8Encode(#255)) + #0#0#0#0;  WriteLn(  HexStr(PLongInt(@S[1])^, 8)  );   S := UTF8Encode(#255) + #0#0#0#0;         // just no ShortString() typecast  WriteLn(  HexStr(PLongInt(@S[1])^, 8)  ); end. These two WriteLn give different results for me. I also get different output here between FPC 3.2.2 and FPC 3.0.4, with the only correct output coming from FPC 3.0.4 in the case of ShortString() typecast, and FPC 3.2.2 giving something weird in both cases. On my system I get the following results: FPC 3.0.4 output: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i 00008FD1    <=  is the only case with correct result (with a ShortString typecast)000000FF        ...FF at least has relation to #255 argument... FPC 3.2.2 output: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i 0000BFC3      ?00000079        ??? I will also note that the setting of DefaultSystemCodePage correctly affects the result of FPC 3.0.4, while in FPC 3.2.2 nothing changes. ---------------------------------------------------------------------- Fibonacci · 2024-12-11 18:16 #255 codepoint after encoding to UTF-8 is c3 bf, FPC 3.2.2 is correct. https://gchq.github.io/CyberChef/#recipe=From_Hex('Auto')Encode_text(' UTF-8%20(65001)')To_Hex('Space',0)&input=ZmY&oenc=65001 UTF8Encode() creates a UTF8String with codepage 65001, then you concat it with string codepage 0 "#0#0#0#0" so there is a conversion resuling in $79. Casting to ShortString resets the codepage to 1250, then there is no conversion. You can call SetCodePage() before concatenation (after UTF8Encode) to avoid conversion. Or, to avoid conversion: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i DefaultSystemCodePage := CP_UTF8; so all strings have the same codepage as the result of UTF8Encode ---------------------------------------------------------------------- tetrastes · 2024-12-11 19:33 Quote from: Fibonacci on December 11, 2024, 07:16:31 pm#255 codepoint after encoding to UTF-8 is c3 bf, FPC 3.2.2 is correct. It depends how compiler interprets #255. It seems that fpc 3.0.4 treats it as AnsiChar, and that OP's CP is 1251: D1 8F is 'я' in utf-8, FF is 'я' in cp1251. FPC 3.2.2 treats #255 as UnicodeChar: 'ÿ' is 00 FF in UTF-16 and C3 BF in UTF-8. Then there is concatenation. It doesn't change the ShortStrings, as they do not have CP information. But for AnsiStrings concatenation results in AnsiString with system CP (cp1251 here), and fpc tries to convert utf-8 to it, so 'ÿ' -> 'y'($79). But more often this results in '?'($3F). ---------------------------------------------------------------------- Avinash · 2024-12-11 21:05 Quote from: tetrastes on December 11, 2024, 08:33:30 pmIt seems that fpc 3.0.4 treats it as AnsiChar, and that OP's CP is 1251: D1 8F is 'я' in utf-8, FF is 'я' in cp1251. FPC 3.2.2 treats #255 as UnicodeChar: 'ÿ' is 00 FF in UTF-16 and C3 BF in UTF-8. Indeed. When I add UnicodeChar(#255) typecast, the result of FPC 3.0.4 becomes exactly the same as 3.2.2 However, none of the listed typecasts: Char(#255), AnsiChar, ShortString, AnsiString changes the result of FPC 3.2.2 Quote from: tetrastes on December 11, 2024, 08:33:30 pmand fpc tries to convert utf-8 to it, so 'ÿ' -> 'y'($79) Is compiler doing right in trying to do what he does? Are these two expressions really supposed to give the same result (as they do now)? Then the quote from the documentation «UTF8Encode convert a widestring or unicodestring to an UTF-8 encoded ansistring» becomes vague. What does it mean, «UTF-8 encoded ansistring»? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  S: String;begin   S := UTF8Encode(UnicodeChar(#255)) + #0#0#0#0;  WriteLn(  HexStr(PLongInt(@S[1])^, 8)  );       //  00000079   S := UnicodeChar(#255) + #0#0#0#0;  WriteLn(  HexStr(PLongInt(@S[1])^, 8)  );       //  00000079 end. ---------------------------------------------------------------------- tetrastes · 2024-12-11 21:34 Quote from: Avinash on December 11, 2024, 10:05:35 pmHowever, none of the listed typecasts: Char(#255), AnsiChar, ShortString, AnsiString changes the result of FPC 3.2.2 Use {$codepage cp1251} QuoteIs compiler doing right in trying to do what he does? Are these two expressions really supposed to give the same result (as they do now)?What are two expressions? QuoteWhat does it mean, «UTF-8 encoded ansistring»?AnsiString has field with CP information, so this is with utf-8 one: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   S := UTF8Encode(#255);  writeln(StringCodePage(S));      // 65001  WriteLn(  HexStr(PLongInt(@S[1])^, 8)  );     // 0000BFC3   S := S + #0#0#0#0;  writeln(StringCodePage(S));     // 1251  WriteLn(  HexStr(PLongInt(@S[1])^, 8) );     // 00000079 ---------------------------------------------------------------------- Avinash · 2024-12-12 00:51 Ok, thanks to tetrastes I understood how it all works, now I'm ready to reformulate why it's a bug. Let's again look at this expression from the point of view of all these codepages: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i S := UTF8Encode(#255) + #0#0#0#0;The documentation states that UTF8Encode returns RawByteString, in other words AnsiString(CP_NONE). And, if I understand it correctly, when concatenated with #0#0#0#0 it should not be converted to anything like a system CP, and the result should be the same as with ShortString() typecast: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i S := ShortString(UTF8Encode(#255)) + #0#0#0#0; But here, instead of AnsiString(CP_NONE), we get AnsiString(CP_UTF8). It seems that the RawByteString type simply doesn't work properly. Quote from: tetrastes on December 11, 2024, 10:34:58 pmUse {$codepage cp1251} Looks like a workaround. It's beyond my understanding why String <=> WideString conversion happens in some situations, but not in others: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  S: ShortString;  U: UnicodeString;begin   S := #$FF;    U := S;                             // there is exist type conversion  WriteLn(  HexStr(Ord(U[1]), 4)  );    // 044F   S := U;                             // there is exist type conversion  WriteLn(  HexStr(Ord(S[1]), 4)  );    // FF     S := UnicodeString(#$044F);           // there is exist type conversion  WriteLn(  HexStr(Ord(S[1]), 4)  );    // FF   U := ShortString(#$FF);               // there is NO type conversion  WriteLn(  HexStr(Ord(U[1]), 4)  );    // FF end. In summary, these things should work much more intuitively than they actually do. ---------------------------------------------------------------------- Thaddy · 2024-12-12 05:44 No, simply do not try to out-smart a compiler. ---------------------------------------------------------------------- tetrastes · 2024-12-12 09:05 Quote from: Avinash on December 12, 2024, 01:51:00 amBut here, instead of AnsiString(CP_NONE), we get AnsiString(CP_UTF8). It seems that the RawByteString type simply doesn't work properly. https://www.freepascal.org/docs- html/current/rtl/system/utf8encode.html: QuoteThe resulting string has code page CP_UTF8.And this is clearly seen: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function UTF8Encode(const s : WideString) : RawByteString;  var    i : SizeInt;    hs : UTF8String;  begin    result:='';    if s='' then    exit;    SetLength(hs,length(s)*3);    i:=UnicodeToUtf8(pchar(hs),length(hs)+1,PWideChar(s),length(s));    if i>0 then      begin        SetLength(hs,i-1);        result:=hs;      end;  end; I think that https://www.freepascal.org/docs- html/current/rtl/system/rawbytestring.html is very unclear and may be even improper: QuoteRawByteString is a single-byte character string which does not have any codepage associated with it. . . . the codepage of the destination is simply set to the codepage of the rawbytestring QuoteIt's beyond my understanding why String <=> WideString conversion happens in some situations, but not in others: It's useful to look at asm code to understand difference Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i # [14] S := #$FF;   movw   $65281,U_$P$PROGRAM_$$_S(%rip)# [16] U := S;                             // there is exist type conversion   leaq   U_$P$PROGRAM_$$_S(%rip),%rax   leaq   U_$P$PROGRAM_$$_U(%rip),%rcx   movq   %rax,%rdx   call   fpc_shortstr_to_unicodestr . . . # [26] U := ShortString(#$FF);               // there is NO type conversion   leaq   U_$P$PROGRAM_$$_U(%rip),%rcx   leaq   .Ld2(%rip),%rdx   call fpc_unicodestr_assign     ; here is simple assigning of constant, the same as U := #$FF;                ; because ShortString(#$FF) is ordinary constant and fpc converts it to UnicodeString constant at compile time. . . .section .rodata.n_.Ld2,"d"   .balign 8.Ld2$strlab:   .short   1200,2   .long   0   .quad   -1,1.Ld2:   .short   255,0 ---------------------------------------------------------------------- Thaddy · 2024-12-12 10:57 It did not occur to you that concatinating shortstrings beyond 255 is silly? or another word starting with S... ---------------------------------------------------------------------- tetrastes · 2024-12-12 13:50 Where do you see shortstrings beyond 255?  You so like that Red Evil... ---------------------------------------------------------------------- Thaddy · 2024-12-12 17:08 Well it started here: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  S: String;begin   //DefaultSystemCodePage := CP_OEMCP;   S := ShortString(UTF8Encode(#255)) + #0#0#0#0;Now explain to me that is sane code. About the most stupid code I have seen in 2024. It deserves a medal. ---------------------------------------------------------------------- tetrastes · 2024-12-12 17:44 Ah, you mean zeroes, not the length? Well, may be it is needed to OP for smth, may be this is for example only. If we change zeroes to f.e. twos, what's the difference for this issue? ---------------------------------------------------------------------- Avinash · 2024-12-12 18:36 Quote from: Thaddy on December 12, 2024, 06:08:04 pmNow explain to me that is sane code.The original task was to get #128..#255 characters of the local codepage in UTF8 sequence. Since WriteLn then outputs the first four bytes of the string, and UTF8Encode will return 2-3 characters (or even 0, who knows), I zero at least four bytes to ensure that the output will be result of UTF8Encode, and not possible garbage. Quote from: tetrastes on December 12, 2024, 10:05:41 amQuoteThe resulting string has code page CP_UTF8. Yes, I am wrong here. I did not look carefully at the description of UTF8Encode, in the old-fashioned way assuming that if the function returns a certain type in the header, then it returns this type, and not anything. This is not a bug in the literal sense, this is a mess of a bug scale. Let's go back to the original example: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  S: ShortString;begin   S := ShortString(UTF8Encode(#255)) + #0#0#0#0;  WriteLn(  HexStr(PLongInt(@S[1])^, 8)  );      // 0000BFC3   S := UTF8Encode(#255) + #0#0#0#0;             WriteLn(  HexStr(PLongInt(@S[1])^, 8)  );      // 00000079 end. You explained it Quote from: tetrastes on December 11, 2024, 08:33:30 pmThen there is concatenation. It doesn't change the ShortStrings, as they do not have CP information. But for AnsiStrings concatenation results in AnsiString with system CP (cp1251 here), and fpc tries to convert utf-8 to it, so 'ÿ' -> 'y'($79). but what is the difference between ShortString and AnsiString(CP_ACP) other than the mechanics of data/memory management? How else is the ShortString codepage interpreted other than the system one or the one specified in DefaultSystemCodePage? Why then is typecasting UTF8 to AnsiString results in $79, but to ShortString $BFC3 ? Next, if typecasting to ShortString works as getting byte sequence of a UTF8 string, then what typecast is needed to get this sequence of bytes as AnsiString, i.e. without the length limit of 255 characters? Okay, let's say #0#0#0#0 has CP_ACP encoding and therefore concatenation results in $79. Then let's try typecasting it to UTF8String: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  S: ShortString;begin   S := UTF8Encode(#255) + UTF8String(#0#0#0#0);    // 0000BFC3  :)  WriteLn(  HexStr(PLongInt(@S[1])^, 8)  ); end. it seems to work. But why then does the following give some strange result? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i S := UTF8String(#255) + UTF8String(#0#0#0#0);      // BFC283C3 ??? :( if we replace UTF8String with UTF8Encode in the second term, then everything is fine again: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i S := UTF8String(#255) + UTF8Encode(#0#0#0#0);      // 0000BFC3 Quote from: tetrastes on December 12, 2024, 10:05:41 amIt's useful to look at asm code to understand the difference We only see that compiler doesn't typecast (that was obvious anyway), not why. Why wouldn't it call fpc_shortstr_to_unicodestr before fpc_unicodestr_assign? He does that in case Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i S := UnicodeString(#$044F);how does a two-byte constant differ from a one- byte one? ---------------------------------------------------------------------- Remy Lebeau · 2024-12-12 23:34 Quote from: Fibonacci on December 11, 2024, 07:16:31 pmUTF8Encode() creates a UTF8String with codepage 65001 The encoding of the characters indeed follows codepage 65001, but the code shown earlier for UTF8Encode() clearly doesn't call SetCodePage() (or equivalent) to set the stored codepage of the returned RawByteString to codepage 65001, which will affect runtime conversions when mixing the RawByteString with other string types. ---------------------------------------------------------------------- Fibonacci · 2024-12-12 23:47 Quote from: Remy Lebeau on December 13, 2024, 12:34:31 amQuote from: Fibonacci on December 11, 2024, 07:16:31 pmUTF8Encode() creates a UTF8String with codepage 65001 The encoding of the characters indeed follows codepage 65001, but the code shown earlier for UTF8Encode() clearly doesn't call SetCodePage() (or equivalent) to set the stored codepage of the returned RawByteString to codepage 65001 Doesnt have to. It has a local variable UTF8String which is of type AnsiString(CP_UTF8), then its assigned to the result of type RawByteString and the CP is preserved. ---------------------------------------------------------------------- Remy Lebeau · 2024-12-13 05:33 Quote from: Fibonacci on December 13, 2024, 12:47:23 amDoesnt have to. It has a local variable UTF8String which is of type AnsiString(CP_UTF8), then its assigned to the result of type RawByteString and the CP is preserved. You're right, I missed that variable. I thought it was updating the Result directly.