Short strings small bug lazarus / 62-General ---------------------------------------------------------------------- Avinash · 2022-11-10 04:25 FPC 3.2.2 Win32 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  S: String; begin   S[0]:= #255;  S := S + #0;  WriteLn(Length(S));  { gives correct 255 }   S[0]:= #255;  S[Length(S) + 1] := #0;  WriteLn(Length(S));  { gives wrong 0,  Turbo Pascal gives 255 } end. Code responsible for this: Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ; [11] S[Length(S) + 1]:= #0;%LINE 11+0                movzx   eax,byte [U_$P$PROGRAM_$$_S]                lea     eax,[eax+1]                movzx   eax,al                mov     byte [U_$P$PROGRAM_$$_S+eax*1],0  And with {$O+} (what, by the way, does it turn on? I can't reproduce its effect by the -O1..4 switches): Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ; [12] S[Length(S) + 1]:= #0;%LINE 12+0                movzx   eax,byte [U_$P$PROGRAM_$$_S]                add     eax,1                and   eax,255                mov     byte [U_$P$PROGRAM_$$_S+eax],0 ---------------------------------------------------------------------- Bart · 2022-11-10 17:47 Quote from: Avinash on November 10, 2022, 05:25:54 amFPC 3.2.2 Win32 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  S: String; begin   S[0]:= #255;  S := S + #0;  WriteLn(Length(S));  { gives correct 255 }   S[0]:= #255;  S[Length(S) + 1] := #0;  WriteLn(Length(S));  { gives wrong 0,  Turbo Pascal gives 255 } end. Absoultey NOT a bug. TP gives range check error at compile time. Only if compiled with range checking off it'll output 255. However, you are accessing invalid memory in this case and therefore the behaviour is therefore undefined. You're just lucky the app doesn't crash. One can guess that fpc treats the index of a shortstring as a byte and this value will overflow: 255+1=0. Some compiler devel can comment on that. Bart ---------------------------------------------------------------------- Thaddy · 2022-11-10 18:06 A case of a "programmer" not counting from zero? The range is #0..#255 which is 256, so if you add/concat something to #255 you have a range error or an overflow. If you want to avoid that, use AnsiString, not shortstring. As Bart wrote, this is absolutely not a bug from the compiler side, it is frankly programmer error. The most important thing is  that after an overflow the compiler's behavior is undefined, hence it can be either rubbish, 0, or 255. Since it is undefined you can not rely on such things. As Bart also wrote, you should use overflow checking, at least during debugging. Wrong code is wrong code.... ---------------------------------------------------------------------- MarkMLl · 2022-11-10 19:39 Is assignment to element zero of a string even legal? I thought that other than the shortstring case it had been retired in favour of SetLength. MarkMLl ---------------------------------------------------------------------- Thaddy · 2022-11-10 20:05 Good question. I oversaw it, but.. element zero stores the length if it is shortstring, and strings start from element 1. Any other pascal string type also stores information below element 1, a.k.a. known as negative offset. ---------------------------------------------------------------------- howardpc · 2022-11-10 20:13 Thaddy. you didn't oversee it, you overlooked it. ---------------------------------------------------------------------- MarkMLl · 2022-11-10 20:30 Quote from: Thaddy on November 10, 2022, 09:05:17 pmGood question. I oversaw it, but.. element zero stores the length if it is shortstring, and strings start from element 1. Any other pascal string type also stores information below element 1, a.k.a. known as negative offset. Yes, but the length was an integer, which shouldn't- according to later versions of Object Pascal- be assignment-compatible with a character (i.e. the # prefix). I agree with Howard's comment about overwhatever, but (wearing my technical author's hat) the word which should be avoided at all costs is "oversight" which in current contexts is /massively/ ambiguous. MarkMLl ---------------------------------------------------------------------- Bart · 2022-11-10 21:41 Quote from: MarkMLl on November 10, 2022, 08:39:55 pmIs assignment to element zero of a string even legal? I thought that other than the shortstring case it had been retired in favour of SetLength. Since this is about TP vs FPC and by default in FPC String = ShortString (unless you have it otherwise specified in fpc.cfg by setting mode or by any other way), setting S[0] is perfectly legal. Bart ---------------------------------------------------------------------- Avinash · 2022-11-10 23:53 It looks like a some service is being provided, but that service is inadequate. It must be either a sensible thing, like Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i          movzx eax, byte ptr [S]         inc al         je @@1         mov byte ptr S[eax], 0  @@1: or it should not exist at all, and the behavior should be as documented in Turbo Pascal Version 7.0 Language Guide, page 55: The first character of a string variable (at index 0) contains the dynamic length of the string; that is, Length(S) is the same as Ord(S[0]). If a value is assigned to the length attribute, the compiler doesn't check whether this value is less than the declared size of the string. It's possible to index a string beyond its current dynamic length. The characters read are random and assignments beyond the current length don't affect the actual value of the string variable. ---------------------------------------------------------------------- dje · 2022-11-11 02:14 Quote from: Avinash on November 11, 2022, 12:53:07 amor it should not exist at all, and the behavior should be as documented in Turbo Pascal Version 7.0 Language Guide, page 55: Yes, and no. Page 55 also states: QuoteThe index expressions select components in each corresponding dimensions of the array. The number of expressions can't exceed the number of index types in the array declaration. The index type for a shortstring seems to be undefined. But, the guide also states: Quote...a value of PChar can be indexed with a singe index expression of type Word. The index expression specifies an offset to add to the character pointer before it's de-referenced.... So, a PChar index is of type word, and yes, FreePascal does this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   PChar(@s)[0] := #255;  PChar(@s)[length(s) + 1] := #0;  writeln(length(s));   Regardless, accessing outside array bounds has be undefined for decades, in all languages. It is processor and implementation dependent. eg: Many 68000 compilers only supported 16 bit indexes for speed reasons. It would be acceptable for a 6502 compiler to wrap 256 byte array accesses by using the 8 bit X/Y index registers, while the 6809 could support 16 bit indexes. Either way, Turbo Pascal made it clear that if you wanted to access behond 256 chars, you should use a PChar, but.... Even then, its limited to 64kb indexes (word's). Another example is, HSPascal (Atari ST) used 16 bit indexes, while PurePascal used 32 bit. One is faster than the other. ---------------------------------------------------------------------- Avinash · 2022-11-11 04:09 Quote from: dje on November 11, 2022, 03:14:01 amPage 55 also states: QuoteThe index expressions select components in each corresponding dimensions of the array. The number of expressions can't exceed the number of index types in the array declaration.It's about matching the number of indexes [I,J,K ..] to the array dimensions. In any case, this mechanism is not friendly to common sense, looks like a bug and generates slower and bulky code (movzx instead of mov or even extra lines of code) for no reason. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc}var  S: String;  X: array [Byte] of Char;  I: Integer;begin  X[I] := #0;  S[I] := #0;   S[Length(S) + 1]:= #0;end. No point in using movzx instead of mov: Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ; [36] X[I] := #0;%LINE 36+0                mov     eax,dword [U_$P$PROGRAM_$$_I]                mov     byte [U_$P$PROGRAM_$$_X+eax*1],0; [37] S[I] := #0;%LINE 37+0                movzx   eax,byte [U_$P$PROGRAM_$$_I]              mov     byte [U_$P$PROGRAM_$$_S+eax*1],0 There is no reason to add the line movzx eax,al here: Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ; [39] S[Length(S) + 1]:= #0;%LINE 39+0                movzx   eax,byte [U_$P$PROGRAM_$$_S]                lea     eax,[eax+1]                movzx   eax,al                mov     byte [U_$P$PROGRAM_$$_S+eax*1],0 ---------------------------------------------------------------------- PascalDragon · 2022-11-11 06:34 Quote from: MarkMLl on November 10, 2022, 08:39:55 pmIs assignment to element zero of a string even legal? I thought that other than the shortstring case it had been retired in favour of SetLength. For ShortString it is legal (Length and SetLength are simply wrappers around an access to element 0 in case of ShortString). Quote from: Avinash on November 11, 2022, 05:09:21 amIn any case, this mechanism is not friendly to common sense, looks like a bug and generates slower and bulky code (movzx instead of mov or even extra lines of code) for no reason. The length value of a ShortString is a Byte and FPC treats it as such when returned from Length(SomeShortString). Thus adding something will overflow if 255 is reached. TP and Delphi on the other hand seem to return the native integer size from Length(SomeShortString) nevertheless and thus adding a value will not overflow. Instead they mask it to 8-bit when assiging it to SomeShortString[0]. The Delphi documentation at least states that Length returns Integer (for FPC it's normally SizeInt except for ShortString). I'll have to try whether that upsets anything if I change the returned length from Byte to SizeInt. Quote from: Avinash on November 11, 2022, 05:09:21 amCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc}var  S: String;  X: array [Byte] of Char;  I: Integer;begin  X[I] := #0;  S[I] := #0;   S[Length(S) + 1]:= #0;end. No point in using movzx instead of mov: Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ; [36] X[I] := #0;%LINE 36+0                mov     eax,dword [U_$P$PROGRAM_$$_I]                mov     byte [U_$P$PROGRAM_$$_X+eax*1],0; [37] S[I] := #0;%LINE 37+0                movzx   eax,byte [U_$P$PROGRAM_$$_I]              mov     byte [U_$P$PROGRAM_$$_S+eax*1],0 There is no reason to add the line movzx eax,al here: Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ; [39] S[Length(S) + 1]:= #0;%LINE 39+0                movzx   eax,byte [U_$P$PROGRAM_$$_S]                lea     eax,[eax+1]                movzx   eax,al                mov     byte [U_$P$PROGRAM_$$_S+eax*1],0 The movzx are necessary, because FPC later on uses it to address memory by passing eax as index and FPC uses the native address width for that. Quote from: Avinash on November 10, 2022, 05:25:54 amAnd with {$O+} (what, by the way, does it turn on? I can't reproduce its effect by the -O1..4 switches): The {$O} switch controls level 2 optimizations. ---------------------------------------------------------------------- Avinash · 2022-11-11 07:20 Quote from: PascalDragon on November 11, 2022, 07:34:18 amThe length value of a ShortString is a Byte and FPC treats it as such when returned from Length(SomeShortString). Thus adding something will overflow if 255 is reached. I can't get overflow in either FPC or TP: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$Q+}  { Overflow checking }var  B: Byte;  I: Integer;  S: String;begin  S[0]:= #255;  I := Length(S) + Byte(10);  WriteLn(I);  {gives 265 in both FPC and TP}   B := 255;  I := B + Byte(10);  WriteLn(I);  {gives 265 in both FPC and TP}end. Also, this does not change anything in the assembler output (and does not apply to the case of a simple assignment to S[J], where movzx also occurs): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i S[Integer(Length(S)) + 1]:= #0; Quote from: PascalDragon on November 11, 2022, 07:34:18 amThe movzx are necessaryBut not the line movzx eax, al ---------------------------------------------------------------------- marcov · 2022-11-11 08:10 The range check is in converting the integer in s[ length(s)+1] back to a byte to fit in S[0]. Not in the calculation of the integer. ---------------------------------------------------------------------- PascalDragon · 2022-11-11 15:27 Quote from: Avinash on November 11, 2022, 08:20:28 amQuote from: PascalDragon on November 11, 2022, 07:34:18 amThe length value of a ShortString is a Byte and FPC treats it as such when returned from Length(SomeShortString). Thus adding something will overflow if 255 is reached. I can't get overflow in either FPC or TP: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$Q+}  { Overflow checking }var  B: Byte;  I: Integer;  S: String;begin  S[0]:= #255;  I := Length(S) + Byte(10);  WriteLn(I);  {gives 265 in both FPC and TP}   B := 255;  I := B + Byte(10);  WriteLn(I);  {gives 265 in both FPC and TP}end. The calculation is done in native platform size. As marcov said, the overflow check will be done at a different location. Quote from: Avinash on November 11, 2022, 08:20:28 amQuote from: PascalDragon on November 11, 2022, 07:34:18 amThe movzx are necessaryBut not the line movzx eax, al Yes, it is: Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     ; [39] S[Length(S) + 1]:= #0;    %LINE 39+0                    ; load length Byte and zero-extend into EAX                    movzx   eax,byte [U_$P$PROGRAM_$$_S]                    ; increment by 1                  lea     eax,[eax+1]                    ; zero extend lowest Byte into EAX                    movzx   eax,al                    ; store 0 at offset EAX of the ShortString                    mov     byte [U_$P$PROGRAM_$$_S+eax*1],0 ---------------------------------------------------------------------- Avinash · 2022-11-12 04:27 Then again – some mechanism that only works for shortstrings? Having no practical sense and pouring movzx'es? Here, no movzx appears: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc}var  X: array [Byte] of Char;  I: Integer;begin  X[I]:= #0;  X[I + 1]:= #0;end.Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ; [6] X[I]:= #0;%LINE 6+0                mov     eax,dword [U_$P$PROGRAM_$$_I]                mov     byte [U_$P$PROGRAM_$$_X+eax*1],0; [7] X[I + 1]:= #0;%LINE 7+0              mov     eax,dword [U_$P$PROGRAM_$$_I]                lea     eax,[eax+1]                mov     byte [U_$P$PROGRAM_$$_X+eax*1],0  In contrast to the strings: Code: ASM  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ; [20] S[I]:= #0;%LINE 20+0                movzx   eax,byte [U_$P$PROGRAM_$$_I]                mov     byte [U_$P$PROGRAM_$$_S+eax*1],0; [21] S[I + 1]:= #0;%LINE 21+0              mov     eax,dword [U_$P$PROGRAM_$$_I]                lea     eax,[eax+1]                movzx   eax,al                mov     byte [U_$P$PROGRAM_$$_S+eax*1],0  In 32-bit code, this compiles into 27 and 32 bytes, which is an 18% difference. ---------------------------------------------------------------------- mika · 2022-11-12 07:25 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program test_rangeOfString;{$mode tp}var s : string;    a : char;begin     s[0]:=#255;     a:='A';     writeln(a,'  len ' ,length(s));     s[length(s)+1]:='B';     writeln(a,'  len ' ,length(s));end. output TP 7 A  len 255 B  len 255 output FPC 3.2.2 A  len 255 A  len 66 As we can see there is "small" bug in Turbo Pascal. FPC do not have to be compatible with old bugs of TP. ---------------------------------------------------------------------- Thaddy · 2022-11-12 07:35 TP handles it always as string[255]. More in general it is not a very good idea to manipulate the length field yourself. ---------------------------------------------------------------------- Josh · 2022-11-12 09:55 Hi I  am not getting this, are we discussing a difference with the result of faulty user code? if you compile with range checking on for tp, and debug mode for fpc/laz then it fails to compile in tp7 and fpc. ---------------------------------------------------------------------- Avinash · 2022-11-12 10:53 If this is the «faulty user code»: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure XBuf(var Buf; Len: Integer);  var    I: Integer;    X: array [0..0] of Char absolute Buf;  begin    for I := 0 to Len-1 do      X[I]:= 'x';  end; var S: String;begin  S := '123456789';  XBuf(S[1], Length(S));  WriteLn(S);end. ...then try looking for [0..0] in the FPC sources, you will find many cases. Returning to the shortstrings, I can well imagine a couple of not meaningless functions: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function NewStrZ(const S: String): PString;var  P: PString;begin  if (@S <> Nil) and (S <> '') then  begin    GetMem(P, Length(S) + 2);    P^ := S;    P^[Length(S) + 1]:= #0;    NewStrZ := P;  end else    NewStrZ := Nil;end; procedure DisposeStrZ(var P: PString);begin  if P <> Nil then  begin    FreeMem(P, Length(P^) + 2);    P := Nil;  end;end; I will also point out once again that in FPC this difference is obtained not in some natural way, but by a special mechanism, as if the programmer always wrote S[Byte(I)] or S[I and $FF], which adds extra instructions to the resulting code. ---------------------------------------------------------------------- Josh · 2022-11-12 11:39 not at pc at the minute, but if memory serves, if using sysutils to get pstring; then string is now ansistring not shortstring; so you cannot access element 0 ---------------------------------------------------------------------- dseligo · 2022-11-12 12:03 Quote from: Josh on November 12, 2022, 10:55:20 amI  am not getting this, are we discussing a difference with the result of faulty user code? Yes, in FPC it doesn't fail in the way as he would want it. ---------------------------------------------------------------------- mika · 2022-11-12 12:43 @Avinash, I start to understand your pain. You don't want to be safeguarded on every iteration with ShortString while range check is off. ---------------------------------------------------------------------- PascalDragon · 2022-11-12 14:52 Quote from: Avinash on November 10, 2022, 05:25:54 amCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  S: String; begin   S[0]:= #255;  S := S + #0;  WriteLn(Length(S));  { gives correct 255 }   S[0]:= #255;  S[Length(S) + 1] := #0;  WriteLn(Length(S));  { gives wrong 0,  Turbo Pascal gives 255 } end. Your code is simply wrong. All three compilers (FPC, Delphi, TP) will raise a range check error when you enable range checks ({$R+}). Due to this this means that the code without range checks is undefined, thus anything is allowed to happen: overwriting memory, crashing, formatting your disks, nuking the world. You can see this with the following code compiled in either TP or Delphi: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program tshortstr; var  i1: Integer;  ss: ShortString;  i2: Integer;begin  i1 := High(Integer);  i2 := High(Integer);  ss[0] := #255;  ss[Length(ss) + 1] := #0;  Writeln(Length(ss), ' ', i1, ' ', i2);end. You will see that the value of i2 is changed. Long story short: this is not a bug and we will not change that behavior. Quote from: Avinash on November 12, 2022, 11:53:07 amIf this is the «faulty user code»: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure XBuf(var Buf; Len: Integer);  var    I: Integer;    X: array [0..0] of Char absolute Buf;  begin    for I := 0 to Len-1 do      X[I]:= 'x';  end; var S: String;begin  S := '123456789';  XBuf(S[1], Length(S));  WriteLn(S);end. ...then try looking for [0..0] in the FPC sources, you will find many cases. And such code will fail with enabled range checking and would also fail if the compiler would generate different code for it. ---------------------------------------------------------------------- Avinash · 2022-11-13 05:07 Quote from: PascalDragon on November 12, 2022, 03:52:32 pmthis is not a bug Well, a wrong settings. It has no practical value, produces overhead at each indexing, and generates unexpected side effects. I read the forum, words like «Declaring an array type always implies declaring two types: a range/index type, and the array type itself. When you index an array, the index will always be converted to the range type.» From here I propose to consider the index type of any array up to 255 elements as a byte, and up to 65535 as a word, which will allow to cram a bunch of movzx into the code and give rise to a lot of effects. Why not? Why should only the string be cursed? Quote from: PascalDragon on November 12, 2022, 03:52:32 pmthe code without range checks is undefined, thus anything is allowed to happen: overwriting memory, crashing, formatting your disks, nuking the world. C++ does not have a range check,  and yet it is not considered that such code is undefined and anything is allowed. ---------------------------------------------------------------------- Thaddy · 2022-11-13 09:05 Quote from: Avinash on November 13, 2022, 06:07:27 amC++ does not have a range check,  and yet it is not considered that such code is undefined and anything is allowed. No, because that is covered in their libraries, not the compiler, in Freepascal the compiler provides that safety. See the "safe" memory functions in std. You are comparing apples with pigeons, not even pears. ---------------------------------------------------------------------- PascalDragon · 2022-11-13 13:47 Quote from: Avinash on November 13, 2022, 06:07:27 amWhy not? Why should only the string be cursed? The behavior will not be changed to support faulty code. Quote from: Avinash on November 13, 2022, 06:07:27 amQuote from: PascalDragon on November 12, 2022, 03:52:32 pmthe code without range checks is undefined, thus anything is allowed to happen: overwriting memory, crashing, formatting your disks, nuking the world. C++ does not have a range check,  and yet it is not considered that such code is undefined and anything is allowed. Pascal is not C++. The rules of C++ do not apply to Pascal. The rules of Pascal apply to Pascal and in Pascal that is a range error and thus any code that is working in such condiitons is undefined behavior. Case closed.