How does the string handler Add a #0 at the end without adding length? lazarus / 31-LCL ---------------------------------------------------------------------- jamie · 2020-03-21 20:39 That is basically the question at hand... how does the #0 terminated get added without adding a length count ? Does the string manager ensures enough extra space to add that without adding count to the LENGHT returned?  If so I can just ensure there is a #0 at the end of the string without memory splash over I guess?  Am I on the right track here?  I only assume length in managed strings are kept in the header of the string.. ---------------------------------------------------------------------- eljo · 2020-03-21 21:00 yesQuote from: jamie on March 21, 2020, 09:39:14 pmThat is basically the question at hand... how does the #0 terminated get added without adding a length count ? Does the string manager ensures enough extra space to add that without adding count to the LENGHT returned?  If so I can just ensure there is a #0 at the end of the string without memory splash over I guess?  Am I on the right track here?  I only assume length in managed strings are kept in the header of the string.. 1) yes the internal memory manager always allocates length+1 chars for a string which is zero filled before returning it so the last char will always be 0. 2) the amount of memory reserved for a string dos not protect you from buffer overruns you still have to be aware of it in your code. 3)concatenating, reducing the length and all those internal string procedures are well debugged for those kind of problems. Sorry I have no idea where the length is kept if at all but as far as understand the managed type it is in an negative position from the one that is reported. ---------------------------------------------------------------------- lucamar · 2020-03-21 21:12 Quote from: eljo on March 21, 2020, 10:00:07 pmSorry I have no idea where the length is kept if at all but as far as understand the managed type it is in an negative position from the one that is reported. The Programmer's guide explains it quite well in Memory Issues - Data Types - String Types, though it doesn't a good job of explaining how the zero-termination is maintained. ---------------------------------------------------------------------- jamie · 2020-03-21 21:25 I looked at the Delphi docs and it seems the manager always adds extra for the null but when performing a LEHGTH check it reports back 1 less than what is actually in the table   There is a info before the pointer that is given to you for the string; Size, Reference count , size of chars and code page..  So there is a total of 4 ,32 bits fields in 32 bit land , the last one being the actual pointer to the start of the string.  So basically a string of length 1 is actually recorded as length 2 in the header..  I guess one could simple use -Pointer math to get to these headers .. Edit:   I guess there are two allocated blocks of memory, one for the HEADER info and the other the actual string, I think the actual string array has the additional mass needed ..  Ether way, it would work.. ---------------------------------------------------------------------- jamie · 2020-03-21 21:51 apparently I've uncovered a bug.. This could have been fixed by now but.... using POS(#0, Pchar(Astring)) always returns 0. It fails to report the position of the trailing #0  I think someone thought it must of been a great idea to first convert this to normal string and then call a managed string operator on it.   For some reason, I thought this worked in the past... maybe it does not in the 64 bit target which I am using at the moment..  But I tested this on a (CONST S:STRING) from a passed parameter...  Pos(#0, Pchar(S)) always returns 0. That is a bug!  Can someone with a higher compiler test that ? I am using 3.0.4-64 ---------------------------------------------------------------------- eljo · 2020-03-21 22:13 can you provide a small app that reproduces the bug? just a console app that writeln's the result of the pos would be enough. ---------------------------------------------------------------------- jamie · 2020-03-21 22:33 I have an old version of Delphi (3) and it fails there too, even the StrPos fails.. Its a simple test. Var  S:String; Begin  S :='Something';  Writeln(Pos(#0,Pchar(S)); End;  Only Pos(#0,S) should return a 0 if not inside the string but when its a PCHAR(S), it should be treated as an array of char and report the first found #0, which could be the end of the string of course but apparently it's using it to mark the end of the string and search no more.  POS should know if you are looking specifically for a #0 then it must be the one at the end.. this only applies if you are doing PCHAR's not other types ---------------------------------------------------------------------- Bart · 2020-03-21 22:40 A PChar terminates with a #0, so a #0 can never be part of a PChar itself (unlike Pascal strings). Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$apptype console}var  S: AnsiString;begin  S := 'foo';  writeln(Pos(#0,PChar(S)));  S := 'foo'#0;  writeln(Pos(#0,PChar(S)));end. Prints: Code: [Select]0 0 This is as expected, and Delphi does exactly the same. Bart ---------------------------------------------------------------------- marcov · 2020-03-21 22:47 Quote from: jamie on March 21, 2020, 11:33:43 pm Only Pos(#0,S) should return a 0 if not inside the string but when its a PCHAR(S), it should be treated as an array of char and report the first found #0, No arrays supported, so it is probably taken as if empty string is in pchar(s). For *nix use there are some deprecated array of string to array of pchar routines in unixutils ( https://www.freepascal.org/docs- html/current/rtl/unixutil/arraystringtoppchar.html ) ---------------------------------------------------------------------- jamie · 2020-03-21 22:55 in my old DELPHI both POS and STRPOS fail however, with FPC 3.0.4 StrPOS works as it should but POS still does not  If you POS(#0, PCHAR(S)); it should work as does STRPOS does...  It's obviously a bug in my old Delphi that got fixed in fpc, at least for the StrPos but still persist for the POS when addressing PChar's   I'll stop using POS for this because it's broken... I'll use StrPos..   POS should only be reporting this with managed strings that are not casted to a Pchar..  POS(#0, AManagedStringThatIsn'tCastedToAPchar) = 0 if not found within the string of said marked length. which is how that should work. To answers Bart's reply, #0 should be reported even though its the end of the string marker.. when doing PCHAR's that is how you find the end of the string.  When POS see's these combinations it should be returning the position found etherway because that string could be sitting in the middle of a large block of memory just waiting to get more added to it and the simplest way to do this is to locate the #0 at the end.  StrPos does what it should be doing we'll stick to that for now until POS gets fixed... ---------------------------------------------------------------------- winni · 2020-03-22 00:02 Hi! As Bart told you: #0 is not part of a Pchar but the END of a pchar. It is not included in the array. Whatever pos, Strpos or any other system function reports: If they return not a zero then they are wrong . Winni ---------------------------------------------------------------------- jamie · 2020-03-22 00:48 I am sorry you don't get it.. POS for a pchar has no overload so what it is doing it first recreating an Ansistring using the NULL (#0) as the end of line marker instead..  It should not be doing that for Pchar(S) etc..  It should be doing what StrLen does.. and That is return the position of the Null..  But StrPos is not part of the System unit but in the Systils.. etc  I won't go on with this any longer, its wrong the way it is now....  End of story. ---------------------------------------------------------------------- winni · 2020-03-22 00:50 Bravo! ---------------------------------------------------------------------- jamie · 2020-03-22 11:53 Allow me to add a failing example that contradicts some other comments about #0 not being in the string is why this is.. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i VarS:String;begin S := 'Test'+#0; Caption := Pos(#0, Pchar(S));  //Fails;  However this does work .. Caption := POS(#0, S); ---------------------------------------------------------------------- Bart · 2020-03-22 11:54 BugFix especially for Jamie: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function Pos(AChar: AnsiChar; APChar: PAnsiChar): SizeInt; overload;begin  if (AChar = #0) then    Result := StrLen(APChar)  else    Result := System.Pos(AChar, APChar);end;  var  S: AnsiString;begin  S := 'foo';  writeln(Pos(#0,PChar(S)));  S := 'foo'#0;  writeln(Pos(#0,PChar(S)));end. Outputs: Code: [Select]3 3 Feel free to be incompatible with the rest of the Pascal community. Bart ---------------------------------------------------------------------- jamie · 2020-03-22 12:10 If you are not willing to fix that it should at least throw an error when it detect this situation.. and if you want to go that way, who ever heard of allowing #0 in a string and call it valid  but yet you can't use a valid Char operation and get what you would expect..  If you don't think this should be fix the code should FAULT at runtime! when ever a case like this pops up.  This operation gives no indication that is isn't doing what's to be expected.  So please, ether we fix this or modify the code so it won't be allowed, at lease this way some other bugs in code can be found that are sitting there dormant and pop out unexpected times with no recourse of where to look! ---------------------------------------------------------------------- PascalDragon · 2020-03-22 12:18 Quote from: jamie on March 22, 2020, 01:48:27 amI am sorry you don't get it.. POS for a pchar has no overload so what it is doing it first recreating an Ansistring using the NULL (#0) as the end of line marker instead..  It should not be doing that for Pchar(S) etc..  It should be doing what StrLen does.. and That is return the position of the Null.. You are wrong. The purpose of StrLen is to determine the length of the passed PChar. That just happens to coincide with the index of the terminating NUL-character. Your PChar is indeed converted to an AnsiString due to there not being an overload for PChar. And this means that the resulting AnsiString will contain StrLen characters which is to mean the PChar without the terminating NUL-character. Thus Pos will never find that terminating NUL, because that NUL is never inside the string. If you're dealing with PChars then please use the functions for PChars (namely StrPos in this case). Quote from: jamie on March 22, 2020, 12:53:17 pmAllow me to add a failing example that contradicts some other comments about #0 not being in the string is why this is.. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i VarS:String;begin S := 'Test'+#0; Caption := Pos(#0, Pchar(S));  //Fails;  However this does work .. Caption := POS(#0, S); That doesn't contradict anything that was said. The layout of the string S is Test#0#0. The first NUL is part of the string (Length(S) will return 5), the second NUL is not part of the string as that is the terminator. Now you cast that string to PChar, that means that the PChar will essentially be Test#0 with StrLen(S) returning 4 and that remaining NUL-character being the terminator. Now if you pass that to Pos that will - as mentioned above - be converted to an AnsiString, but this time with the layout Test#0 where Length will be 4 and the NUL-character will not be part of the string. If you're mixing Pascal and NUL-terminated you need to know what you're doing! This is all by design. ---------------------------------------------------------------------- Bart · 2020-03-22 12:22 Why do you think that you are the only person that is right in this discussion and everybody else including the fpc devels and Borland and Embarcadero are wrong? As explained before a #0 is always the terminator for a PChar. Since Pos() will only search to the end of the PChar not including the terminator, since this is not part of the actual data it can only ever return 0 as result, as it cannot ever find a #0 inside a PChar. Pascal strings were designed to be able to have any char inside it, including #0. The has been exploited in may programs to store extra data inside a string, knowing that the OS would never display them with any cooked (as opposed to raw) IO procedure (writeln etc.). I.e. one of my old programs used it as a start of a signature in the first record of a datafile. Bart ---------------------------------------------------------------------- MarkMLl · 2020-03-22 12:56 Quote from: jamie on March 21, 2020, 10:51:39 pmusing POS(#0, Pchar(Astring)) always returns 0. It fails to report the position of the trailing #0 That's not a bug, it's a feature. Pos() works on Pascal-style strings and the result can't exceed the length, Length('A') is 1 irrespective of whether it's got a trailing \0 internally for convenience when treating it as a C-style character array. I think what you've not allowed for is that there are two distinct lengths involved: 1) The number of valid characters stored in the string (or entries in a list etc.). 2) The amount of storage allocated by the memory manager, which is likely to be some multiple of the physical word/alignment size. The requirement for \0 beyond the end of a string merely means that the difference between (1) and (2) is never zero, whatever you might like to believe >:-) MarkMLl ---------------------------------------------------------------------- jamie · 2020-03-22 13:20 Say as you will .. I am not trying to be adamant about it , its just that is how I've been brought up to code.. in my mind this is ridiculous.  Please force the function to FAULT!!!!!!!!!!! under these conditions... I would wager it would uncover lots of bugs. In the mean time I have created an overload function which works as it should, I'll put that in my ExtHelper unit I include now in just about everything I do..  I tried to inline it because its such a short function but for what ever reason the compile wants to call it as a non inline.  So I resorted to a Assembler call using a string search which works fine, except it won't be cross platform. Work in that later.. That in itself was a task also... Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Function POS(aChar :Char; APChar:Pchar):Integer; overload; inline;Begin  Result:=0;  While (aPChar<>Nil)and(aPChar[Result]<>aChar)do Inc(Result);end;    That won't inline, I did ensure the {$INLINE ON} is in there, too. but in anycase the Assembler one seems to work..  Thanks to all and please don't let me upset the PASCAL community.  Apparently it would be worst than a nuclear explosion. ---------------------------------------------------------------------- 440bx · 2020-03-22 13:34 Quote from: jamie on March 21, 2020, 10:51:39 pmusing POS(#0, Pchar(Astring)) always returns 0. It fails to report the position of the trailing #0 Pos(#0, Pchar(S)) always returns 0. That is a bug! It isn't a bug.  You'll find that the equivalent C function (strstr) behaves exactly the same (returns a pointer to the beginning of the null terminated array, not a pointer to the null.) The reason it isn't a bug is because the null (#0) is not considered a character.  It is a sentinel that marks the end of the character sequence.  Functions that are designed to return character positions are not going to return the position of the non-character sentinel. you can try this to convince yourselfCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  const  ntdll = 'ntdll.dll'; function StrPos  (SearchIn     : pchar;                SearchFor    : pchar)       : pchar;   cdecl; external ntdll name 'strstr';     { !! renamed       } var  s : packed array[0..2] of char = ('a', 'b', 'c');  p : pchar;   i : integer;  n : integer;  BEGIN  p := StrPos(s, #0);  // use NTDLL's implementation of StrPos (strstr)   n := 0;  for i := 0 to high(s) + 1 do  begin    if s[i] = #0 then break;    inc(n);  end; On exit StrPos will have p point to the first character of s, not the fourth one.  That is the same as 0 (zero) if instead of returning a pointer it returned an index as the Pascal function does. if you want the position of the terminator/sentinel then you can code your own loop, like the "for" in the above code or, simpler, a call to lstrlen will return the index because when it says zero, it means the zeroth character.  For the array above, it would return 3, and since the array is zero based, that is also the index of the null terminator, i.e s[3]. There is no bug but, it obviously bugs you. ---------------------------------------------------------------------- marcov · 2020-03-22 13:50 Quote from: jamie on March 22, 2020, 12:53:17 pmAllow me to add a failing example that contradicts some other comments about #0 not being in the string is why this is.. In Pascal #0's are allowed to be in the string. So that #0 of yours is included in the string and after it comes the terminating #0, which is not part of the string. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  S := 'Test'+#0;  .... will basically be Test#0#0 with length 5 QuoteCaption := POS(#0, S); Will search data at s for #0 with a maximum of 5 characters. (a strnpos() in C terms) Yes. And now do I posex starting on the position after the found #0 to find the terminating one. That won't work, because that one is outside the string, so it will simply return false because there are no more chars in the string to search. ---------------------------------------------------------------------- PascalDragon · 2020-03-22 14:01 Quote from: Bart on March 22, 2020, 01:22:36 pmAs explained before a #0 is always the terminator for a PChar. Since Pos() will only search to the end of the PChar not including the terminator, since this is not part of the actual data it can only ever return 0 as result, as it cannot ever find a #0 inside a PChar. Small clarification: Pos does not search inside the PChar, but inside the AnsiString converted from the PChar, which is a small, but important difference. Quote from: MarkMLl on March 22, 2020, 01:56:54 pmQuote from: jamie on March 21, 2020, 10:51:39 pmusing POS(#0, Pchar(Astring)) always returns 0. It fails to report the position of the trailing #0 That's not a bug, it's a feature. Pos() works on Pascal-style strings and the result can't exceed the length, Length('A') is 1 irrespective of whether it's got a trailing \0 internally for convenience when treating it as a C-style character array. I think what you've not allowed for is that there are two distinct lengths involved: 1) The number of valid characters stored in the string (or entries in a list etc.). 2) The amount of storage allocated by the memory manager, which is likely to be some multiple of the physical word/alignment size. The requirement for \0 beyond the end of a string merely means that the difference between (1) and (2) is never zero, whatever you might like to believe >:-) MarkMLl In fact it's even three sizes: your 1) and 2) plus as a 3) the size of the string plus header (containing the length, the code page and the reference count) and terminating NUL-character. Quote from: jamie on March 22, 2020, 02:20:43 pm Please force the function to FAULT!!!!!!!!!!! under these conditions... I would wager it would uncover lots of bugs.                                                It can not and in fact must not fault, because it is behaving exactly as designed! ---------------------------------------------------------------------- jamie · 2020-03-22 14:18 Without getting ROCKS thrown at me, then why can't you FORCE a RUNTIME fault when a #0 is in the search criteria  for PCHAR types ? At lease this would give the coder some what not to do instead of a silent fail. ---------------------------------------------------------------------- marcov · 2020-03-22 14:24 Quote from: jamie on March 22, 2020, 03:18:39 pmWithout getting ROCKS thrown at me, then why can't you FORCE a RUNTIME fault when a #0 is in the search criteria  for PCHAR types ? At lease this would give the coder some what not to do instead of a silent fail. The need to typecast should be the warning. The typecast says you know what your are doing. ---------------------------------------------------------------------- PascalDragon · 2020-03-22 14:47 Quote from: jamie on March 22, 2020, 03:18:39 pmWithout getting ROCKS thrown at me, then why can't you FORCE a RUNTIME fault when a #0 is in the search criteria  for PCHAR types ? At lease this would give the coder some what not to do instead of a silent fail. What you fail to grasp is that Pos here never sees a PChar. It only sees an AnsiString, because the PChar argument is implicitely cast to an AnsiString before being handed to Pos. Thus there can not be any handling of that. Quote from: marcov on March 22, 2020, 03:24:01 pmQuote from: jamie on March 22, 2020, 03:18:39 pmWithout getting ROCKS thrown at me, then why can't you FORCE a RUNTIME fault when a #0 is in the search criteria  for PCHAR types ? At lease this would give the coder some what not to do instead of a silent fail. The need to typecast should be the warning. The typecast says you know what your are doing. Well, to be fair, jamie casts to a PChar, probably assuming there's a PChar-compatible overload and the compiler instead converts it to a AnsiString. Then again all this is by design. The terminating NUL-character is not considered part of the string, thus even with a PChar-turned- AnsiString this behavior is correct. ---------------------------------------------------------------------- MarkMLl · 2020-03-22 17:33 Quote from: PascalDragon on March 22, 2020, 03:01:47 pmIn fact it's even three sizes: your 1) and 2) plus as a 3) the size of the string plus header (containing the length, the code page and the reference count) and terminating NUL-character. I was assuming that fixed overheads were included in the (2) - (1) difference. MarkMLl