const string vs. string lazarus / 62-General ---------------------------------------------------------------------- jollytall · 2021-12-20 07:53 I have a very weird error, but maybe somebody knows the answer: I have a simple string splitting function: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function ParseString(const aString : string; const aDelimiter : string; var aBefore : string; var aAfter : string) : boolean;  var    lPosition : int64;  begin  lPosition := PosEx(aDelimiter,aString);  result := lPosition > 0;  if not result then    begin    aBefore := aString;    aAfter := '';    exit;    end;  aBefore := Copy(aString,1,lPosition - 1);  aAfter := Copy(aString, lPosition + aDelimiter.Length);  end;I call it with aString generated from Laz2_DOM (i.e. the UTF-8 compatible) including UTF-8 characters. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i s := Node.NodeValue;After the split aBefore is OK, but aAfter turns accented characters into garbage. If I copy the very same string (printed from my program the original and copied manually) into a string likeCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i s := 'as far as I can tell, the same text öüóőúéáűí'; and call the function with this string then both aBefore and aAfter are OK. I checked both inside the function, and visually in both cases I see exactly the same text in aString, still the aAfter result is different To make things even more strange, if I change the function declaration to Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function ParseString(aString : string; const aDelimiter : string; var aBefore : string; var aAfter : string) : boolean; i.e. only remove the const from aString, but leaving const in front of aDelimiter, it works OK. Any idea? ---------------------------------------------------------------------- Zvoni · 2021-12-20 09:27 1) Why in blazes don't you just use Copy2SymbDel from StrUtils? Or can your Delimiter be longer than 1 Char? https://www.freepascal.org/docs- html/rtl/strutils/copy2symbdel.html 2) Just tested with Copy2SymbDel: Confirmed: using his Test-String above i get Garbage for the Umlauts on the Console Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Project1;Uses StrUtils;Var  s:String;  r:String;begin  s := 'as far as I can tell, the same text öüóőúéáűí';  r:=Copy2SymbDel(s,',');  Writeln(s);  // returns -->  the same text öüóÅ'úéáűí  Writeln(r);end. ---------------------------------------------------------------------- jollytall · 2021-12-20 09:50 Yes, my delimiter can be longer than 1 char. And I also need both the before and after part (if not, I use another function). What is strange - as mentioned - that the aBefore is not ruined, only aAfter. So 'őőőabcdááá' with delimiter 'bc' would split into 'őőőa' and dgarbage. And then again, what is even worse, that if the const is not in front of it, it works well for any string and even with const in it, it works for strings I assign value to, but not with strings assigned through a node, even when they seem to be the same. My guess is that somewhere in the "metadata" (how to call it, how to test it?) of string there is a difference. ---------------------------------------------------------------------- jollytall · 2021-12-20 09:57 One more thought. I guess it is somehow related to the fact, how a position is identified, as byte position or utf codepoint. Maybe the metadata (if that exists) is different for the two assignment methods and therefore the second copy used works different. When there is no const, it probably makes an internal copy from the passed parameter to an internal string variable (although for string it should only be a reference count increase until the content is not changed - so it should not change much) and that "sorts-out" the UTF identification somehow. But it is only guessing. ---------------------------------------------------------------------- marcov · 2021-12-20 10:07 If you are on windows, you have to enable some utf8 measures. Keep in mind that the LCL switches to utf8, but console apps not. ---------------------------------------------------------------------- Zvoni · 2021-12-20 10:09 Found something: https://wiki.freepascal.org/Character_and_string_types insert a {$codepage utf8}  directive Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Project1;{$codepage utf8}Uses StrUtils;Var  s:String;  r:String;begin  s := 'as far as I can tell, the same text öüóőúéáűí';  r:=Copy2SymbDel(s,',');  Writeln(s);  // returns -->   the same text öüóőúéáűí ---> CORRECT!  Writeln(r);  // returns -->  as far as I can tellend. Forgot: FPC320/LAZ2.0.12 32-Bit on Win10-64Bit EDIT: Just tested your function with {$codepage utf8}: works ---------------------------------------------------------------------- Martin_fr · 2021-12-20 10:57 Do you by any chance call "ParseString" something like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  ParseString(MyText, ',', FoundBefore, MyText) So you get the remainder back in "MyText"? Because in that case Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i aAfter := Copy(aString, lPosition + aDelimiter.Length);"aAfter" is the same string as "aString" => and therefore this operation modifies "aString". But if "aString" is declared "const", then that is a violation. And you are lucky it doesn't just crash.... https://www.freepascal.org/docs- html/current/ref/refsu67.html#x183-20700014.4.4 QuoteSpecifying a parameter as Constant is giving the compiler a hint that the contents of the parameter will not be changed by the called routine. In other words, you promise to the compiler that "aString" will not be changed. Not be changed by any means: neither direct, nor indirect. If the compiler can spot, that you break the promise, it will tell you. If you break it, and the compiler is not able to detect it, your code fails. As it says: QuoteThis allows the compiler to perform optimizations which it could not do otherwiseIf you broke the promise, you tricked the compiler to do things that do not work (i.e. you made it "otherwise", but did not tell the compiler). Same applies, if you pass a global var or field in your class, and modify that global var/field. Or if by any other way, you get access to the variable passed as const, or to its content. ("content": Afaik for objects (instances of a class), only the object variable (I.e. the hidden pointer) is treated as "const", the fields are permitted to be changed  / For arrays, I am not sure). Also it is not just strings. All managed types (strings/array/...) are affected by this. Even none managed types (records, even simple integer, so for "integer" you would need "constref") can be affected. Just with those you are a lot more likely, that it will work due to pure luck. ---------------------------------------------------------------------- PascalDragon · 2021-12-20 12:48 Quote from: Zvoni on December 20, 2021, 10:27:23 am2) Just tested with Copy2SymbDel: Confirmed: using his Test-String above i get Garbage for the Umlauts on the Console Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Project1;Uses StrUtils;Var  s:String;  r:String;begin  s := 'as far as I can tell, the same text öüóőúéáűí';  r:=Copy2SymbDel(s,',');  Writeln(s);  // returns -->  the same text öüóÅ'úéáűí  Writeln(r);end. And if you do a Writeln(s) before the Copy2SymbDel you'll see the same 'garbage' on the console as afterwards. That is because by default the file has codepage CP1252. If you want to handle it as UTF-8 you either need to use the $codepage directive or the file needs an UTF-8 BOM. ---------------------------------------------------------------------- Zvoni · 2021-12-20 13:18 PD, yeah, found that out, too. See Post #5 ---------------------------------------------------------------------- Josh · 2021-12-20 13:47 hi lazutf8 has some useful routines Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Project1;uses lazutf8;var s,aft,bef:string;  p,l:longint;begin  s:='as far as I can tell, the same text öüóőúéáűí';  p:=UTF8Pos(',',s);  l:=utf8length(s);  aft:=UTF8Copy(s,1,p-1);  bef:=utf8copy(s,p+1,l-p);  writeln(s);  writeln(aft);  writeln(bef);  readln;end.   produces Quoteas far as I can tell, the same text öüóoúéáuí as far as I can tell  the same text öüóoúéáuí ---------------------------------------------------------------------- winni · 2021-12-20 14:56 Hi! For the sake of lazyness - let the machine work. Instead of Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i bef:=utf8copy(s,p+1,l-p);  you can do Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i bef:=utf8copy(s,p+1,maxint); utf8copy checks the parameters before it copies. Winni ---------------------------------------------------------------------- jollytall · 2021-12-20 16:24 Quote from: marcov on December 20, 2021, 11:07:36 amIf you are on windows, you have to enable some utf8 measures. Keep in mind that the LCL switches to utf8, but console apps not. No, I am on Linux. Quote from: Martin_fr on December 20, 2021, 11:57:11 amDo you by any chance call "ParseString" something like this: OMG. Sometimes I indeed do this, but I am not sure it really happens as you say: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; {$mode objfpc}{$H+} var s: string; procedure a(const aIn : string; var aOut : string);  begin  s := 'Second value';  writeln(aIn);  aOut := 'Third value';  writeln(aIn);  end; begin s := 'First value';a(s, s);writeln(s);readln;end.and the result is Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i First valueFirst valueThird valueBased on what you wrote and what is in the manual you linked, the first output should already be "Second value" as I change the underlying global s behind aIn before the writeln, but it does not happen. Furthermore, in your logic, the second output should be "Third value", what I am not even sure. In this case I do not change the variable behind aIn. I always thought that in this case a new variable is created for aOut and once the procedure is completed, it is only then assigned to the calling s, so it cannot get crazy. Anyway, regardless what the theory is, in practice this is not failing either. Quote from: josh on December 20, 2021, 02:47:49 pmhi lazutf8 has some useful routines Thanks, I had checked them, but few comments. I am a bit reluctant, as in fpc lazutils is not a standard dependency, so why to complicate. Also, it says everywhere that string is fully utf-8 compliant (except the length, what is byte length, not character length and so a copy can cut a character at a wrong place), but (especially because my aDelimiter is always simple characters) Pos and Pos based cutting should be OK. And last but definitely not least, I still do not understand, what is the difference between s := 'my string'; and s := Node.NodeValue; especally because before calling the failed ParseString function, I already touch s many times. ---------------------------------------------------------------------- Martin_fr · 2021-12-20 16:56 Quote from: jollytall on December 20, 2021, 05:24:28 pmQuote from: Martin_fr on December 20, 2021, 11:57:11 amDo you by any chance call "ParseString" something like this: OMG. Sometimes I indeed do this, but I am not sure it really happens as you say: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure a(const aIn : string; var aOut : string);  begin  s := 'Second value';  writeln(aIn);  aOut := 'Third value';  writeln(aIn);  end; begins := 'First value';a(s, s);end.and the result is Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i First valueFirst valueThird valueBased on what you wrote and what is in the manual you linked, the first output should already be "Second value" as I change the underlying global s behind aIn before the writeln, but it does not happen. Furthermore, in your logic, the second output should be "Third value", what I am not even sure. In this case I do not change the variable behind aIn. I always thought that in this case a new variable is created for aOut and once the procedure is completed, it is only then assigned to the calling s, so it cannot get crazy. Anyway, regardless what the theory is, in practice this is not failing either. Nope. "aIn"  is a pointer to the memory that contains 'First Value'. "aOut" is a pointer to a managed-pointer (aka AnsiString) to that memory. "aIn" is not the same pointer as "s", it just points to the memory. "aOut" is a pointer to "s" When you assign to "s" ("s := '....';" => you do not (necessarily [1]) change the memory containing 'First Value'. You point "s" to new memory, with new content. [1] => under certain circumstances, when the new value is shorter (or only a tiny bit longer, in certain boundaries), and a few other conditions are met, the new value may get stored in the "current" memory. Then you would change the memory, and keep the pointer. Now => "s" is originally assigned a constant. That memory with the constant is there forever. It will never change. Therefore => the pointer in "aIn" still points to that memory. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i s := copy('First value', 1, 5+random(2));Not a constant. Now the output depends, on what happens with the memory. Assigning to "s" may or may not change the old memory. - you try playing around by adding/removing -gh (aka heaptrc from the debug options. - or adding something like "for i := 1 to 100 do StringOfChar('x',i mod 32);" which will hopefully re-use the memory... You can also test (the non-constant) by replacing Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i       //s := 'Second value';      s[1]:='X'; Which will be predictable. ---- You also have to make sure, not to have any other var, to which you assigned the string. Doing in the main body Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i begins := copy('First value', 1, 5+random(2));s2 := s; // add a 2nd ref => protects the memorya(s, s); And all changes. Changing "aOut" from var, to a normal (no var/const/out) param, also adds another ref... ---------------------------------------------------------------------- BeniBela · 2021-12-20 17:37 Quote from: Martin_fr on December 20, 2021, 11:57:11 amDo you by any chance call "ParseString" something like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  ParseString(MyText, ',', FoundBefore, MyText) So you get the remainder back in "MyText"? Because in that case Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i aAfter := Copy(aString, lPosition + aDelimiter.Length);"aAfter" is the same string as "aString" => and therefore this operation modifies "aString". But if "aString" is declared "const", then that is a violation. And you are lucky it doesn't just crash.... That should really show a warning in that case ---------------------------------------------------------------------- Martin_fr · 2021-12-20 18:00 QuoteCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function ParseString(const aString : string; const aDelimiter : string; var aBefore : string; var aAfter : string) : boolean;Quote from: BeniBela on December 20, 2021, 06:37:53 pmQuote from: Martin_fr on December 20, 2021, 11:57:11 amDo you by any chance call "ParseString" something like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  ParseString(MyText, ',', FoundBefore, MyText) That should really show a warning in that case Well technically that warning would be wrong. Because *currently* afaik the below is "save". Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   TextCopy := MyText;  ParseString(MyText, ',', FoundBefore, MyText) - I have no idea if this is by documentation. I.e., if this is future save. - I would not suggest to do this, because you couldn't booby trap your code in a more effective way. ---------------------------------------------------------------------- BeniBela · 2021-12-21 13:22 Quote from: Martin_fr on December 20, 2021, 07:00:09 pmWell technically that warning would be wrong. That is not a problem. FPC shows all kind of warnings about things that are fine (e.g. variables assigned but not used, unhandled enums in case ) I also wonder if this is documented to be safe: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function test(const s: string): string;//...var a: string;a := test(a); ---------------------------------------------------------------------- jollytall · 2021-12-21 16:19 This must be. I use it all the time, e.g.: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i s := Trim(s);That would be very stupid if you had to do: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i q := Trim(s);s := q; ---------------------------------------------------------------------- Josh · 2021-12-21 16:34 Hi I thought constants were that. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var f:string=' fred'; function test(const s:string):string;begin  s:=trim(s);  <- compile fails as expectedend; begin  f:=test(f);end; Error: Can't assign values to const variable apologies if i mis understand ---------------------------------------------------------------------- jollytall · 2021-12-22 07:26 I understood BeniBela's original example as the s := Test(s) is OUTSIDE of the function, (marked //...) . I used Trim as another example that I call in s := Trim(s); style. It is not inside the theoretical Test function. If it is INSIDE the function, then it is obviously a compile error as noone can assign to a const parameter. Also, BeniBala's example would be a nice infinite loop (stack overflow) even if no const is used, so I am sure, he meant outside, like your line #9. The question, whether that is documented to be safe, vs. using the return value as a parameter. So, to me it is still a question, ifCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function Test(const s: string) : string;  begin  result := s + s;  end;//...Str := Test(Str); is safe (I think it must be, even if it would not be documented), then why wouldCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure Test(const s: string; var q : string);  begin  q := s + s;  end;//...Str := Test(Str);not be safe (back to Martin_fr's original comment). ---------------------------------------------------------------------- Martin_fr · 2021-12-22 10:21 In the below "AnsiString" should be read as "any managed string type" (afaik: aka LongString, but not ShortString). Quote from: BeniBela on December 21, 2021, 02:22:55 pmI also wonder if this is documented to be safe: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function test(const s: string): string;//...var a: string;a := test(a);  Well, the question is awkwardly worded: "documented to be safe" => Usually, it's the opposite. Something may be documented as "not supported" or any other attribute indicating that it can not used, or at least can not be used safely. => For example: "a := 100;" is not (explicitly) documented as safe. It is save, because the documentation does not say the opposite. "const param" are attributed as "the user giving the compiler a hint that " => that is the user has checked that . And found it to be so. Consequently, if it is not (if the hint is *incorrect*), then the result (resulting compiled code) can not be expected to be *correct*. About "a := test(a)". There is an order in that statement: 1) evaluate the right hand side expression. (In this case call "test(a)" 2) assign the result to "a" (2) is executed after (1). => This "execution order" can also be seen in the following code.  - I am not sure if/where this behaviour (of the code below) is documented (i.e. explicitly).  - But afaik, it is at least given by not having "the opposite" documented. As for the "execution order", there is no documentation that "adding a var param" will change that execution order. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var MyGlobal: TSomething;procedure Foo(a: TSomething); // not const/var/out or constrefbegin  a := SomeValue;  if MyGlobal = SomeValue then     DoBadThings;end;function Bar: TSomething;begin  result := SomeValue;  if MyGlobal = SomeValue then     DoBadThings;end; begin  MyGlobal := GoodValue;  Foo(MyGlobal);  MyGlobal := Bar;end; Neither Foo nor Bar will call "DoBadThings". "result" in Bar, is not a reference to "MyGlobal", even if the result will eventually be stored there. => So far, I would consider the above as "the documented behaviour" => even though I have not checked how much of this is explicitly mentioned, and how much is given by not being mentioned as "the opposite". => All of this, is equally true for TSomething=Integer and TSomething=AnsiString. => Except, in case of "TSomething = array of TWhatever;": If you did in Foo "a[n] := xxx", then that affects "MyGlobal[n]". Afaik that is documented somewhere (though I have not checked). And the doc afaik specifies, that if you do not want that then (in such cases) you have to call "SetLength(a, lenght(a);". SetLength will ensure that the dynamic array in "a" has a refcount of 1 (one). For AnsiString that is not necessary, as AnsiString implements "copy on write" (which dynamic array do not). And now (again) back to "a := test(a)": As given above: The result assignment to  "a :=" is only done after the call to "test" (same as the call to "Bar" above) has returned. So while in "test" or "Bar" the value of "a" is not affected by the assignment. The fact that "a" is an AnsiString does not matter here (it's not documented to be different. There documentation touching this, if you check for "copy on write". But it would not affect the "execution order"). Yes, it is true that functions returning "AnsiString" (and also some other types, including afaik dynamic-array) have the "result" passed in has "hidden argument", and that therefore "result" in such functions (internally) is a "var param". => But that is an internal implementation detail. => I am not aware, of any documentation saying that this "internal detail" changes other documented behaviour. Therefore (I consider the) behaviour of "a:=test( {const param} a);" as documented to be save. About "that is an internal detail": I am not sure that this is anywhere in the doc. IIRC, I personally gained that info from statements made by core developers on the mail list. Should indeed it be found somewhere on the doc, then I would expect that such part of the doc, would make it quite clear to be an "implementation detail", and not to have functional impact on how compiled code behaves. From my own research (and possible also from further statements of core developers), I also know that in the case of "a:=test(a);" (if "a" is an AnsiString, or afaik any managed type) the value passed in for the hidden result parameter, is actually not "a" but a temp variable (that then afterwards will be assigned to "a"). I do not know, if there currently is any case where FPC would optimize this temp var away. But if it does (or would do in future), it would (given the above) not be allowed to change behaviour.