dynamic array bug ? lazarus / 62-General ---------------------------------------------------------------------- Nitorami · 2018-06-26 14:57 Can someone explain the behaviour of this code ? Setlength () is expected to set newly allocated array elements to zero. However when applied to "result" from within a function, it initialises result with the contents of the previously processed array !?!?. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc}{$OPTIMIZATION OFF}{$R+,S+,Q+} type  Vector = array of integer; function NewVector (len: longint): Vector;begin//  result := nil; //that solves the issue... but why ?  SetLength (result,len);end; procedure Print (const V: Vector);var i: integer;begin  for i := 0 to high(V) do write (V[i]:4);  writeln;end; var  A,B: Vector;begin  A := NewVector(3);  A[0] := 1;  A[1] := 4;  A[2] := 9;   B := NewVector(3);   writeln ('Address of A is: ',ptrint(A));  write   ('Content of A is: ');  Print (A);   writeln ('Address of B is: ',ptrint(B));  write   ('Content of B is: ');  Print (B);end.  Output: QuoteAddress of A is: 4897328 Content of A is:    1   4   9 Address of B is: 4897360 Content of B is:    1   4   9 Initialising "result" to nil solves the issue, but I don't know why. Thought that "result" may still contain the address of A, so setlength() would find an existing array at this place and do nothing, i.e. B would be allocated at the same address as A. Although that would be a bug, it would explain that the content of A and B are the same. However the addresses of A and B are different, so B is in fact created as a new instance, however filled with the content of A. This is very weird. I guess it is a bug, but maybe someone knows better. Windows, FPC 3.0.4. ---------------------------------------------------------------------- turrican · 2018-06-26 15:05 {$OPTIMIZATION OFF} Is that preventing to initialize local stack variable maybe? ---------------------------------------------------------------------- Nitorami · 2018-06-26 15:09 It does not matter whether optimisation is ON or OFF, I just chose the most conservative settings to demonstrate it is not related to optimization. ---------------------------------------------------------------------- turrican · 2018-06-26 15:11 Quote from: Nitorami on June 26, 2018, 05:09:11 pmIt does not matter whether optimisation is ON or OFF, I just chose the most conservative settings to demonstrate it is not related to optimization. Sorry, my fault... ---------------------------------------------------------------------- howardpc · 2018-06-26 15:13 Using an uninitialised variable and then finding it contains (or points to) non-random data is not a compiler bug, but a programmer failing. In this simple case your uninitialised Result happens to pick up the 'same' data consistently. But this is an implementation detail, and cannot be relied on. Result is undefined until you assign a value to it. ---------------------------------------------------------------------- turrican · 2018-06-26 15:13 That can be related to stack registers... but idk why. ---------------------------------------------------------------------- turrican · 2018-06-26 15:17 Quote from: howardpc on June 26, 2018, 05:13:19 pmUsing an uninitialised variable and then finding it contains (or points to) non-random data is not a compiler bug, but a programmer failing. In this simple case your uninitialised Result happens to pick up the 'same' data consistently. But this is an implementation detail, and cannot be relied on. Result is undefined until you assign a value to it. That's true and is a good practice to initializate variables but in this case (for example) Delphi compiler works fine. I tried it. ---------------------------------------------------------------------- Nitorami · 2018-06-26 15:43 QuoteUsing an uninitialised variable and then finding it contains (or points to) non-random data is not a compiler bug, but a programmer failing. In this simple case your uninitialised Result happens to pick up the 'same' data consistently. But this is an implementation detail, and cannot be relied on. Result is undefined until you assign a value to it. Not agreed in this case. The documentation does not tell me to initialise the argument of Setlength before using it. SetLength itself does the initialisation. ---------------------------------------------------------------------- turrican · 2018-06-26 15:52 Quote from: Nitorami on June 26, 2018, 05:43:08 pmQuote SetLength itself does the initialisation. +1 ---------------------------------------------------------------------- Nitorami · 2018-06-26 15:57 I agree that the issue may have to do with initialisation, and it is quite possible that "result" within a function body is not automatically initialised for reasons of performance. But if that causes a dynamic array allocation to go wrong, it is a bug. The compiler itself should take care of the initialisation of managed variables. ---------------------------------------------------------------------- Martin_fr · 2018-06-26 15:57 Quote from: Nitorami on June 26, 2018, 05:43:08 pmNot agreed in this case. The documentation does not tell me to initialise the argument of Setlength before using it. SetLength itself does the initialisation. Well the issue, as far as I understand it, is: Dynamic arrays and strings are so called managed types. They are always in some kind of initialized state (i.e. either nil, or valid data (old data can be valid)). Those types are refcounted (they are pointers to data that includes a refcount). If a dynamic array goes out of scope, its refcount is reduced, and it is freed. so in case of "var a: array of ...", a will be "nil" to start with (in practice, not necessarily documented) . Because there is no old data available, so there is only nil. Then SetLength will initialize the entries, because all entries are new. Doing Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SetLength(a,2);a[1]:=99;SetLength(a,20);will keep the value of a[1]. In other words: - SetLenght initializes new entries in the array - SetLenght does NOT initialize the array itself Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i TIntegerArray = array of integer;Procedure Foo(var a: TIntegerArray)var param, will get the value passed in. SetLength will keep existing values. And here is the final bit, that is less known. For managed types "Result" is internally handled as a var param. ------------------------- That is the technical bit. As for what the expected behaviour should be.... I haven't checked all the docs. But in general, ALL variables should be initialized before use. I am not aware that managed types are excluded in the doc (in practice they often are, but its about the doc) As above,  SetLenght initializes the entries, not the array. So initialization would be   Result := nil;   Result := SomeOtherArray; --- EDIT https://www.freepascal.org/docs-html/rtl/system/setlength.html Doesn't say anything about initializing the entries either. ---------------------------------------------------------------------- Nitorami · 2018-06-26 16:07 That sounds reasonable, but it means I would always have to know whether a variable is initialised or not. My understanding so far was that global variables are not automatically initialised, so something like Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var A: TVector;SetLength (A,3) would be likely to fail, but in fact it it always worked for me. EDIT: It may be that the compiler automatically initialises globally defined managed types to nil, but then we have a dangerous inconsistency. It can hardly be expected from the user to know that only in this single case he needs to set the argument of SetLength to nil before using it. Particularlay where the compiler knows quite well that "result" is a dynamic array, and could do the job automatically. ---------------------------------------------------------------------- engkin · 2018-06-26 16:41 Might be easier to see the problem with this tiny replacement: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure NewVector_real (var Result: Vector; len: longint);begin  SetLength (Result, len);end; In the first call Result was nil. In the second call Result had the address of the previous call. Since it is not nil, its contents are copied. ---------------------------------------------------------------------- wk_ · 2018-06-26 17:52 Hi, Looks like a glaring bug to me. BTW: when modelling "result" as an additional parameter, it should be an out parameter, since from a language point of view you cannot pass anything in when calling a function (iow. "result" always is a fresh variable). In addition, when returning from the function, result goes out of scope, hence the value it holds should be invalidated/destroyed (ref count -1). Finally, since in the program the function is once called to assign to variable A and then to variable B, even a procedure with result as var parameter would not lead to the observed result. Cheers,  Will ---------------------------------------------------------------------- Nitorami · 2018-06-26 18:43 I think so, too. Filed a bug report id=33912. ---------------------------------------------------------------------- taazz · 2018-06-26 19:09 Quote from: Nitorami on June 26, 2018, 05:43:08 pmQuoteUsing an uninitialised variable and then finding it contains (or points to) non-random data is not a compiler bug, but a programmer failing. In this simple case your uninitialised Result happens to pick up the 'same' data consistently. But this is an implementation detail, and cannot be relied on. Result is undefined until you assign a value to it. Not agreed in this case. The documentation does not tell me to initialise the argument of Setlength before using it. SetLength itself does the initialisation. Incorrect. SetLength either initializes or re-initializes the contents. In short if the argument passed is an already initialized variable it tries to preserve the existing data if possible (ee on expanding but not on shrinking). So yes it requires a proper initialized variable to decide if it needs to preserve existing data or not. ---------------------------------------------------------------------- Nitorami · 2018-06-26 19:31 QuoteIncorrect. SetLength either initializes or re-initializes the contents. In short if the argument passed is an already initialized variable it tries to preserve the existing data if possible (ee on expanding but not on shrinking). So yes it requires a proper initialized variable to decide if it needs to preserve existing data or not. I understand that. However the compiler knows that the function shall return a dynamic array, hence it could as well set implicitly result := nil before the Setlength() call. It does not do so, and that is all but clear. There are possibly reasons for that behaviour, but this is a rather dangerous construct within a language praised as "safe": You may use setlength(x) anywhere in your code without initialising x, ONLY within a function it goes bonkers. What is the logic of that ? Related bug report 0030321 already claims this behaviouer is poorly documented; however even if documented, I would find it difficult to accept. The report report says "fixed in 3.1.1" but from Jonas' response to my report 0030321 it is not quite clear if actually the code has been changed, or only the documentation. ---------------------------------------------------------------------- taazz · 2018-06-26 19:40 Quote from: Nitorami on June 26, 2018, 09:31:04 pmQuoteIncorrect. SetLength either initializes or re-initializes the contents. In short if the argument passed is an already initialized variable it tries to preserve the existing data if possible (ee on expanding but not on shrinking). So yes it requires a proper initialized variable to decide if it needs to preserve existing data or not. I understand that. However the compiler knows that the function shall return a dynamic array, hence it could as well set implicitly result := nil before the Setlength() call. It does not do so, and that is all but clear.    true. Quote from: Nitorami on June 26, 2018, 09:31:04 pmThere are possibly reasons for that behaviour, but this is a rather dangerous construct within a language praised as "safe": safe not fool proof. Quote from: Nitorami on June 26, 2018, 09:31:04 pmYou may use setlength(x) anywhere in your code without initialising x, ONLY within a function it goes bonkers. What is the logic of that ? Incorrect. It is unsafe to use setlength in uninitialized variables in any place, this may lead to overflow conditions. Never do that. Quote from: Nitorami on June 26, 2018, 09:31:04 pmRelated bug report 0030321 already claims this behaviouer is poorly documented; however even if documented, I would find it difficult to accept. The report report says "fixed in 3.1.1" but from Jonas' response to my report 0030321 it is not quite clear if actually the code has been changed, or only the documentation. Yeah your logic is a bit hard to accept for me too, but if it ends up with (at least) better documentation no matter what solution is implemented its enough for me. I'm already initializing everything my self already and neve rely on the compiler to do so. In fact if tomorrow the compiler decided that no variable is initialized for speed reasons my code is safe. ---------------------------------------------------------------------- Thaddy · 2018-06-26 19:45 Only the docs ---------------------------------------------------------------------- Nitorami · 2018-06-26 19:46 QuoteI'm already initializing everything my self already and neve rely on the compiler to do so. In fact if tomorrow the compiler decided that no variable is initialized for speed reasons my code is safe. Which means if you have something like Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var a: array of integer;SetLength (a,100);  you would shove an "a := nil" between any such statements ? ---------------------------------------------------------------------- taazz · 2018-06-26 19:49 Quote from: Nitorami on June 26, 2018, 09:46:16 pmQuoteI'm already initializing everything my self already and neve rely on the compiler to do so. In fact if tomorrow the compiler decided that no variable is initialized for speed reasons my code is safe. Which means if you have something like Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var a: array of integer;SetLength (a,100);  you would shove an "a := nil" between any such statements ? usually Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  a: array of integer =nil;begin  SetLength (a,100); ---------------------------------------------------------------------- Nitorami · 2018-06-26 19:58 Ok, but taking that a step further, we might also ask whether the compiler will tomorrow still fill the array with zeroes, so we would prudently add something like fillchar (a[0], 100*sizeof(integer), 0);... ---------------------------------------------------------------------- taazz · 2018-06-26 20:03 Quote from: Nitorami on June 26, 2018, 09:58:14 pmOk, but taking that a step further, we might also ask whether the compiler will tomorrow still fill the array with zeroes, so we would prudently add something like fillchar (a[0], 100*sizeof(integer), 0);... I already have a unit with procedures that do that for a number of standard and custom data types. I'm not as rigorous in its use as I'm with initializing variables, but its used occasionally, I do like to be sure that the state of my memory is known especially if it does not involve some other form of initialization (eg load from file). ---------------------------------------------------------------------- ASerge · 2018-06-26 20:11 It's not a bug, just non-optimal generated code. Maybe just to be more safe (unnecessarily safe). First, what does the SetLength function do? Pseudocode: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure SetLength(var A: dynamic_array; NewLen: SizeInt);begin  if RefCount(A) <> 1 then    A := MakeCopy(A); // all other reference not change, A has RefCount = 1  ReallocInPlace(A, NewLen * SizeOf(A[0]));  if TypeOf(A[0]) contains ref types then    initialize new membersend;Second, FPC use temp variable, and this is look like bug: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i // A := NewVector(3);NewVector(3, Temp); // before call Temp is nilA := Temp; // after call A and Temp are the same, refcount = 2//...// B := NewVector(3);NewVector(3, Temp); // before call Temp has refcount 2, as result (see SelLength above) makes a copy of the arrayB := Temp; // after call A refcount=1, B and temp are the same with ref count = 2 Delphi does not use a temporary variable in this case.