Dynamic array extensions ? lazarus / 64-FPC development ---------------------------------------------------------------------- Nitorami · 2018-06-06 21:11 I would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short: - support for array constructors using "[...]" syntax - support for Insert(), Delete() and Concat() - support for "+" operator - support for dynamic array constants (and variable initializations) Sounds ok, however: The "+" operator is effectively an alias for concat(), and it will be an intrinsic, which means that existing "+" operator overloads for dynamic arrays will no longer compile. Intrinsic operators cannot be overloaded, hence it will no longer be possible to custom define the "+" operator to do a mathematical elementwise addition of numerical vectors/matrices. With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied. What is your opinion on that  ? ---------------------------------------------------------------------- Phil · 2018-06-07 01:13 Quote from: Nitorami on June 06, 2018, 11:11:25 pmI would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short: - support for array constructors using "[...]" syntax - support for Insert(), Delete() and Concat() - support for "+" operator - support for dynamic array constants (and variable initializations) What is your opinion on that  ? Sounds like Pascal array syntax is getting dragged into the 21st century so it's more like other languages. Take the Swift MLPClassifier example givenhere (under Interpreter): https://github.com/tensorflow/swift/blob/master/Usage.md That gives several examples of constant arrays being declared within the code that uses the arrays. I assume something like that will be available in Pascal. If so, that should make for cleaner code, less code. I'm kind of indifferent to operator overloading. Since you already have to declare a function that gets called when you use the overloaded operator, maybe just make that function available in the interface section so it can be used in case "+" is lost. The Swift example also uses operator overloading for tensor arithmetic. So something like this: tanh(matmul(x, w1) + b1) Could be written without operator overloading like this: tanh(add(matmul(x, w1), b1)) That still seems pretty clear. After all, you can't get away completely from calling functions when you're doing this kind of stuff. I suppose you could overload operators for both tanh and matmul if there were obvious math symbols that made sense. Are there limitations on what characters you can overload? In Swift, I believe you can use any Unicode character as an overloaded operator. In fact, they actually overload ⊗ for matrix multiplication, so you could write this: tanh(x ⊗ w1 + b1) Interesting that they don't use that overloaded operator in their own example. Perhaps cooler heads prevailed. In my Pascal interface to TensorFlow I overloaded the arithmetic operators too, per Swift, but I don't think I would miss them if they weren't available. Perhaps just rejoice in the "lower-ceremony" syntax for arrays and make peace with the loss of an overloaded "+" in this one case. ---------------------------------------------------------------------- taazz · 2018-06-07 02:34 Quote from: Nitorami on June 06, 2018, 11:11:25 pmI would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short: - support for array constructors using "[...]" syntax - support for Insert(), Delete() and Concat() - support for "+" operator - support for dynamic array constants (and variable initializations) Sounds ok, however: The "+" operator is effectively an alias for concat(), and it will be an intrinsic, which means that existing "+" operator overloads for dynamic arrays will no longer compile. Intrinsic operators cannot be overloaded, hence it will no longer be possible to custom define the "+" operator to do a mathematical elementwise addition of numerical vectors/matrices. With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied. What is your opinion on that  ? on one hand it nice not to have to write those pesky init routines, on the other I have no idea why it is needed to become an intrinsic, until I get some insight on that I can't really comment. I guess it makes things both better and worst. Its better because now + becomes standard, stable, when I see it on piece of code that I debug, I know exactly what it does, it is worst well I guess you know why. Now you have to type all kinds of staff and your code moves a bit farther away from math symbolism. You could say I'm on the fence on that. ps Oh by the way becoming like swift is a bad thing not a good one (just kidding) ---------------------------------------------------------------------- Phil · 2018-06-07 02:53 Quote from: taazz on June 07, 2018, 04:34:03 amOh by the way becoming like swift is a bad thing not a good one (just kidding) You'll probably find this interesting, on why Google selected Swift over Python (considering that Python rules in ML). https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensor Flow.md I'm looking to see whether Pascal makes sense to use with TensorFlow. ---------------------------------------------------------------------- Thaddy · 2018-06-07 07:19 Well the logical thing for dynamic arrays is really concat after all, because it not necessarily holds data that you can perform vector math on. For use as vectors you can write a type helper Add(), something like: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Type   TMyIntvector = array of array of integer;    TMyIntVectorHelper = type helper for TMyIntVector    class function Add(const a,b:TMyIntVector):TMyIntVector;static;    function Add(const b:TMyIntVector):TMyIntVector;  end;Still, how do you retrieve the dimensions easily? (also goes for the operators on vector cells) These must be known and furthermore dynamic arrays can be ragged arrays, which complicates it even further. Ragged: Given an array of array of integer, you can do this:   Setlength(a,3,5);   Setlength(a[0],3);   Setlength(a[4],2); ---------------------------------------------------------------------- marcov · 2018-06-07 07:28 Quote from: Thaddy on June 07, 2018, 09:19:09 amWell the logical thing for dynamic arrays is really concat after all, because it not necessarily holds data that you can perform vector math on. You could invert that logic, and say that contrary to strings because dynamic arrays CAN contain data you can do math on, you reserve that operator for its mathematical meaning. And for appending you can write a type helper Append() Then you keep the association between static and dynamic arrays instead of strings and dynamics arrays.  Also, Strings are commonly concatenated, but general arrays much less often, while running an operator on every element of an array is more common. ---------------------------------------------------------------------- Thaddy · 2018-06-07 07:56 That was my initial opinion. That I reversed because of the complexity. Either way is fine with me but I understand the ratio. ---------------------------------------------------------------------- circular · 2018-06-13 13:04 Where can we say that we want to keep the + operator as it is? ---------------------------------------------------------------------- marcov · 2018-06-13 13:25 I updated my answer ---------------------------------------------------------------------- GAN · 2018-06-13 19:43 Quote from: circular on June 13, 2018, 03:04:19 pmWhere can we say that we want to keep the + operator as it is? I agree. ---------------------------------------------------------------------- hnb · 2018-06-13 21:30 Quote from: GAN on June 13, 2018, 09:43:50 pmQuote from: circular on June 13, 2018, 03:04:19 pmWhere can we say that we want to keep the + operator as it is? I agree. probably the only solution is combo of both : https://bugs.freepascal.org/my_view_page.phphttp://lists.freepascal.or g/mailman/listinfo/fpc-devel/ while I like most of improvements, the support for "+" operator seems strange even for me (I generally like new things)... Also strange is breaking backward compatibility (which AFAIK is one of priority) with questionable gain. ---------------------------------------------------------------------- schuler · 2018-06-14 07:29 Hi Nitorami, These improvements are really fantastic! I would like to take the opportunity to share what I miss: faster math vector operations. Some vectors I use have 10000 elements that are summed/multiplied with other vectors. I ended coding it in assembler (the full code has more than 4000 lines): TNeuralFloatArrPtr is a pointer to the first element of a single precision floating point dynamic array.  Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure AVXMulAdd(PtrA, PtrB: TNeuralFloatArrPtr; MulOp: TNeuralFloat; NumElements: integer);var  MulOpPtr: pointer;  localNumElements, MissedElements: integer;begin  localNumElements := (NumElements div 4) * 4;  MissedElements := NumElements - localNumElements;  if localNumElements > 0 then  begin    MulOpPtr := Addr(MulOp);  asm  mov ecx, localNumElements  mov rax, PtrB  mov rdx, MulOpPtr   VBROADCASTSS ymm5, [rdx]  mov rdx, PtrA   push rcx  shr ecx,5  // number of large iterations = number of elements / 32  jz @SkipLargeAddLoop @LargeAddLoop:  {$IFDEF AVX2}  vmovups ymm0, [rdx]  vmovups ymm1, [rdx+32]  vmovups ymm2, [rdx+64]  vmovups ymm3, [rdx+96]   vfmadd231ps ymm0, ymm5, [rax]  vfmadd231ps ymm1, ymm5, [rax+32]  vfmadd231ps ymm2, ymm5, [rax+64]  vfmadd231ps ymm3, ymm5, [rax+96]  {$ELSE}  vmulps  ymm0, ymm5, [rax]  vmulps  ymm1, ymm5, [rax+32]  vmulps  ymm2, ymm5, [rax+64]  vmulps  ymm3, ymm5, [rax+96]   vaddps  ymm0, ymm0, [rdx]  vaddps  ymm1, ymm1, [rdx+32]  vaddps  ymm2, ymm2, [rdx+64]  vaddps  ymm3, ymm3, [rdx+96]  {$ENDIF}   vmovups [rdx],    ymm0  vmovups [rdx+32], ymm1  vmovups [rdx+64], ymm2  vmovups [rdx+96], ymm3   add rax, 128  add rdx, 128  dec ecx  jnz @LargeAddLoop @SkipLargeAddLoop:  vzeroupper   pop rcx  and ecx,$0000001F  jz @EndAdd  shr ecx, 2 // number of small iterations = (number of elements modulo 16) / 4 @SmallAddLoop:   movups  xmm2, [rax]  movups  xmm4, [rdx]   mulps   xmm2, xmm5  addps   xmm4, xmm2   movups  [rdx], xmm4   add rax, 16  add rdx, 16   dec ecx  jnz @SmallAddLoop @EndAdd:  end  [    'RAX', 'RCX', 'RDX',    'ymm0', 'ymm1', 'ymm2', 'ymm3', 'ymm4', 'ymm5'  ];  end; // of if   if MissedElements>0 then  begin    PtrA^[localNumElements] += MulOp*PtrB^[localNumElements];    if MissedElements>1 then    begin    PtrA^[localNumElements+1] += MulOp*PtrB^[localNumElements+1];      if MissedElements>2 then PtrA^[localNumElements+2] += MulOp*PtrB^[localNumElements+2];    end;  end;end; In the case that you are interested in the full source code, please let me know. Wish everyone happy coding ---------------------------------------------------------------------- VTwin · 2018-06-14 10:57 Quote from: Nitorami on June 06, 2018, 11:11:25 pmWith the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied. Oh dear! I consistently use operator overloading on vectors and matrixes. Seems like going backwards to rewrite everything as functions or container classes. If I add two vectors I don't want them concatenated!! :/ ---------------------------------------------------------------------- circular · 2018-06-14 16:35 Nitorami, could you point out the FPC e-mail that announces those changes? http://lists.freepascal.org/pipermail/fpc-devel/ ---------------------------------------------------------------------- Nitorami · 2018-06-14 19:15 @circular: Of course. http://free-pascal-general.1045716.n5.nabble.com/Feature-announcement- Dynamic-array-extensions-td5731410.html You may have to register with your email address. @schuler: 1. I think you did not really read the original post; I did not advertize these "improvements", which in my view are unnecessary as part of the core language, as it is trivial to implement them manually. I wanted to hear other user's opinion on the developer's plan to implement the + operator as intrinsic for concatenation of dynamic strings, making it unavailable for simple vector addition: I use vector addition a lot but never needed concatenation. 2. Thanks for the code. For my current project the normal non- vectorised math is sufficient, although optimisation is always tempting. My own knowledge of assembler is poor, and I appreciated the native support for vectorised math which had once been introduced (I think by switch $Sv). But unfortunately it has not been maintained any further and is now broken, at least under windows :-( ---------------------------------------------------------------------- circular · 2018-06-14 20:18 I read the FP thread. Seems there has been a debate over there. If I understand correctly: Code: Delphi  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var a: array of integer;begin  a := [1,2,3];  a += 1;          // will give a = [1,2,3,1] instead a = [2,3,4]  a := [1,2,3];  a += [4,5,6];  // will give a = [1,2,3,4,5,6] instead of a = [5,7,9]end; Correct? ---------------------------------------------------------------------- Nitorami · 2018-06-15 08:23 Yes, that seems to be the plan - the "+" operator acting as concatenation, defined as intrinsic in the language so it can't be overloaded anymore, e.g. to do numeric addition instead. ---------------------------------------------------------------------- schuler · 2018-06-15 08:43 Nitorami, Sorry. I evidently didn't understand your post first time I read. I do have concerns with "+". I think that everyone that uses Free Pascal for math will have concerns. In my case, it won't brake any code although I'm looking for another symbol in the keyboard that could be used for concat. What about & for concatenation? BTW, is this concatenation or union? If it is union, I might suggest || instead. ---------------------------------------------------------------------- Nitorami · 2018-06-15 09:16 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i BTW, is this concatenation or union? If it is union, I might suggest || instead. They meant concatenation, probably for similarity between ansistrings and dynamic arrays. Other operators have been proposed, read more in the mailing list discussion http://free-pascal- general.1045716.n5.nabble.com/Feature-announcement-Dynamic-array- extensions-td5731410.html But I don't understand why we need an intrinsic operator at all. I would leave it to the programmer to define one, whether it is for concatenation, addition, or whatever else is required. ---------------------------------------------------------------------- circular · 2018-06-15 20:46 @schuler Agreed that "&" would be nice, like in VB. It could be also used in strings: Code: Delphi  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var s: string; i: integer;begin  i := 36;  s := 'hello number ' & i; // would yield 'hello number 36'end; ---------------------------------------------------------------------- Bart · 2018-06-16 10:59 OMG. This is Pascal, it is strongtyped. If you want such syntax, please use Basic. Bart ---------------------------------------------------------------------- circular · 2018-06-16 13:47 ---------------------------------------------------------------------- lainz · 2018-06-16 14:03 You can just add the + to convert automatically to string. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Operator + (s : string; i : integer) z : string;begin  z := s + IntToStr(i);end; {$R *.lfm} { TForm1 } procedure TForm1.Button1Click(Sender: TObject);var  s: string;  i: integer;begin  s := 'Hello world ';  i := 10;  s := s + i; // 'Hello world 10'  ShowMessage(s);end; ---------------------------------------------------------------------- circular · 2018-06-16 17:02 Indeed that's cool ---------------------------------------------------------------------- Abelisto · 2018-06-17 19:26 @circular, @lainz Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i WriteStr(s, 'Hello number ', 10, '!'); ---------------------------------------------------------------------- circular · 2018-06-18 15:10 @abelisto Thanks I did not know this one ---------------------------------------------------------------------- lainz · 2018-06-18 17:05 Quote from: Abelisto on June 17, 2018, 09:26:48 pm@circular, @lainz Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i WriteStr(s, 'Hello number ', 10, '!'); Looks really good =) I can format float as well with it. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i WriteStr(s, 'hello cañón 人物 ', 10.5678:2:2, '!'); ---------------------------------------------------------------------- PascalDragon · 2018-06-20 20:30 Quote from: Nitorami on June 06, 2018, 11:11:25 pmI would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short: - support for array constructors using "[...]" syntax - support for Insert(), Delete() and Concat() - support for "+" operator - support for dynamic array constants (and variable initializations) Sounds ok, however: The "+" operator is effectively an alias for concat(), and it will be an intrinsic, which means that existing "+" operator overloads for dynamic arrays will no longer compile. Intrinsic operators cannot be overloaded, hence it will no longer be possible to custom define the "+" operator to do a mathematical elementwise addition of numerical vectors/matrices. With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied. What is your opinion on that  ? Addendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch. If the modeswitch is active then the "+" operator overload is not possible and the compiler will warn if such an overload is in scope. If the modeswitch is not set, then everything is as before. The modeswitch is enabled by default in Delphi modes, because that's why the feature was added. To disable the modeswitch there use this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode Delphi}{$modeswitch ArrayOperators-} // Note the "-" ---------------------------------------------------------------------- Thaddy · 2018-06-20 20:32 Neat solution! ---------------------------------------------------------------------- Nitorami · 2018-06-21 19:45 QuoteAddendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch. Thank you. Guess that is an acceptable solution. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$modeswitch ArrayOperators-} // Note the "-"Why is it "-" in this case rather than "OFF" ? ---------------------------------------------------------------------- taazz · 2018-06-21 20:00 Quote from: Nitorami on June 21, 2018, 09:45:49 pmQuoteAddendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch. Thank you. Guess that is an acceptable solution. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$modeswitch ArrayOperators-} // Note the "-"Why is it "-" in this case rather than "OFF" ? As far as I understand, + is an alias to on,  - an alias to off, (or vice versa) they are interchangeable, he only drawn your attention to the - because it is easy to miss, if he had used off instead then the note would not have been necessary. ---------------------------------------------------------------------- VTwin · 2018-06-22 16:43 Quote from: Nitorami on June 21, 2018, 09:45:49 pmQuoteAddendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch. Thank you. Guess that is an acceptable solution. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$modeswitch ArrayOperators-} // Note the "-"Why is it "-" in this case rather than "OFF" ? https://www.freepascal.org/docs-html/prog/progsu105.html QuoteThe syntax is as follows: {$MODESWITCH XXX}  {$MODESWITCH XXX+}  {$MODESWITCH XXX-} The first two will switch on feature XXX, the last one will switch it off. A mode switch seems like a good solution. Thanks! ---------------------------------------------------------------------- PascalDragon · 2018-06-22 20:36 Quote from: Nitorami on June 21, 2018, 09:45:49 pmQuoteAddendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch. Thank you. Guess that is an acceptable solution. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$modeswitch ArrayOperators-} // Note the "-"Why is it "-" in this case rather than "OFF" ? Because modeswitches currently don't support ON/OFF, only +/- (the latter can also be used with other switches, like e.g. $ScopedEnums). ---------------------------------------------------------------------- ASBzone · 2018-06-22 23:31 Quote from: PascalDragon on June 20, 2018, 10:30:14 pmAddendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch. If the modeswitch is active then the "+" operator overload is not possible and the compiler will warn if such an overload is in scope. If the modeswitch is not set, then everything is as before. The modeswitch is enabled by default in Delphi modes, because that's why the feature was added. To disable the modeswitch there use this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode Delphi}{$modeswitch ArrayOperators-} // Note the "-" Awesome.  Much appreciated. ---------------------------------------------------------------------- PascalDragon · 2018-06-23 11:49 Quote from: PascalDragon on June 22, 2018, 10:36:52 pmBecause modeswitches currently don't support ON/OFF, only +/- (the latter can also be used with other switches, like e.g. $ScopedEnums).  And now FPC also supports ON and OFF in modeswitches