Compiler Generated Temporary Interface References lazarus / 64-FPC development ---------------------------------------------------------------------- Coxy · 2021-01-26 13:12 I'm having problems with temporary interface references created by the compiler. Here's a simplified example, Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit Unit1; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls; type   TMyInterface = interface    procedure DoSometing;  end;   TMyOtherIntf = interface(TMyInterface)  end;   { TMyClass }   TMyClass = class(TInterfacedObject,TMyOtherIntf)  public    destructor Destroy;override;    procedure DoSometing;  end;   { TForm1 }   TForm1 = class(TForm)    Button1: TButton;    procedure Button1Click(Sender: TObject);  private    function GetMyInterface:TMyOtherIntf;  public   end; var  Form1: TForm1; implementation {$R *.lfm} { TForm1 } procedure TForm1.Button1Click(Sender: TObject);var  myIntf : TMyInterface;begin  MyIntf:=GetMyInterface;   MyIntf:=nil;    // Should cause destructor of TMyClass to be called.end;        // Place breakpoint here function TForm1.GetMyInterface: TMyOtherIntf;begin  result := TMyClass.Create;end; { TMyClass } destructor TMyClass.Destroy;begin  inherited Destroy;        // Place breakpoint hereend; procedure TMyClass.DoSometing;begin end; end.  If you place breakpoints where indicated and run this example you'll find the "end;" breakpoint is triggered before the one in the destructor. This is probably caused by compiler creating temporary variable to hold "GetMyInterface" interface value "TMyOtherIntf" which is then converted to "TMyInterface", it being tidied up in the routines exit code. Now I'm implementing non reference counted interfaces with my own "_AddRef", "_Release" and "QueryInterface" routines, and have a "FreeInterface" routine similar to "FreeAndNil" that free the underlying TObject. The tidying up of temporary interface references in routine exit code causes an access violation. Example, Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure ClearArray;var  i : integer;  intf : TMyInterface;begin  for i:=0 to FIntfCount do  begin    intf:=GetInterface(i);    FreeInterface(intf);  end;end;  Is there a way to have compiler tidy temporary interfaces references immediately after assignment to "intf" rather than in routine exit code. I know about using "{$interfaces CORBA}", but I wish to mix both reference counted and non reference counted interfaces in the same unit. Free Pascal: 3.2.0 OS: Windows 10 ---------------------------------------------------------------------- marcov · 2021-01-26 13:29 Easiest is to look at the generated assembly code for the method (compile with -al) If you handtuned your interface handling in Delphi, beware that doesn't mean it is safe if it "works". Read this: https://stackoverflow.com/questions/9592654/what-are-the- differences-between-implementation-of-interfaces-in-delphi-and- laza/9623502#9623502 ---------------------------------------------------------------------- Coxy · 2021-01-26 13:50 I stepped through assembler code in debugger to identify use of temporary interface variable and clean up in routine exit code. For me it works in Delphi and fails in FPC. Coding routines like "FreeAndNil" and "FreeInterface" become problematic because of the compilers creation and use of temporary variables. One workaround in code is to do the following, Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure ClearArray;var  i : integer;  intf : TMyInterface;  procedure SetIntf;  begin     intf:=GetInterface(i);  end;begin  for i:=0 to FIntfCount do  begin    SetIntf;    FreeInterface(intf);  end;end;  Here the temporary interface variable is tidied up in the sub method exit code, resolving the problem. But this is a workaround not a fix. ---------------------------------------------------------------------- marcov · 2021-01-26 14:23 Quote from: Coxy on January 26, 2021, 02:50:59 pmI stepped through assembler code in debugger to identify use of temporary interface variable and clean up in routine exit code. For me it works in Delphi and fails in FPC. Coding routines like "FreeAndNil" and "FreeInterface" become problematic because of the compilers creation and use of temporary variables. Then your routines are wrong. If you do it right, then it works with all cases. If you depend on the lifetime of the variable the code is simply wrong, as the URL I gave explains, working in Delphi is not necessarily correct code. If you still have questions, please create a self contained (preferably console) example of what goes wrong, and we'll look if it is a coding error or a compiler bug (however, that is unlikely) ---------------------------------------------------------------------- Coxy · 2021-01-26 15:04 Here's some example code. See routine FreeInterface, comment/uncomment highlighted lines to reproduce problem. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit Unit1; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls; type   IMyInterface = interface    ['{CEC89718-0B2D-4BA5-886F-03FF11955644}']    function GetSelf:TObject;    function _ReferenceCounted:boolean;  end;   { TMyInterfacedObjectNoRefCount }   TMyInterfacedObjectNoRefCount = class(TObject, IUnknown, IMyInterface)  private  protected    function _AddRef:longint;{$IFDEF LINUX}cdecl{$ELSE}stdcall{$ENDIF};    function _Release:longint;{$IFDEF LINUX}cdecl{$ELSE}stdcall{$ENDIF};    function QueryInterface({$IFDEF FPC_HAS_CONSTREF}constref{$ELSE}const{$ENDIF} IID: TGUID; out Obj): {$IFDEF LINUX}Longint;cdecl{$ELSE}HResult;stdcall{$ENDIF};  public   function GetSelf:TObject;virtual;    function _ReferenceCounted:boolean;virtual;  end;   TMyInterface = interface  procedure DoSometing;  end;   TMyOtherIntf = interface(TMyInterface)  end;   { TMyClass }   TMyClass = class(TMyInterfacedObjectNoRefCount,TMyOtherIntf)  public    destructor Destroy;override;    procedure DoSometing;  end;   { TForm1 }   TForm1 = class(TForm)    Button1: TButton;    procedure Button1Click(Sender: TObject);  private    function GetMyInterface:TMyOtherIntf;  public   end; var  Form1: TForm1; implementation {$R *.lfm}  procedure FreeInterface({$IFDEF FPC_HAS_CONSTREF}constref{$ELSE}const{$ENDIF} X);{$IFDEF AUTOREFCOUNT}beginend;{$ELSE}var  obj : TObject;  p : ^Pointer;  a1 : IUnknown;  a2 : IMyInterface;  procedure GetIMyInterface;  begin    // The following assignment in FPC causes RecCount to be incremented twice    // while DCC only once. This is because FPC increments with the call to    // QueryInterface (DCC doesn't). The temp result is released (decremented)    // on exit from this nested method. So the net effect is only one increment.    // If this assignment were done in the main routine the temp result would be    // released in the tidy up for FreeInterface after the object had been freed    // causing an AV.    // This method works in both FPC and DCC. Except in routine that call    // FreeInterface that also have a temporary reference to this interface.    a2 := a1 as IMyInterface;  end;begin  a1 := IUnknown(X);  if not assigned(a1) then    exit;   try//    a2 := a1 as IMyInterface;     // Uncomment for AV    GetIMyInterface;            // Uncomment for work  except    exit;  end;  if not a2._ReferenceCounted then  begin    obj := a2.GetSelf;     a1 := nil;    a2 := nil;     obj.Free;     p := Pointer(@X);    p^ := nil;  end  else    a1 := nil;end; { TMyInterfacedObjectNoRefCount } function TMyInterfacedObjectNoRefCount._AddRef: longint; stdcall;begin  result := -1end; function TMyInterfacedObjectNoRefCount._Release: longint; stdcall;begin  result := -1end; function TMyInterfacedObjectNoRefCount.QueryInterface(constref IID: TGUID; out  Obj): HResult; stdcall;const  E_NOINTERFACE = HResult($80004002);begin  if GetInterface(IID, Obj) then Result := 0 else Result := E_NOINTERFACE;end; function TMyInterfacedObjectNoRefCount.GetSelf: TObject;begin  result := self;end; function TMyInterfacedObjectNoRefCount._ReferenceCounted: boolean;begin  result := false;end; {$ENDIF}  { TForm1 } procedure TForm1.Button1Click(Sender: TObject);var  myIntf : TMyInterface;begin  MyIntf:=GetMyInterface;   FreeInterface(MyIntf);end; function TForm1.GetMyInterface: TMyOtherIntf;begin  result := TMyClass.Create;end; { TMyClass } destructor TMyClass.Destroy;begin  inherited Destroy;end; procedure TMyClass.DoSometing;begin end; end. ---------------------------------------------------------------------- PascalDragon · 2021-01-27 08:18 Quote from: Coxy on January 26, 2021, 04:04:42 pmSee routine FreeInterface, comment/uncomment highlighted lines to reproduce problem. Mixing class and interface references like that is always a slippery slope and thus the general advice is: don't. Just don't. You're essentially relying on implementation defined behaviour that can for example change between compiler versions or depending on the optimization settings. Quote from: Coxy on January 26, 2021, 02:12:25 pmI know about using "{$interfaces CORBA}", but I wish to mix both reference counted and non reference counted interfaces in the same unit. The {$Interfaces xxx} directive is local so you can decide for each interface declaration inside a unit whether it should be a reference counted one or not. What is however not possible is to inherit a reference counted interface from a non reference counted one and vice versa. ---------------------------------------------------------------------- Coxy · 2021-01-27 09:39 QuoteThe {$Interfaces xxx} directive is local so you can decide for each interface declaration inside a unit whether it should be a reference counted one or not. What is however not possible is to inherit a reference counted interface from a non reference counted one and vice versa. Thanks for pointing this out, I didn't know this. QuoteMixing class and interface references like that is always a slippery slope and thus the general advice is: don't. Just don't. You're essentially relying on implementation defined behaviour that can for example change between compiler versions or depending on the optimization settings. My point about the compilers use of temporary interface references, possible having unintended consequences, is not confined to example "FreeInterface". I've updated my original example to highlight the issue. Place breakpoint on "end;" of "Button1Click" and also in destructor of "TMyClass", and run code. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit Unit2; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls; type   IMyInterface = interface    ['{7E741756-BA4F-4BB2-8AFF-5805586D3B53}']    procedure DoSometing;  end;   IMyOtherIntf = interface(IMyInterface)    ['{50897EE1-6BBE-431B-920A-EA3CFDCDF763}']  end;   { TMyClass }   TMyClass = class(TInterfacedObject,IMyOtherIntf,IMyInterface)  protected    FId : string;  public    constructor Create(pId:string);    destructor Destroy;override;     procedure DoSometing;  end;   { TForm2 }   TForm2 = class(TForm)    Button1: TButton;    procedure Button1Click(Sender: TObject);  private    function GetMyInterface:IMyInterface;    function GetMyOtherIntf:IMyInterface;  public   end; var  Form2: TForm2; implementation {$R *.lfm} { TForm1 } procedure TForm2.Button1Click(Sender: TObject);var  myInterface : IMyInterface;  myOtherIntf : IMyOtherIntf;begin  myInterface:=GetMyInterface;  myOtherIntf:=GetMyOtherIntf as IMyOtherIntf;   myInterface:=nil;    // Should cause destructor of TMyClass to be called.  myOtherIntf:=nil;    // Should cause destructor of TMyClass to be called.   // Something that writes out application state. end;        // Place breakpoint here function TForm2.GetMyInterface: IMyInterface;begin  result := TMyClass.Create('IMyInterface');end; function TForm2.GetMyOtherIntf: IMyInterface;begin  result := TMyClass.Create('IMyOtherIntf');end; { TMyClass } constructor TMyClass.Create(pId: string);begin  FId:=pId;end; destructor TMyClass.Destroy;begin  // Do something that changes application state.   inherited Destroy;        // Place breakpoint hereend; procedure TMyClass.DoSometing;begin end; end.  First breakpoint reached is in destructor with item id "IMyInterface", next breakpoint is "end;" of button click handler, final breakpoint is destructor called for "IMyOtherIntf". This would lead to "IMyOtherIntf" application state update being lost. I know this is a contrived example, but it illustrates what could happen in a concrete application. Given this code and knowledge they are the final references to their interfaces, Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   myInterface:=nil;  myOtherIntf:=nil;  I would expect outcomes and timing of side effects, calling of destructors, for both assignments to be consistent and not depend on build type or optimization. ---------------------------------------------------------------------- marcov · 2021-01-27 10:10 The two interfaces are two references to the same object. So the object gets refcounted twice, and then with the :=nil decreased twice and finally deallocated. The exact moment is implementation defined, and you shouldn't rely on it. I see no problem. Keep in mind that the interfaces don't hold state, the underlying object does. Or do I understand you wrong? ---------------------------------------------------------------------- Coxy · 2021-01-27 10:52 QuoteThe two interfaces are two references to the same object. So the object gets refcounted twice, and then with the :=nil decreased twice and finally deallocated. It doesn't matter both interfaces are implemented by the same object type, it could be rewritten to use unrelated and different interfaces and implementing objects to produce same behaviour. QuoteThe exact moment is implementation defined, and you shouldn't rely on it. I see no problem. The manual https://www.freepascal.org/docs-html/ref/refse50.html says  "When the reference count reaches zero, usually the instance of the class that implements the interface, is freed.". I known no time is specified here but I take it to be immediate and in flow of code, not as soon as is convenient and possibly out of sequence, if the latter is the case then documentation should explicitly state this so application logic can reflect this. QuoteOr do I understand you wrong? My purpose here is to improve the compiler. Its possible use of temporary items, of what ever kind, should not be visible to or govern application development. This example shows this not to be the case. ---------------------------------------------------------------------- PascalDragon · 2021-01-27 12:31 Quote from: Coxy on January 27, 2021, 11:52:33 amQuoteThe exact moment is implementation defined, and you shouldn't rely on it. I see no problem. The manual https://www.freepascal.org/docs-html/ref/refse50.html says  "When the reference count reaches zero, usually the instance of the class that implements the interface, is freed.". I known no time is specified here but I take it to be immediate and in flow of code, not as soon as is convenient and possibly out of sequence, if the latter is the case then documentation should explicitly state this so application logic can reflect this. And the information that the object instance is freed when the reference count reaches zero is true (at least for those inheriting from TInterfacedObject or those classes correctly implementing a reference counting scheme similar to TInterfacedObject). What the documentation explicitly does not mention is where additional references might come from (namely temporary variables). This is by choice and thus the behaviour when the class instance will be freed is implementation defined. The only guarantee is that it will be freed as long as it isn't kept artificially alive (e.g. by a global variable or a cycle between interface references). Quote from: Coxy on January 27, 2021, 11:52:33 amQuoteOr do I understand you wrong? My purpose here is to improve the compiler. Its possible use of temporary items, of what ever kind, should not be visible to or govern application development. This example shows this not to be the case. But they are visible to application development. The same is true for other managed types like strings as well. Though it's indeed best visible with interfaces. ---------------------------------------------------------------------- Coxy · 2021-01-27 13:16 QuoteBut they are visible to application development. Not sure what you mean here, temporary variables created by compiler are not visible to application developer. QuoteAnd the information that the object instance is freed when the reference count reaches zero is true (at least for those inheriting from TInterfacedObject or those classes correctly implementing a reference counting scheme similar to TInterfacedObject). What the documentation explicitly does not mention is where additional references might come from (namely temporary variables). This is by choice and thus the behaviour when the class instance will be freed is implementation defined. The only guarantee is that it will be freed as long as it isn't kept artificially alive (e.g. by a global variable or a cycle between interface references). I think you are saying this is a language feature, which I would disagree with. Temporary items are a compiler construct, a means to an end, not part of the language, their use, and side effects of their use, should be hidden from application developers and not impact on application design. ---------------------------------------------------------------------- BeniBela · 2021-01-27 21:52 Quote from: marcov on January 26, 2021, 03:23:03 pm If you depend on the lifetime of the variable the code is simply wrong, as the URL I gave explains, working in Delphi is not necessarily correct code. Of the variable? But with temporary interface references, there is no variable Quote from: Coxy on January 27, 2021, 11:52:33 amThe manual https://www.freepascal.org/docs-html/ref/refse50.html says  "When the reference count reaches zero, usually the instance of the class that implements the interface, is freed.". It also says "Whenever the variable goes out of scope, the reference count is automatically decreased" That seems to imply, that is does not decrease the reference count before the variable goes out of scope In fact, it does not decrease it until the end of the scope: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i type ti = interfaceprocedure dop;end;type TSMart = class(TInterfacedObject,ti)  n: string;  procedure dop;  constructor Create(an: string);  destructor Destroy; override;end; procedure TSMart.dop;begin  writeln();end; constructor TSMart.Create(an: string);begin  n := anend; destructor TSMart.Destroy;begin  writeln('destroy ', N);  inherited Destroy;end; procedure test;var a,b,c,d:IUnknown;begin  a := TSMart.Create('a');  (ti(TSMart.Create('0'))).dop;  b := TSMart.Create('b');  (ti(TSMart.Create('1'))).dop;  c := TSMart.Create('c');  (ti(TSMart.Create('2'))).dop;  d := TSMart.Create('d');  (ti(TSMart.Create('3'))).dop;  writeln('x');end; ---------------------------------------------------------------------- Coxy · 2021-01-28 07:33 I'm not concerned with ARC life cycle management as such, compiler temporary items behave identically to application ones and in line with the language definition. Compile generated code is equivalent to this (for example code posted earlier), Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm2.Button1Click(Sender: TObject);var  myInterface : IMyInterface;  myOtherIntf : IMyOtherIntf;  compilerTemp : IMyInterface;begin  myInterface:=GetMyInterface;  compilerTemp:=GetMyOtherIntf;  myOtherIntf:=compilerTemp as IMyOtherIntf;   myInterface:=nil;    // Should cause destructor of TMyClass to be called.  myOtherIntf:=nil;  // Should cause destructor of TMyClass to be called.   // Something that writes out application state.   compilerTemp:=nil;end;        // Place breakpoint here This code effects execution follow. If the compiler, instead, generated this, Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm2.Button1Click(Sender: TObject);var  myInterface : IMyInterface;  myOtherIntf : IMyOtherIntf;  compilerTemp : IMyInterface;begin  myInterface:=GetMyInterface;  compilerTemp:=GetMyOtherIntf;  myOtherIntf:=compilerTemp as IMyOtherIntf;  compilerTemp:=nil;   myInterface:=nil;    // Should cause destructor of TMyClass to be called.  myOtherIntf:=nil;    // Should cause destructor of TMyClass to be called.   // Something that writes out application state. end;      // Place breakpoint here Then the code would behave identically with or without the use of temporary items. If there's a reason why the compiler handles temporary items in this way I willing to accept it, but would like to know what it is. ---------------------------------------------------------------------- marcov · 2021-01-28 09:02 Quote from: Coxy on January 27, 2021, 02:16:36 pmI think you are saying this is a language feature, which I would disagree with. Temporary items are a compiler construct, a means to an end, not part of the language, their use, and side effects of their use, should be hidden from application developers and not impact on application design. The language typically allows the room for the compiler to use it or not. This is what "implementation defined" means. And afaik the rule is that the interface reference stays defined for the statement it was last used in. How long it persists after that, is implementation defined, and assumptions on it are by definition wrong. ---------------------------------------------------------------------- marcov · 2021-01-28 09:12 Quote from: BeniBela on January 27, 2021, 10:52:44 pmIt also says "Whenever the variable goes out of scope, the reference count is automatically decreased" That is the whole point, the scope is not fixated in detail for the programmer. In practice it can be everything from after the statement of last use to the end of the procedure/method, depending how the compiler divides the code up into blocks of statements that belong together.  Some compilers deallocate temps quickly, which is something that dates back to old (16-bit) compilers, to keep pressure of the limited stack. (and keep in mind that with old static (short)string types that was more deeply felt. A recursive procedure with two by-value shortstring parameters and some local variables could run out of 16-bits worth of stack space rather quickly) Some let them linger longer because it is more efficient to do it all at once, usually at the end of the method. ABI rules also play a role. ---------------------------------------------------------------------- Coxy · 2021-01-28 09:55 As I said before, I'm just trying to improve the compiler for developers. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Some compilers deallocate temps quickly, which is something that dates back to old (16-bit) compilers, to keep pressure of the limited stack. (and keep in mind that with old static (short)string types that was more deeply felt. A recursive procedure with two by-value shortstring parameters and some local variables could run out of 16-bits worth of stack space rather quickly) Some let them linger longer because it is more efficient to do it all at once, usually at the end of the method. ABI rules also play a role. Removed  [It seems the compiler is reducing determinism for developers at the cost of efficiency.] NEW *** It seems the compiler is increasing efficiency at the cost of reducing determinism for developers. *** As a side note, Delphi doesn't seem to exhibit this behaviour, if I compile in {$mode delphi} what should I expect in terms of compatibility of execution? ---------------------------------------------------------------------- PascalDragon · 2021-01-28 12:35 Quote from: Coxy on January 28, 2021, 10:55:26 amAs a side note, Delphi doesn't seem to exhibit this behaviour, if I compile in {$mode delphi} what should I expect in terms of compatibility of execution? We only provide compatibility for documented behaviour. The point of release of temporary variables is not documented in the Delphi documentation thus we don't provide compatibility here. ---------------------------------------------------------------------- Coxy · 2021-01-28 14:11 I've created a bug report for this id: 0038415 I'll post back the outcome. ---------------------------------------------------------------------- Coxy · 2021-01-28 17:59 Apparently this has been raised as an issue before, earliest one being id: 9472 back in 2007, now having Resolution of "won't fix"! My bug report, id: 38415, has therefore been flagged as "resolved". I have suggested compiler emitting a warning of temporary interface reference use so application developers can be aware of them and, if needed, explicitly create temporary variables themselves negating compilers need for them. ---------------------------------------------------------------------- nanobit · 2021-01-28 18:36 Very recently the LCL was hit by such temp-ref issue: https://bugs.freepascal.org/view.php?id=36151 Sequence in TCustomFormEditor.deleteComponent(): 1) (AForm as INonFormDesigner).LookupRoot := nil; // new temp ref 2) call to PropertyEditorHook.PersistentDeleted; contains some very hidden Application.ProcessMessages which destroys objects 3) release of temp ref at method-end (crash in destructor) ---------------------------------------------------------------------- Coxy · 2021-01-28 20:46 QuoteVery recently the LCL was hit by such temp-ref issue: https://bugs.freepascal.org/view.php?id=36151 Well hopefully as more application problems relating to this issue arise the devs will consider modifying compiler functionality. Documented behaviour does not necessarily equate to usability.