Improving Appstore stacktraces on fpc_* functions lazarus / 43-Android ---------------------------------------------------------------------- BeniBela · 2017-11-11 23:13 When an app crashes, Google's store provides a stacktrace. Unfortunately it is often incomplete, especially it seems to stops on fpc_ functions. Now for example I got a two line stacktrace for my app, which appears to be SYSTEM$_$TOBJECT_$__$$_INHERITSFROM$TCLASS$$BOOLEAN and  fpc_do_is. Somehow  fpc_do_is, the is operator crashed. Probably I called it on an already freed object, but how can someone find that? Searching and examining  every usage of the is operator would take forever. So how can you compile everything that it shows the caller of fpc_* functions? I already compiled my project with -O1. Would it help to compile the rtl  with -O1? Or even  -O-  ? ---------------------------------------------------------------------- engkin · 2017-11-12 00:40 Compiling with -al will give you the line number where fpc_do_is is used. ---------------------------------------------------------------------- BeniBela · 2017-11-12 13:58 Quote from: engkin on November 12, 2017, 01:40:25 amCompiling with -al will give you the line number where fpc_do_is is used. Well that lists 196 usages. None with an obvious bug That does not help to list the caller in the next stacktrace Perhaps the function needs a better stackframe around it. Or there is a tool to create new functions fpc_do_is1 ... fpc_do_is196, and the replace the calls to those ---------------------------------------------------------------------- engkin · 2017-11-12 15:31 Quote from: BeniBela on November 12, 2017, 02:58:39 pmOr there is a tool to create new functions fpc_do_is1 ... fpc_do_is196, and the replace the calls to those I doubt a tool exists, but I like the idea. I tried the following: Add a unit with fpc_do_is1 ... fpc_do_is196. I declared them as: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function fpc_do_is1(aclass : tclass;aobject : tobject) : boolean; compilerproc;function fpc_do_is2(aclass : tclass;aobject : tobject) : boolean; compilerproc;...implementation function fpc_do_is1(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS1']; compilerproc;  begin     Result:=assigned(aobject) and assigned(aclass) and       aobject.inheritsfrom(aclass);  end; function fpc_do_is2(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS2']; compilerproc;  begin     Result:=assigned(aobject) and assigned(aclass) and       aobject.inheritsfrom(aclass);  end;...Compile your project using custom assembler (I used -al -s -Aas)Modify calls to fpc_do_is in the *.s files to use your numbered callsRun PPAS Should be easy automate the process with some code. ---------------------------------------------------------------------- BeniBela · 2018-01-20 10:17 I did not want to mess with the assembly, so I removed the "is" operator from my Pascal source. A dozens in every update, till there are no more calls to fpc_do_is anywhere It is still crashing on fpc_do_is The caller must be some other fpc function ---------------------------------------------------------------------- taazz · 2018-01-20 10:55 Quote from: BeniBela on January 20, 2018, 11:17:34 amI did not want to mess with the assembly, so I removed the "is" operator from my Pascal source. A dozens in every update, till there are no more calls to fpc_do_is anywhere It is still crashing on fpc_do_is The caller must be some other fpc function the is operator should never raise an exception. That leads me to believe that its more probable that you have a buffer overrun (or similar) problem. any chance to unwind the stack to the origin? that might give you a path to follow but at this point I would try to find an emulator/device that has the problem and remote debug it. ---------------------------------------------------------------------- Thaddy · 2018-01-20 11:07 First thing that comes to mind to check is the use of interfaces (uses is internally) may be mix up between interface var en class var? ---------------------------------------------------------------------- engkin · 2018-01-20 14:25 Quote from: taazz on January 20, 2018, 11:55:57 amthe is operator should never raise an exception. I am not sure I follow you here. is operator ends up calling InheritsFrom which in turn plays with pointers: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i       class function TObject.InheritsFrom(aclass : TClass) : Boolean;         var           vmt: PVmt;         begin           if assigned(aclass) then           begin               vmt:=PVmt(self);               while assigned(vmt) and (vmt <> PVmt(aclass)) do                 vmt := vmt^.vParent;               InheritsFrom := (vmt = PVmt(aclass));           end           else             inheritsFrom := False;        end; The highlighted line is an evil one. Imagine vmt holds a random number. An invalid address. Actually we can produce it: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure Test;var  sl:TStringList;begin  sl := TStringList($12345678);   if sl is TStringList then    WriteLn('sl is TStringList')  else    WriteLn('sl is not TStringList');end;This code should crash. It does on my system. Edit: Maybe is operator should be protected with try except: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function fpc_do_is(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS']; compilerproc;  begin    try     fpc_do_is:=assigned(aobject) and assigned(aclass) and       aobject.inheritsfrom(aclass);    except on EAccessViolation do      Result := False;    end;  end; ---------------------------------------------------------------------- BeniBela · 2018-01-20 15:34 Quote from: taazz on January 20, 2018, 11:55:57 amThat leads me to believe that its more probable that you have a buffer overrun (or similar) problem. any chance to unwind the stack to the origin? that might give you a path to follow but at this point I would try to find an emulator/device that has the problem and remote debug it. The app store does not give the stack and I could never reproduce the issue anywhere... The app stores tries unwinding to create a backtrace, but clearly it fails after the second function Quote from: Thaddy on January 20, 2018, 12:07:54 pmFirst thing that comes to mind to check is the use of interfaces (uses is internally) may be mix up between interface var en class var? Is with interfaces  usually becomes fpc_intf_is Quote from: engkin on January 20, 2018, 03:25:53 pmMaybe is operator should be protected with try except: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function fpc_do_is(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS']; compilerproc;  begin    try     fpc_do_is:=assigned(aobject) and assigned(aclass) and       aobject.inheritsfrom(aclass);    except on EAccessViolation do      Result := False;    end;  end; Or perhaps try except is the problem. There is a use of fpc_do_is in fpc_catches. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   try    try      raise exception.Create('..');    except on e: Exception do begin      e.free;      raise;    end;    end;  except on e: Exception do begin    writeln('b');  end;  end;  gives exactly that backtrace (see attachment) Amd the big problem are the ??, especially the one in the third row GDB figures out that it should continue to fpc_catches, but the app store can't. It aborts on the first ?? How can you modify fpc_do_is/fpc_catches to not cause ?? in backtraces ---------------------------------------------------------------------- BeniBela · 2018-01-21 00:46 I have found my problem :/ Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   except on e: Exception do begin    ...    raise e;  end;     But still how do you prevent the holes in the stacktrace between fpc_do_is and fpc_catches? If fpc_catches was in the backtrace the issue would have been much easier to find.  (actually the app store handles it even better than fpc by showing inheritsfrom and fpc_do_is. The default backtrace on linux just shows inheritsfrom ) ---------------------------------------------------------------------- Martin_fr · 2018-01-21 00:58 If you do not want the "holes" then you must compile the rtl and packages without optimization. Those functions are optimized, and have their stack frame dropped. That is, the code that generates a frame on the stack. There may be: {$OPTIMIZATION STACKFRAME} If this is in the rtl code, you may need to remove it. ---------------------------------------------------------------------- taazz · 2018-01-21 01:14 Quote from: engkin on January 20, 2018, 03:25:53 pmQuote from: taazz on January 20, 2018, 11:55:57 amthe is operator should never raise an exception. I am not sure I follow you here. is operator ends up calling InheritsFrom which in turn plays with pointers: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i       class function TObject.InheritsFrom(aclass : TClass) : Boolean;         var           vmt: PVmt;         begin           if assigned(aclass) then           begin               vmt:=PVmt(self);               while assigned(vmt) and (vmt <> PVmt(aclass)) do                 vmt := vmt^.vParent;               InheritsFrom := (vmt = PVmt(aclass));           end           else             inheritsFrom := False;        end; The highlighted line is an evil one. Imagine vmt holds a random number. An invalid address. Actually we can produce it: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure Test;var  sl:TStringList;begin  sl := TStringList($12345678);   if sl is TStringList then    WriteLn('sl is TStringList')  else    WriteLn('sl is not TStringList');end;This code should crash. It does on my system. Edit: Maybe is operator should be protected with try except: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function fpc_do_is(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS']; compilerproc;  begin    try     fpc_do_is:=assigned(aobject) and assigned(aclass) and       aobject.inheritsfrom(aclass);    except on EAccessViolation do      Result := False;    end;  end;the idea of the is operator is to safely return a true/false answer with out raising exceptions. It is used on classes, interfaces and pointers. I have no idea how it is implemented in fpc but the idea behind it is something along the lines of Code: [Select] default answer is  no;   if isobject(InputVar) and isclass(InputClass) then result := Tclass(InputVar).InheritsFrom(InputClass)   else   if isInterface(Inputvar) and isInterface(inputClass) then result := Supports(InputVar,InputClass)   else .... The line shown should never execute if the inputvar has no vmt or the InputClass is an interface. While the compiler can (and should) decide at compile time which xxxx_do_is to call based on the parameters (overload?) there must be a small runtime check for validity as well. for me if it raises an exception the its a bug, as opposed to inheritsfrom which I expect to get an exception in the same case. then again I'm OT (again) in this thread. Quote from: BeniBela on January 20, 2018, 04:34:16 pmHow can you modify fpc_do_is/fpc_catches to not cause ?? in backtraces you build the rtl/fcl with debug info I guess. ---------------------------------------------------------------------- engkin · 2018-01-21 03:04 Quote from: taazz on January 21, 2018, 02:14:42 amthe idea of the is operator is to safely return a true/false answer with out raising exceptions. It is used on classes, interfaces and pointers. I have no idea how it is implemented in fpc but the idea behind it is something along the lines of Code: [Select] default answer is  no;   if isobject(InputVar) and isclass(InputClass) then result := Tclass(InputVar).InheritsFrom(InputClass)   else   if isInterface(Inputvar) and isInterface(inputClass) then result := Supports(InputVar,InputClass)   else .... FPC for isobject(InputVar) checks that InputVar is not nil. The same applies to isclass(InputClass). That's why both operator is and InheritsFrom could crash when invalid pointers are used. Quote from: taazz on January 21, 2018, 02:14:42 am[snip]for me if it raises an exception the its a bugThat why I suggested catching EAccessViolation. Quote from: BeniBela on January 21, 2018, 01:46:12 amI have found my problem :/ Congrats!! ---------------------------------------------------------------------- taazz · 2018-01-21 18:18 Quote from: engkin on January 21, 2018, 04:04:32 amQuote from: taazz on January 21, 2018, 02:14:42 amthe idea of the is operator is to safely return a true/false answer with out raising exceptions. It is used on classes, interfaces and pointers. I have no idea how it is implemented in fpc but the idea behind it is something along the lines of Code: [Select] default answer is  no;   if isobject(InputVar) and isclass(InputClass) then result := Tclass(InputVar).InheritsFrom(InputClass)   else   if isInterface(Inputvar) and isInterface(inputClass) then result := Supports(InputVar,InputClass)   else .... FPC for isobject(InputVar) checks that InputVar is not nil. The same applies to isclass(InputClass). That's why both operator is and InheritsFrom could crash when invalid pointers are used. Quote from: taazz on January 21, 2018, 02:14:42 am[snip]for me if it raises an exception the its a bugThat why I suggested catching EAccessViolation. That would be a blanket statement yes. ---------------------------------------------------------------------- BeniBela · 2018-03-03 18:33 Quote from: engkin on November 12, 2017, 01:40:25 amCompiling with -al will give you the line number where fpc_do_is is used. I think I always put -al there for debugging and removed  it for the release. Although I am not sure how it was months ago. But with the previous update I forgot to remove -al. And then the app store reported another eleven crashes: java.lang.UnsatisfiedLinkError on aarch64 OnePlus OnePlus3. Now I have removed -al again and the reports stopped coming Logical conclusion: besides giving numbers -al damages the .so file