Making sense of pparser, pastree and pscanner - help needed.
lazarus / 64-FPC development

----------------------------------------------------------------------
JernejL · 2025-01-16 13:53

I have tried to use unit parser included in fpc. followed instructions
here:

https://wiki.freepascal.org/fcl-passrc

I used TSimpleEngine from there, and some other sources to see how
resolver,scanner and parser work together.

However, if a class file uses {$include} directive, it cannot find
includes:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
game_Actors.pas(10,24) Error: Could not find include file
'compiler.inc'
I cannot figure out why as i did set BaseDirectory and add
AddIncludePath to TStreamResolver where the include file is. Does
anybody know how to set this up properly? Since fpdoc uses it, i
assume there is a way to make this work.

for example, just drop a memo, button and opendialog onto form:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  unit
Unit1; {$mode objfpc}{$H+} interface uses        Classes, SysUtils,
Forms, Controls, Graphics, Dialogs, Buttons, StdCtrls, pparser,
pastree, pscanner, strutils; type  // maybe better / different example
https://github.com/fpc/FPCSource/blob/fd677978e7661b22a15fb3a9bce3f5c4
b2ae8134/utils/fpdoc/unitdiff.pp#L61         { TForm1 }  TForm1 =
class(TForm)                BitBtn1: TBitBtn;                Memo1:
TMemo;                OpenDialog1: TOpenDialog;               
procedure BitBtn1Click(Sender: TObject);         private        
public         end;  TSimpleEngine = class(TPasTreeContainer) public 
 function CreateElement(AClass: TPTreeElement; const AName: String;  
 AParent: TPasElement; AVisibility: TPasMemberVisibility;     const
ASourceFilename: String; ASourceLinenumber: Integer): TPasElement;   
 override;   function FindElement(const AName: String): TPasElement;
override; end;   var        Form1: TForm1;     procedure
ParseFile(const FileName: string); implementation function
TSimpleEngine.CreateElement(AClass: TPTreeElement; const AName:
String; AParent: TPasElement; AVisibility: TPasMemberVisibility; const
ASourceFilename: String; ASourceLinenumber: Integer):
TPasElement;begin Result := AClass.Create(AName,
AParent); Result.Visibility := AVisibility; Result.SourceFilename :=
ASourceFilename; Result.SourceLinenumber :=
ASourceLinenumber;end; function TSimpleEngine.FindElement(const AName:
String): TPasElement;begin { dummy implementation, see
TFPDocEngine.FindElement for a real example } Result :=
nil;end;  procedure ParseFile(const FileName: string);var  I:
Integer;    Var     R : TStreamResolver;     S : TPascalScanner;     P
: TPasParser;     M : TPasModule;     C : TPasTreeContainer;       
 ms: TMemoryStream; begin    R:= TStreamResolver.Create;   try    
 ms:= Tmemorystream.create();     ms.LoadFromFile(filename);    
 r.BaseDirectory:= ExtractFilePath(filename);     r.AddIncludePath(
ExtractFilePath(filename) );    
 R.AddStream(ExtractFileName(filename), ms);    
 S:=TPascalScanner.Create(R);       try     
 S.OpenFile(ExtractFileName(filename));        C:=
TPasTreeContainer.Create();       try         C.InterfaceOnly:=True;  
       P:=TPasParser.Create(S, R, C);         try         
 P.ParseUnit(M);         finally           P.Free;         end;     
 finally         C.Free;       end;     finally       S.Free;   
 end;   finally     R.Free;   end;  end; {$R *.lfm} { TForm1
} procedure TForm1.BitBtn1Click(Sender: TObject);begin   if
opendialog1.execute = false then exit;  
ParseFile(opendialog1.FileName); end; end.

----------------------------------------------------------------------
dsiders · 2025-01-16 18:16

Quote from: JernejL on January 16, 2025, 02:53:56 pmI have tried to
use unit parser included in fpc. followed instructions here:

https://wiki.freepascal.org/fcl-passrc

I used TSimpleEngine from there, and some other sources to see how
resolver,scanner and parser work together.

However, if a class file uses {$include} directive, it cannot find
includes:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
game_Actors.pas(10,24) Error: Could not find include file
'compiler.inc'
I cannot figure out why as i did set BaseDirectory and add
AddIncludePath to TStreamResolver where the include file is. Does
anybody know how to set this up properly? Since fpdoc uses it, i
assume there is a way to make this work.

for example, just drop a memo, button and opendialog onto form:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  unit
Unit1; {$mode objfpc}{$H+} interface uses        Classes, SysUtils,
Forms, Controls, Graphics, Dialogs, Buttons, StdCtrls, pparser,
pastree, pscanner, strutils; type  // maybe better / different example
https://github.com/fpc/FPCSource/blob/fd677978e7661b22a15fb3a9bce3f5c4
b2ae8134/utils/fpdoc/unitdiff.pp#L61         { TForm1 }  TForm1 =
class(TForm)                BitBtn1: TBitBtn;                Memo1:
TMemo;                OpenDialog1: TOpenDialog;               
procedure BitBtn1Click(Sender: TObject);         private        
public         end;  TSimpleEngine = class(TPasTreeContainer) public 
 function CreateElement(AClass: TPTreeElement; const AName: String;  
 AParent: TPasElement; AVisibility: TPasMemberVisibility;     const
ASourceFilename: String; ASourceLinenumber: Integer): TPasElement;   
 override;   function FindElement(const AName: String): TPasElement;
override; end;   var        Form1: TForm1;     procedure
ParseFile(const FileName: string); implementation function
TSimpleEngine.CreateElement(AClass: TPTreeElement; const AName:
String; AParent: TPasElement; AVisibility: TPasMemberVisibility; const
ASourceFilename: String; ASourceLinenumber: Integer):
TPasElement;begin Result := AClass.Create(AName,
AParent); Result.Visibility := AVisibility; Result.SourceFilename :=
ASourceFilename; Result.SourceLinenumber :=
ASourceLinenumber;end; function TSimpleEngine.FindElement(const AName:
String): TPasElement;begin { dummy implementation, see
TFPDocEngine.FindElement for a real example } Result :=
nil;end;  procedure ParseFile(const FileName: string);var  I:
Integer;    Var     R : TStreamResolver;     S : TPascalScanner;     P
: TPasParser;     M : TPasModule;     C : TPasTreeContainer;       
 ms: TMemoryStream; begin    R:= TStreamResolver.Create;   try    
 ms:= Tmemorystream.create();     ms.LoadFromFile(filename);    
 r.BaseDirectory:= ExtractFilePath(filename);     r.AddIncludePath(
ExtractFilePath(filename) );    
 R.AddStream(ExtractFileName(filename), ms);    
 S:=TPascalScanner.Create(R);       try     
 S.OpenFile(ExtractFileName(filename));        C:=
TPasTreeContainer.Create();       try         C.InterfaceOnly:=True;  
       P:=TPasParser.Create(S, R, C);         try         
 P.ParseUnit(M);         finally           P.Free;         end;     
 finally         C.Free;       end;     finally       S.Free;   
 end;   finally     R.Free;   end;  end; {$R *.lfm} { TForm1
} procedure TForm1.BitBtn1Click(Sender: TObject);begin   if
opendialog1.execute = false then exit;  
ParseFile(opendialog1.FileName); end; end.   
I'm no expert, but...

 I believe the include path needs to be relative to the directory
where the source file is located. And, if it's the same directory the
include path needs to be '.'.

----------------------------------------------------------------------
JernejL · 2025-01-17 08:46

Seems like i need to set both relative and absolute to make any
progress, which is odd.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
 r.BaseDirectory:= ExtractFilePath(filename);     r.AddIncludePath(
'.' );     r.AddIncludePath( ExtractFilePath(filename) ); 
There is no more complaint about include missing, but it does die
somewhere else:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Project
rttiprocessor raised exception class 'EStringListError' with
message:List index (-1) out of bounds
I guess i will need to compile RTL with debug symbols to find out
where this problem is.

stack trace is only partially helpful, no proper line numbers:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i #0
fpc_raiseexception at :0#1
CLASSES$_$TSTRINGS_$__$$_ERROR$ANSISTRING$LONGINT at :0#2
CLASSES$_$TSTRINGLIST_$__$$_CHECKINDEX$LONGINT at :0#3
CLASSES$_$TSTRINGLIST_$__$$_GET$LONGINT$$ANSISTRING at :0 #4 PSCANNER$
_$TSTREAMRESOLVER_$__$$_FINDSTREAM$ANSISTRING$BOOLEAN$$TSTREAM at :0#5
PSCANNER$_$TSTREAMRESOLVER_$__$$_FINDSTREAMREADER$ANSISTRING$BOOLEAN$$
TLINEREADER at :0#6 PSCANNER$_$TSTREAMRESOLVER_$__$$_FINDINCLUDEFILE$A
NSISTRING$$TLINEREADER at :0#7
PSCANNER$_$TPASCALSCANNER_$__$$_HANDLEINCLUDEFILE$ANSISTRING at :0#8
PSCANNER$_$TPASCALSCANNER_$__$$_HANDLEINCLUDE$ANSISTRING$$TTOKEN at
:0#9
PSCANNER$_$TPASCALSCANNER_$__$$_HANDLEDIRECTIVE$ANSISTRING$$TTOKEN at
:0#10 PSCANNER$_$TPASCALSCANNER_$__$$_DOFETCHTOKEN$$TTOKEN at :0#11
PSCANNER$_$TPASCALSCANNER_$__$$_FETCHTOKEN$$TTOKEN at :0

----------------------------------------------------------------------
JernejL · 2025-01-20 10:45

Apparently the parser itself is rather poorly implemented and does not
support initialize operator within records.

QuoteUnknown operator type: Initialize at token "Identifier
Initialize" in file game_Actors.pas at line 175 column 24 line:175
column:24 file:game_Actors.pas
Exception at 00455023: EPasWriter:

I will report this as a bug but i guess i'll need to go with some
other solution in the meantime at least.

----------------------------------------------------------------------
JernejL · 2025-01-20 11:42

apparently i was an idiot with that.

Still, in the end i found out that the parser has no support for -Sc /
{$COPERATORS ON}

  //EparserError on C++ style
  //X+=Y;      { Same as X := X+Y, needs -Sc command line switch}
  //x-=y;
  //X/=2;      { Same as X := X/2, needs -Sc command line switch}
  //x*=y;

----------------------------------------------------------------------
Thaddy · 2025-01-20 12:09

Good, I like that, because {$COPERATORS ON} is silly.
A candidate for removal, were it not that it breaks a minority of
code, because some people actually use it.

----------------------------------------------------------------------
JernejL · 2025-01-23 14:16

Quote from: Thaddy on January 20, 2025, 01:09:44 pmGood, I like that,
because {$COPERATORS ON} is silly.
A candidate for removal, were it not that it breaks a minority of
code, because some people actually use it.


I figured out the latest version of pscanner from trunk works well and
supports COPERATORS, plus i managed to get most of it working based on
pasrewrite example.

I will now write my own paswrite-based module so i can generate what i
need (i need to generate rpc mock classes).

Regarding {$COPERATORS ON}:  I guess you never wrote a lot of math
intensive code to need it (like for game).
Repeating tons of multi-property expressions is not very fun, i love
this feature as it makes code much more readable.