A FreePascal Parser in C++ lazarus / 64-FPC development ---------------------------------------------------------------------- Rochus · 2023-07-28 13:50 In case anyone is interested, here is a parser for FreePascal 3.2.2 written in C++: https://github.com/rochus-keller/FreePascal/ It is already able to parse the ~700 files of the FP compiler (which is what I'm mostly interested in). I will also improve compatibilty with the RTL at least, but the next step is a navigator/x-referencer as implemented e.g. for https://github.com/rochus-keller/LisaPascal/. There are still open questions concerning the language and I will come back here with these at a later time. ---------------------------------------------------------------------- Thaddy · 2023-07-28 18:09 It would probably be much easier to write a C++ parser in Pascal.... ---------------------------------------------------------------------- Rochus · 2023-07-28 18:35 Maybe, but personally I aspire on https://github.com/rochus- keller/Oberon/ on the long run; until then I'm most efficient with C++. ---------------------------------------------------------------------- 440bx · 2023-07-28 19:09 I just wanted to mention, that is an impressive amount of work and dedication. ---------------------------------------------------------------------- Rochus · 2023-07-28 19:33 Thanks. ---------------------------------------------------------------------- simone · 2023-07-28 20:37 Thanks for this remarkable work and for the other projects related to the languages of the Pascal family that I found in your github repository. In particular, I really appreciated the complete and up-to-date fpc grammar specified in EBNF. I think it's the only one. Curiously, I'm the initiator of the thread that you mention on the project page. I was just looking for a grammar in EBNF. It was the year 2016. Thank you. ---------------------------------------------------------------------- Rochus · 2023-07-28 21:03 Thanks for the feedback which I appreciate. Concerning the grammar please keep in mind that it is optimized for parser generation and also is likely to need additional modifications to cope with the RTL and FCL code. To make it suitable for a parser I had to apply a couple of tricks to avoid LL(k) lookahead; for example, I had to merge record and array constant with factor (i.e. the '(' expression ')' part), which has to be considered by the parser/validator. ---------------------------------------------------------------------- marcov · 2023-07-28 21:54 Consider running the testsuite. It contains many grammar oddities. ---------------------------------------------------------------------- Rochus · 2023-07-28 22:02 Thanks. I'm actually already satisfied when it successfully parses the "regular" source code; currently this is the case with the FP compiler source code (with a few syntax error fixes), which is what I'm mostly interested in at the moment. It's not my goal to replace the FP compiler, so I can live with the subset of the language used by the FP compiler sources. Compatibilty with all existing source code including oddities might be possible, but a lot of work. ---------------------------------------------------------------------- Curt Carpenter · 2023-07-28 23:10 Curios as to the motivation driving this impressive work. ---------------------------------------------------------------------- Rochus · 2023-07-28 23:22 The idea is to use it to analyse and possibly reuse the FP compiler backend. I'm not a big fan of LLVM because of its monstrous size and complexity, and FPC looks like a lean implementation which supports each and every architecture. Another idea is to translate parts of the huge FP libraries to Oberon+. But details and feasibility are not yet clear. So far it was helpful to explore the features of the current language version. ---------------------------------------------------------------------- ccrause · 2023-07-30 06:01 Quote from: Rochus on July 29, 2023, 01:22:33 amSo far it was helpful to explore the features of the current language version. In your opinion, are there elements of the Pascal grammar that can be simplified or improved?  For example the syntax of procedure/function modifiers. ---------------------------------------------------------------------- Rochus · 2023-07-30 09:16 Well, you see that I'm currently using C++ (albeit in a moderate, rather conservative form) and spend time with Oberon+ where the focus is explicitly on simplicity. FreePascal is definitely a big language, actually several languages in one, apparently with a very tolerant compiler (supporting an even bigger language with variations than the ones specified). So if you ask me, whether this could be simpler, then yes, but only in theory, since there already is a lot of code which depends on this big language and it's many variations. This means that when you start to "simplify" the language, i.e. by allowing let's say "proc" instead of the redundant "function" and "procedure", or "return expression;" than "result := expression; exit;", then in reality you just add yet another variation, because the compiler has to keep compatibility with all existing code. This makes overall complexity even bigger. ---------------------------------------------------------------------- korba812 · 2023-07-30 10:39 Quote from: Rochus on July 30, 2023, 11:16:30 am... "return expression;" than "result := expression; exit;"...This is already done through the Exit function: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Exit(expression); ---------------------------------------------------------------------- Rochus · 2023-07-30 10:47 Thanks for the hint. Pseudo functions/procedures are yet another dimension to explore in addition to syntax. This will keep me busy for a while. ---------------------------------------------------------------------- marcov · 2023-07-30 11:19 Quote from: Rochus on July 30, 2023, 11:16:30 am This means that when you start to "simplify" the language, i.e. by allowing let's say "proc" instead of the redundant "function" and "procedure", or "return expression;" than "result := expression; exit;", then in reality you just add yet another variation, because the compiler has to keep compatibility with all existing code. This makes overall complexity even bigger. Actually modula2 did this. No need to be terse in this age of assisting AI IDEs, "procedure" is fine enough. p.s. Note that FPC doesn't have a formal frontend-backend separation, and trying to interface the backend from a different frontend would be a chore in any language. But using a different one surely won't make it easier. ---------------------------------------------------------------------- Rochus · 2023-07-30 11:38 Quote from: marcov on July 30, 2023, 01:19:18 pmNote that FPC doesn't have a formal frontend-backend separation, and trying to interface the backend from a different frontend would be a chore in any language. But using a different one surely won't make it easier. Well, I assume that at least the code generator uses the same intermediate representation for all supported architectures. Otherwise the compiler would hardly be manageable. And I also assume that this is also the representation used in the optimization phase. But that's what I intend to explore. And I will also do performance measurements to find out what the optimizer can do. ---------------------------------------------------------------------- marcov · 2023-07-30 12:12 Quote from: Rochus on July 30, 2023, 01:38:24 pmQuote from: marcov on July 30, 2023, 01:19:18 pmNote that FPC doesn't have a formal frontend-backend separation, and trying to interface the backend from a different frontend would be a chore in any language. But using a different one surely won't make it easier. Well, I assume that at least the code generator uses the same intermediate representation for all supported architectures From what I know (I'm not a compiler developer) it is a multi pass tree representation followed by something called asm list which is a target specific representation closer to generated assembler. The so called high level optimizer works on the tree representation and the asm/peephole optimizer on asmlists QuoteOtherwise the compiler would hardly be manageable. And I also assume that this is also the representation used in the optimization phase. But that's what I intend to explore. And I will also do performance measurements to find out what the optimizer can do. It's decent, but it depends of course what you compare it with. Corporate behemoths like LLVM and GCC, or other standalone community driven compilers ? ---------------------------------------------------------------------- Rochus · 2023-07-30 12:28 Quote from: marcov on July 30, 2023, 02:12:41 pm... it depends of course what you compare it with. First and foremost with my existing Oberon+ backends: https://github.com/rochus-keller/Oberon/tree/master/testcases/Are-we- fast-yet ---------------------------------------------------------------------- tr_escape · 2023-07-30 12:33 https://github.com/rochus-keller/FreePascal/blob/9ba7e276b3f6dc5b605cf e78a6fce3ff182894d5/FpTokenType.cpp#L1044 if or implementation  is it fast way? ---------------------------------------------------------------------- marcov · 2023-07-30 12:49 Quote from: tr_escape on July 30, 2023, 02:33:55 pmhttps://github.com/rochus-keller/FreePascal/blob/9ba7e276b3f6dc5b605 cfe78a6fce3ff182894d5/FpTokenType.cpp#L1044 if or implementation  is it fast way? I'd expect as soon as such a deep nested IF only has one possible candidate, a mem compare for the remainder would be faster. ---------------------------------------------------------------------- marcov · 2023-07-30 12:50 Quote from: Rochus on July 30, 2023, 02:28:04 pmQuote from: marcov on July 30, 2023, 02:12:41 pm... it depends of course what you compare it with. First and foremost with my existing Oberon+ backends: https://github.com/rochus-keller/Oberon/tree/master/testcases/Are-we- fast-yet IMHO one file math problems are bad benchmarks that favour heavily optimizing compilers. Unless you typically write one file math problmes :-) ---------------------------------------------------------------------- Rochus · 2023-07-30 13:23 > IMHO one file math problems are bad benchmarks that favour heavily optimizing compilers. Microbenchmarks alone tend to produce non-representative and random results; that's why the Are-we-fast-yet suite also includes a couple of bigger applications and also rules for the implementations to support fair comparisons. If you just look at single microbenchmarks in the reference result tables the overall result would be different, with a higher error margin. > a mem compare for the remainder would be faster. I didn't understand the argument of the fellow; but apparently it is about the lexer performance, not the generated code. I didn't measure and optimize compiler performance yet. Your proposal depends on the optimization features of the C++ compiler; if it doesn't inline the code, then you essentially replace the if-conditions by a procedure call which again does if-conditions. But it is pointless to discuss this without specific measurements. ---------------------------------------------------------------------- marcov · 2023-07-30 14:23 Quote from: Rochus on July 30, 2023, 03:23:27 pmif it doesn't inline the code, then you essentially replace the if-conditions by a procedure call which again does if-conditions. But it is pointless to discuss this without specific measurements. But you might save up multiple .size checks etc. And small, constant memcmp's might also be inlined. Note that I also only looked because of tr_escape's comment, and that is what directly struck me. ---------------------------------------------------------------------- Rochus · 2023-07-31 16:13 Quote from: tr_escape on July 30, 2023, 02:33:55 pmif or implementation is it fast way? Quote from: marcov on July 30, 2023, 02:49:12 pmBut you might save up multiple .size checks etc. And small, constant memcmp's might also be inlined. I did measurements today comparing the keyword recognizer if-tree with the memcmp approach using the LisaPascal project. Please find the two versions (generated files) as attachments. There is no difference in performance! Actually there is a slight difference in that the memcmp version is about 1% slower than the if-tree version, but I assume the measurement error is higher than that. How did I measure: I run the LisaPascal parser driver (see https://github.com/rochus- keller/LisaPascal/blob/master/main.cpp) on the full Lisa source directory. 360 pascal files are parsed, 347 of which without errors. I measure the wall time required to parse the 360 files, which is one run. I did six runs with each configuration and calculated the average. - The run with the if-tree version takes 6547 ms on average on my EliteBook 2530. - The run with the memcmp version takes 6614 ms on average. You might have notized that I also replaced the QByteArray with bare const char* and len, the performance gain of which is barely measurable (~0.4%). ---------------------------------------------------------------------- colby · 2023-08-01 13:42 Assuming no performance changes, what would even be the point of "fixing" this?  I suppose the deeply nested if looks bad (I don't like long lines, either), but the file is automatically generated, and it could be changed to use logical-and in the if clause, rather than nesting ifs.  Compare: https://codeberg.org/triplescripts/trplkt/src/commit/23f26d3ad4be5705b 5282caab76cf3f10d677605/src/TokenScanner.src#L545 ---------------------------------------------------------------------- Rochus · 2023-08-01 14:31 Quote from: colby on August 01, 2023, 03:42:45 pmwhat would even be the point of "fixing" this? I was just curious given the claims of the fellows, since intuition usually is an unreliable advisor when it comes to performance. People might then be a little more careful with their claims when they see the measured values. Personally I don't care much how the generated code looks; the present one is decently readable and efficient. Also replacing the if-tree by an AND expression generates about the same amount of machine code.