anonymous functions in the highlighter of the Editor lazarus / 9-Editor ---------------------------------------------------------------------- Martin_fr · 2023-07-13 18:28 Unfortunately I only was made aware of it after 3.0 branch => so all this is for 4.0. The Pas-HL doesn't deal well with anonymous functions(in git main, it does have a basic implementation). However, I don't use anonymous functions myself. So I am not experiencing the up and downs of how the HL works or might work, nor am I reminded of it and therefore not likely to have many ideas of what could be considered... The basics - how the HL worked till now. The HL tries to continue, if code seems incorrect. It tries to find the best way to continue. While editing code, it happens that begin/end keywords are added, and until the matching other end is written, they are unbalanced. The following is for some people a common case Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure Foo;begin  if a then begin    c := a; // writing the conditional code, not yet added the "end"  b := 1;end; procedure Bar;beginend; In that case the HL would see "procedure Bar". It would know it is not nested (because the previous procedure had had its code begun after the "begin"). So the HL would decide the previous procedure must be finished, and assume the missing "end". That works very well... Until you have anonymous functions. Currently the HL tries the following new method. It checks if after the "procedure" is a name or not (thereby deciding if it should deem it anonymous or not). But that comes with new problems. We can't know what is a name, we can only to the reverse. The HL looks for known identifiers that would NOT be a name: "begin", "var", "const". Or "(" for an argument list, ":" for functions (";" in case?). Still not enough: cdecl, stdcall, inline .... 1) The list probably isn't complete. reports or patches welcome 2) The HL is not meant to look ahead => that could cause new issues (it currently tries to find ONE next identifier, skipping space and comment, up to 1000 lines (might get reduced to less lines). 3) While typing the words may not be complete "procedure beg" is a named procedure, until the "in" of "begin" is typed in.... As for (3): Yes, it could accept all partial versions of each potentially allowed keyword/identifier... But that are a lot, and then cuts into the functionality of closing a function with a missing "end". Even "inline" and "cdecl" are allowed as names. So for starters.... Any opinion or idea... Doesn't mead it will be implemented. But maybe something good turns up. ---------------------------------------------------------------------- Peter H · 2023-07-13 20:02 I myself solved the highlighting problem with macros: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$macro on}{$define proc:= procedure}{$define func:= function}....   procedure test2(h: int64);  begin    queue(proc    begin      Writeln(txt);    end);  end; .....  This means, in anonymous procedures and functions, the words "proc" and "func" are ignored by the highlighter. As far as begin...end highlighting goes, this works for me and I anyway prefer the shorter form, especially with one-line lambdas. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  queue(proc begin Writeln(txt); end); This solution is of course not for everybodys taste.... I do not know about the highlighter. Cant ist simply ignore "procedure" and "function"? It could ignore these words, but remember their position. If it finds a "begin" it could look if there is a preceding "function" or "procedure" and highlight this too. If it must highlight these words, and if it cannot read forward it could read backwards and highlight the remembered preceding "function" or "procedure" only then, when it has found a "begin" because functions and procedures always have a "begin", when the code section starts. ---------------------------------------------------------------------- Martin_fr · 2023-07-13 20:42 Quote from: Peter H on July 13, 2023, 10:02:00 pmI do not know about the highlighter. Cant ist simply ignore "procedure" and "function"? Well, for my workflow (the example code) I want the HL to see procedure, and I want it to assume "new top level" procedure. About scanning forward. - Within one line => no problem. - Into the next line... Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ...// anonymousprocedure (*     *)begin  It is simple to scan forward => that happens now. But it will not deliver a consistent result. Try it: 1) Add "foo" in front of begin (don't edit anything else) => it is still recognized as a anonymous proc. 2) add a space before/after "procedure (on that line) => no, it is no longer an anon proc (because the HL can see the name) 3) remove the name => no change 4) do a space again => change Actually, just testing => this crashes... BUt once I fix the crash, the above is what would happen. The HL only scans the changed line, and lines after that. It needs to be extended, to know when it needs to go backward, and how far. Of course can be done => but is tricky. The HL is triggered after each single keystroke => it needs to be somewhat efficient about speed. Not impossible, but a lot more work than just a small change. So likely for the foreseeable future that is not offering this. ---- Then the question is: How consistent should it be? ---------------------------------------------------------------------- Peter H · 2023-07-13 20:59 I must admit, I did never see it working during typing, and I never missed this. Occasionally when I have heavily edited my code, I get confused and then I click "begin" or "end" to see the overall structure and to see where I have forgotten something... Or when the "begin" and the "end" span more than one page on the screen. If the code is heavily incomplete or malformed then you must implement a mindreading feature.... ;-) ---------------------------------------------------------------------- Martin_fr · 2023-07-13 21:28 Here is a (contrived) example; Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i //(*program Project1;{//*)unit foo;interfaceimplementation//*} procedure foo;var a: integer; a: integer; a: integer; a: integer; a: integer; a: integer;begin  if a = 0 then begin foo;  end  else    if a = 0 then begin foo;  end  else      if a = 0 then begin foo;  end  else        if a = 0 then begin foo;  end  else          if a = 0 then begin foo;  end  else            if a = 0 then begin foo;  end  else foo;end;   copy and paste the procedure, so you get 500.000 lines of code. Then hit return, between the // and (* in the first line => now you should get a short delay. Yes that is extrem. But the point is, most other editing in that unit is fine. It's fast, despite the amount of lines. And well that can be fixed, and I had plans to do that for over 10 years.... (still having them). But it explains why I am carefully adding new features in a way that I may later have to fix. And yes, in most cases scanning a bit ahead wont be noticeable. Even the extra battery drain for laptop users will be low. So to find a good solution it is useful to collect - ideas - workflows (what is actually needed, what can help, what can be ignored As for the latter, I have had quite a few surprises when I got feedback on other features. About how some people would use something, that would have never occurred to me. There are way to many workflows out there. There will have to be limited scan ahead.... But what is more and what is less important for the HL to color/fold/outline... QuoteI did never see it working during typing - If you have the outline enabled => add a begin somewhere in the code.   And you should see all the begins (and there outlines) below will change color, as they are one nest level deeper. - Open a comment, or string => color changes immediately - If you have the vertical line that indicates the end of a procedure, when you add a begin, watch how it move to the next "procedure", because the "end" no longer closes the current procedure, but the next procedure will account for the missing end. Not all of that is always useful (not to everyone). But some is... ---------------------------------------------------------------------- Peter H · 2023-07-13 21:57 I am sorry I do not remember to have enabled outline, I do not know what it is. ;-( I modified the code a little bit and what I see when I click "begin", is this in the attachment. So, if it has found "procedure" or "function" after a "begin" which is not satisfied with a matching "end", then it should ignore "function" and "procedure". This could work. Of course I do not know your code structure and planning so this is just an uneducated idea. ---------------------------------------------------------------------- Peter H · 2023-07-13 22:29 Another possibility is: It has found a begin, which is yet not satisfied. Then it finds "procedure" or "function". Then it could highlight "function/procedure" and then ignore and skip everything until it finds a corresponding "begin" and it highlights this too. When there are nested anonymous procedures, this cycle is iteratively continued. If there is no following begin, then the code structure is too uncompleted and cannot be outlined. In other words: Presume that after "function" or "procedure" a "begin" must follow and ignore everything inbetween und do this recursively. ---------------------------------------------------------------------- Peter H · 2023-07-13 23:04 I have now enabled "outline" and I see this. Apparently it has skipped the first begin. This is too colorful for me, probably I will disable it again....  (Lazarus:3.99, FPC:3.3.1) ---------------------------------------------------------------------- Martin_fr · 2023-07-14 06:32 Quote from: Peter H on July 13, 2023, 11:57:12 pmSo, if it has found "procedure" or "function" after a "begin" which is not satisfied with a matching "end", then it should ignore "function" and "procedure". Have you tested with latest git main branch? (Not 3.0. fixes) ---------------------------------------------------------------------- Martin_fr · 2023-07-14 06:51 So the crash is because of the scanning ahead (scanning lines ahead)..... And it will take more time to fix. So for now, until I have the time, it will need to be limited to the current line. Which will likely mean, nested procedures will only be ok, if they have the opening "(" for params, or the "begin" (or maybe but not sure) the "cdecl") on the SAME line as the procedure keyword. Maybe an option for a more lenient approach can be done. But the default setting must see a new procedure as the end of the previous procedure, unless it can make sure that it is anonymous. And it only knows for sure, if it can find (in the ONE line) that the procedure has no name. I'll fix that on the weekend. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure foo;begin   a := procedure begin // will be OK / begin on same line  end;   a := procedure() // will be OK / ( on the same line   begin    end;   a := function : // will be OK /    : on the same line   integer   begin    end;   a := procedure // will NOT be OK / will end "foo"    begin    end; end; The last one can not be ignored. It is an essential feature of the current HL. Maybe it can be turned off by option.... ---------------------------------------------------------------------- Peter H · 2023-07-14 08:31 Quote from: Martin_fr on July 14, 2023, 08:32:17 amQuote from: Peter H on July 13, 2023, 11:57:12 pmSo, if it has found "procedure" or "function" after a "begin" which is not satisfied with a matching "end", then it should ignore "function" and "procedure". Have you tested with latest git main branch? (Not 3.0. fixes) I use always fpcupdeluxe and choose "trunk" and "GIT" there, both for Lazarus and FPC. It is not a production system, my interest is more to explore and test the new features. About once in a week I update or when there is a specific reason. ---------------------------------------------------------------------- Martin_fr · 2023-07-14 19:02 Quote from: Peter H on July 14, 2023, 10:31:43 amAbout once in a week I update or when there is a specific reason. Well, some changes were made yesterday. So if you haven't you should update. If you have, odd, because what you show, is what I got *before* those changes... --- Since there is a crash, a further fix will be soon. ---------------------------------------------------------------------- Peter H · 2023-07-14 21:20 No, I updated maybe a week ago. Currently I cannot update, Lazarus Trunk apparently does not compile. I try later again. ---------------------------------------------------------------------- Martin_fr · 2023-07-15 12:57 Ok, I fixed the crash. But that meant for now, that "scan ahead" is limited to the current line. For now, this was meant to be an easy and quick fix. As I am quite busy with other stuff. So a better implementation will be some day later. Shortcomings can be collected as an issue on the tracker, so that when I get to it, I will have a list (otherwise lots of stuff likely will just be overlooked) So currently the following how it currently works: * The HL does not depend on "modeswitch anonymous proc" (will be added later / this might affect inc files, where the switch is not in the inc file) * While in a code block (after begin), the HL will see "procedure" as the begin of the next top level procedure (assuming unmatched begin/end). * Except if, the keyword procedure/function is on the same line followed by  - :  (function only)  - (  This indicates there was no name, and then it is an anonymous proc. (Currently also ; though that can not actually happen)  - begin var const type label  Again no name means anonymous proc.  - inline cdecl stdcall (and some others) this could actually be a name, but are more likely to be the proc modifier... So currently they are accepted. * All of those will be detected if there is a comment in between, but they are still on the same line: "procedure {foo} (a: bool) begin" will see the "(". In any other case, including when the word after "procedure" has only been partly typed yet, the procedure is not an anonymous proc. Obviously, this could be greatly improved by - checking more than one token ahead. - checking the next lines too (but that needs rework throughout SynEdit) It is also possible that a future option may be added, to tell the HL: Procedure inside a begin/end should default to anon (unless proven otherwise). But as indicated, currently (and without such an option) the use of "procedure" to detect unbalanced begin/end, and therefore the keyword "procedure" closes all open begin/end and starts a new procedure block, that is by intention. And that will be kept as default. (Thereby meaning "procedure" is only anon, if it can be proven to be anon / or at least be show highly likely) Bit of history.... The HL (like afaik the compiler) was originally written as "single scan" without any way to look ahead. Back in the day, that was also helping to keep it fast / today (most) PC are fast enough to do more. (Though I haven't tested limits on raspberry like devices / and Laptops still use up battery). In any case the "no look ahead rule" is no longer a "can't be done", rather more a "do with care". But because of the history, other code depends on that, and must be tested/fixed. That is, code (outside the HL) can cache info about lines in the text (including HL/fold data). And such code assumes that if line 25 of the text changed, then all caches up to line 24 are still valid. And with scan ahead for anon procs, that was not the case and the code crashed. Hence adding more look ahead will have to wait, until sufficient time is available. Also detecting and storing the modeswitch, may need some rework. There are some easy ways to add that, but they would need some testing, maybe some rework. ---------------------------------------------------------------------- Peter H · 2023-07-15 15:15 This might be out of topic, but currently I cannot install FPC+Lazarus trunk using fpcupdeluxe on win64. I see various compiling errors, the first is: C:\fpc2\lazarus\components\leakview\dbginforeader.pas(923) Fatal: (10026) There were 1 errors compiling module, stopping Neither updating nor a total fresh install works. This is since yesterday evening until now. I do not think this has to do with the highlighter code modifications. I go on with this, when it works again. ---------------------------------------------------------------------- Martin_fr · 2023-07-15 16:52 My fpc trunk is about 3 weeks old, and that works. This could be an issue in either lazarus or fpc. In any case, it needs to be reported on the bug tracker. (It might even depend on settings like -On or -g or any other => In that case you could be in a small group of people getting the error, and if you don't report it, it might be a long time. If "Fatal" was the only error, then that might be the compiler has crashed, in which case it would be a bug in fpc. ---------------------------------------------------------------------- Martin_fr · 2023-07-15 16:53 In any case there is no hurry, since I got other stuff todo first anyway. ---------------------------------------------------------------------- Peter H · 2023-07-16 18:30 Quote from: Martin_fr on July 14, 2023, 09:02:37 pmQuote from: Peter H on July 14, 2023, 10:31:43 amAbout once in a week I update or when there is a specific reason. Well, some changes were made yesterday. So if you haven't you should update. If you have, odd, because what you show, is what I got *before* those changes... --- Since there is a crash, a further fix will be soon. Ok, I was now able to test it. (The aforementioned  bug is resolved. It was more a missing overload in the compilers code then a bug) It works as you said with procedure(), but also works with procedure(. And it works with my macros proc and func. I do not know your code and do not know if this is possible. This is a quick idea: Could it not treat "procedure ", that means the word "procedure" with an appended (white)space, treat in the same way as "procedure(", this means treat "(" in the same way as a whitespace? I do of course not know, if this causes other problems, but so far I think about problems, I see none. The round brackets have no meaning for overall code structure. ---------------------------------------------------------------------- Martin_fr · 2023-07-16 18:51 Quote from: Peter H on July 16, 2023, 08:30:09 pmI do not know your code and do not know if this is possible. This is a quick idea: Could it not treat "procedure ", that means the word "procedure" with an appended space, treat in the same way as "procedure(", this means treat "(" in the same way as a space? Well, there are probably a few "mini" heuristics that could be added. Like if it is after ":=" => that probably catches a lot. (And can be done, even if the ":=" is on another line. I might still do that... I hadn't done it, because if the rules (heuristics) become to complex, then you get more bug reports like "is sometimes broken, but no idea how to reproduce" => and that costs more time to establish if it is a known "not yet working" or a "new bug". Just having leading-spaces on the line is not working. While not particularly likely, the following is possible (and could be in  some ones  coding style) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$Ifdef WithFoo}  procedure  Foo;  begin    //...  end;{$Endif}  However, by default this should act as safeguard for unterminated "begin" blocks. (Again, if later an option is added, then the logic can be different, for those who prefer anon-func over safeguard) ---------------------------------------------------------------------- Peter H · 2023-07-17 05:45 I did some experiments. When I add misplaced, eventually unbalanced "(" characters, this does apparently not disturb Code structure highlighting. And I believe, it shouldnt. This is ok. So if the word "procedure" is always treated as if it where "procedure(", this should work always. Apparently the highlighter does not care about misplaced round brackets this does not disturb it and I believe this behaviour is ok. If the code structure is uncomplete or faulty (missing begin for example) then the highlighter should display this corrupted structure, so that I see it. This is ok. If the structure is buggy, this bug should be made as obvious as possible. When I make complex modification it can happen, that I inadvertendly delete a "begin" or "end" or add one too much. Then the highlighther should display this as it is, especially when it is wrong, because then I can easier see and  fix it. ---------------------------------------------------------------------- Peter H · 2023-07-17 07:34 I made some code with intentional misplaced round brackets and this works. So if the highlighter would assume, after procedure or function always a () follows, (even when there is none) then it should work for all procedures and functions, anonymous or not. ---------------------------------------------------------------------- Martin_fr · 2023-07-17 08:29 The HL does not care about the extra (misplaced) brackets. It stops looking at the first "(" => because if there is nothing (except space/comment) between the keyword "procedure" and the "(" then this proves there is no name for the procedure. Non-anonymous procedures must however have a name. "procedure Foo()" That name comes before the "(". So the "(" proves it's anonymous. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure Foo()();Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i begin {...} a:=procedure ()(); The HL has no defined behaviour for the 2nd "()". It will ignore it, and get to a defined state at the next "begin" (or "var"...) In the above the 2nd "()" do not call the proc. There first would need to be a "begin". However the HL does not currently do any special handling for "(...)" invoking a proc call. In fact "foo(1)" the HL does not know that foo is a prog. It could be a typecast. ---------------------------------------------------------------------- Peter H · 2023-07-17 18:24 Quote from: Martin_fr on July 17, 2023, 10:29:45 amNon-anonymous procedures must however have a name. "procedure Foo()" That name comes before the "(". So far I can see it works when () is appended to each procedure and function. It works also in class declarations. You can test this without programming if you globally replace in the editor procedure -> procedure() and function -> function() For the overall code structure and outlining it does not matter if a procedure/function has a name or not and probably should not matter. See this: