Disable parsing of include files? [solved] lazarus / 9-Editor ---------------------------------------------------------------------- creaothceann · 2018-11-29 18:36 I use compiler directives to (un-)define symbols and to include files over several levels. Code: [Select]case i of         00:  {$define Phase00}  {$i I_en}  {$undef Phase00};         01:  {$define Phase01}  {$i I_en}  {$undef Phase01};         02:  {$define Phase02}  {$i I_en}  {$undef Phase02};         03:  {$define Phase03}  {$i I_en}  {$undef Phase03};         04:  {$define Phase04}  {$i I_en}  {$undef Phase04};         05:  {$define Phase05}  {$i I_en}  {$undef Phase05};         06:  {$define Phase06}  {$i I_en}  {$undef Phase06};         07:  {$define Phase07}  {$i I_en}  {$undef Phase07};         08:  {$define Phase08}  {$i I_en}  {$undef Phase08};         09:  {$define Phase09}  {$i I_en}  {$undef Phase09};         10:  {$define Phase10}  {$i I_en}  {$undef Phase10};         11:  {$define Phase11}  {$i I_en}  {$undef Phase11};         12:  {$define Phase12}  {$i I_en}  {$undef Phase12};         13:  {$define Phase13}  {$i I_en}  {$undef Phase13};         14:  {$define Phase14}  {$i I_en}  {$undef Phase14};         15:  {$define Phase15}  {$i I_en}  {$undef Phase15};         16:  {$define Phase16}  {$i I_en}  {$undef Phase16};         17:  {$define Phase17}  {$i I_en}  {$undef Phase17}; end;Code: [Select]case o of         {$define n}  {$i I_o}  {$undef n}         {$define e}  {$i I_o}  {$undef e} endCode: [Select]{$ifdef n} $00 {$else} $0100 {$endif}:    begin  {$define i_BRK}  {$i am_Stack_BRK_COP}                 {$undef i_BRK}  end; {$ifdef n} $01 {$else} $0101 {$endif}:    begin  {$define i_ORA}  {$i am_DirectIndexedIndirect}         {$undef i_ORA}  end; {$ifdef n} $02 {$else} $0102 {$endif}:    begin  {$define i_COP}  {$i am_Stack_BRK_COP}                 {$undef i_COP}  end; {...} {$ifdef n} $FF {$else} $01FF {$endif}:    begin  {$define i_SBC}  {$i am_AbsoluteLongX}                 {$undef i_SBC}  end; Naturally this makes compilation slow, but it also makes editing slow. In my test program (see below), loading the project, editing the unit, and editing the include files (if they're loaded into the IDE) incurs pause(s) of several seconds. My guess is that it's for syntax highlighting. Is that unavoidable (and I have to edit these files with another editor), or can I speed up working with these files? ---------------------------------------------------------------------- sash · 2018-11-29 19:13 But why do you need to include the same file 18 times? ---------------------------------------------------------------------- lucamar · 2018-11-29 19:43 Quote from: sash on November 29, 2018, 08:13:11 pmBut why do you need to include the same file 18 times? One may assume that what gets really included depends on the previous $DEFINE. But yes, it looks weird: I had to stop and think ETA  Quote from: creaothceann on November 29, 2018, 07:36:43 pmIs that unavoidable (and I have to edit these files with another editor), or can I speed up working with these files? That ... let's call it "style" ... of programming will inevitably slow down any "smart" editor like the Lazarus one. Notepad style editors won't care because they don't have to parse your source to try to help you. ---------------------------------------------------------------------- wp · 2018-11-29 19:46 A very strange programming style for Pascal. I get the impression you are confusing Pascal with an interpreted script. Why don't you use variables to control program flow? ---------------------------------------------------------------------- creaothceann · 2018-11-29 20:10 Quote from: sash on November 29, 2018, 08:13:11 pmBut why do you need to include the same file 18 times?The included files contain different sections that are activated by the PhaseXX defines. This is for an emulator (a bytecode interpreter). There are 256 opcodes, and each opcode can be 2..9 cycles. 1 cycle is 2 phases. Also, the CPU can be in 2 modes (indicated by n and e). So I have to differentiate between up to 18*2*256 = 9216 phase handlers; each of them is actually only a few lines of code (and many are empty). I have most of that code already written, now I have to put it together somehow though and turn it into a working program. Quote from: wp on November 29, 2018, 08:46:05 pmA very strange programming style for Pascal.Yeah, it's a bit closer to C etc. than to Pascal. Including compilation times. Quote from: wp on November 29, 2018, 08:46:05 pmI get the impression you are confusing Pascal with an interpreted script.I'm deliberately using $ifdefs here, this is supposed to be resolved at compile time. (And no, inline doesn't always work - already tried that.) Quote from: wp on November 29, 2018, 08:46:05 pmWhy don't you use variables to control program flow?Because maintaining that state adds to the system requirements and we don't have 10GHz CPUs yet. But this is actually not solely for running the program: this is for storing the code that is easily accessible. I don't know yet if my approach at the main loop is going to be the fastest one, so I put the code into include files to be able to try several designs without huge rewrites. Now it turns out that each added or removed character slows down the editor to a crawl. Sigh. ---------------------------------------------------------------------- creaothceann · 2018-11-29 20:12 Quote from: lucamar on November 29, 2018, 08:43:57 pmThat style of programming will inevitably slow down any "smart" editor like the Lazarus one. Notepad style editors won't care because they don't have to parse your source to try to help you.Yeah, that's what I've been thinking. But maybe there's a checkbox somewhere that disables it? I don't want to change and recompile Lazarus just for that. ---------------------------------------------------------------------- lucamar · 2018-11-29 20:23 You'd have to disable well over half of what makes the editor distinctive: most of CodeTools, interface<->implementation jumps, etc. I may be wrong (never found myself in a situation like this) but maybe it will go quicker if you have the include files opened in an editor window.( ??? ) ---------------------------------------------------------------------- creaothceann · 2018-11-29 20:43 Quote from: lucamar on November 29, 2018, 09:23:09 pmI may be wrong (never found myself in a situation like this) but maybe it will go quicker if you have the include files opened in an editor window. (???)An external editor? Because right now I'm editing them directly in Lazarus. ---------------------------------------------------------------------- wp · 2018-11-29 20:44 Quote from: creaothceann on November 29, 2018, 09:10:37 pmI'm deliberately using $ifdefs here, this is supposed to be resolved at compile time.But how are you going to resolve this at compile time, i is only known at runtime. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i case i of      00:  {$define Phase00}  {$i I_en}  {$undef Phase00};        01:  {$define Phase01}  {$i I_en}  {$undef Phase01};   ... ---------------------------------------------------------------------- sash · 2018-11-29 20:48 Quote from: creaothceann on November 29, 2018, 09:10:37 pmBecause maintaining that state adds to the system requirements and we don't have 10GHz CPUs yet. So your "optimization" is for the sake of elimination of 2-3 nested cases/ifs ? But wait, you still use cases/ifs, which are evaluated at runtime. ---------------------------------------------------------------------- creaothceann · 2018-11-29 21:06 Quote from: wp on November 29, 2018, 09:44:46 pmQuote from: creaothceann on November 29, 2018, 09:10:37 pmI'm deliberately using $ifdefs here, this is supposed to be resolved at compile time.But how are you going to resolve this at compile time, i is only known at runtime. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i case i of      00:  {$define Phase00}  {$i I_en}  {$undef Phase00};        01:  {$define Phase01}  {$i I_en}  {$undef Phase01};   ... I mean that these are compiler directives, not script directives. Quote from: sash on November 29, 2018, 09:48:11 pmSo your "optimization" is for the sake of elimination of 2-3 nested cases/ifs ? But wait, you still use cases/ifs, which are evaluated at runtime. Yes, currently my approach is to use case-of on the current phase (will be in the local variable i), then use case-of on the current mode and opcode (stored in a Word as (Mode SHL 8) OR Opcode). So there will be one case-of selecting between 18 code paths, and 18 nested case-ofs each selecting between 2*256 code paths. So while having many case-ofs isn't ideal, I think this is going to work the best with the CPU branch predictors. But I don't know yet because I can't run any benchmarks yet. ---------------------------------------------------------------------- Martin_fr · 2018-11-29 21:38 Quote from: creaothceann on November 29, 2018, 07:36:43 pmMy guess is that it's for syntax highlighting. The main highlighting works per file, and single pass. It will not care about includes. But codetools do. That includes the "lowlighting" of inactive ifdef blocks. That will be a lot in your case. But more important, it is used for codecompletion and all that. Not sure how well that works in your project. What you could try.... Move the include files to a different folder. Add this path in a place where codetools does not find it (not sure, maybe fpc.cfg? or a wrapper script around fpc). Then codetools should not be able to find the files. Even if open, they would be seen as independent files. Dont know if it will help... ---------------------------------------------------------------------- sash · 2018-11-29 22:10 QuoteSo while having many case-ofs isn't ideal, I think this is going to work the best with ... My point that you are gaining nothing: you still need to traverse and evaluate your control-flow nodes (case-of) the same number of times as you would without your "include hell". The only benefit you could probably have (what depends on amount of repetitive fragments) - a more compact source, what is irrelevant to runtime optimizations, because binary code is still generated for each {$include}, but adds drawbacks as poorly maintainable code (IMO) and IDE performance issues. ---------------------------------------------------------------------- lucamar · 2018-11-29 22:32 With so many code paths your best bet, IMHO, is to use a jump table-- or two--built on startup or at compile-time. That will probably be quicker than using case statements. ETA: Sorry, didn't see this: Quote from: creaothceann on November 29, 2018, 09:43:24 pmQuote from: lucamar on November 29, 2018, 09:23:09 pmI may be wrong (never found myself in a situation like this) but maybe it will go quicker if you have the include files opened in an editor window. (???)An external editor? Because right now I'm editing them directly in Lazarus. No, I meant opened in the Lazarus editor; but if you're alredy doing that ... ---------------------------------------------------------------------- creaothceann · 2018-11-29 22:42 Quote from: Martin_fr on November 29, 2018, 10:38:54 pmWhat you could try.... Move the include files to a different folder. Add this path in a place where codetools does not find it (not sure, maybe fpc.cfg? or a wrapper script around fpc). Then codetools should not be able to find the files. Even if open, they would be seen as independent files. Don't know if it will help...Alright. I guess I'll just remove the directory from the "include files" path in the project settings, and add it to a batch file for compiling the project (via lazbuild?). Or something like that. This should allow me to still use the IDE for basic editing. Quote from: sash on November 29, 2018, 11:10:29 pmyou still need to traverse and evaluate your control-flow nodes (case-of) the same number of times as you would without your "include hell".Yes, but that's not the sole decisive factor. Not all CPU operations are equal. An ADD instruction that reads a random memory address is slower than dozens of correctly predicted if/case-of jumps. In my case, the first case-of follows a simple pattern: most phases are executed in order. Phase 6 follows 5 follows 4 follows 3 etc. ... The second case-of would benefit from modern branch prediction that exploits correlations between the opcodes of the emulated program (e.g. instruction ADC is often preceded by instruction CLC). Quote from: sash on November 29, 2018, 11:10:29 pmThe only benefit you could probably have (what depends on amount of repetitive fragments) - a more compact source, what is irrelevant to runtime optimizations, because binary code is still generated for each {$include}, but adds drawbacks as poorly maintainable code (IMO) and IDE performance issues.A more compact source is very important to me if the alternative is writing thousands of lines by hand. Maybe I'll find a more maintainable way, too. Quote from: lucamar on November 29, 2018, 11:32:25 pmWith so many code paths your best bet, IMHO, is to use a jump table--or two--built on startup or at compile-time. That will probably be quicker than using case statements.Afaik case-of is already compiled into a jump table if it has more than a dozen cases or so. I did dabble with computed GOTOs, but they need you to list every label and label address, and CPUs >= Haswell shouldn't need this these days anyway. Hopefully. ---------------------------------------------------------------------- Martin_fr · 2018-11-29 23:05 Quote from: creaothceann on November 29, 2018, 11:42:17 pmA more compact source is very important to me if the alternative is writing thousands of lines by hand. Maybe I'll find a more maintainable way, too. Include files are not the only way. 1) You can write and run a preprocessor (project options > Compiler command > Execute before) 2) You can use generics 3) (I never said that, I deny it if anyone says I did...) {$macro on}    not sure that will do much good in your case 4) For repeated editing tasks, look into editor macros (well, you still get the code duplication..., just some tool to maintain it)    http://wiki.lazarus.freepascal.org/Editor_Macros_PascalScript ---------------------------------------------------------------------- Martin_fr · 2018-11-29 23:26 Quote from: creaothceann on November 29, 2018, 09:10:37 pmQuote from: wp on November 29, 2018, 08:46:05 pmI get the impression you are confusing Pascal with an interpreted script.I'm deliberately using $ifdefs here, this is supposed to be resolved at compile time. (And no, inline doesn't always work - already tried that.) If you are "fighting" a speed problem, there may be other ways. If you are not sure where time goes, get a linux environment and use callgrind (part of valgrind). Dont just try to squeeze micro optimizations (like inline, or pointers). See why some code is called more often, and if it can be changed. You may be able to build very optimized "case like" jump tables at runtime (once during initialization). But that is all theoretical, I dont know the codebase.... Also review how you organize and store data. Every cache miss costs time. Sometimes compacter data (even if it requires an extra statement) can be faster. But that is hard to predict again. - If you run with threads: there are currently speed improvements in the works, in fpc trunk. - If you work with managed types (strings/dyn array), check "implicit exceptions" and turning them off. ---------------------------------------------------------------------- engkin · 2018-11-30 03:04 I did not read the whole thread, so my apology if I repeated someone's answer. Use macros: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit U_Test;{$Macro ON}{$define macro_I_en:={$i I_en}}...case i of        00:  {$define Phase00}  macro_I_en  {$undef Phase00};        01:  {$define Phase01}  macro_I_en  {$undef Phase01};        02:  {$define Phase02}  macro_I_en  {$undef Phase02};... According to FPC your sample produces more than 9000 lines. ---------------------------------------------------------------------- creaothceann · 2018-11-30 09:27 @engkin: see above. Quote from: Martin_fr on November 30, 2018, 12:05:53 am3) (I never said that, I deny it if anyone says I did...) {$macro on}    not sure that will do much good in your caseAlready tried that: ("$define Phase := 1" etc.), seemed to be slightly slower than just defining and undefining symbols. But I might have to use it later anyway. Quote from: Martin_fr on November 30, 2018, 12:26:02 amIf you are not sure where time goes, get a linux environment and use callgrind (part of valgrind). Dont just try to squeeze micro optimizations (like inline, or pointers). See why some code is called more often, and if it can be changed.Yeah, probably going to set up a Linux installation at some point. Quote from: Martin_fr on November 30, 2018, 12:26:02 amAlso review how you organize and store data. Every cache miss costs time. Sometimes compacter data (even if it requires an extra statement) can be faster. But that is hard to predict again.The virtual CPU's state is packed, fits into a cache line (64 bytes). The hot codepaths hopefully fit into the instruction cache, it might be faster to use non-inlined functions instead of unrolling everything. If I'm lucky, the whole thing with all the loaded data will fit into L3 caches. ---------------------------------------------------------------------- engkin · 2018-11-30 18:18 Quote from: creaothceann on November 30, 2018, 10:27:50 amQuote from: Martin_fr on November 30, 2018, 12:05:53 am3) (I never said that, I deny it if anyone says I did...) {$macro on}    not sure that will do much good in your caseAlready tried that: ("$define Phase := 1" etc.), seemed to be slightly slower than just defining and undefining symbols. But I might have to use it later anyway. Did you try to define just {$i I_en} as a macro, as in my post? That will prevent the IDE from going through the tree of included files and bring the editing speed back to normal. ---------------------------------------------------------------------- creaothceann · 2018-11-30 18:44 Quote from: engkin on November 30, 2018, 07:18:17 pmDid you try to define just {$i I_en} as a macro, as in my post?Oh wow, that worked. Thanks! (I'm surprised that the first "}" doesn't end the macro definition and that the IDE doesn't use it.)