[HN Gopher] A Faster Lexer in Go
       ___________________________________________________________________
        
       A Faster Lexer in Go
        
       Author : sophiewang
       Score  : 48 points
       Date   : 2022-05-05 06:13 UTC (1 days ago)
        
 (HTM) web link (eli.thegreenplace.net)
 (TXT) w3m dump (eli.thegreenplace.net)
        
       | mhh__ wrote:
       | Free speedups are always appreciated but when benchmarking some
       | not-optimized-at-all lexers, they were still so fast that I
       | couldn't really justify optimizing them.
       | 
       | Unless you're parsing genuinely enormous amounts of code (or
       | latency bound) I find that most compilers are far too slow in
       | other areas to justify optimizing the lexer versus fixing
       | architectural issues in (say) semantic analysis.
       | 
       | A lot of parts of compilers are non-local, and involve lots of
       | pointer chasing, which means avoiding repeated work is very
       | important. In the D compiler there a bunch of checks where we
       | stop you shooting yourself in the foot, I think generating all of
       | these tree-walkers from a specification (so you can walk it all
       | in one go) would probably be a decent speedup.
        
         | woodruffw wrote:
         | In this case, the speedup would probably be during LLVM's build
         | itself: that's when TableGen inputs are parsed as part of
         | codegen for big blobs of C++. Improvements to lexing speed here
         | _could_ actually produce some build time improvements, since
         | TableGen 's outputs tend to have a high dependent degree in the
         | dependency graph.
         | 
         | (That being said, I have no idea if LLVM would merge a TableGen
         | lexer written in Go.)
        
           | mhh__ wrote:
           | The question prior to any of that is how many lines of
           | tablegen LLVM has
           | 
           | According to a cloc definition I just did:
           | LLVM Table Gen            files: 1041                blank:
           | 75684                comment: 69272                code:
           | 480047
        
       ___________________________________________________________________
       (page generated 2022-05-06 23:00 UTC)