[HN Gopher] Rebuilding Babel: The Tokenizer
       ___________________________________________________________________
        
       Rebuilding Babel: The Tokenizer
        
       Author : pgayed
       Score  : 22 points
       Date   : 2022-12-07 07:42 UTC (1 days ago)
        
 (HTM) web link (www.nan.fyi)
 (TXT) w3m dump (www.nan.fyi)
        
       | rbtprograms wrote:
       | This is way more in depth of an evaluator than I've ever written,
       | but it's a fun project. I'm a largely self taught dev and build
       | an evaluator was an illuminating project for me. Mine was just a
       | math parser but this is a cool project!
        
       | dilap wrote:
       | FWIW, a tokenizer is not really _necessary_. It (arguably) makes
       | things cleaner  / easier, but you can also just write a parser
       | that operates directly on a byte stream, if you feel so inclined,
       | and it turns out fine.
        
         | kccqzy wrote:
         | Yes I agree. The article actually says the purpose is
         | 
         | > To separate the logic that handles a language's microsyntax
         | from the logic that handles regular syntax.
         | 
         | But everything is just syntax. This distinction between
         | microsyntax and regular syntax is really just for convenience;
         | and sometimes it's not that much convenience.
        
       ___________________________________________________________________
       (page generated 2022-12-08 23:01 UTC)