Post B4QML7sk6E8vETxpyq by lethalbit@vt.social
 (DIR) More posts by lethalbit@vt.social
 (DIR) Post #B4QLwrvvkD2gopt98q by lethalbit@vt.social
       0 likes, 0 repeats
       
       So here is a question to people who have done language front-ends.When collecting token span information (e.g. start/stop byte/grapheme offsets into the file) do you /also/ keep track of the normalized line number/grapheme column?Or do you re-work that out when you need to emit them.Should both be stored together, e.g. each span contains the normalized line number and grapheme column with it as context, or should it be cached some ancillary table for lookup?Thoughts?
       
 (DIR) Post #B4QML7sk6E8vETxpyq by lethalbit@vt.social
       0 likes, 0 repeats
       
       In general I think that throwing away that information, and then re-constructing it as needed might be the way to go.As that way you can keep the span element smaller, and you can likely write specialized decently fast/vectorized machinery for converting from a byte/grapheme offset into a line/col
       
 (DIR) Post #B4QMOyfCJcJlngStY8 by JoshJers@mastodon.gamedev.place
       0 likes, 0 repeats
       
       @lethalbit last one I did, I tracked the indices of every line start in a separate table, and then the absolute text span for every token in units and code points (which should have been grapheme clusters instead)On occasion when I needed a line number it just became a binary search in the line start indices list and then column is just the count off of thatWas less fiddly than tracking all those values per token