[HN Gopher] Towards A Token-Free Future In NLP (2022)
       ___________________________________________________________________
        
       Towards A Token-Free Future In NLP (2022)
        
       Author : optimalsolver
       Score  : 5 points
       Date   : 2023-01-30 19:14 UTC (3 hours ago)
        
 (HTM) web link (peltarion.com)
 (TXT) w3m dump (peltarion.com)
        
       | f_devd wrote:
       | I've actually been working on a Unicode only code prediction LM,
       | and it already works pretty well the main issues with Unicode
       | models as also found in the article is the large sequence length
       | required compared a sentencePiece or BPE tokenizer. The current
       | direction to resolve this seems to be to use structured state
       | spaces (S4/DSS) models which scale linearly along sequence length
       | compared to O(n^2) for transformers. I haven't read the
       | charformer paper yet but if it does a dynamic pooling of the
       | tokens it could be a promising step to have this same
       | functionality in transformer models.
        
       | deepsquirrelnet wrote:
       | I work in an industry where most of the text is not natural --
       | super wide vocabulary, lots of symbols and abbreviations. I built
       | some utf-8 tokenized models that do very well in comparison to
       | sentence/word piece tokenizers.
       | 
       | Unless your corpus fits the tokenizer vocabulary very well, I
       | think there are many cases where "token free" models can be
       | advantageous. You only have to determine how to deal with the
       | expansion of the input dimension and how it might blow up your
       | attention costs.
        
       ___________________________________________________________________
       (page generated 2023-01-30 23:02 UTC)