[HN Gopher] Pile-T5
       ___________________________________________________________________
        
       Pile-T5
        
       Author : tosh
       Score  : 53 points
       Date   : 2024-04-15 15:37 UTC (7 hours ago)
        
 (HTM) web link (blog.eleuther.ai)
 (TXT) w3m dump (blog.eleuther.ai)
        
       | gwern wrote:
       | Now that's a blast from the past - T5! I always thought it was
       | underused, but it's also from so long ago now, and even Google
       | has moved past it to UL2 etc AFAIK. What's the use-case here for
       | reproducing it? Aren't there already many good code models?
        
         | emadm wrote:
         | I figured it would be interesting to see scale and for the
         | image models
        
         | brizii wrote:
         | my group is currently working on a T5 model (and tokenizer) for
         | html, as there are very few (if any) tokenizers that work well
         | with HTML!
         | 
         | You can try using GPT4's tokenizer on your own HTML inputs
         | below [1] ... there's definitely room for improvement!
         | 
         | [1] https://tiktokenizer.vercel.app
        
         | dartos wrote:
         | Blast from the past? Like 2 years ago?
        
           | michaelt wrote:
           | Gather round, children, and let Grandpa tell you a tale from
           | way back in the day, when we thought 24GB was a lot of
           | VRAM...
        
             | bevekspldnw wrote:
             | My nearly new 48GB A6000 is basically an ancient artifact
             | relative to the exploding size of MoE requirements.
        
           | p1esk wrote:
           | T5 was released in 2019.
        
             | stavros wrote:
             | Isn't that two years ago?
        
               | littlestymaar wrote:
               | Two years already? This locked-down year of 2020 really
               | messes up with me perception of time.
        
         | euclaise wrote:
         | A lot of embedding models are built on top of T5's encoder,
         | this offers a new option
         | 
         | The modularity of the enc-dec approach is useful - you can
         | insert additional models in between (e.g. A diffusion model),
         | you can use different encoders for different modalities, etc
        
       | tosh wrote:
       | a new T5
       | 
       | * llama tokenizer
       | 
       | * the pile dataset
       | 
       | * trained on 2T tokens (2 times of og T5)
       | 
       | * better, & especially better at coding
        
       | bevekspldnw wrote:
       | I was using it for translation with somewhat wobbly results.
       | Would love to see a translation fine tune.
        
       | AIorNot wrote:
       | Can someone provide some context? please explain what this is?
       | thanks
        
       | adt wrote:
       | https://lifearchitect.ai/models-table/
        
       ___________________________________________________________________
       (page generated 2024-04-15 23:02 UTC)