[HN Gopher] Electra: Pre-Training Text Encoders as Discriminator...
       ___________________________________________________________________
        
       Electra: Pre-Training Text Encoders as Discriminators Rather Than
       Generators (2020)
        
       Author : luu
       Score  : 59 points
       Date   : 2024-07-16 00:18 UTC (2 days ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | cs702 wrote:
       | Good work by well-known reputable authors.
       | 
       | The gains in training efficiency and compute cost versus widely
       | used text-encoding models like RoBERTa and XLNet are significant.
       | 
       | Thank you for sharing this on HN!
        
       | adw wrote:
       | (2020)
        
       | trhway wrote:
       | Reminds somewhat parallel from the classic expert systems - human
       | experts shine at discrimination, and that is one of the most
       | efficient methods of knowledge eliciting from them.
        
       | visarga wrote:
       | LOL, I was reading the abstract and remembering there used to be
       | a paper like that. Then I look at the title and see it was from
       | 2020. For a moment I thought someone plagiarised the original
       | paper.
       | 
       | Unfortunately BERT models are dead. Even the cross between BERT
       | and GPT - the T5 architecture (encode-decoder) is rarely used.
       | 
       | The issue with BERT is that you need to modify the network to
       | adapt it to any task by creating a prediction head, while decoder
       | models (GPT style) do every task with tokens and never need to
       | modify the network. Their advantage is that they have a single
       | format for everything. BERT's advantage is the bidirectional
       | attention, but apparently large size decoders don't have an issue
       | with unidirectionality.
        
         | jszymborski wrote:
         | BERT and T5 models are slowly consuming the computational
         | biology field, so they certainly aren't dead to all.
        
         | johnsutor wrote:
         | BERT isn't dead for smaller tasks (think NER, Sentiment
         | Analysis) where low latency is needed.
        
         | nickpsecurity wrote:
         | There's also articles for pre-training BERT models on hardware
         | resources a small lab could afford. Those are still useful,
         | too, even if not highly competitive. So, they could still have
         | value for low-cost, small, model development.
        
         | hiddencost wrote:
         | BERT is alive and well for most commercial uses of NLP.
         | 
         | If you're running 100k QPS through the model with a budget of
         | 0.1 cents per query, you aren't going to be using a GPT model
         | for classification.
        
       ___________________________________________________________________
       (page generated 2024-07-18 23:11 UTC)