[HN Gopher] Data2vec 2.0: Highly efficient self-supervised learn...
       ___________________________________________________________________
        
       Data2vec 2.0: Highly efficient self-supervised learning for vision,
       speech, text
        
       Author : ashvardanian
       Score  : 70 points
       Date   : 2022-12-13 18:03 UTC (4 hours ago)
        
 (HTM) web link (ai.facebook.com)
 (TXT) w3m dump (ai.facebook.com)
        
       | bilsbie wrote:
       | What would be some use cases for this?
       | 
       | How does it work?
        
         | nl wrote:
         | > What would be some use cases for this?
         | 
         | It can do image recognition, speech recognition or text
         | classification (eg, is this positive or negative sentiment)
         | using the same model architecture (trained on different data
         | each time)
         | 
         | It's very competitive in each of those fields with the existing
         | state of the art. This is interesting because usually the
         | models for each of those fields are different.
         | 
         | > How does it work?
         | 
         | It's trained by passing sequences of data (pixels sequentially,
         | words in order, speech as a wav file) with part of each
         | sequence masked out. The model has to learn to correctly guess
         | what is in that masked area.
         | 
         | The innovation is that it instead of predicting the masked area
         | directly, it tries to predict what the representation of that
         | masked area is in the Neural Network itself. That's extremely
         | unusual (I don't think I've seen that before) and I'll need to
         | study it more to completely understand why that explains the
         | better performance.
         | 
         | It's also very fast to train.
        
         | notsoprocoder wrote:
         | I've only scanned the blog but have read OG Data2Vec paper.
         | 
         | Data2Vec presents an architecture that performs well across the
         | main benchmarks for vision, speech and text. The architecture
         | is a variation on the transformer network with slight tweaks
         | for each learning modality. Data2Vec 2 seems to be more a more
         | efficient variant.
         | 
         | In terms of applications, data2vec gives a single reliable
         | architecture for each approach. Whereas before you may have
         | used a CNN for vision and a transformer for text etc.
         | 
         | Additionally, this research is building towards multi-modal
         | learning where an architecture could be trained on images, text
         | and speech to learn about a topic. (But to my knowledge there
         | isn't anything ground breaking in this space yet).
        
           | bilsbie wrote:
           | Thanks! Nice write up
           | 
           | I think it would help if I knew of a concrete task I could
           | use this for.
        
           | m00x wrote:
           | Not only does it perform better as a generalist, it also
           | groups embeddings for the the multiple modalities of the same
           | concept close together (keeps bird sounds close to bird image
           | in its internal representation, etc). This way it can learn
           | in different dimensions, just like humans do.
           | 
           | You could ask it to sing like a bird in text, and it could
           | respond in image + sound, or just sound. This allows for
           | better general understanding of things. Deepmind did the same
           | with Gato:
           | https://www.deepmind.com/publications/a-generalist-agent
        
         | itake wrote:
         | I wish they would provide an "Explain like I am 5" to everyone
         | that doesn't know ML.
        
       ___________________________________________________________________
       (page generated 2022-12-13 23:01 UTC)