[HN Gopher] Contrastive Representation Learning
       ___________________________________________________________________
        
       Contrastive Representation Learning
        
       Author : gk1
       Score  : 78 points
       Date   : 2022-08-19 14:09 UTC (8 hours ago)
        
 (HTM) web link (lilianweng.github.io)
 (TXT) w3m dump (lilianweng.github.io)
        
       | mountainriver wrote:
       | another incredible article by Lilian Weng, never ceases to
       | impress and enlighten
        
       | hadrianpaulo wrote:
       | Are there techniques for contrastive learning that's also
       | applicable to tabular data?
        
       | cs702 wrote:
       | Nice job! This is a fantastic resource for anyone interested in
       | using contrastive methods for inducing AI/ML models to learn to
       | embed data in a space such that samples considered similar stay
       | close to each other (e.g., as measured by cosine or Euclidean
       | distance) while dissimilar ones stay far apart. Self-supervised
       | contrastive methods, in particular, can be remarkably useful when
       | none of the samples in your data are labeled and you want your
       | model to discover structure.
        
       | fxtentacle wrote:
       | I'm surprised that this doesn't mention cross-entropy, the
       | contrastive loss function used by Facebook's wav2vec2 XLS-R
       | pretraining paper and by OpenAI's CLIP.
        
         | canjobear wrote:
         | Contrastive losses arise from using methods like NCE (mentioned
         | in the post) to approximate cross entropy loss when the
         | partition function is intractable.
        
       | roknovosel wrote:
       | Great read, thanks for sharing. Would love to see the natural
       | language + code mixed in there :)
       | 
       | I've been interested in contrastive learning for a while, mainly
       | as a means to train semantic code search models. OpenAI released
       | a great paper on this topic called Text and Code Embeddings by
       | Contrastive Pre-Training[1] that outlines the approach. I've used
       | it as a base to build https://codesearch.ai [2] with pretty good
       | results.
       | 
       | [1] https://arxiv.org/pdf/2201.10005.pdf [2]
       | https://sourcegraph.com/notebooks/Tm90ZWJvb2s6MTU1OQ==
        
       ___________________________________________________________________
       (page generated 2022-08-19 23:01 UTC)