[HN Gopher] Transformers for software engineers
       ___________________________________________________________________
        
       Transformers for software engineers
        
       Author : datkinson
       Score  : 57 points
       Date   : 2022-04-01 21:43 UTC (1 days ago)
        
 (HTM) web link (blog.nelhage.com)
 (TXT) w3m dump (blog.nelhage.com)
        
       | blueflow wrote:
       | I hoped this would be an explanation of electricity, magnetism,
       | the Lorentz force and induction.
        
       | sthatipamala wrote:
       | Thanks for this writeup! Software engineers should definitely
       | learn about Transformers. They are a significant step-change in
       | the advancement of deep learning. They've proven useful in
       | multiple domains outside of natural language, including computer
       | vision and audio.
       | 
       | This opens up the world of multi-modal models which represent
       | concepts by considering multiple types of input. The most popular
       | one has been OpenAI's CLIP [0]
       | 
       | If anyone's interested, I run a podcast and did an episode on
       | Transformer models and their implications [1]. Check it out!
       | 
       | [0] https://openai.com/blog/clip/ [1]
       | https://www.youtube.com/watch?v=Kb0II5DuDE0
        
         | dontreact wrote:
         | Just a fun technical note:
         | 
         | CLIP would be possible with any language and vision encoder.
         | 
         | It turns out that transformers were the most efficient encoder
         | choices available at the time but most likely the approach
         | would have given interesting results with a resnet and some
         | sort of convolutional language encoder. In fact the paper has a
         | resnet as one model for the vision side.
        
       | chaosprint wrote:
       | Very good content! It's interesting that you use Rust here.
        
         | f311a wrote:
         | I think it's better to use other languages for pseudo code.
         | Rust is not the easiest language to read and understand for
         | people who never used it.
        
         | nelhage wrote:
         | (author here)
         | 
         | I actually did a v0 writeup in Go, but I wanted a language that
         | had a bit more powerful type system and more support for a
         | fluent/functional style in some of the expressions. I was
         | optimizing for what I know and felt was highly expressive; I've
         | since gotten a bunch of feedback from people who are interested
         | in the content but not comfortable reading Rust, so perhaps it
         | wasn't the best choice.
        
       ___________________________________________________________________
       (page generated 2022-04-02 23:00 UTC)