[HN Gopher] Vision Transformers Need Registers
       ___________________________________________________________________
        
       Vision Transformers Need Registers
        
       Author : cscurmudgeon
       Score  : 67 points
       Date   : 2024-05-11 18:08 UTC (4 hours ago)
        
 (HTM) web link (openreview.net)
 (TXT) w3m dump (openreview.net)
        
       | superkuh wrote:
       | "Attention sinks" for vision models?
        
       | ttul wrote:
       | According to the paper, the "registers" are additional learnable
       | tokens that are appended to the input sequence of a Vision
       | Transformer model during training.
       | 
       | They are added after the patch embedding layer, with a learnable
       | value, similar to the [CLS] token and then at the end of the
       | Vision Transformer, the register tokens are discarded, and only
       | the [CLS] token and patch tokens are used as image
       | representations.
       | 
       | The register tokens provide a place for the model to store,
       | process and retrieve global information during the forward pass,
       | without repurposing patch tokens for this role.
       | 
       | Adding register tokens removes the artifacts and high-norm
       | "outlier" tokens that otherwise appear in the feature maps of
       | trained Vision Transformer models.
       | 
       | Using register tokens leads to smoother feature maps, improved
       | performance on dense prediction tasks, and enables better
       | unsupervised object discovery compared to the same models trained
       | without the additional register tokens.
       | 
       | This is a neat result. For just a 2% increase in inference cost,
       | you can significantly improve ViT model performance. Close to a
       | free lunch.
        
         | kadushka wrote:
         | Interesting. One other potential benefit is an easier
         | quantization of the activations.
        
         | macleginn wrote:
         | There was an attempt to add several CLS tokens to BERT, with
         | less spectacular results: https://arxiv.org/pdf/2210.05043
        
         | swyx wrote:
         | are there lessons here for regular (non vision) transformers?
         | sounds close to attention sinks/pause tokens?
        
         | johntb86 wrote:
         | For these tokens you first need to unembed the result of the
         | final layer, the re-embed the resulting token on the next pass.
         | Has anyone investigated passing the raw output of one pass to
         | the input of the next?
        
       | richdougherty wrote:
       | Related? "Let's Think Dot by Dot: Hidden Computation in
       | Transformer Language Models" https://arxiv.org/abs/2404.15758
       | 
       | > Chain-of-thought responses from language models improve
       | performance across most benchmarks. However, it remains unclear
       | to what extent these performance gains can be attributed to
       | human-like task decomposition or simply the greater computation
       | that additional tokens allow. We show that transformers can use
       | meaningless filler tokens (e.g., '......') in place of a chain of
       | thought to solve two hard algorithmic tasks they could not solve
       | when responding without intermediate tokens. However, we find
       | empirically that learning to use filler tokens is difficult and
       | requires specific, dense supervision to converge. We also provide
       | a theoretical characterization of the class of problems where
       | filler tokens are useful in terms of the quantifier depth of a
       | first-order formula. For problems satisfying this
       | characterization, chain-of-thought tokens need not provide
       | information about the intermediate computational steps involved
       | in multi-token computations. In summary, our results show that
       | additional tokens can provide computational benefits independent
       | of token choice. The fact that intermediate tokens can act as
       | filler tokens raises concerns about large language models
       | engaging in unauditable, hidden computations that are
       | increasingly detached from the observed chain-of-thought tokens.
       | 
       | > In this work, we demonstrate that transformers trained on the
       | next-token prediction objective can achieve improved performance
       | on certain tasks when given filler tokens, achieving perfect
       | accuracy whereas the no-filler, immediate-answer setting achieves
       | only low accuracy.
       | 
       | --
       | 
       | I wonder if we could get benefits from adding special
       | computation/register tokens to text LLMs?
       | 
       | More discussion:
       | 
       | - https://news.ycombinator.com/item?id=40182695
       | 
       | - https://www.reddit.com/r/LocalLLaMA/comments/1cf2w5a/transfo...
        
       | joaquincabezas wrote:
       | I was at ICLR and this was one of the best of this year, it was
       | also evident during the poster session. Congrats to the authors!!
        
       ___________________________________________________________________
       (page generated 2024-05-11 23:00 UTC)