https://huggingface.co/papers/2308.01544

Hugging Face's logo Hugging Face
[                    ]

  * Models
  * Datasets
  * Spaces
  * Docs
  * Solutions
  * Pricing
  * 
  * -----------------------------------------------------------------
  * Log In
  * Sign Up

Papers
arxiv:2308.01544

Multimodal Neurons in Pretrained Text-Only Transformers

Published on Aug 3
* Featured in Daily Papers on Aug 4
 
[ ] Upvote
9

  * [1675335466]
  * [1670231290]
  * [5b95d2509d]
  * [56cc620425]
  * [1651037596]
  * [yfb9vWVpca]
  * [1675778487]
  * [241e6051f8]
  * +1

Authors:
[e4df93ad4a] Sarah Schwettmann ,
Neil Chowdhury ,
Antonio Torralba

Abstract

Language models demonstrate remarkable capacity to generalize
representations learned in one modality to downstream tasks in other
modalities. Can we trace this ability to individual neurons? We study
the case where a frozen text transformer is augmented with vision
using a self-supervised visual encoder and a single linear projection
learned on an image-to-text task. Outputs of the projection layer are
not immediately decodable into language describing image content;
instead, we find that translation between modalities occurs deeper
within the transformer. We introduce a procedure for identifying
"multimodal neurons" that convert visual representations into
corresponding text, and decoding the concepts they inject into the
model's residual stream. In a series of experiments, we show that
multimodal neurons operate on specific visual concepts across inputs,
and have a systematic causal effect on image captioning.

View arXiv page View PDF

Community

EditPreview
[                    ]
[                    ]
[                    ]
Upload images, audio, and videos by dragging in the text input,
pasting, or clicking here.
Tap or paste here to upload images
[                    ]
Comment

* Sign up or log in to comment

 
[ ] Upvote
9

  * [1675335466]
  * [1670231290]
  * [5b95d2509d]
  * [56cc620425]
  * [1651037596]
  * [yfb9vWVpca]
  * [1675778487]
  * [241e6051f8]
  * [c6b7cec309]
  * 

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2308.01544 in a model README.md to link it from
this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2308.01544 in a dataset README.md to link it from
this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2308.01544 in a Space README.md to link it from
this page.
Company
(c) Hugging Face
TOS Privacy About Jobs  
Website
Models Datasets Spaces Pricing Docs