https://openreview.net/forum?id=2dnO3LLiJ1

Toggle navigationOpenReview.net
[                    ]
  * Login

Open Peer Review. Open Publishing. Open Access. Open Discussion. Open
Recommendations. Open Directory. Open API. Open Source.
x

Vision Transformers Need Registers

Download PDF

Timothee Darcet, Maxime Oquab, Julien Mairal, Piotr Bojanowski

Published: 16 Jan 2024, Last Modified: 12 Apr 2024ICLR 2024 oral
EveryoneRevisionsBibTeX
Code Of Ethics: I acknowledge that I and all co-authors of this work
have read and commit to adhering to the ICLR Code of Ethics.
Keywords: representation, vision, transformer, register, SSL, CLIP,
attention, attention map, interpretability, DINO, DINOv2
Submission Guidelines: I certify that this submission complies with
the submission instructions as described on https://iclr.cc/
Conferences/2024/AuthorGuide.
TL;DR: We find artifacts in ViT features. We add new tokens
("registers") that fix this issue.
Abstract: Transformers have recently emerged as a powerful tool for
learning visual representations. In this paper, we identify and
characterize artifacts in feature maps of both supervised and
self-supervised ViT networks. The artifacts correspond to high-norm
tokens appearing during inference primarily in low-informative
background areas of images, that are repurposed for internal
computations. We propose a simple yet effective solution based on
providing additional tokens to the input sequence of the Vision
Transformer to fill that role. We show that this solution fixes that
problem entirely for both supervised and self-supervised models, sets
a new state of the art for self-supervised visual models on dense
visual prediction tasks, enables object discovery methods with larger
models, and most importantly leads to smoother feature maps and
attention maps for downstream visual processing.
Anonymous Url: I certify that there is no URL (e.g., github page)
that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no
acknowledgement section in this submission for double blind review.
Primary Area: unsupervised, self-supervised, semi-supervised, and
supervised representation learning
Submission Number: 3647
Loading

  * About OpenReview
  * Hosting a Venue
  * All Venues

  * Contact
  * Feedback
  * Sponsors
  * Join the Team

  * Frequently Asked Questions
  * Terms of Use
  * Privacy Policy

  * About OpenReview
  * Hosting a Venue
  * All Venues
  * Sponsors
  * Join the Team

  * Frequently Asked Questions
  * Contact
  * Feedback
  * Terms of Use
  * Privacy Policy

OpenReview is a long-term project to advance science through improved
peer review, with legal nonprofit status through Code for Science &
Society. We gratefully acknowledge the support of the OpenReview
Sponsors. (c) 2024 OpenReview

x

Send Feedback

Enter your feedback below and we'll get back to you as soon as
possible. To submit a bug report or feature request, you can use the
official OpenReview GitHub repository:
Report an issue

[                    ]
Select a topic or type what you need help with
[                    ]
[                    ]
[                    ]
[                    ]
[                    ]
[                    ]
CancelSend
x

BibTeX Record

Click anywhere on the box above to highlight complete record
Done