https://arxiv.org/abs/2404.04125

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2404.04125
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computer Vision and Pattern Recognition

arXiv:2404.04125 (cs)
[Submitted on 4 Apr 2024 (v1), last revised 8 Apr 2024 (this version,
v2)]

Title:No "Zero-Shot" Without Exponential Data: Pretraining Concept
Frequency Determines Multimodal Model Performance

Authors:Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma,
Philip H.S. Torr, Adel Bibi, Samuel Albanie, Matthias Bethge
View a PDF of the paper titled No "Zero-Shot" Without Exponential
Data: Pretraining Concept Frequency Determines Multimodal Model
Performance, by Vishaal Udandarao and 7 other authors
View PDF HTML (experimental)

    Abstract:Web-crawled pretraining datasets underlie the impressive
    "zero-shot" evaluation performance of multimodal models, such as
    CLIP for classification/retrieval and Stable-Diffusion for image
    generation. However, it is unclear how meaningful the notion of
    "zero-shot" generalization is for such multimodal models, as it
    is not known to what extent their pretraining datasets encompass
    the downstream concepts targeted for during "zero-shot"
    evaluation. In this work, we ask: How is the performance of
    multimodal models on downstream concepts influenced by the
    frequency of these concepts in their pretraining datasets? We
    comprehensively investigate this question across 34 models and
    five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M,
    LAION-400M, LAION-Aesthetics), generating over 300GB of data
    artifacts. We consistently find that, far from exhibiting
    "zero-shot" generalization, multimodal models require
    exponentially more data to achieve linear improvements in
    downstream "zero-shot" performance, following a sample
    inefficient log-linear scaling trend. This trend persists even
    when controlling for sample-level similarity between pretraining
    and downstream datasets, and testing on purely synthetic data
    distributions. Furthermore, upon benchmarking models on
    long-tailed data sampled based on our analysis, we demonstrate
    that multimodal models across the board perform poorly. We
    contribute this long-tail test set as the "Let it Wag!" benchmark
    to further research in this direction. Taken together, our study
    reveals an exponential need for training data which implies that
    the key to "zero-shot" generalization capabilities under
    large-scale training paradigms remains to be found.

Comments: Extended version of the short paper accepted at DPFM,
          ICLR'24
Subjects: Computer Vision and Pattern Recognition (cs.CV);
          Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:  arXiv:2404.04125 [cs.CV]
          (or arXiv:2404.04125v2 [cs.CV] for this version)
          https://doi.org/10.48550/arXiv.2404.04125
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Vishaal Udandarao [view email]
[v1] Thu, 4 Apr 2024 17:58:02 UTC (37,862 KB)
[v2] Mon, 8 Apr 2024 21:14:43 UTC (37,863 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled No "Zero-Shot" Without Exponential
    Data: Pretraining Concept Frequency Determines Multimodal Model
    Performance, by Vishaal Udandarao and 7 other authors
  * View PDF
  * HTML (experimental)
  * TeX Source
  * Other Formats

license icon view license
Current browse context:
cs.CV
< prev   |   next >
new | recent | 2404
Change to browse by:
cs
cs.CL
cs.LG

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] GotitPub Toggle
Gotit.pub (What is GotitPub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack