https://arxiv.org/abs/2312.14135

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2312.14135
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computer Vision and Pattern Recognition

arXiv:2312.14135 (cs)
[Submitted on 21 Dec 2023 (v1), last revised 26 Dec 2023 (this
version, v2)]

Title:V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

Authors:Penghao Wu, Saining Xie
Download a PDF of the paper titled V*: Guided Visual Search as a Core
Mechanism in Multimodal LLMs, by Penghao Wu and 1 other authors
Download PDF HTML (experimental)

    Abstract:When we look around and perform complex tasks, how we
    see and selectively process what we see is crucial. However, the
    lack of this visual search mechanism in current multimodal LLMs
    (MLLMs) hinders their ability to focus on important visual
    details, especially when handling high-resolution and visually
    crowded images. To address this, we introduce V*, an LLM-guided
    visual search mechanism that employs the world knowledge in LLMs
    for efficient visual querying. When combined with an MLLM, this
    mechanism enhances collaborative reasoning, contextual
    understanding, and precise targeting of specific visual elements.
    This integration results in a new MLLM meta-architecture, named
    Show, sEArch, and TelL (SEAL). We further create V*Bench, a
    benchmark specifically designed to evaluate MLLMs in their
    ability to process high-resolution images and focus on visual
    details. Our study highlights the necessity of incorporating
    visual search capabilities into multimodal systems. The code is
    available this https URL.

Comments: Project page with code: this https URL
Subjects: Computer Vision and Pattern Recognition (cs.CV)
Cite as:  arXiv:2312.14135 [cs.CV]
          (or arXiv:2312.14135v2 [cs.CV] for this version)
          https://doi.org/10.48550/arXiv.2312.14135
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Saining Xie [view email]
[v1] Thu, 21 Dec 2023 18:55:06 UTC (5,704 KB)
[v2] Tue, 26 Dec 2023 15:20:45 UTC (5,708 KB)
Full-text links:

Access Paper:

    Download a PDF of the paper titled V*: Guided Visual Search as a
    Core Mechanism in Multimodal LLMs, by Penghao Wu and 1 other
    authors
  * Download PDF
  * PostScript
  * Other Formats
  * HTML (experimental)

[by-4]
Current browse context:
cs.CV
< prev   |   next >
new | recent | 2312
Change to browse by:
cs

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack