https://arxiv.org/abs/2303.13375

Skip to main content
Cornell University
We gratefully acknowledge support from
the Simons Foundation and member institutions.
 
arxiv logo > cs > arXiv:2303.13375
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2303.13375 (cs)
[Submitted on 20 Mar 2023]

Title:Capabilities of GPT-4 on Medical Challenge Problems

Authors:Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean
Carignan, Eric Horvitz
Download a PDF of the paper titled Capabilities of GPT-4 on Medical
Challenge Problems, by Harsha Nori and 4 other authors
Download PDF

    Abstract: Large language models (LLMs) have demonstrated
    remarkable capabilities in natural language understanding and
    generation across various domains, including medicine. We present
    a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on
    medical competency examinations and benchmark datasets. GPT-4 is
    a general-purpose model that is not specialized for medical
    problems through training or engineered to solve clinical tasks.
    Our analysis covers two sets of official practice materials for
    the USMLE, a three-step examination program used to assess
    clinical competency and grant licensure in the United States. We
    also evaluate performance on the MultiMedQA suite of benchmark
    datasets. Beyond measuring model performance, experiments were
    conducted to investigate the influence of test questions
    containing both text and images on model performance, probe for
    memorization of content during training, and study probability
    calibration, which is of critical importance in high-stakes
    applications like medicine. Our results show that GPT-4, without
    any specialized prompt crafting, exceeds the passing score on
    USMLE by over 20 points and outperforms earlier general-purpose
    models (GPT-3.5) as well as models specifically fine-tuned on
    medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM
    540B). In addition, GPT-4 is significantly better calibrated than
    GPT-3.5, demonstrating a much-improved ability to predict the
    likelihood that its answers are correct. We also explore the
    behavior of the model qualitatively through a case study that
    shows the ability of GPT-4 to explain medical reasoning,
    personalize explanations to students, and interactively craft new
    counterfactual scenarios around a medical case. Implications of
    the findings are discussed for potential uses of GPT-4 in medical
    education, assessment, and clinical practice, with appropriate
    attention to challenges of accuracy and safety.

Comments: 33 pages, 15 figures
Subjects: Computation and Language (cs.CL); Artificial Intelligence
          (cs.AI)
Cite as:  arXiv:2303.13375 [cs.CL]
          (or arXiv:2303.13375v1 [cs.CL] for this version)
          https://doi.org/10.48550/arXiv.2303.13375
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Harsha Nori [view email]
[v1] Mon, 20 Mar 2023 16:18:38 UTC (97 KB)
Full-text links:

Download:

    Download a PDF of the paper titled Capabilities of GPT-4 on
    Medical Challenge Problems, by Harsha Nori and 4 other authors
  * PDF
  * Other formats

(license)
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2303
Change to browse by:
cs
cs.AI

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export bibtex citation Loading...

Bibtex formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Mendeley logo Reddit logo ScienceWISE logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flowers
Influence Flowers (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack