https://arxiv.org/abs/2207.05221

close this message

Accessible arXiv

Do you navigate arXiv using a screen reader or other assistive
technology? Are you a professor who helps students do so? We want to
hear from you. Please consider signing up to share your insights as
we work to make arXiv even more open.

Share Insights
Skip to main content
Cornell University
We gratefully acknowledge support from
the Simons Foundation and member institutions.
 
arxiv logo > cs > arXiv:2207.05221
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2207.05221 (cs)
[Submitted on 11 Jul 2022]

Title:Language Models (Mostly) Know What They Know

Authors:Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan,
Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova
DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy
Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman
, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson,
Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse,
Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark,
Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, Jared Kaplan
Download PDF

    Abstract: We study whether language models can evaluate the
    validity of their own claims and predict which questions they
    will be able to answer correctly. We first show that larger
    models are well-calibrated on diverse multiple choice and true/
    false questions when they are provided in the right format. Thus
    we can approach self-evaluation on open-ended sampling tasks by
    asking models to first propose answers, and then to evaluate the
    probability "P(True)" that their answers are correct. We find
    encouraging performance, calibration, and scaling for P(True) on
    a diverse array of tasks. Performance at self-evaluation further
    improves when we allow models to consider many of their own
    samples before predicting the validity of one specific
    possibility. Next, we investigate whether models can be trained
    to predict "P(IK)", the probability that "I know" the answer to a
    question, without reference to any particular proposed answer.
    Models perform well at predicting P(IK) and partially generalize
    across tasks, though they struggle with calibration of P(IK) on
    new tasks. The predicted P(IK) probabilities also increase
    appropriately in the presence of relevant source materials in the
    context, and to the presence of hints towards the solution of
    mathematical word problems. We hope these observations lay the
    groundwork for training more honest models, and for investigating
    how honesty generalizes to cases where models are trained on
    objectives other than the imitation of human writing.

Comments: 23+17 pages
Subjects: Computation and Language (cs.CL); Artificial Intelligence
          (cs.AI); Machine Learning (cs.LG)
Cite as:  arXiv:2207.05221 [cs.CL]
          (or arXiv:2207.05221v1 [cs.CL] for this version)
          https://doi.org/10.48550/arXiv.2207.05221
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Jared Kaplan [view email]
[v1] Mon, 11 Jul 2022 22:59:39 UTC (1,249 KB)
Full-text links:

Download:

  * PDF
  * Other formats

[by-4]
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2207
Change to browse by:
cs
cs.AI
cs.LG

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export bibtex citation Loading...

Bibtex formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Mendeley logo Reddit logo ScienceWISE logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code & Data

Code and Data Associated with this Article

[ ] arXiv Links to Code Toggle
arXiv Links to Code & Data (What is Links to Code & Data?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
( ) Related Papers

Recommenders and Search Tools

[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)
( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs and how to get involved.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack