https://arxiv.org/abs/2410.03608

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2410.03608
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Artificial Intelligence

arXiv:2410.03608 (cs)
[Submitted on 4 Oct 2024]

Title:TICKing All the Boxes: Generated Checklists Improve LLM
Evaluation and Generation

Authors:Jonathan Cook, Tim Rocktaschel, Jakob Foerster, Dennis
Aumiller, Alex Wang
View a PDF of the paper titled TICKing All the Boxes: Generated
Checklists Improve LLM Evaluation and Generation, by Jonathan Cook
and 4 other authors
View PDF HTML (experimental)

    Abstract:Given the widespread adoption and usage of Large
    Language Models (LLMs), it is crucial to have flexible and
    interpretable evaluations of their instruction-following ability.
    Preference judgments between model outputs have become the de
    facto evaluation standard, despite distilling complex,
    multi-faceted preferences into a single ranking. Furthermore, as
    human annotation is slow and costly, LLMs are increasingly used
    to make these judgments, at the expense of reliability and
    interpretability. In this work, we propose TICK (Targeted
    Instruct-evaluation with ChecKlists), a fully automated,
    interpretable evaluation protocol that structures evaluations
    with LLM-generated, instruction-specific checklists. We first
    show that, given an instruction, LLMs can reliably produce
    high-quality, tailored evaluation checklists that decompose the
    instruction into a series of YES/NO questions. Each question asks
    whether a candidate response meets a specific requirement of the
    instruction. We demonstrate that using TICK leads to a
    significant increase (46.4% $\to$ 52.2%) in the frequency of
    exact agreements between LLM judgements and human preferences, as
    compared to having an LLM directly score an output. We then show
    that STICK (Self-TICK) can be used to improve generation quality
    across multiple benchmarks via self-refinement and Best-of-N
    selection. STICK self-refinement on LiveBench reasoning tasks
    leads to an absolute gain of $+$7.8%, whilst Best-of-N selection
    with STICK attains $+$6.3% absolute improvement on the real-world
    instruction dataset, WildBench. In light of this, structured,
    multi-faceted self-improvement is shown to be a promising way to
    further advance LLM capabilities. Finally, by providing
    LLM-generated checklists to human evaluators tasked with directly
    scoring LLM responses to WildBench instructions, we notably
    increase inter-annotator agreement (0.194 $\to$ 0.256).

          Artificial Intelligence (cs.AI); Computation and Language
Subjects: (cs.CL); Human-Computer Interaction (cs.HC); Machine
          Learning (cs.LG)
Cite as:  arXiv:2410.03608 [cs.AI]
          (or arXiv:2410.03608v1 [cs.AI] for this version)
          https://doi.org/10.48550/arXiv.2410.03608
          Focus to learn more
          arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jonathan Cook [view email]
[v1] Fri, 4 Oct 2024 17:09:08 UTC (514 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled TICKing All the Boxes: Generated
    Checklists Improve LLM Evaluation and Generation, by Jonathan
    Cook and 4 other authors
  * View PDF
  * HTML (experimental)
  * TeX Source
  * Other Formats

license icon view license
Current browse context:
cs.AI
< prev   |   next >
new | recent | 2024-10
Change to browse by:
cs
cs.CL
cs.HC
cs.LG

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] GotitPub Toggle
Gotit.pub (What is GotitPub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack