https://arxiv.org/abs/2201.06796

close this message
arXiv smileybones icon

Giving Week!

Show your support for Open Science by donating to arXiv during Giving
Week, April 25th-29th.

DONATE
Skip to main content
Cornell University
We gratefully acknowledge support from
the Simons Foundation and member institutions.
 
arxiv logo > cs > arXiv:2201.06796
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Human-Computer Interaction

arXiv:2201.06796 (cs)
[Submitted on 18 Jan 2022 (v1), last revised 25 Jan 2022 (this
version, v2)]

Title:CoAuthor: Designing a Human-AI Collaborative Writing Dataset
for Exploring Language Model Capabilities

Authors:Mina Lee, Percy Liang, Qian Yang
Download PDF

    Abstract: Large language models (LMs) offer unprecedented
    language generation capabilities and exciting opportunities for
    interaction design. However, their highly context-dependent
    capabilities are difficult to grasp and are often subjectively
    interpreted. In this paper, we argue that by curating and
    analyzing large interaction datasets, the HCI community can
    foster more incisive examinations of LMs' generative
    capabilities. Exemplifying this approach, we present CoAuthor, a
    dataset designed for revealing GPT-3's capabilities in assisting
    creative and argumentative writing. CoAuthor captures rich
    interactions between 63 writers and four instances of GPT-3
    across 1445 writing sessions. We demonstrate that CoAuthor can
    address questions about GPT-3's language, ideation, and
    collaboration capabilities, and reveal its contribution as a
    writing "collaborator" under various definitions of good
    collaboration. Finally, we discuss how this work may facilitate a
    more principled discussion around LMs' promises and pitfalls in
    relation to interaction design. The dataset and an interface for
    replaying the writing sessions are publicly available at this
    https URL.

Comments:  Published as a conference paper at CHI 2022
Subjects:  Human-Computer Interaction (cs.HC); Computation and
           Language (cs.CL)
Cite as:   arXiv:2201.06796 [cs.HC]
           (or arXiv:2201.06796v2 [cs.HC] for this version)
           https://doi.org/10.48550/arXiv.2201.06796
           Focus to learn more
           arXiv-issued DOI via DataCite
Related    https://doi.org/10.1145/3491102.3502030
DOI:       Focus to learn more
           DOI(s) linking to related resources

Submission history

From: Mina Lee [view email]
[v1] Tue, 18 Jan 2022 07:51:57 UTC (1,906 KB)
[v2] Tue, 25 Jan 2022 05:29:58 UTC (1,907 KB)
Full-text links:

Download:

  * PDF
  * Other formats

[by-4]
Current browse context:
cs.HC
< prev   |   next >
new | recent | 2201
Change to browse by:
cs
cs.CL

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

DBLP - CS Bibliography

listing | bibtex
Mina Lee
Percy Liang
Qian Yang
a export bibtex citation Loading...

Bibtex formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Mendeley logo Reddit logo ScienceWISE logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code & Data

Code and Data Associated with this Article

[ ] arXiv Links to Code Toggle
arXiv Links to Code & Data (What is Links to Code & Data?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
( ) Related Papers

Recommenders and Search Tools

[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)
( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs and how to get involved.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack