https://arxiv.org/abs/2406.09490

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2406.09490
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2406.09490 (cs)
[Submitted on 13 Jun 2024]

Title:Newswire: A Large-Scale Structured Database of a Century of
Historical News

Authors:Emily Silcock, Abhishek Arora, Luca D'Amico-Wong, Melissa
Dell
View a PDF of the paper titled Newswire: A Large-Scale Structured
Database of a Century of Historical News, by Emily Silcock and
Abhishek Arora and Luca D'Amico-Wong and Melissa Dell
View PDF HTML (experimental)

    Abstract:In the U.S. historically, local newspapers drew their
    content largely from newswires like the Associated Press.
    Historians argue that newswires played a pivotal role in creating
    a national identity and shared understanding of the world, but
    there is no comprehensive archive of the content sent over
    newswires. We reconstruct such an archive by applying a
    customized deep learning pipeline to hundreds of terabytes of raw
    image scans from thousands of local newspapers. The resulting
    dataset contains 2.7 million unique public domain U.S. newswire
    articles, written between 1878 and 1977. Locations in these
    articles are georeferenced, topics are tagged using customized
    neural topic classification, named entities are recognized, and
    individuals are disambiguated to Wikipedia using a novel entity
    disambiguation model. To construct the Newswire dataset, we first
    recognize newspaper layouts and transcribe around 138 millions
    structured article texts from raw image scans. We then use a
    customized neural bi-encoder model to de-duplicate reproduced
    articles, in the presence of considerable abridgement and noise,
    quantifying how widely each article was reproduced. A text
    classifier is used to ensure that we only include newswire
    articles, which historically are in the public domain. The
    structured data that accompany the texts provide rich information
    about the who (disambiguated individuals), what (topics), and
    where (georeferencing) of the news that millions of Americans
    read over the course of a century. We also include Library of
    Congress metadata information about the newspapers that ran the
    articles on their front pages. The Newswire dataset is useful
    both for large language modeling - expanding training data beyond
    what is available from modern web texts - and for studying a
    diversity of questions in computational linguistics, social
    science, and the digital humanities.

Comments: arXiv admin note: text overlap with arXiv:2306.17810,
          arXiv:2308.12477
Subjects: Computation and Language (cs.CL); General Economics
          (econ.GN)
Cite as:  arXiv:2406.09490 [cs.CL]
          (or arXiv:2406.09490v1 [cs.CL] for this version)
          https://doi.org/10.48550/arXiv.2406.09490
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Melissa Dell [view email]
[v1] Thu, 13 Jun 2024 16:20:05 UTC (3,050 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled Newswire: A Large-Scale Structured
    Database of a Century of Historical News, by Emily Silcock and
    Abhishek Arora and Luca D'Amico-Wong and Melissa Dell
  * View PDF
  * HTML (experimental)
  * TeX Source
  * Other Formats

license icon view license
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2024-06
Change to browse by:
cs
econ
econ.GN
q-fin
q-fin.EC

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] GotitPub Toggle
Gotit.pub (What is GotitPub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack