https://arxiv.org/abs/1806.06237

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > stat > arXiv:1806.06237
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Statistics > Machine Learning

arXiv:1806.06237 (stat)
[Submitted on 16 Jun 2018 (v1), last revised 15 Nov 2019 (this
version, v2)]

Title:PeerReview4All: Fair and Accurate Reviewer Assignment in Peer
Review

Authors:Ivan Stelmakh, Nihar B. Shah, Aarti Singh
Download a PDF of the paper titled PeerReview4All: Fair and Accurate
Reviewer Assignment in Peer Review, by Ivan Stelmakh and 1 other
authors
Download PDF

    Abstract: We consider the problem of automated assignment of
    papers to reviewers in conference peer review, with a focus on
    fairness and statistical accuracy. Our fairness objective is to
    maximize the review quality of the most disadvantaged paper, in
    contrast to the commonly used objective of maximizing the total
    quality over all papers. We design an assignment algorithm based
    on an incremental max-flow procedure that we prove is
    near-optimally fair. Our statistical accuracy objective is to
    ensure correct recovery of the papers that should be accepted. We
    provide a sharp minimax analysis of the accuracy of the
    peer-review process for a popular objective-score model as well
    as for a novel subjective-score model that we propose in the
    paper. Our analysis proves that our proposed assignment algorithm
    also leads to a near-optimal statistical accuracy. Finally, we
    design a novel experiment that allows for an objective comparison
    of various assignment algorithms, and overcomes the inherent
    difficulty posed by the absence of a ground truth in experiments
    on peer-review. The results of this experiment as well as of
    other experiments on synthetic and real data corroborate the
    theoretical guarantees of our algorithm.

          Machine Learning (stat.ML); Data Structures and Algorithms
Subjects: (cs.DS); Information Theory (cs.IT); Machine Learning
          (cs.LG)
Cite as:  arXiv:1806.06237 [stat.ML]
          (or arXiv:1806.06237v2 [stat.ML] for this version)
          https://doi.org/10.48550/arXiv.1806.06237
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Ivan Stelmakh [view email]
[v1] Sat, 16 Jun 2018 12:42:04 UTC (115 KB)
[v2] Fri, 15 Nov 2019 02:15:04 UTC (184 KB)
Full-text links:

Download:

  * Download a PDF of the paper titled PeerReview4All: Fair and
    Accurate Reviewer Assignment in Peer Review, by Ivan Stelmakh and
    1 other authors
    PDF
  * Other formats

(license)
Current browse context:
stat.ML
< prev   |   next >
new | recent | 1806
Change to browse by:
cs
cs.DS
cs.IT
cs.LG
math
math.IT
stat

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack