https://www.nature.com/articles/d41586-021-02895-8 Skip to main content Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript. Advertisement Advertisement Nature * View all journals * Search * My Account Login * Explore content * About the journal * Publish with us Subscribe * Sign up for alerts * RSS feed 1. nature 2. news 3. article * NEWS * 26 October 2021 Giant, free index to world's research papers released online Catalogue of billions of phrases from 107 million papers could ease computerized searching of the literature. * Holly Else 1. Holly Else View author publications You can also search for this author in PubMed Google Scholar * Share on Twitter * Share on Facebook * Share via E-Mail You have full access to this article via your institution. Download PDF Carl Malamud Technologist Carl Malamud.Credit: Smita Sharma In a project that could unlock the world's research papers for easier computerized analysis, an American technologist has released online a gigantic index of the words and short phrases contained in more than 100 million journal articles -- including many paywalled papers. The catalogue, which was released on 7 October and is free to use, holds tables of more than 355 billion words and sentence fragments listed next to the articles in which they appear. It is an effort to help scientists use software to glean insights from published work even if they have no legal access to the underlying papers, says its creator, Carl Malamud. He released the files under the auspices of Public Resource, a non-profit corporation in Sebastopol, California that he founded. Malamud says that because his index doesn't contain the full text of articles, but only sentence snippets up to five words long, releasing it does not breach publishers' copyright restrictions on the re-use of paywalled articles. However, one legal expert says that publishers might question the legality of how Malamud created the index in the first place. Some researchers who have had early access to the index say it's a major development in helping them to search the literature with software -- a procedure known as text mining. Gitanjali Yadav, a computational biologist at the University of Cambridge, UK, who studies volatile organic compounds emitted by plants, says she aims to comb through Malamud's index to produce analyses of the plant chemicals described in the world's research papers. "There is no way for me -- or anyone else -- to experimentally analyse or measure the chemical fingerprint of each and every plant species on Earth. Much of the information we seek already exists, in published literature," she says. But researchers are restricted by lack of access to many papers, Yadav adds. Malamud's 'General Index', as he calls it, aims to address the problems faced by researchers such as Yadav. Computer scientists already text mine papers to build databases of genes, drugs and chemicals found in the literature, and to explore papers' content faster than a human could read. But they often note that publishers ultimately control the speed and scope of their work, and that scientists are restricted to mining only open-access papers, or those articles they (or their institutions) have subscriptions to. Some publishers have said that researchers looking to mine the text of paywalled papers need their authorization. And although free search engines such as Google Scholar have -- with publishers' agreement -- indexed the text of paywalled literature, they only allow users to search with certain types of text queries, and restrict automated searching. That doesn't allow large-scale computerized analysis using more specialized searches, Malamud says. Terabytes of data Malamud's project is his latest venture in a career spent releasing locked-up information for free access online -- often in the face of legal challenges. He originally focused on publishing government-produced legal and financial information. But more recently he has turned his attention to opening up the scientific literature. He began with a project to allow scientists to text mine -- but not read -- a giant store of research papers he's holding on a server in India; an idea he says he's still working on. The General Index now allows anyone to mine scientific works, but it doesn't have its own web-search portal, so if scientists want to search it they will have to download its files and develop their own programs. Malamud is hoping that users will make any search engines they create available to others. In its compressed format the catalogue totals almost 5 terabytes, and then expands to 38 terabytes. As well as sentence fragments, the files also include tables of nearly 20 billion keywords in the literature, and tables of a paper's title, authors and DOI (article identifier), so that users can track down a full paper if they have access to read it. Michael Carroll, a legal researcher at the American University Washington College of Law, says that distributing the index should be legal worldwide because the files do not copy enough of an underlying article to infringe the publisher's copyright -- although laws vary by country. "Copyright does not protect facts and ideas, and these results would be treated as communication of facts derived from the analysis of the copyrighted articles," he says. The only legal question, Carroll adds, is whether Malamud's obtaining and copying of the underlying papers was done without breaching publishers' terms. Malamud says that he did have to get copies of the 107 million articles referenced in the index in order to create it; he declined to say how, but emphasizes that researchers will not have access to the full texts of the papers, which are stored in a secured, undisclosed location in the United States. "I am very confident that what I'm doing is legal. We are not doing this to provoke a lawsuit, we are doing it to advance science," he says. Nature contacted six publishers about the General Index for this article: all but one declined to comment. In a statement, Springer Nature said that the company supports open-research initiatives that use technology and algorithms to meet the needs of researchers. "We have seen some initiatives run into trouble, however, when the necessary rights have not been secured to enable their sustainability," the statement added. (Springer Nature publishes this journal; Nature's news team is editorially independent of its publisher.) Another legal researcher, Arul George Scaria at Delhi's National Law University, says that any publishers that tried to use copyright laws to prevent researchers from using the General Index "would eventually be disappointed". The release of the index, Scaria says, is a "major development for the wealth of information it has unlocked from those 107 million journal articles". doi: https://doi.org/10.1038/d41586-021-02895-8 Related Articles * [d41586-021] The plan to mine the world's research papers Subjects * Computer science * Databases * Publishing Latest on: Computer science How AI is helping the natural sciences How AI is helping the natural sciences Career Guide 13 OCT 21 Skilful precipitation nowcasting using deep generative models of radar Skilful precipitation nowcasting using deep generative models of radar Article 29 SEP 21 Biologically informed deep neural network for prostate cancer discovery Biologically informed deep neural network for prostate cancer discovery Article 22 SEP 21 Databases A guide to the Nature Index A guide to the Nature Index Nature Index 27 OCT 21 How to make your research reproducible How to make your research reproducible Career Column 21 OCT 21 How the world's biggest brain maps could transform neuroscience How the world's biggest brain maps could transform neuroscience News Feature 06 OCT 21 Publishing Thousands of new and improved mineral symbols Correspondence 26 OCT 21 This digital-hygiene routine will protect your scholarship This digital-hygiene routine will protect your scholarship World View 26 OCT 21 Predatory publishers' latest scam: bootlegged and rebranded papers Predatory publishers' latest scam: bootlegged and rebranded papers Comment 26 OCT 21 Nature Careers Jobs * Physics and Chemistry Postdocs Los Alamos National Laboratory (LANL) Los Alamos, NM, United States * Assistant Professor - Climate Dynamics and Modeling - (FAC001997) University of Houston (UH) Houston, TX, United States * Assistant or Associate Professor Chemical Extraction of Critical Metals NSERC Canada Research Chair (CRC) Tier 2 The Robert M. Buchan Department of Mining Queen's University at Kingston Queen's University Kingston, Canada * Staff Scientist St. Jude Children's Research Hospital (St. Jude) Memphis, TN, United States You have full access to this article via your institution. Download PDF Related Articles * [d41586-021] The plan to mine the world's research papers Subjects * Computer science * Databases * Publishing Advertisement Sign up to Nature Briefing An essential round-up of science news, opinion and analysis, delivered to your inbox every weekday. Email address [ ] [ ] Yes! Sign me up to receive the daily Nature Briefing email. I agree my information will be processed in accordance with the Nature and Springer Nature Limited Privacy Policy. Sign up * Close Nature Briefing Sign up for the Nature Briefing newsletter -- what matters in science, free to your inbox daily. Email address [ ] Sign up [ ] I agree my information will be processed in accordance with the Nature and Springer Nature Limited Privacy Policy. Close Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing Explore content * Research articles * News * Opinion * Research Analysis * Careers * Books & Culture * Podcasts * Videos * Current issue * Browse issues * Collections * Subjects * Follow us on Facebook * Follow us on Twitter * Subscribe * Sign up for alerts * RSS feed About the journal * Journal Staff * About the Editors * Journal Information * Our publishing models * Editorial Values Statement * Awards * Journal Impact * Contact * Editorial policies * History of Nature * Send a news tip Publish with us * For Authors * For Referees * Submit manuscript Search Search articles by subject, keyword or author [ ] Show results from [All journals] Search Advanced search Quick links * Explore articles by subject * Find a job * Guide to authors * Editorial policies Nature (Nature) ISSN 1476-4687 (online) ISSN 0028-0836 (print) nature.com sitemap Nature portfolio * About us * Press releases * Press office * Contact us * * * Discover content * Journals A-Z * Articles by subject * Nano * Protocol Exchange * Nature Index Publishing policies * Nature portfolio policies * Open access Author & Researcher services * Reprints & permissions * Research data * Language editing * Scientific editing * Nature Masterclasses * Nature Research Academies Libraries & institutions * Librarian service & tools * Librarian portal * Open research * Recommend to library Advertising & partnerships * Advertising * Partnerships & Services * Media kits * Branded content Career development * Nature Careers * Nature Conferences * Nature events Regional websites * Nature Africa * Nature China * Nature India * Nature Italy * Nature Japan * Nature Korea * Nature Middle East Legal & Privacy * Privacy Policy * Use of cookies * Manage cookies/Do not sell my data * Legal notice * Accessibility statement * Terms & Conditions * California Privacy Statement Springer Nature (c) 2021 Springer Nature Limited