https://www.nature.com/articles/d41586-025-01266-x Skip to main content Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript. Advertisement Advertisement Nature * View all journals * Search * Log in * Explore content * About the journal * Publish with us * Subscribe * Sign up for alerts * RSS feed 1. nature 2. news 3. article * NEWS * 25 April 2025 Huge reproducibility project fails to validate dozens of biomedical studies Unique reproducibility effort in Brazil focuses on common methods rather than a single field -- and prompts call for reform. By * Rodrigo de Oliveira Andrade 1. Rodrigo de Oliveira Andrade View author publications You can also search for this author in PubMed Google Scholar * Twitter * Facebook * Email You have full access to this article via your institution. Two female researchers wearing full PPE sit working at extraction units in the lab, with their faces reflected in the glass A replication drive focused on results that lean on three methods commonly used in biomedical research in Brazil. Credit: Mauro Pimentel/AFP/Getty In an unprecedented effort, a coalition of more than 50 research teams has surveyed a swathe of Brazilian biomedical studies to double-check their findings -- with dismaying results. The teams were able to replicate the results of less than half of the tested experiments^1. That rate is in keeping with that found by other large-scale attempts to reproduce scientific findings. But the latest work is unique in focusing on papers that use specific methods and in examining the research output of a specific country, according to the research teams. The results provide an impetus to strengthen the country's science, the study's authors say. "We now have the material to start making changes from within -- whether through public policies or within universities," says Mariana Boechat de Abreu, a metascience researcher at the Federal University of Rio de Janeiro (UFRJ) in Brazil and one of the coordinators of the project. The work was posted on 8 April to the bioRxiv preprint server and has not yet been peer reviewed. Ambitious undertaking The massive experiment was coordinated by the Brazilian Reproducibility Initiative, a collaborative effort launched in 2019 by researchers at the UFRJ. The scientists wanted to assess publications "based on methods, rather than research area, perceived importance or citation counts", de Abreu says. And they wanted to do so on a large scale. Ultimately, 213 scientists at 56 laboratories in Brazil were involved in the work. The project unfolded during the COVID-19 pandemic, which brought numerous logistical challenges. And teams disagreed about how closely to follow the tested protocols. "It was like trying to turn dozens of garage bands, each with its own way of playing, into an orchestra," says project coordinator Olavo Bohrer Amaral, a physician at the UFRJ. [d41586-025] Reproducibility trial: 246 biologists get different results from same data sets The authors began by reviewing a random sample of life-sciences articles to determine the most common biomedical research methods used in Brazil, ensuring that any biomedical lab interested in joining the project would be capable of reproducing the experiments. They ended up selecting three of these methods: an assay of cell metabolism, a technique for amplifying genetic material and a type of maze test for rodents. Then the authors randomly selected biomedical papers that relied on those methods and were published from 1998 to 2017 by research teams in which at least half the contributors had a Brazilian affiliation. The collaborators initially chose a subset of 60 papers for replication, guided by factors such as whether a paper included certain statistical information. Three labs tested each experiment, and an independent committee judged which of those tests was a valid replication. The coalition performed 97 valid replication attempts of 47 experiments. Falling short The authors judged a paper's replicability by five criteria, including whether at least half of the replication attempts had statistically significant results in the same direction as the original paper. Only 21% of the experiments were replicable using at least half of the applicable criteria. The authors also found that the effect size -- the magnitude of the observed impact in the experiments -- was, on average, 60% larger in the original papers than in the experimental follow-ups, indicating that published results tend to overestimate the effects of the interventions tested. Enjoying our latest content? Login or create an account to continue * Access the most recent journalism from Nature's award-winning team * Explore the latest features & opinion covering groundbreaking research Access through your institution or Sign in or create an account # Continue with Google # Continue with ORCiD doi: https://doi.org/10.1038/d41586-025-01266-x References 1. Amaral, O. B. et al. Preprint at bioRxiv https://doi.org/10.1101/ 2025.04.02.645026 (2025). 2. Open Science Collaboration. Science 349, aac4716 (2015). Article PubMed Google Scholar Download references Reprints and permissions Related Articles * [d41586-025] What reproducibility crisis? New research protocol yields ultra-high replication rate * [d41586-025] 'Doing good science is hard': retraction of high-profile reproducibility study prompts soul-searching * [d41586-025] Reproducibility trial publishes two conclusions for one paper * [d41586-025] Peer-replication model aims to address science's 'reproducibility crisis' Subjects * Scientific community * Research management * Publishing * Medical research Latest on: Scientific community My 'woke DEI' grant has been flagged for scrutiny. Where do I go from here? My 'woke DEI' grant has been flagged for scrutiny. Where do I go from here? Career Column 25 APR 25 Hundreds more NSF grants terminated after agency director resigns Hundreds more NSF grants terminated after agency director resigns News 25 APR 25 How Trump's attack on universities is putting research in peril How Trump's attack on universities is putting research in peril News 24 APR 25 Research management Europe must become a research epicentre as US system gets undermined Correspondence 22 APR 25 Defend scientific integrity and academic freedom Correspondence 22 APR 25 'Totally broken': how Trump 2.0 has paralysed work at US science agencies 'Totally broken': how Trump 2.0 has paralysed work at US science agencies News 17 APR 25 Publishing Science sleuths flag hundreds of papers that use AI without disclosing it Science sleuths flag hundreds of papers that use AI without disclosing it News 24 APR 25 China overtakes the United States in cancer research output China overtakes the United States in cancer research output Nature Index 23 APR 25 Cancer vaccine momentum builds, but US funding cuts raise concerns Cancer vaccine momentum builds, but US funding cuts raise concerns Nature Index 23 APR 25 Nature Careers Jobs * Associate professor in computational biology The Department of Molecular Biology at Umea University is seeking an associate professor in the field of computational biology. Umea, Sweden Umea University [] * Faculty Positions at Institute of Physics (IOP), Chinese Academy of Sciences IOP is China's premier research institution in condensed matter physics and related fields. Beijing, China Institute of Physics (IOP), Chinese Academy of Sciences (CAS) [] * Faculty Positions at SUSTech Department of Biomedical Engineering We seek outstanding applicants for full-time tenure-track/tenured faculty positions. Positions are available for both junior and senior-level. Shenzhen, Guangdong, China Southern University of Science and Technology (Biomedical Engineering) [] * Full Professorship (W3) for "Experimental Psychopathology" (f/m/ d) The Medical Faculty Mannheim of Heidelberg University invites applications for the Full Professorship (W3) for "Experimental Psychopathology" (... Mannheim, Baden-Wurttemberg (DE) Medizinische Fakultat Mannheim der Universitat Heidelberg [] * Full Professorship (W3) for "Experimental Psychotherapy" (f/m/d) The Medical Faculty Mannheim of Heidelberg University invites applications for the newly-established Full Professorship (W3) for "Experimental Ps... Mannheim, Baden-Wurttemberg (DE) Medizinische Fakultat Mannheim der Universitat Heidelberg [] Related Articles * [d41586-025] What reproducibility crisis? New research protocol yields ultra-high replication rate * [d41586-025] 'Doing good science is hard': retraction of high-profile reproducibility study prompts soul-searching * [d41586-025] Reproducibility trial publishes two conclusions for one paper * [d41586-025] Peer-replication model aims to address science's 'reproducibility crisis' Subjects * Scientific community * Research management * Publishing * Medical research Advertisement Sign up to Nature Briefing An essential round-up of science news, opinion and analysis, delivered to your inbox every weekday. Email address [ ] [ ] Yes! Sign me up to receive the daily Nature Briefing email. I agree my information will be processed in accordance with the Nature and Springer Nature Limited Privacy Policy. Sign up Close Nature Briefing Sign up for the Nature Briefing newsletter -- what matters in science, free to your inbox daily. Email address [ ] Sign up [ ] I agree my information will be processed in accordance with the Nature and Springer Nature Limited Privacy Policy. Close Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing Explore content * Research articles * News * Opinion * Research Analysis * Careers * Books & Culture * Podcasts * Videos * Current issue * Browse issues * Collections * Subjects * Follow us on Facebook * Follow us on Twitter * Subscribe * Sign up for alerts * RSS feed About the journal * Journal Staff * About the Editors * Journal Information * Our publishing models * Editorial Values Statement * Journal Metrics * Awards * Contact * Editorial policies * History of Nature * Send a news tip Publish with us * For Authors * For Referees * Language editing services * Submit manuscript Search Search articles by subject, keyword or author [ ] Show results from [All journals] Search Advanced search Quick links * Explore articles by subject * Find a job * Guide to authors * Editorial policies Nature (Nature) ISSN 1476-4687 (online) ISSN 0028-0836 (print) nature.com sitemap About Nature Portfolio * About us * Press releases * Press office * Contact us Discover content * Journals A-Z * Articles by subject * protocols.io * Nature Index Publishing policies * Nature portfolio policies * Open access Author & Researcher services * Reprints & permissions * Research data * Language editing * Scientific editing * Nature Masterclasses * Research Solutions Libraries & institutions * Librarian service & tools * Librarian portal * Open research * Recommend to library Advertising & partnerships * Advertising * Partnerships & Services * Media kits * Branded content Professional development * Nature Careers * Nature Conferences Regional websites * Nature Africa * Nature China * Nature India * Nature Italy * Nature Japan * Nature Middle East * Privacy Policy * Use of cookies * Your privacy choices/Manage cookies * Legal notice * Accessibility statement * Terms & Conditions * Your US state privacy rights Springer Nature (c) 2025 Springer Nature Limited