(C) PLOS One
This story was originally published by PLOS One and is unaltered.
 . . . . . . . . . .



Developing artificial intelligence tools for institutional review board pre-review: A pilot study on ChatGPT’s accuracy and reproducibility [1]

['Yasuko Fukataki', 'Health Data Science', 'Department Of Social Science', 'Graduate School Of Medicine', 'Hokkaido University', 'Sapporo', 'Biostatistics', 'Data Management', 'Sapporo Medical University', 'Wakako Hayashi']

Date: 2025-07 

Abstract This pilot study is the first phase of a broader project aimed at developing an explainable artificial intelligence (AI) tool to support the ethical evaluation of Japanese-language clinical research documents. The tool is explicitly not intended to assist document drafting. We assessed the baseline performance of generative AI—Generative Pre-trained Transformer (GPT)-4 and GPT-4o—in analyzing clinical research protocols and informed consent forms (ICFs). The goal was to determine whether these models could accurately and consistently extract ethically relevant information, including the research objectives and background, research design, and participant-related risks and benefits. First, we compared the performance of GPT-4 and GPT-4o using custom agents developed via OpenAI’s Custom GPT functionality (hereafter “GPTs”). Then, using GPT-4o alone, we compared outputs generated by GPTs optimized with customized Japanese prompts to those generated by standard prompts. GPT-4o achieved 80% agreement in extracting research objectives and background and 100% in extracting research design, while both models demonstrated high reproducibility across ten trials. GPTs with customized prompts produced more accurate and consistent outputs than standard prompts. This study suggests the potential utility of generative AI in pre-institutional review board (IRB) review tasks; it also provides foundational data for future validation and standardization efforts involving retrieval-augmented generation and fine-tuning. Importantly, this tool is intended not to automate ethical review but rather to support IRB decision-making. Limitations include the absence of gold standard reference data, reliance on a single evaluator, lack of convergence and inter-rater reliability analysis, and the inability of AI to substitute for in-person elements such as site visits.

Author summary In this pilot study, we examined whether ChatGPT could accurately and consistently extract key elements from clinical research protocols and informed consent forms written in Japanese. These elements include the research objectives and background, research design, and participant-related risks and benefits, all essential for ethical review. This study is part of a larger project aiming to develop an explainable AI tool to assist institutional review board (IRB) members in evaluating clinical research documents, but not to automate ethical review. As a prerequisite for implementing advanced technologies such as document retrieval and task-specific adaptation, this study evaluated the basic capabilities of current large language models. We compared standard and customized prompts for GPT-4 and GPT-4o and found that GPT-4o, when guided by customized prompts, produced more accurate and consistent outputs. However, study limitations remain, including the absence of gold standard reference data, reliance on a single evaluator, and lack of convergence and inter-rater reliability analysis. This study offers important early insights into how generative AI could assist the ethical evaluation of clinical research documents written in Japanese.

Citation: Fukataki Y, Hayashi W, Nishimoto N, Ito YM (2025) Developing artificial intelligence tools for institutional review board pre-review: A pilot study on ChatGPT’s accuracy and reproducibility. PLOS Digit Health 4(6): e0000695. https://doi.org/10.1371/journal.pdig.0000695 Editor: Po-Chih Kuo, National Tsing-Hua University: National Tsing Hua University, TAIWAN Received: November 8, 2024; Accepted: June 5, 2025; Published: June 30, 2025 Copyright: © 2025 Fukataki et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: All data underlying the findings in this study are fully available in the Supporting Information files (S1 to S11 Files). These include source texts, GPT outputs, evaluation tables, and the mock IRB review form. No personal or identifying information is included. S1 to S10 Files are available from Figshare at: https://doi.org/10.6084/m9.figshare.28737845 S11 File (Mock IRB Evaluation Form) is available from Figshare at: https://doi.org/10.6084/m9.figshare.28838291 Funding: The author(s) received no specific funding for this work. Competing interests: The authors have declared that no competing interests exist.

Introduction An ethical review of clinical research is essential to ensure both participant safety and research reliability. Committees responsible for ethical review evaluate the appropriateness of study protocols and establish standards to safeguard human rights and safety. However, the review processes of institutional review boards (IRBs) worldwide, as well as certified review boards (CRBs) in Japan, face several persistent challenges. For instance, review comments from Japan’s CRBs show substantial variation in terms of level of detail, structural organization, and evaluative consistency. Some CRBs use free-form formats, in which statements may be extremely brief (e.g., “no issues”) or reflect differing evaluative perspectives and address different elements, even among reviewers in the same field. This variability stems from the absence of a uniform structure. In contrast, other CRBs produce reviews that are clearly organized by key elements such as research objectives, study design, and risk–benefit assessment. These disparities have prompted calls to standardize and harmonize the quality of ethical reviews [1–7]. Across different ethics review committees internationally, considerable variation has been reported in revision instructions and review durations for clinical research protocols and informed consent forms (ICFs) [8–10]. Such inter-committee variability undermines the consistency and transparency of the review process and often stems from differing interpretations of ethical standards and shortages in expert human resources. Delays in the review process can also postpone study initiation and, in turn, the introduction of new therapies and medical technologies. Moreover, if revision comments based on ethical standards are inadequate or inappropriate, they may lead to insufficient or improper amendments to protocols and ICFs, potentially compromising participant safety. Recent advances in large language models (LLMs), machine learning, deep learning, and generative artificial intelligence (AI) have shown promise in medical domains such as image analysis, electronic health record interpretation, and virtual health assistants [11–17]. LLMs, a type of deep learning-based generative AI, are capable of performing natural language processing (NLP) tasks [18]. ChatGPT, a widely known example of generative AI, has been trained on a large corpus of text and is capable of performing complex natural language tasks [19]. For instance, ChatGPT achieved scores at or above the passing threshold on publicly available questions from the United States Medical Licensing Examination (USMLE) [20] and the Japanese National Medical Licensing Examination [21,22], demonstrating its potential in high-level medical knowledge tasks. Despite these promising capabilities, generative AI outputs are known to be prone to bias and hallucinations, raising ethical concerns [23–26]. In healthcare applications, AI transparency and reliability are crucial [24,25], and explainability is increasingly emphasized [23,26]. Explainable AI refers to technologies that enable AI to present the reasoning behind its outputs, thereby increasing the trustworthiness of the system. Notably, few studies have examined the application of generative AI to the IRB review process itself. One key barrier is the high confidentiality of documents such as study protocols and ICFs, which makes it inappropriate to analyze them using cloud-based generative AI models (e.g., ChatGPT, Gemini, Copilot). Furthermore, IRB reviews typically involve complex deliberation by multiple experts with extensive experience and knowledge [1,27–29], and it remains unclear whether AI can adequately replicate such multifaceted processes. Godwin et al. developed an AI tool to support the drafting of IRB application documents, seeking to improve the efficiency of document-preparation processes [30]. However, this approach is limited to assisting the creation of application materials and is not intended to support the process of reviewing submitted documents. In contrast, the present project aims to develop an explainable AI tool to support the IRB document review process. Specifically, the system is designed to provide both the evaluation results—namely, which key elements the AI identified as critical for comprehensive IRB review, including research objectives and background, study design, participant-related risks and benefits, and other domain-specific considerations such as data management, statistical methodology, clinical expertise, and pharmacological perspectives—and the reasoning behind those evaluations. The goal is to assist IRB members in their decision-making and to improve the efficiency and consistency of the document review process. Importantly, this project does not aim to automate the IRB review; rather, it is based on the premise that AI should serve as a supporting tool for human reviewers. The implementation of such AI support tools will eventually require advanced techniques such as retrieval-augmented generation (RAG) and fine-tuning (FT). However, it is first necessary to evaluate whether existing generative AI models can accurately and reproducibly extract key information from Japanese clinical research documents. Without this foundational capability, subsequent technological enhancements would lack reliability. Accordingly, this article reports a foundational pilot study within our broader project. The overall structure of the project and the role of this study are illustrated in Fig 1. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 1. Project Overview and the Position of this Study in Developing an AI-based IRB Review Support System. https://doi.org/10.1371/journal.pdig.0000695.g001 This figure illustrates the broader framework for building an explainable AI tool to support document review in IRB processes. This pilot study serves as the first phase, evaluating the baseline accuracy and reproducibility of generative AI models in extracting and summarizing key components—namely, research objectives and background, research design, and participant-related risks and benefits—from clinical research protocols and ICFs. In this study, we examined whether ChatGPT could accurately and reproducibly extract and summarize these key elements from Japanese-language documents. The findings are intended to serve as foundational data for the future implementation of RAG and FT technologies. Notably, this study does not evaluate the ability of AI to make judgments based on review criteria or to explain those judgments; such evaluation is planned in subsequent phases of the project. We conducted two primary comparisons: (1) a performance comparison between GPT-4 and GPT-4o; (2) a reproducibility comparison between customized GPTs (modified via prompt engineering, without FT) and standard prompts.

Materials and methods Study overview This study evaluated the accuracy and reproducibility of outputs generated by GPT models (OpenAI, San Francisco, CA, USA) in the document review of research protocols and ICFs. With a view toward future implementations of AI in supporting document-based IRB reviews in Japan, we focused on processing Japanese-language protocols using prompts written in Japanese. Two types of comparisons were conducted: a performance comparison between GPT-4 and GPT-4o (the latest model as of May 2024); a reproducibility comparison between customized GPTs and standard prompts, The primary KPIs were the accuracy and reproducibility of information extraction by AI for three elements: research objectives and background; research design; participant-related risks and benefits. These evaluation categories were preliminarily defined by focusing on three key elements considered important in IRB document review—namely, research objectives and background, research design, and participant-related risks and benefits. This pilot study specifically assessed the accuracy and reproducibility of extracting and summarizing these limited elements. Since the study targeted only a subset of information typically reviewed by IRB members, the KPIs used here are provisional and will be refined through future studies. No FT or parameter adjustments were applied to the models. Instead, evaluation was conducted solely using prompt engineering with the default configuration provided by OpenAI. Data source Mock review materials publicly available from the Ministry of Health, Labour and Welfare of Japan were used for this study [34]. These materials were created for training and education on IRB review and include fictional protocols and corresponding mock evaluation results across 10 disease domains [35–38]. Specifically, the dataset includes: Ten research projects covering oncology (n = 3), gastrointestinal diseases (n = 1), rheumatoid arthritis (n = 1), vaccines (n = 1), medical devices (n = 1), cardiovascular diseases (n = 1), pediatric psychiatric disorders (n = 1), and cognitive impairment (n = 1).

Each set contains research protocols, explanatory documents, monitoring manuals, audit manuals, and expert reviewer reports (mock IRB assessments). The validity of the mock evaluations is not disclosed, so the materials were used as reference standards, not as absolute ground truth. For this study, we selected and analyzed the following protocol: “Phase II trial of trastuzumab for patients with advanced solid tumors without standard treatment options” (Mock Review No. 1) Model selection and configuration We evaluated the performance of the GPT-4 and GPT-4o models. GPT-4 is a conventional high-performance model with strong capabilities across a range of text generation tasks [39]. GPT-4o, released on May 13, 2024, offers twice the speed and 50% lower cost compared to GPT-4 Turbo. GPT-4o also achieves state-of-the-art performance in speech recognition, multilingual processing, and visual capabilities. In particular, improvements in multilingual capabilities allow for higher accuracy when processing non-English texts [40]. Given these characteristics, we considered it valuable to verify the accuracy improvements offered by GPT-4o. We did not apply FT or model parameter adjustments. Instead, we used the GPTs feature provided by OpenAI [41] to develop customized GPTs tailored to extract specific categories of information (Table 9). PPT PowerPoint slide

PNG larger image

TIFF original image Download: Table 9. Overview of the Four Customized GPTs and Their Prompt Structures. https://doi.org/10.1371/journal.pdig.0000695.t009 Evaluation process and comparison Procedure for GPTs trials. To assess accuracy and reproducibility, the following procedure was implemented: Start a new chat session

Each trial was initiated in a new session to eliminate any influence from previous interactions. Select model and GPT

GPT-4 or GPT-4o was selected along with one of the four customized GPTs (GPTs-1 to GPTs-4). Upload document

The target research protocol or ICF was uploaded, then outputs were automatically generated using the embedded prompts. Save output

Each result was saved with a name format such as “GPTs_name_trialNumber.” All outputs were evaluated after completing 10 trials for each GPT. Repeat process

Each GPT was run 10 times with both models, totaling 80 trials. Procedure for standard prompt trials. For comparison, we also conducted evaluations using standard prompts. Start a new chat session

Each trial was conducted in an independent session. Model selection

The GPT-4o model was used. Upload document and enter prompt

The research protocol or ICF was uploaded. The following prompt (in Japanese) was entered:

“You are an IRB reviewer. Please extract the participant-related benefits for participants from the following research protocol.” Save output

Each result was saved as “StandardPrompt_trialNumber” and evaluated after all 10 trials were completed. Repeat process

A total of 10 trials were performed. Evaluation criteria for accuracy and reproducibility All outputs were manually reviewed by one evaluator based on predefined criteria (Tables 6–8) using the following process: 1. Accuracy. Outputs were classified as “consistent,” “partially consistent,” or “inconsistent” using the definitions presented in Tables 6 and 7. 2. Reproducibility. Reproducibility was determined based on the criteria presented in Table 8, using the number of trials rated as “consistent” or “partially consistent” per model (out of 10). The extracted text and its translation, GPT-generated outputs, results from standard prompts, and the mock IRB evaluation form are provided as supporting information (S1 to S11 Files) to ensure transparency and reproducibility.

Acknowledgments We would like to express our gratitude to Associate Professor Kenji Hirata from the Department of Diagnostic Imaging, Graduate School of Medicine, Hokkaido University, for providing valuable insights related to this project. We also thank Professor Shiro Hinotsu from the Division of Biostatistics and Data Management, Sapporo Medical University, for dedicating his time to discussing this topic with us. We thank ChatGPT (OpenAI) for assisting with the English translation of Japanese-language research materials, including manuscript drafts, mock protocols, and output examples. All AI-generated translations were carefully reviewed and verified by the authors. We also thank Editage (Cactus Communications) for providing final English proofreading. The authors take full responsibility for the accuracy and integrity of the manuscript content.

[END]
---
[1] Url: https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000695

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/