(C) PLOS One
This story was originally published by PLOS One and is unaltered.
 . . . . . . . . . .



Significant reduction in manual annotation costs in ultrasound medical image database construction through step by step artificial intelligence pre-annotation [1]

['Fu Zheng', 'The Ultrasound Diagnosis Department', 'The Hospital Of Joint Logistics Support Force Of Pla', 'Ningbo City', 'Zhejiang Province', 'Liu Xingming', 'Xu Juying', 'Tao Mengying', 'Yang Baojian', 'Shan Yan']

Date: 2025-09 

Abstract This study investigates the feasibility of reducing manual image annotation costs in medical image database construction by utilizing a step by step approach where the Artificial Intelligence model (AI model) trained on a previous batch of data automatically pre-annotates the next batch of image data, taking ultrasound image of thyroid nodule annotation as an example. The study used YOLOv8 as the AI model. During the AI model training, in addition to conventional image augmentation techniques, augmentation methods specifically tailored for ultrasound images were employed to balance the quantity differences between thyroid nodule classes and enhance model training effectiveness. The study found that training the model with augmented data significantly outperformed training with raw images data. When the number of original images number was only 1,360, with 7 thyroid nodule classifications, pre-annotation using the AI model trained on augmented data could save at least 30% of the manual annotation workload for junior physicians. When the scale of original images number reached 6,800, the classification accuracy of the AI model trained on augmented data was very close with that of junior physicians, eliminating the need for manual preliminary annotation.

Author summary Building high-quality medical image databases for training artificial intelligence (AI) applications often requires labor-intensive manual annotation by clinicians, which is also the case in ultrasound imaging. Additionally, class imbalances (e.g., rare vs. common nodule types) further exacerbate the complexity of this process.This study addresses these challenges by introducing a stepwise AI pre-annotation strategy combined with ultrasound-specific data augmentation to streamline thyroid nodule database construction. Using the YOLOv8 model, we developed a workflow where AI iteratively pre-annotates new image batches after training on prior data. Crucially, we augmented ultrasound images with simulated artifacts like defocus, acoustic shadows, and sidelobe effects, which are common in clinical practice but often overlooked in traditional augmentation, to improve model robustness and balance class representation. Our results show that this approach reduces junior physicians’ manual annotation workload by over 30% for small datasets (1,360 images) and achieves accuracy comparable to human annotators for larger datasets (6,800 + images), enabling fully automated preliminary labeling. These findings demonstrate how domain-specific AI training can dramatically lower costs and accelerate database expansion while maintaining clinical relevance. This method not only benefits thyroid nodule analysis but also provides a scalable framework for other ultrasound-based diagnostic tasks, bridging the gap between AI development and real-world clinical workflows.

Citation: Zheng F, XingMing L, JuYing X, MengYing T, BaoJian Y, Yan S, et al. (2025) Significant reduction in manual annotation costs in ultrasound medical image database construction through step by step artificial intelligence pre-annotation. PLOS Digit Health 4(6): e0000738. https://doi.org/10.1371/journal.pdig.0000738 Editor: Miguel Ángel Armengol de la Hoz, Fundación Progreso y Salud: Junta de Andalucia Consejeria de Salud y Familias Fundacion Progreso y Salud, SPAIN Received: January 8, 2025; Accepted: May 19, 2025; Published: June 30, 2025 Copyright: © 2025 Zheng et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: The data is available at the URL: https://www.scidb.cn/anonymous/VmZVWlZq DOI: 10.57760/sciencedb.23119 Funding: This work was supported by the Ningbo Natural Science Foundation (Grant No. 2022J321 to FZ). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The URL of the funder website is: https://kjj.ningbo.gov.cn/ Competing interests: The authors have declared that no competing interests exist.

1. Introduction The application of artificial intelligence (AI) in the medical field is booming, and AI-assisted medical image classification and recognition are gradually being deeply integrated into clinical practice, widely used in Computed Tomography (CT), Magnetic Resonance Imaging (MRI), pathological diagnosis, and ultrasound imaging-aided diagnosis [1–4], becoming a valuable assistant to doctors. AI models require targeted training to perform related tasks. AI model training methods are generally divided into unsupervised learning [5], semi-supervised learning [6], and supervised learning [7] based on whether supervision is present. For medical image classification tasks, the predicting results of supervised learning is closer to classification habits in clinical practice. Supervised learning requires medical images to be annotated in advance, rely on meticulously and professionally annotation work. The quality of image data annotation directly affects model performance [8]. The annotation of ultrasound medical image data often requires preliminary annotation by junior physicians followed by review by senior physicians to ensure the accuracy of annotation information. Therefore, doctors’ experience directly determines the quality of image annotation, and experienced doctors are scarce and busy with clinical work, resulting in high costs for manual annotation, which limits the scale of image databases. The performance of AI is related to the size of the training data set; thus, how to reduce manual annotation costs and rapidly expand the database scale is a critical issue affecting AI model quality. In medical imaging datasets, severe class imbalance poses a significant challenge to model training efficiency. Specifically, while common disease categories may account for over 30% of cases (e.g., benign thyroid nodules), rare pathological findings (such as medullary thyroid carcinoma) often constitute less than 0.1% of the total data set. This extreme distribution disparity forces models to prioritize majority class features during optimization, leading to compromised recognition capability for under-represented categories [9,10]. Studies have shown that data augmentation can effectively improve model training efficiency [11]. These augmentation methods are widely used for images captured by cameras, but the generation principles of ultrasound image data differ from those of camera-captured images. Therefore, how to specifically augment ultrasound images is an urgent problem to be solved to enhance training effectiveness with a fixed database scale. Current research indicates that the accuracy of AI models in image annotation and classification can surpass manual annotation [12], making it a feasible solution to let AI models assist in annotation while learning. This study takes thyroid nodule ultrasound images as an example to explore the feasibility of using AI to pre-annotate images to save manual annotation costs when dealing with small sample sizes and significant quantity differences between classes in ultrasound image data.

4. Discussion This study demonstrates the feasibility of significantly reducing manual annotation costs in thyroid nodule ultrasound classification through a phased AI pre-annotation strategy combined with ultrasound-specific data augmentation techniques. These findings provide a novel technical pathway for efficient medical image database construction, particularly offering practical value for resource-limited small and medium-sized healthcare institutions. Compared to traditional end-to-end annotation paradigms, the iterative pre-annotation mechanism proposed in this study exhibits unique advantages. When the original data set reached 6,800 images, the model achieved classification accuracy very close to junior physicians (76.9% vs. 73.85%) (Fig 13 ). The junior physicians annotation accuracy is aligning with findings from Christiansen et al. [23] in ovarian cancer ultrasound research and findings from Cheng et al. [24] in hepatocellular nodular lesions research. However, our phased training strategy reduced the required junior doctors’ annotation volume for model optimization by approximately 30% at the first round and 100% at the fifth round(as the AI model replaced the junior doctor to annotate the images). Notably, ultrasound-specific augmentation methods—including simulated defocus, acoustic shadows and sidelobe artifacts—effectively addressed the limitations of conventional geometric transformations in medical image enhancement. Compared to the method used in Jingjiao’s universal ultrasound foundation model (USFM) research [25], our approach provides a novel framework for ultrasound image enhancement based on the physical mechanisms of ultrasound artifact generation. Additionally, in contrast to Generative Adversarial Networks (GANs) [26], our method achieves batch image generation with minimal computational resources. In this study, accelerated by an NVIDIA 4090 GPU, we generated over 10,000 images per minute. When compared to the Synthetic Minority Over-sampling Technique (SMOTE) [27], which synthesizes new samples by linearly interpolating neighboring minority-class instances in the feature space (potentially producing unrealistic synthetic images), our method generates images that better align with real-world ultrasound data in terms of anatomical plausibility and artifact patterns. The reduction in annotation workload revealed in this study carries dual clinical significance: 1) It alleviates the review burden on physicians, allowing greater focus on complex case diagnoses; 2) It establishes a sustainable framework for continuous database expansion. Our approach demonstrates superior marginal cost reduction while maintaining annotation quality (validated through senior physician review). Remarkably, while junior physicians showed <50% accuracy in identifying 4B/4C/5-class nodules, the augmented model improved sensitivity for these rare categories up to 31% (the 5-class nodules), suggesting AI’s potential value in detecting high-risk nodules. Despite the aforementioned strengths, this study has several limitations that warrant consideration. 1) Single-Center Data Source: The data were exclusively derived from a single hospital, limiting generalizability. Although images from 10 different type ultrasound machines across four brands were included to mitigate inter-device variability, this approach may not fully represent all commercially available systems. Future multicenter studies incorporating data from diverse institutions are planned to enhance reliability and external validity. 2) Static Image Analysis: The study focused solely on static 2D grayscale ultrasound images. While these provide high-resolution snapshots, dynamic video sequences—critical for real-world clinical lesion identification—were not evaluated. To address this gap, we intend to integrate video-derived frames into subsequent research to better simulate clinical workflows. 3) Limited Ultrasound Modalities: Only conventional 2D grayscale imaging was assessed, excluding advanced modalities such as shear wave elastography, contrast-enhanced ultrasound, and Doppler flow/power imaging. Whether our methodology applies to these techniques requires systematic validation in future investigations. 4) Generalizability of Augmentation Methods: The image augmentation strategies developed here may not extend to CT or MRI datasets due to fundamental differences in artifact generation mechanisms (e.g., beam-hardening artifacts in CT vs. susceptibility artifacts in MRI). Domain-specific adaptation will be necessary for cross-modality applications. 5)This study, using thyroid nodule classification in ultrasound images as an example, future research can explore similar tasks such as breast nodule classification, lymph node classification, and liver nodule classification in ultrasound images, Because they share the same imaging principles and artifact generation mechanisms.

5. Conclusion This study validates the feasibility of a closed-loop “AI-assisted annotation → human verification → model iteration” system, offering novel insights to overcome medical AI’s data bottleneck. Ultrasound image tailored Image data augmentation can reduce the impact of poor model training performance caused by imbalance across ultrasound image categories. Through step-by-step annotation, training, and using the step-by-step trained models to assist in image annotation and bounding box selection, the efficiency of pre-annotation by junior doctors can be improved. Ultimately, this approach can replace the pre-annotation work of junior doctors, significantly reducing the cost of data construction.

[END]
---
[1] Url: https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000738

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/