(C) PLOS One
This story was originally published by PLOS One and is unaltered.
 . . . . . . . . . .



VASCilia is an open-source, deep learning-based tool for 3D analysis of cochlear hair cell stereocilia bundles [1]

['Yasmin M. Kassim', 'Department Of Cell', 'Developmental Biology', 'University Of California San Diego', 'La Jolla', 'California', 'United States Of America', 'David B. Rosenberg', 'Samprita Das', 'Xiaobo Wang']

Date: 2026-01 

One notable feature of ZFT-Net is that it does not produce prediction fluctuation errors, which we named “switches”. This consistency in its predictions offers a high degree of confidence, making ZFT-Net our primary choice for VASCilia. As described in Sect 4.3 , the preparation of the data and the configuration of CNN played a crucial role in achieving these results. We speculate that ZFT-Net outperforms the other well-known networks because our images are downsampled to 256x256 resolution. Deeper networks tend to lose spatial characteristics in the process, which negatively impacts their performance. The architecture of ZFT-Net, on the other hand, is better suited to preserve these critical spatial features, resulting in better performance across all metrics.

Errors are color-coded and oriented to reflect the different cases in the confusion matrix. GT0, GT1, and GT2 correspond to the Pre-Cellular Zone (PCZ), Cellular Clarity Zone (CCZ), and Noise Saturation Zone (NSZ), respectively. At the bottom of the figure, we provide an example of the first subject’s prediction, visualized through a barplot. It shows a smooth transition in prediction from 0 (PCZ), to 1 (CCZ), to 2 (NSZ) without any fluctuation errors, except for two minor boundary errors that are likely due to annotator subjectivity. For ZFT-Net, Conv represents Convolution layers, BN stands for Batch Normalization, ReLU is the Rectified Linear Unit activation function, Pool refers to Pooling layers, and FC denotes Fully Connected layers.

The results highlight that ZFT-Net outperforms all other architectures across all evaluation metrics. Notably, ZFT-Net achieved a switch count of 0, meaning its predictions consistently transitioned from class 0 (pre-cellular zone) to class 1 (cellular clarity phase), and finally to class 2 (noise saturation zone), without fluctuations or misclassifications. This indicates the model’s robustness in maintaining a stable and accurate prediction sequence. Source data are provided in Supplementary Information S1 Data .xlsx.

In this section, we present the performance and evaluation of our custom network, ZFT-Net, along with comparisons to other commonly used architectures such as ResNet10, ResNet18, DenseNet121, and EfficientNet. Our goal was to classify image frames within each 3D z-stack into three zones: the pre-cellular zone (PCZ), the cellular clarity zone (CCZ) and the noise saturation zone (NSZ) to focus segmentation and analysis on the portion of the z-stack that contains clear cellular structures. The results shown in Table 1 and Fig 2 highlight the performance of the networks in metrics such as accuracy, precision, recall, specificity, F1 score and error rate, and prediction stability. We found ZFT-Net produced the best results. All metrics were averaged across all three classes and calculated using true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN) derived from the confusion matrix.

During inference, we corrected the orientation of the stereocilia bundles by predicting the rotation angle for each image in the tested stack using the pre-trained model. For each frame, the model produced class scores corresponding to possible angles of rotation. The predicted scores were averaged across all frames in the stack to ensure that the final decision accounted for the model’s predictions for every frame. The angle corresponding to the highest average score (class × 5 ° ) was selected as the final predicted orientation. To achieve proper alignment, the images were rotated by minus the predicted angle.

After data preparation and augmentation Sect 4.4 , our training dataset resulted in 72 folders (classes), each containing 1,831 images. We experimented with several networks, and DenseNet121 yielded the best performance; see Table 2 .

The orientation of 3D cochlear image stacks is highly variable due to inconsistencies in imaging parameters, tissue handling, and the complex spiral structure of the cochlea. Automated orientation of cochlear hair cell image stacks can significantly accelerate batch processing, which is essential for efficiently handling large datasets. To address this challenge, we generated training data for PCPAlignNet, which aligns each 3D stack to orient the hair cell bundles along the planar polarity axis.

[A] Box plot comparing the IoU alignment of model predictions with Human 1 and Human 2, and the agreement between Human 1 and Human 2. Statistical tests indicate that the model predictions are significantly better aligned with both Human 1 (p = 0.0002) and Human 2 (p = 0.0000) compared to the agreement between the two annotators. Notably, there is no significant difference between the model’s alignment with Human 1 and Human 2 (p = 0.0879). [B] A density plot (KDE) estimates the distribution of IoU values by smoothing the data with small curves (kernels) at each point. The y-axis shows the density, representing the relative likelihood of observing the IoU values at different points on the x-axis. The density curves for Model vs. Human 1 and Model vs. Human 2 show considerable overlap, indicating similar IoU distributions between the model and both annotators. In contrast, Human 1 versus Human 2 shows less overlap, suggesting greater variability between the annotators’ manual segmentation.

The dataset has been annotated by five different human annotators. To put the accuracy of our predictions in a more meaningful context, we investigated the margin of error between annotators on the same stack. We found that the average IoU for 47 instances from a single stack between two annotators is 0.70. Strikingly, the model achieved a higher average IoU of 0.76 with the first annotator and 0.74 with the second annotator. We speculate this is due to the model’s ability to effectively average the experience of multiple annotators (five annotators in our case). These results also highlight the difficulty of achieving accuracies beyond a certain amount, as even two human expert annotators only agreed 70% on pixel-based 3D annotations. In addition, Paired t-tests revealed that the model’s predictions are significantly better aligned with both Annotator 1 (t = 3.7399, p = 0.0002) and Annotator 2 (t = 7.8134, p = 0.0000) compared to the agreement between the two annotators. No significant difference was found between the model’s alignment with Annotator 1 and Annotator 2 (t = –1.7431, p = 0.0879), indicating that the model generalizes equally well across both sets of annotations ( Fig 7A and 7B ). We speculate that the model may be averaging the annotators’ varying styles effectively, and that these results underscore the robustness of the model in producing reliable and consistent predictions, which, in some cases, surpass the level of agreement between human expert annotators.

We examined the margin of error between two annotators for a single 3D stack over 47 instances. The average overlap was 0.70 between two annotators, 0.74 between one annotator and the prediction, and 0.76 between the second annotator and the prediction.

Fig 6. Comparison of raw crops (first row), GT masks for the first human annotator (second row), GT masks for the second human annotator (third row), and the Predicted masks (fourth row) for 10 different crops from the same stack, the IoU score underneath the crops indicate the score between the 3D GT masks and the 3D predicted masks.

The metrics are evaluated across two categories: typical cases that do not present extreme challenges and complex cases affected by factors such as contrast variations, noise, overlapping bundles, and close proximity between them. Source data are provided in Supplementary Information S1 Data .xlsx.

Fig 5A indicates that the average F1-measure and accuracy are 99.4% and 98.8%, respectively, at an IoU of 0.5 for typical test set. In contrast, Fig 5C reveals that at the same IoU level, the average F1-measure and Accuracy decrease to 95.7% and 91.9% for complex test set. Maintaining the F1 measure and accuracy above 90% demonstrates that the algorithm effectively handles the significant challenges present in the images. We can observe from Fig 5B and 5D that the total number of TPs remains high until the IoU reaches 0.8 and 0.72, respectively. Beyond these thresholds, there is a noticeable increase in both FPs and FNs. When you observe that TPs remain high until IoU thresholds of 0.8 and 0.72, this indicates that the algorithm is quite effective at correctly identifying and matching relevant objects or features in the dataset up to these points. The increasing errors (FPs and FNs) at higher IoU thresholds indicate that the algorithm struggles with precision and recall balance as criteria become more strict. Fig 6 showcases ten crops of stereocilia bundles: the first row displays the raw crops, the second and third rows features ground truth manual annotations made by two human annotators in the open-source Computer Vision Annotation Tool (CVAT [ 36 ]), and the fourth row presents our predicted 3D volumes for each bundle. It is evident that most bundles achieve IoU scores between 0.7 and 0.8, which visually suggest near-perfect alignment.

After completing the training process, we employed the trained model to segment each bundle in the images from the testing set. To evaluate performance, we calculate the Intersection over Union (IoU) for 3D volumes by comparing predicted and ground truth segmentation masks. It is defined as , where represents the volume of the intersection of the two volumes, and ∪ represents the volume of their union. In addition, we calculate the F1 measure, accuracy, precision, and recall for the predictions against the ground truth. The evaluation process iterates through unique labels found in both ground-truth and predicted volumes, excluding the background label, and computes the IoU for the corresponding labels. This method tracks TP, FP, and FN across varying IoU thresholds. For instance, if one 3D bundle in the ground truth overlaps with two bundles in the predictions, the algorithm considers the one with the largest overlap as a TP and the unmatched one as a false positive. We perform evaluations across a range of IoU thresholds, from 0.1 to 1 in steps of 0.05. In object detection, we do not consider the True Negative (TN) because it refers to correctly labeling background pixels or non-object pixels as such. Since the background or non-object areas can be vast, focusing on TNs would skew the performance metric towards the most common class (background), which does not provide useful information about the model’s ability to detect objects of interest.

The variation in IDs across frames reflects the independent detection process in each 2D frame. Subsequently, a multi-object assignment algorithm intervenes to reconcile these IDs, effectively re-assigning them to maintain consistency across frames. This step is crucial for reconstructing accurate 3D objects, ensuring that each bundle retains a consistent ID throughout all frames.

Panel D presents a scenario where intra-row bundles are tightly clustered; however, the algorithm efficiently distinguishes each bundle, highlighting its ability to resolve complex spatial relationships within the samples. In Panel E , despite the raw image being notably dark, the algorithm remains effective in detecting and segmenting individual bundles.

We partitioned the dataset into training, testing, and validation groups at the stack level to avoid data leakage and mixing frames from different stacks. The training set includes 30 stacks, while the validation contains 5 stacks. Ten stacks are withheld for testing and evaluation. These stacks are categorized into two types: typical and complex cases. Six stacks fall into the typical category, featuring images that are easier to segment due to less overlap among the bundles and well-separated bundle rows. The remaining four stacks are classified as complex cases. These contain structures that pose significant challenges for segmentation due to various factors. The objective of this partitioning is to evaluate the decrease in performance when processing challenging cases and check the robustness of the detection algorithm. Fig 3A – 3E illustrate the model’s proficiency in accurately detecting the cells and demonstrate the model’s effectiveness in overcoming the inherent challenges associated with this task. As a representative single-stack example, Fig 4 shows 13 frames from one stack segmented in 2D; a multi-object assignment algorithm then reconstructs each 3D object. Details are provided in Sect 4.6 .

Our method utilizes 2D detection on individual frames, which is then followed by reconstructing the 3D object through a multi-object assignment algorithm. For detailed methodology, refer to Sect 4.6 .

2.4 VASCilia computational tools and measurements

The Napari plugin equips users with essential measurements and deep learning-based tools tailored for analyzing cochlear hair cell stereocilia bundles. Users can accurately obtain up to 15 different 2D and 3D measurements, including volume, surface area, and centroids of segmented regions. Beyond these fundamental measurements, the plugin includes specialized metrics critical for hair cell research, such as calculating stereocilia bundle heights, predicting the region of the cochlea from which the stack is taken (Base, Middle, Apex), classifying hair cells (e.g. inner vs. outer hair cell subtypes), measuring fluorescence signal within each bundle, and determining bundle orientation with respect to the planar polarity axis of the cochlea. These features are designed to support the most common goals of the hair cell imaging research community.

2.4.2 Fluorescence intensity measurements. Understanding signal levels whether they are one or multiple proteins or a particular stain is crucial in various research fields, including cochlear function. The plugin enables the user to get a precise quantification for the signal to allow researchers to gain deeper insights into its function and potential involvement in hearing loss. The plugin has a mechanism to calculate both the mean and total fluorescence intensity, subtracting the background intensity (optional) to correct for ambient noise. This is achieved by superimposing the 3D segmented masks onto the corresponding fluorescent image slices and aggregating the fluorescent intensities across the z-stack. The resulting intensities are normalized using max normalization and plotted, allowing for comparative analysis across different cell types such as inner hair cells (IHCs) and outer hair cells (OHCs) in various categories (OHC1, OHC2, and OHC3). Data for each cell type is stored and visualized through histogram that display the mean and total intensity distributions to provide insights and observation about the protein or signal expressions across the cochlear architecture. To enable further downstream analysis, users will obtain plots for each cell type, resulting in a total of 20 outputs per stack (10 plots and 10 CSV files). See Fig 11 for the total intensity histogram bars for all cells in three modes, illustrated in Sect 3, from the Napari plugin. The plugin can also be used to analyze and generate plots for other proteins in different channels as needed by the study. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 11. The plot demonstrates the utility and ease with which users can compare data between bar plots and actual cellular images within the plugin. Top: Segmented bundles; class legend (IHC WT = yellow, OHC WT = red). Green numbers mark each bundle’s ID. Middle: Per-bundle fluorescence (phalloidin) for three depth-aggregation modes: Mode 1:All layers (use each bundle’s full depth); Mode 2: Common depth only (truncate to depth across bundles); Mode 3: Pad to max depth (pad shallower bundles to depth by repeating the weakest-layer value, then sum). Bottom: Galleries from four example cells (shared same depth). Across cells, IHC bundles show consistently higher fluorescence than OHC bundles, reflecting greater F-actin signal. Source data are provided in Supplementary Information S1 Data.xlsx. https://doi.org/10.1371/journal.pbio.3003591.g011

2.4.3 Case Study 1: Differences in stereocilia bundle heights and fluorescence signals (phalloidin and anti-EPS8, WT vs. Eps8 KO). Analysis of bundle heights and phalloidin signal across cochlear regions and hair cell types in neonatal P5 WT and Eps8 KO mice (Fig 10A): To challenge our approach and demonstrate its utility, we conducted a study involving 18 cochlear stacks from neonatal P5 mice, with samples from three wild type (WT) animals and three Eps8 KO mice, which have previously been shown to have abnormally short stereocilia [37]. Each animal’s cochlea was analyzed across three regions (Base, Middle, and Apex), and within each region, we studied two cell types (IHCs and OHCs). See Fig 10A (top row) for a detailed analysis of stereocilia bundle height variations between WT and KO. The violin plots were generated using data processed through the VASCilia plugin. Consistent with prior studies, VASCilia showed a gradient of increasing bundle heights along the base-mid-apex tonotopic axis for both WT and Eps8 KO mice, with Eps8 KO mice showing significantly shorter bundles than WT. S1 and S2 Tables summarize bundle height distributions (mean, SD, median, N) and the pairwise tonotopic differences. They show a robust Base→Middle→Apex increase in WT (IHC/OHC) and KO OHCs, with the only non-significant step being Base vs. Middle in KO IHC (Holm-adjusted tests). See Fig 10A (bottom row) for the results of our challenge case study on the accumulated phalloidin signal between WT and Eps8 KO mice in three regions (Base, Middle, and Apex) for two cell types (IHC and OHC). Intensities were computed as the 3D sum within each bundle and global Min-Max normalized across all WT+KO samples to place values on a scale of 0 to 1. In this analysis, we applied mode 2, discussed in Sect 3, separately to each dataset. For a given data set D, we set n to the largest depth shared by all bundles within D; equivalently, . This dataset-specific choice of n ensures that within each data set, all bundles contribute the same number of layers. The IHC WT groups showed the highest normalized intensities in the Base region (median 0.84), with a tonotopic gradient of decreasing intensity from Base to Apex across all groups. KO groups exhibited a significant reduction in intensity compared to WT, with the OHC KO Apex group showing the lowest median intensity of 0.11. See S3 and S4 Tables, which report descriptive statistics (mean, SD, median, N) and pairwise contrasts (Δ, 95% CI, Hedges’ g, Holm-adjusted p-values); significant comparisons are indicated. Analysis of anti-EPS8 signal in mid-cochlear inner hair cells of mature P30 WT and Eps8 KO mice (Fig 10B): Quantitative analysis of stereocilia bundle morphology and anti-EPS8 signal in postnatal day 30 (P30) inner hair cells (IHCs) revealed pronounced structural and molecular differences between wild type (WT) and Eps8 knockout (KO) mice. WT bundles (n = 14) exhibited a mean length of 3.59 ± (SD; range: 3.12–4.28 ), whereas Eps8 KO bundles (n = 15) were significantly shorter, measuring 1.22 ± (SD; range: 0.90–1.53 ; Welch’s t = 25.89, df = 21.29, p = 1.4 × 10−17). To evaluate bundle-specific anti-EPS8 labeling intensity, total fluorescence intensity was quantified within the same segmented IHC bundles. WT bundles exhibited a mean total intensity of 0.616 ± 0.059 (SD), while Eps8 KO bundles showed markedly lower values of 0.094 ± 0.012. Normalization was performed by dividing each bundle’s total intensity by the maximum total intensity observed across both WT and KO groups. This reduction was highly significant (Welch’s t = 8.63, df = 14.16, p = 5.1 × 10−7). These findings demonstrate that the loss of Eps8 leads to a substantial decrease in both the length of the stereocilia and the anti-Eps8 signal, consistent with its essential role in the development and maintenance of the stereocilia F-actin architecture.

2.4.4 Case Study 2: Fluorescence-based bundle texture analysis and classification in Cdh23−/− and Cdh23+/− hair cells. Quantifying bundle disorganization (Cdh23−/− vs. Cdh23+/−) (Fig 12A): Here we present an illustrative case study demonstrating VASCilia’s utility beyond bundle-height and intensity measurements. We analyzed 3D image stacks from Cdh23v-6J/v-6J mice, referred here as Cdh23−/− [39] to quantify texture differences (Gray-level Co-occurrence Matrix (GLCM) and Local Binary Pattern (LBP)) between genotypes. We compared heterozygous controls (Cdh23+/−; n = 187 crops) with homozygous knockouts (Cdh23−/−; n = 176 crops). Cdh23+/− bundles were well organized, whereas Cdh23−/− bundles were visibly disorganized, consistent with the expected phenotype. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 12. This figure presents the texture analysis displayed at the top as panel (A), and the anti-CDH23 fluorescence quantification analysis included at the bottom as panel (B). Source data are provided in Supplementary Information S1 Data.xlsx. (Panel A) [Top row]: t-SNE of texture features [I to IV] and PCA projection show clear genotype separation in feature space, with crops from the two genotypes providing visual representation. (Panel A) [I to V]: GLCM energy by genotype (Cdh23−/− bundles have lower energy than Cdh23+/−, reflecting less uniform, more complex textures; differences are statistically significant); GLCM correlation by genotype (Cdh23−/− bundles exhibit lower correlation than Cdh23+/−, indicating weaker spatial regularity; differences are statistically significant); GLCM contrast by genotype (Cdh23−/− bundles show higher contrast than Cdh23+/−, consistent with greater local intensity variation; differences are statistically significant); GLCM homogeneity by genotype (Cdh23−/− bundles show reduced homogeneity relative to Cdh23+/−, consistent with more uneven local texture; differences are statistically significant); and Random forest feature importance highlights key GLCM/LBP drivers. (Panel B): Anti-CDH23 immunofluorescence quantification analysis for P7 IHC and OHC stereocilia bundles from a mid-cochlear location. Each data point represents a value obtained from an individual bundle, normalized to the maximum value within the dataset. Exemplar images on the right show stereocilia bundles stained with phalloidin (white) and anti-CDH23 antibody (magenta), with the automatically generated stereocilia bundle mask overlaid. https://doi.org/10.1371/journal.pbio.3003591.g012 To quantify these differences, we extracted texture features from segmented bundle crops using VASCilia. Specifically, we computed GLCM features—contrast, correlation, energy, and homogeneity—and LBP [40] histograms for each crop. Cdh23−/− samples had higher median contrast (136.88 vs. 74.76), consistent with more heterogeneous texture; lower correlation (0.98 vs. 0.99), reflecting weaker spatial regularity; lower energy (0.0147 vs. 0.0157), indicating less uniformity; and lower homogeneity (0.1368 vs. 0.1782), consistent with greater local contrast. Two-sided Mann–Whitney U tests showed highly significant differences across all GLCM features (p<0.0001). Several LBP bins (0, 1, 2, 5, 6, 7, 9) also differed between genotypes (p<0.01), reflecting shifts in local texture patterns. Dimensionality reduction (PCA and t-SNE) visually revealed a clear separation between genotypes in feature space. To quantitatively validate this observation, we trained a random forest on the extracted features (training n = 290; test n = 73), which achieved 94.5% accuracy on held-out crops (4/73 misclassified). The most important features aligned with those that were statistically significant (Fig 12A); representative examples are shown in S5 Fig. Anti-CDH23 fluorescence quantification analysis (Cdh23−/− vs. Cdh23+/− (Fig 12B): To examine how loss of cadherin-23 affects anti-CDH23 staining, we quantified total fluorescence intensity in IHCs and OHCs bundles from Cdh23−/− and Cdh23+/− mice. Data were obtained from 7 Cdh23−/− and 6 Cdh23+/− datasets, all from the mid-cochlear location. Values were min–max scaled using the global all cells range across both genotypes and cell types. As shown in Fig 12B, across all bundles, Cdh23−/− cells (n = 112) measured 0.202 ± 0.012 SEM, whereas Cdh23+/− cells (n = 76) had a mean normalized intensity of SEM. By cell type, IHC bundles were highest: Cdh23−/− (n = 34) SEM vs. Cdh23+/− (n = 24) 0.621 ± 0.040 SEM. OHC bundles showed the same trend: Cdh23−/− (n = 78) 0.137 ± 0.007 SEM vs. Cdh23+/− (n = 52) SEM. These results demonstrate a robust, genotype-dependent reduction in anti-CDH23 bundle fluorescence in Cdh23−/− hair cells across both IHCs and OHCs, consistent with cadherin-23 being required to maintain normal stereocilia organization and F-actin integrity in the mid-cochlea.

2.4.5 Tonotopic position prediction (BASE, MIDDLE, APEX). One of the fascinating features of cochlear hair cell bundles is the so-called “tonotopic” organization, wherein the bundle heights follow a pattern of increasing lengths as a function of position along the length of the cochlea, see Fig 1. Since the frequency of sound detected follows a similar pattern along the length of the cochlea, we were motivated to generate a model that can accurately assess the tonotopic position of the imaged region of the cochlea. To conduct this task, we trained a classification CNN. For more details about the training and architecture, see Sect 4.8. This method has demonstrated robust performance, achieving a subject-based accuracy of 97%; 28 out of 29 stacks were correctly identified. The single observed misclassification in our testing data involved a stack from the MIDDLE being incorrectly predicted as BASE, which we speculate was due to the close proximity and overlapping characteristics of these two cochlear regions. The covariance matrix for the regions BASE, MIDDLE, and APEX is in Fig 13. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 13. left: covariance matrix for model prediction, others: Grad-CAM Visualization for Base, Middle, and Apex; Resized and Overlaid Response from the Last Convolutional Layer Highlighting Focus on Bundles During Decision Making. Source data are provided in Supplementary Information S1 Data.xlsx. https://doi.org/10.1371/journal.pbio.3003591.g013 To gain insights into the focus areas of our model during prediction, we employed Gradient-weighted Class Activation Mapping (Grad-CAM), a powerful visualization tool used to explain deep learning-based predictions. Grad-CAM helps visually identify which parts of an image are pivotal for a model’s decision-making process. Grad-CAM works by capturing the gradients of the target label, which is the correct class in this scenario, as they propagate back to the final convolutional layer just before the softmax. It then uses these gradients to weigh the convolutional feature maps from this layer, generating a heatmap that visually emphasizes the image regions most critical for predicting the class label. Grad-CAM visualization is shown in Fig 13.

[END]
---
[1] Url: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3003591

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/