(C) PLOS One
This story was originally published by PLOS One and is unaltered.
 . . . . . . . . . .



MMSpa is a deep learning-based tool that enhances the identification of spatial domains in spatial transcriptomics studies [1]

['Yi Liu', 'Institute Of Fundamental', 'Frontier Sciences', 'University Of Electronic Science', 'Technology Of China', 'Chengdu', 'Yixiao Zhai', 'Pinglu Zhang', 'Zhongguancun Academy', 'Beijing']

Date: 2026-01 

Spatial transcriptome (ST) technologies have transformed the study of tissue structure by retaining the spatial distribution of gene expression. One major challenge in accurately identifying spatial domains is to extract domain-related information from spatial locations and gene expression. Here, we propose MMSpa, a masked graph attention autoencoder framework specifically designed to improve spatial domain identification. MMSpa incorporates an edge-removal strategy to construct an enhanced spatial graph to fundamentally address cross-domain interference and characterize clearer domain boundaries. By focusing on masked gene expression reconstruction, MMSpa learns stable latent representations that capture core biological features, facilitating the identification of similar spatial subdomains and detecting domain differences across biological samples at the same developmental stage. Comparative analyses using ST datasets from multiple ST technologies and platforms demonstrated that MMSpa outperforms existing methods across various accuracy assessments. Notably, MMSpa excels in challenging scenarios involving highly heterogeneous and complex tissues, and can reveal finer-grained functional tissue domains obscured by other methods. This superior capability positions MMSpa as a powerful tool for uncovering new biological insights and compensating for the lack of spatial annotation in histopathology.

Funding: The work was supported by the National Natural Science Foundation of China (Nos. 62131004, 62450002, 62425107 to Q.Z., No.62573090 to X.L.), Zhejiang Provincial Natural Science Foundation of China (No. LD24F020004 to Q.Z.). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data Availability: All spatial transcriptomics datasets were obtained from publicly available repositories, with detailed sources provided in S1 Table . Python source code for MMspa is available on GitHub ( https://github.com/LYxiaotai/MMSpa ) and Zenodo (DOI: https://doi.org/10.5281/zenodo.15385685 ). Reproducible analysis code and the source data underlying the figures are openly accessible via Zenodo (DOI: https://doi.org/10.5281/zenodo.17451775 ).

Copyright: © 2026 Liu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

To this end, we proposed MMSpa, a graph attention (GAT) autoencoder framework featuring two masking strategies: masked feature reconstruction and re-mask decoding. The GAT module in MMSpa enables adaptive learning of local spatial neighbors, while the two masking strategies enhance model robustness, resulting in stable latent representations that capture more core biological information. Additionally, MMSpa adopts an edge removal strategy to construct an enhanced spatial graph, which can also be considered the noisy edges masking strategy, making the spatial graph more specific for the domain identification task and facilitating clearer delineation of domain boundaries. By employing mask strategies and integrating gene expression data with the enhanced spatial graph, MMSpa learns stable latent representations that improve spatial domain identification and downstream analyses, such as tissue structure visualization, Uniform Manifold Approximation and Projection (UMAP) visualization, spatial trajectory inference, pseudotime analysis, and discovery of domain-specific marker genes. We benchmarked MMSpa against nine advanced methods using 21 ST datasets generated by 10× Visium [ 1 ], Stereo-seq [ 2 ], STARmap [ 3 ], osmFISH [ 4 ], and MERFISH [ 5 ] platforms with different spatial resolutions. MMSpa consistently outperforms nine existing methods across various accuracy assessment metrics. When applied to human breast cancer datasets, MMSpa successfully distinguishes between tumor and healthy regions, revealing a similarity in the biological state between the biological states of the tumor edge and the surrounding healthy tissue. More importantly, we demonstrated MMSpa’s superior ability in uniquely identifying functional regions obscured in other methods at finer scales, whether applied to healthy or diseased tissues, which directly enhances MMSpa’s potential for uncovering new biological insights.

Recently, the spatial methods based on graph neural networks (GNNs) have gained attention, such as SpaceFlow [ 20 ], conST [ 21 ], GraphST [ 22 ], SEDR [ 23 ], STAGATE [ 24 ], MAEST [ 25 ], SpaMask [ 26 ], and stCMGAE [ 27 ]. They begin by constructing a spatial graph that incorporates spatial information, and then designing different GNN modules to learn low-dimensional latent representations that integrate both spatial information and gene expression. These representations are subsequently utilized for domain identification and downstream analyses. However, these GNN-based methods struggle to simultaneously consider the following key points for domain identification: (1) constructing a spatial graph with well-defined neighborhoods for each spot. Most existing methods rely solely on the spatial proximity principle, which may not be well-suited for the specific requirements of the domain identification task and can limit the ability to accurately characterize domain boundaries, especially in highly heterogeneous or highly complex tissues, (2) ensuring the stability of latent representations, as a good representation is expected to stable with more core biological information (See “ Discussion ”), and (3) adaptively learning feature similarities among neighboring spots, capturing deeper biological features embedded in highly sparse ST datasets for identifying fine-grained sub-domains. Overall, accurately identifying spatial domains remains a significant challenge.

Spatial transcriptome (ST) sequencing technologies (e.g., 10× [ 1 ], Stereo-Seq [ 2 ], STARmap [ 3 ], osmFISH [ 4 ], MERFISH [ 5 ], etc.) enable access to gene expression at different spatial locations within tissues [ 6 ]. The additional spatial location information provided by ST data allows tissues to be segmented into specific spatial regions, representing higher-order tissue structures or spatial domains with similar gene expression patterns and spatial coherence [ 7 – 10 ]. Identifying spatial domains has become a fundamental initial step in ST data analysis [ 11 – 15 ] and is crucial for downstream analysis [ 8 ], such as visualizing tissue structures [ 16 ], discovering domain-specific marker genes [ 17 ], and exploring spatial features in development and disease [ 18 , 19 ].

Results

Overview of MMSpa MMSpa first adopts an edge removal strategy to construct an enhanced spatial graph by using spatial coordinates and the spatial gene expression matrix. Specifically, MMSpa constructs an initial spatial graph based on spatial coordinates, connecting spots that are physically close to each other. Simultaneously, MMSpa constructs an opponent spatial graph, based on the distance calculated by the spatial gene expression matrix, and spots furthest from the center spot are connected to the center (Fig 1A). Then, the common edges between the initial spatial graph and the opponent spatial graph are considered noisy edges. MMSpa removes these noisy edges from the initial spatial graph to obtain the final enhanced spatial graph (Fig 1A and 1B) (see “Methods”). The edge removal strategy is specifically designed to better characterize domain boundaries, as boundary spots and their physical neighbors may not necessarily belong to the same domain. By removing noisy edges from the initial spatial graph, the enhanced spatial graph can be more specific for the domain identification task and help characterize clearer boundaries (See “Discussion”). PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 1. Overview of MMSpa. (A) MMSpa begins with the construction of the enhanced spatial graph. The initial spatial graph is constructed based on spatial coordinates. Simultaneously, the opponent spatial graph is constructed based on spatial gene expression. By removing the common edges on the initial and opponent spatial graphs from the initial spatial graph, an enhanced spatial graph can be obtained. This enhanced spatial graph serves as the input for subsequent steps. (B) The enhanced spatial graph and spatial gene expression are used as inputs. Initially, a masking strategy is applied to obtain masked spatial gene expression. MMSpa then feeds the enhanced spatial graph and masked spatial gene expression into a graph attention encoder. Subsequently, a re-masking strategy is employed to obtain masked latent representations. Finally, MMSpa reconstruct the gene expressions of the initially masked spots. (C) The obtained latent representations can be applied to spatial domain identification and other downstream analyses. MMSpa also has unique potential for uncovering new biological insights. https://doi.org/10.1371/journal.pbio.3003580.g001 Subsequently, MMSpa randomly chooses a proportion of spots and masks their gene expression. The enhanced spatial graph and the masked gene expressions are then fed into a GAT encoder to generate latent representations. Then, MMSpa re-masks a proportion of spots’ latent representations. Based on these masked latent representations and a GAT decoder, MMSpa reconstructs the gene expressions of the initially masked spots (Fig 1B). Unlike existing methods, this masked feature reconstruction strategy enables MMSpa to learn stable latent representations and capture essential biological information embedded in dependencies and features among spots (See “Discussion”). Furthermore, the re-mask strategy enhances MMSpa’s learning capability (See “Discussion”). Utilizing GAT modules, MMSpa adaptively learns the varying importance between spots with their neighbors based on local spatial context. Ultimately, the obtained latent representations can be used for domain identification, UMAP visualization, trajectory inference, and Pseudo-Spatiotemporal Map (pSM) analysis (Fig 1C). The uniquely identified functional domains by MMSpa directly enhance its potential for discovering new biological insights, including finding differentially expressed genes that other methods miss, and discovering new enriched biological processes and pathways.

Benchmarking MMSpa with nine existing methods We conducted a benchmarking analysis to compare the performance of MMSpa with 9 existing methods (SpaceFlow [20], conST [21], GraphST [22], SEDR [23], STAGATE [24], stCMGAE [27], SpaMask [26], SpaDo [28], and MAEST [25]) by using the classical human dorsolateral prefrontal cortex (DLPFC) dataset [10] (S1 Table). The DLPFC dataset has a total of 12 slices, each slice has clear boundaries and has been previously annotated into four or six cortical layers along with white matter (WM) [10]. First, we quantitatively assessed the accuracy of MMSpa and nine other methods in identifying spatial domains. Considering the annotations as ground truth, we compared domain identification results to the ground truth using three accuracy assessment metrics, including the adjusted rand index (ARI), normalized mutual information (NMI), and Purity (See “Methods”). These metrics evaluate the similarity between the domain identification results and the expected annotations, with higher scores indicating greater accuracy. Across all 12 slices, MMSpa achieved higher median scores than all nine compared methods for all three metrics (Fig 2A). PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 2. Benchmarking MMSpa with nine existing methods in the human DLPFC tissues. (A) Boxplots show the quantitative performance of MMSpa compared to the nine methods in domain identification accuracy across all 12 DLPFC tissue slices. The x-axis of each boxplot displays the ARI, NMI, and Purity scores, respectively. (B) Visualization of manual annotations for slice 151674. (C) Domain identification results for slice 151674 obtained by MMSpa and the other nine methods. (D), (E), and (F) Show UMAP visualization, PAGA trajectory graph, and Pseudo-Spatiotemporal Map (pSM) for slice 151674 based on the latent representations from MAEST, SpaMask, stCMGAE, conST, and MMSpa. Other methods’ visualizations are provided in S1B Fig. The underlying data for this figure can be found at https://doi.org/10.5281/zenodo.17451775. https://doi.org/10.1371/journal.pbio.3003580.g002 To gain more details about the domain identification results, we examined the DLPFC slice 151674 with six layers and the WM (Fig 2B and 2C). We found that SpaceFlow and STAGATE only identified WM, and failed to accurately characterize the remaining layers. SEDR and GraphST were roughly close to the expected annotation layer shapes, but each had limitations: SEDR did not correctly recover Layer 1 to Layer 3, and GraphST exhibited serrated domain boundaries. Similarly, the boundaries between layers obtained by SpaDo also exhibit a jagged pattern. We further focused on the results of three other methods that also utilize the “masking” technique in GNNs. We observed that MAEST and SpaMask had different degrees of irregular patchy contamination between layers (such as Cluster 5 and Cluster 7 in MAEST, Cluster 7 in SpaMask), resulting in unclear shapes and indistinct boundaries for each layer. In contrast, only stCMGAE and MMSpa were able to clearly delineate the cortical layers according to expected shapes, with each layer being accurately aligned in the tissue slice, especially for Layers 1–3. However, MMSpa demonstrated superior performance in predicting finer details of biological structures. Precise measurement of cortical layer thickness is crucial for diagnosing and studying various neurodegenerative and psychiatric conditions [29]. In the WM layer, which features a sharp inflection point, MMSpa accurately recovers this feature, while stCMGAE only predicts part of the WM region (Cluster 2) and the inflection point appears rounded. When combining different clusters, both stCMGAE and MMSpa can accurately predict the WM region (stCMGAE: Cluster 2 + Cluster 7, MMSpa: Cluster 5), as well as Layer 4/5 (stCMGAE: Cluster 3, MMSpa: Cluster 3), and Layer 6 (stCMGAE: Cluster 5, MMSpa: Cluster 2 + Cluster 7). In this instance, MMSpa still provides more precise biological structure predictions, especially the thickness predictions across different layers. In fact, the Layer 6 region is thicker, while Layers 4/5 are thinner. Compared to stCMGAE, MMSpa predicts the thickness of Layer 6 and Layer 4/5 more accurately, aligning better with the original biological structure. As expected, MMSpa also achieves a higher accuracy score. Interestingly, although the visual appearance of conST seems superior at first glance, it receives a lower accuracy score. The key distinction is that ARI, NMI, Purity assess the overall consistency between the predicted results and ground truth across the entire tissue section, while human vision tends to focus on large, uniform regions of color. As a result, the domain predictions made by conST, with its “large color block” effect, may give the impression of a “clear structure”. However, the global metrics that rigorously evaluate the alignment of the entire structure, which may differ from the initial visual impression. The higher score of MMSpa indicates that it performs better in terms of maintaining average consistency across all spots in the tissue slice. This is particularly noticeable in areas that may be less visually distinct, where the MMSpa can better capture finer biological details. For example, MMSpa generates much sharper and more compact boundaries between layers, while conST shows diffuse and fragmented ones. Additionally, MMSpa can more accurately recover the laminar thickness, such as Layer 1 (conST: Cluster 2, MMSpa: Cluster 4) and Layer 3 (conST: Cluster 6, MMSpa: Cluster 6). MMSpa also demonstrates superior capability in capturing the precise morphological features of cortical layers, such as the inflection point in the WM layer and Layer 2. Overall, although conST may give a clearer visual impression of layered structures at first glance, this may be a visual deception effect. Detailed biological discussions showed that MMSpa has significant advantages in global quantitative metrics (with an ARI approximately 28.6% higher than conST) (S1A Fig) and reconstruction of fine biological structures (Fig 2B). We believe MMSpa provides a more comprehensive and biologically relevant domain identification. The results for all other slices are shown in S2–S4 Figs. Then, we compared the latent representations obtained from different methods through different downstream analyses, including the UMAP, trajectory inference (PAGA) [30], and pSM analysis (see “Methods”). Here, we used slice 151674 as an example. Applying UMAP [31] to the latent representations, we generated the 2D visualization of all spots colored by the domain annotations (Figs 2D and S1B). The UMAP plots revealed that most existing methods could only distinguish spots in WM from those in other layers. Notably, STAGATE, SpaDo, and methods with the “masking” strategy achieved clearer separation of spots in different layers (except MAEST), consistent with the hierarchical structure of the cortical layers [32]. We found that they also correctly recovered the inside-out developmental sequence from Layer 1 to Layer 6 and accurately depicted the similarities between adjacent layers (Fig 2E). These findings align with the temporal order of cortical layer development and the characteristics of neighboring layers [32]. Subsequently, we used pSM to conduct a more detailed analysis of the cortex development (Figs 2F and S1B). We found that MMSpa, STAGATE, stCMGAE, and SpaMask presented a smoother color trend, better matching the chronological order of layer development (from WM, Layer 6 to Layer 1) (Fig 2F). However, neither stCMGAE nor SpaMask can accurately depict the shape of the transition from WM to layer 6, indicating the competitiveness of MMSpa when compared to other methods that also use masking strategies. Additionally, while STAGATE showed similar pSM patterns to MMSpa, it performed poorly in domain identification. In contrast, MMSpa excelled in both domain identification and other downstream analyses. Similar findings can be observed in slices 151672 and 151508 (S5–S7A Figs). Finally, we evaluated the robustness of MMSpa in comparison to the top five competitive deep learning methods (STAGATE, SEDR, GraphST, conST, and SpaceFlow) by varying the hyperparameter for the number of nearest neighbors (K), which controls the extent of local spatial smoothing and significantly influences model performance. Our results showed that, as K varied, the ARI scores of all six methods exhibited slight fluctuations across the 12 DLPFC slices. Notably, regardless of K, the median ARI score of MMSpa consistently outperformed the other five methods (S7B Fig). Overall, these comprehensive results underscore the superiority of MMSpa, it not only significantly enhances domain identification accuracy compared to existing methods but also better captures biological phenomena through its latent representations in downstream analyses.

Extension application for MMSpa to identify spatial domains in multiple spatial slices We conducted an extension application for MMSpa to identify spatial domains in multiple spatial slices. We performed additional preprocessing on the input data. Specifically, assume there are slices to be processed for the multi-slice domain identification task. First, we denote the gene expression matrices for each slice as to , and their spatial adjacency matrices as to . We then concatenate the gene expression matrices along the spot dimension to obtain a joint gene expression matrix : Similarly, we create a block-diagonal adjacency matrix from the spatial adjacency matrices: Finally, we use and as inputs for MMSpa to perform domain identification across the multi-slice dataset. We applied MMSpa for multi-slice domain identification on the DLPFC dataset. The DLPFC dataset consists of 12 slices from 3 adult samples, with 4 adjacent slices per sample: Sample1 with #151673, #151674, #151675, and # 151676.

Sample2 with #151669, #151670 #151671, and # 151672.

Sample3 with #151507, #151508 #151509, and # 151510. We performed multi-slice domain identification for each of the 3 sample groups. Among the methods we compared for single-slice domain identification, considering that SpaDo was initially designed for multi-slice domain identification, and GraphST, STAGATE, and SpaMask have also mentioned that they can be extended for multi-slice domain identification. We compared the performance of MMSpa in multi-slice domain identification with these methods by calculating domain identification accuracy metrics: ARI, NMI, and Purity (S5 Table). As shown in the results, MMSpa demonstrates the ability to identify spatial domains in multiple spatial slices. Specifically, for Sample 1, MMSpa outperforms the other methods in all three metrics. In the application of Sample 2, although MMSpa is lower than STAGATE by 2.7% in the NMI metric, it still outperforms in both ARI and Purity. For Sample 3, MMSpa maintains the highest accuracy in the Purity metric, with only a minor 0.5% difference in NMI compared to STAGATE. Notably, MMSpa and STAGATE show similar performance in the other two samples, with MMSpa performing slightly better. As for the ARI value, although the ARI value of MMSpa is slightly lower than SpaDo’s, it is worth noting that SpaDo’s better ARI performance in Sample 3 does not extend to the other samples. MMSpa outperforms SpaDo on all three metrics for the other two samples, with the largest accuracy difference reaching 11.7% (e.g., for Sample 1, MMSpa ARI = 0.590, SpaDo ARI = 0.473).

Ablation studies To verify the contributions of the main components in MMSpa, we conduct several ablation studies on six datasets with manual annotations from five platforms, including 10× Visium datasets (the DLPFC dataset with 12 slices and human breast cancer dataset), the Stereo-Seq dataset (E9.5 mouse embryo dataset from slice #E9.5_E2_S3), the STARmap mouse cortex dataset, the osmFISH mouse somatosensory cortex dataset, and the MERFISH mouse hypothalamus dataset. As shown in S16 Fig, we first tested whether the performance of MMSpa benefited from the edge removal strategy (S1 Note). The results show that the enhanced spatial graph provides a significant advantage over the initial spatial graph in each platform dataset application, with an absolute improvement of 3.6%–8.5% in ARI, 1.7%–5.8% in NMI, and 1.8%–3.6% in Purity of the six datasets. Considering that the DLPFC dataset includes 12 slices, we conducted a slice-wise analysis and observed that each slice’s three metrics improved to varying degrees (S2 Table). These improvements demonstrated the importance of the edge removal strategy in each platform dataset application, which plays a crucial role in enhancing the specificity of the spatial graph for the domain identification task. Particularly, in the application to the human breast cancer dataset, the median scores of the three metrics (ARI, NMI, and Purity) increased by 8.0%, 2.1%, and 3.4%, respectively, demonstrating the utility of this edge removal strategy in the identification of tissue domains with heterogeneous cell types (S16 Fig). We then examined the influence of the masking strategies (S1 Note). MMSpa employs two masking strategies: one that masks the gene expression matrix before the encoder (masked feature reconstruction), and another that re-masks the encoder’s output before the decoder. Removing the masking of the gene expression matrix before the encoder means that the model reconstructs the entire gene expression matrix. Our results showed that the accuracy of the six datasets from five platforms dropped by 6.0%–14.4% in ARI, 2.1%–13.4% in NMI, and 2.7%–8.5% in Purity when the gene expression matrix was not masked before the encoder (S16 Fig). This suggests that the denoising (masked) graph autoencoder framework with masked feature reconstruction significantly enhances domain identification performance in ST datasets with different resolutions. The improvement in domain identification accuracy for each 12 slices of the DLPFC dataset further supports this conclusion (S2 Table). For the re-mask strategy, the removal of this strategy caused various degrees of decrease in accuracy in each dataset from the five platforms. In particular, in the application of imaging-based platforms (STARmap, osmFISH, and MERFISH) ST datasets, the ARI, NMI, and Purity decreased by 2.0% to 10.9%, 1.0%–9.4%, and 2.6%–4.6%, respectively (S16 Fig). A similar trend could be found in each 12 slices of the DLPFC dataset (S2 Table), causing the median ARI, NMI, Purity scores of the DLPFC dataset decreased by 10.0%, 7.4%, and 4.9%, respectively (S16 Fig). The re-mask strategy can be regarded as a form of regulation, which can further encourage the encoder to learn the latent representations. Moreover, we investigated the impact of the edge removal strategy and masked feature reconstruction on the downstream analysis (UMAP, trajectory inference, and pSM analysis) of latent representations. We conducted the ablation studies on slices 151674, 151672, and 151508 of the DLPFC dataset. Compared to the edge removal strategy (S17A Fig), the masked feature reconstruction strategy had a more substantial impact on downstream analyses (S17B Fig). After removing the masked feature reconstruction strategy, all three downstream analyses were affected to varying degrees, leading to a reduction in the ability of latent representations to capture biosignatures, with trajectory inference and pSM being the most impacted (S17B Fig). This suggests that stable latent representations encompass more biometric features, which not only improve domain identification performance but also significantly contribute to downstream analyses, while the masked feature reconstruction strategy is essential for extracting more stable and core information from ST data.

[END]
---
[1] Url: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3003580

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/