(C) PLOS One
This story was originally published by PLOS One and is unaltered.
 . . . . . . . . . .



Time-series visual representations for sleep stages classification [1]

['Rebeca Padovani Ederli', 'Institute Of Computing', 'University Of Campinas', 'Unicamp', 'Campinas', 'Sp', 'Didier A. Vega-Oliveros', 'Department Of Science', 'Technology', 'Federal University Of Sao Paulo']

Date: 2025-06 

Polysomnography is the standard method for sleep stage classification; however, it is costly and requires controlled environments, which can disrupt natural sleep patterns. Smartwatches offer a practical, non-invasive, and cost-effective alternative for sleep monitoring. Equipped with multiple sensors, smartwatches allow continuous data collection in home environments, making them valuable for promoting health and improving sleep habits. Traditional methods for sleep stage classification using smartwatch data often rely on raw data or extracted features combined with artificial intelligence techniques. Transforming time series into visual representations enables the application of two-dimensional convolutional neural networks, which excel in classification tasks. Despite their success in other domains, these methods are underexplored for sleep stage classification. To address this, we evaluated visual representations of time series data collected from accelerometer and heart rate sensors in smartwatches. Techniques such as Gramian Angular Field, Recurrence Plots, Markov Transition Field, and spectrograms were implemented. Additionally, image patching and ensemble methods were applied to enhance classification performance. The results demonstrated that Gramian Angular Field, combined with patching and ensembles, achieved superior performance, exceeding 82% balanced accuracy for two-stage classification and 62% for three-stage classification. A comparison with traditional approaches, conducted under identical conditions, showed that the proposed method outperformed others, offering improvements of up to 8 percentage points in two-stage classification and 9 percentage points in three-stage classification. These findings show that visual representations effectively capture key sleep patterns, enhancing classification accuracy and enabling more reliable health monitoring and earlier interventions. This study highlights that visual representations not only surpass traditional methods but also emerge as a competitive and effective approach for sleep stage classification based on smartwatch data, paving the way for future research.

Funding: Part of the results presented in this work was obtained through the project “Hub of Artificial Intelligence in Health and Wellbeing - Viva Bem,” funded by Samsung Eletrônica da Amazônia Ltda., within the scope of the Information Technology Law 8.248/91. There was no additional external funding received for this study. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2025 Padovani Ederli et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

The transformation of time series into visual representations, combined with deep learning techniques, has proven effective for both classification tasks, highlighting the versatility of the proposed methodology. This strategy advances the understanding of sleep patterns, representing a promising avenue for future research.

This study addresses two distinct classification tasks: (1) two-stage classification (sleep/wake) and (2) sleep stages classification (wake/NREM/REM). The primary goal of binary classification is to distinguish between wake and sleep periods, making it useful for applications that require basic sleep detection, such as large-scale monitoring of sleep-wake patterns or the initial assessment of sleep disorders. On the other hand, the sleep stages classification aims to identify more detailed patterns by separating sleep into NREM and REM stages, which is essential for more in-depth clinical analyses, such as detecting specific sleep disorders (e.g., sleep apnea or insomnia) and assessing sleep quality based on the sleep-wake cycle architecture.

Although the transformation of smartwatch data into images is established in fields like HAR, its application in sleep stage classification remains underexplored. Addressing this gap, this study aims to classify sleep stages by leveraging smartwatch sensor data transformed into visual representations and applying deep learning techniques. The methodology uses the publicly available Sleep Accel dataset [ 25 ], containing accelerometer and heart rate data from Apple Watch devices, annotated with PSG-based sleep stages. Time series data were transformed into image representations, including RP, GAF, MTF, and spectrograms, to capture temporal and spatial patterns. Additionally, images were divided into patches, enabling the classification models to focus on local details. These models, combined with ensemble techniques, demonstrated improved prediction accuracy. Performance was evaluated using balanced accuracy and Cohen’s kappa coefficient. Comparisons with traditional methods, such as raw data models and feature extraction approaches, highlighted the advantages of visual representations.

Image-based methods have achieved remarkable results in human activity recognition (HAR). For instance, integrating GAF, RP, and MTF into a convolutional model enhanced gymnastics action recognition [ 22 ]. Similarly, spectrograms of inertial and biological signals have improved the classification of activity intensity levels [ 23 ]. Gesture recognition studies have also utilized GAF and MTF, achieving high accuracy in classifying wrist movements related to food intake [ 24 ].

Transforming time series into visual representations has proven to be an effective technique for sensor data analysis. This approach converts one-dimensional time series into two-dimensional images, allowing the direct application of deep learning models. Representations like Recurrence Plots (RP) [ 18 ], Gramian Angular Fields (GAF) [ 19 ], Markov Transition Fields (MTF) [ 19 ], and spectrograms capture diverse aspects such as temporal features, state transitions, and phase space representations [ 20 , 21 ].

The rise of 2D-CNNs has introduced a promising alternative. These networks are designed to detect local patterns, such as edges, textures, and shapes, making them highly effective for computer vision tasks. Pooling layers reduce data dimensionality while preserving crucial features, and the hierarchical structure of 2D-CNNs enables the analysis of visual patterns in two-dimensional data [ 17 ].

Despite advancements, existing approaches have limitations. Manual feature extraction often requires domain expertise, is sensitive to noise, and fails to capture complex relationships within the data [ 3 , 16 ]. In contrast, raw data processing struggles with high dimensionality, noise, and difficulty identifying temporal and spatial patterns, leading to compromised interpretability [ 3 , 14 , 15 ].

Studies have explored various devices and approaches to classify sleep stages. Single sensors, such as photoplethysmography (PPG), have been used to extract features and achieve promising results with classifiers like SVM [ 11 ]. Other studies combined wavelets with RF, revealing that variables like age and sleep periods influence performance [ 12 ]. Smartwatches equipped with accelerometers and PPG have demonstrated their potential as viable PSG alternatives by applying recurrent neural networks for classification [ 13 ]. Additionally, methods directly processing raw data with models such as LSTMs have shown success in analyzing activity and heart rate data [ 14 ]. Deep networks trained on multimodal PSG data further highlight the potential of end-to-end learning directly from raw inputs [ 15 ].

Sensor data, often organized as time series, have been widely used in various sleep-related applications [ 6 – 8 ]. A common approach involves extracting features from these time series and using classical machine learning algorithms like Support Vector Machine (SVM), K-Nearest Neighbors (KNN), and Random Forest (RF). With advancements in deep learning, two prominent methodologies for processing raw data have gained attention. The first uses Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM) [ 9 ] and Gated Recurrent Unit (GRU) [ 10 ], which are designed to capture temporal patterns and long-term dependencies. The second employs 1-dimensional convolutional neural networks (1D-CNNs), effective for learning local patterns in sequential data.

Polysomnography (PSG) is the gold standard for evaluating sleep stages in clinical settings, providing detailed data from multiple physiological signals. However, PSG presents notable challenges, including high equipment costs, patient discomfort, and the need for monitoring in controlled environments, which may disrupt natural sleep [ 4 ]. Smartwatches have emerged as a practical, less invasive alternative for home sleep monitoring. These devices, equipped with sensors such as accelerometers and heart rate monitors, enable continuous data collection, promoting habit adjustments and health improvements [ 5 ].

The sleep stages are essential for maintaining health and diagnosing sleep disorders. According to the guidelines of the American Academy of Sleep Medicine (AASM), sleep stages are classified into five distinct categories: wake, NREM N1, NREM N2, NREM N3, and REM, each with specific physiological characteristics [ 1 ]. Sleep cycles, composed of NREM and REM stages, occur every 90 to 120 minutes throughout the night and are critical for restorative sleep. Alterations in these patterns often indicate the presence of sleep disorders [ 2 ]. To simplify analysis, some studies group these stages into four (wake, light sleep, deep sleep, and REM), three (wake, NREM, and REM), or two stages (wake and sleep) [ 3 ].

Materials and methods

This section describes the publicly available database, methodology, and evaluation metrics adopted in this work.

Overview of visual representations for sleep stages classification A time series is a set of observations collected sequentially over time, according to a specific sampling rate. In this work, time series are obtained for the entire duration of the sleep recording, with sampling rates varying according to the sensor used. These data, which may have timestamps in different formats, are analyzed in periods of 30 seconds, called “epochs”. The time series of PSG recordings are standardized into 30-second epochs, each with a corresponding label for a sleep stage. Accelerometer data from smartwatches are also time series, measuring the device’s acceleration over time. The start and end times of the time series from different types of data (smartwatch and PSG) must be synchronized. This paper investigates using visual representations of smartwatch data to classify sleep stages. The approach involves methods that incorporate both the spatial and temporal aspects of the data, integrating accelerometer (ACC) and heart rate (HR) information from smartwatches. The classification of sleep stages was simplified into classifying sleep/wake (binary classification) and wake/NREM/REM (sleep stages classification). Notably, the data are analyzed in real-time, meaning only the data available up to the current moment are considered, demonstrating the practical application of the research. To provide a high-level overview of the proposed method, Fig 2 illustrates the general pipeline followed in this study. The process starts with raw time series data from smartwatches, which are transformed into visual representations. The final output consists of predictions for two scenarios: two-stage (wake/sleep) and sleep stages classification (wake/NREM/REM). PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 2. General pipeline of the proposed method. The process begins with time series data, which are transformed into visual representations. The final output consists of predictions for two scenarios: two-stage (wake/sleep) and sleep stages classification (wake/NREM/REM). https://doi.org/10.1371/journal.pone.0323689.g002 A more detailed methodology breakdown is presented in Fig 3, where the illustrated process is applied to both classification scenarios. The transformation of raw data into images was applied using techniques found in some related works: RP, GAF, MTF, and spectrograms. The images were generated from ACC data and HR rate data separately, and ensemble techniques were also performed to combine the classifications obtained with each data type. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 3. Proposed methodology scheme. The process begins with transforming raw accelerometer (ACC) data (pink) and heart rate (HR) data (blue) into visual representations. These visual representations serve as the input for training and validation. An ACC and HR data ensemble is created and validated (gray). The images are divided into patches used as inputs for their respective training sessions. Validation is carried out based on the ensemble results obtained from all patches. Finally, the ACC + HR ensemble is performed and validated again after obtaining the ensemble results from the patches (gray). https://doi.org/10.1371/journal.pone.0323689.g003 Using images to perform classification tasks allows the application of a technique that divides the original image into sub-images or patches. As shown in Fig 3, the images of different representations (RP, GAF, MTF, and spectrograms), in addition to being processed in the format in which they were initially generated, are also divided into patches. Ensembles are applied to combine the predictions of the models trained with the following inputs: original ACC and HR data, patches of ACC data, patches of HR data, and patches of ACC data and patches of HR data. The following subsections present the details of the methodology regarding data preparation, representations, patches, and ensembles.

Data preparation A measurement refers to a single data point collected from sensors. These measurements form a time series, a sequence of data points recorded at regular intervals. Before generating the images, the raw motion and heart rate data were interpolated. Each image was created using 600 measurements [25], resulting in 600 600 pixels, except for spectrograms, with varied dimensions. For processing as input into a network, the images are resized to 224 224 pixels. Each image corresponds to a 30-second window by the PSG recording standard used in the literature. Therefore, each image is classified as sleep or wake in the binary classification problem and as Wake, NREM, or “REM” in the three-stage classification. The different image representations were obtained for each accelerometer axis (x, y, and z), and an RGB image combining x, y, and z is generated to support information from all three axes. This strategy is addressed in related works [26], and it is essential to highlight that using images generated from individual axes does not take advantage of all the motion information [27]. Since the heart rate data consists of a single value (in bpm), the images are generated in grayscale (without performing RGB composition).

Patches The technique of dividing an image into sub-images, or patches, is a common approach in image visualization and machine learning applications [35, 36], primarily aimed at improving focus on local details of the image. This allows the model to learn finer features that might be missed when observing the original image. As Fig 6 depicts, each image generated by different representations (600 600 pixels or other dimensions, in the case of spectrograms) is divided into nine patches, and each patch (224 224 pixels, with approximately 16% overlap between patches to avoid losing local information) is treated as an independent input for the training process. After training, the patches can be regrouped to make predictions about the entire image. A model is trained on each patch to predict the class of the corresponding patch region. Then, these predictions are combined to obtain the complete segmented image prediction. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 6. “Wake” images generated with GAF and accelerometer data. Example of an original image (600 600) and examples of patches (224 224 each). https://doi.org/10.1371/journal.pone.0323689.g006

Ensembles Ensemble techniques combine the predictions of multiple models to improve the robustness and accuracy of the final prediction, aiming to leverage individual decisions and mitigate the drawbacks of each model. In this context, an ensemble uses the predictions of models trained with: original images of accelerometer data + original images of heart rate data: two models (one for each type of sensor);

patches of accelerometer data: nine models (one for each patch);

patches of heart rate data: nine models (one for each patch);

patches of accelerometer data + patches of heart rate data: 18 models (nine models for accelerometer data patches + nine models for heart rate data patches). These techniques are helpful in complex problems where more than a single model may be required to capture all the nuances of the data [37]. The ensembles of simple averaging, weighted averaging, simple network, and deep features are described below. Simple averaging. In the simple averaging ensemble, the predictions of each classifier are combined by calculating the arithmetic mean of the predictions. For the binary classification problem, this means calculating the mean of the predicted probabilities for the classes “Sleep” and “Wake” by the classifiers. For the three-class classification, the mean of the predictions for each class is calculated individually. This type of ensemble was applied to combine the predictions of models trained with: original images from accelerometer data + original images from heart rate data;

patches from accelerometer data;

patches from heart rate data; and

patches from accelerometer data + patches from heart rate data. Weighted averaging. Weighted averaging is a variation of the simple averaging method, where each classifier’s prediction contributes a different weight to the final prediction. The weights are usually assigned based on each classifier’s performance. We generated 1000 random sets of n weights to find the best combination in each case. This type of ensemble was applied to combine the predictions of models trained with: original images from accelerometer data + original images from heart rate data (n = 2 number of sensors);

patches from accelerometer data (n = 9 number of patches);

patches from heart rate data (n = 9 number of patches); and

patches from accelerometer data + patches from heart rate data (n = 9 Simple network. In this method, the classifiers’ predictions are input to a simple neural network, which learns the best way to combine these predictions. The predicted probabilities for the classes of interest by a classifier trained with patches are the inputs to this network, providing the final prediction. To maintain only one simple network, this ensemble was applied only with the predictions of models trained with: patches from accelerometer data; and

patches from heart rate data. Using models trained only with patches, multiple predictions contribute to a more informative input, unlike using accelerometer and heart rate data, where only two models would contribute. This makes a simple network less advantageous for combining only two predictions due to the additional effort not justified by the problem’s complexity. Deep features. In this type of ensemble, feature vectors are extracted from the deep layers of each classifier and combined to be used as input for a final model. This final model is trained to make the final prediction using these combined features, leveraging the data representations provided by the different classifiers. This type of ensemble was applied to combine the predictions of models trained with original images from accelerometer data + original images from heart rate data. Although the diversity of information provided by the various deep feature vectors from patches can be informative, it also introduces significant complexity to the modeling process. This complexity manifests in the data dimensions to be processed. Working with only two deep feature vectors generated from accelerometer and heart rate data simplifies the modeling process.

Training and validation For training, we employed transfer learning using the EfficientNet-B0 model, a specific variant of the EfficientNet family [38], pre-trained on the extensive ImageNet dataset [39]. The strategy includes freezing some of the initial layers of these networks to preserve the learned generic features while the deeper layers are adapted to the specific dataset. This adaptation was carried out by adding a dense network at the end of the architecture, a process known as fine-tuning, allowing fine adjustments of the network parameters to fit the classes of interest better. EfficientNet-B0 was chosen due to its balance between high accuracy and computational efficiency, making it well-suited for tasks involving image classification. Additionally, other architectures, such as ResNet network [40], were also tested. However, EfficientNet-B0 consistently provided better performance in terms of accuracy and training time for the specific dataset used in this study. The decision to freeze 90% of the initial layers of the EfficientNet-B0 network was based on a layer-freezing experiment. Different percentages of layers were frozen, ranging from 50% to 100%, and their impact on model performance was analyzed. Freezing 90% of the layers yielded the best trade-off, preserving general features while allowing the deeper layers to adapt to the dataset. Through these layer-freezing tests, it was established to use 90% of the first layers of the EfficientNet-B0 network frozen. Additionally, a dense layer of size 512 with dropout and a dense layer of size 256 with dropout were added. The k-fold cross-validation technique was applied, with k = 5, dividing the dataset into five distinct partitions to ensure that each sample was not used for both training and validation to evaluate the robustness and generalization of the models. We recall that data from the same subject were not simultaneously used for training and validation. Unlike stratified cross-validation, which preserves label distributions in each fold, we opted for a random split to maintain the natural variability of sleep stage transitions in real-world sleep patterns. To determine whether the random split introduced significant discrepancies, we analyzed the class distribution in each split. Table 2 presents the percentage of Wake, NREM, and REM samples in the training and validation sets for sleep stage classification. For sleep/wake classification, where Sleep includes both NREM and REM, the distribution for Wake remains the same, while Sleep corresponds to the sum of NREM and REM. The results indicate that the overall distribution remains stable across splits, particularly in the training data, ensuring a balanced representation during model learning. While the validation distribution shows some variability, particularly in Split 1 and Split 2 for Wake and Split 5 for REM, this reflects real-world sleep data, where sleep stages are inherently imbalanced across different nights and individuals. Since the training data maintains a consistent distribution and the model is evaluated across multiple folds, the impact of these variations is minimized. Thus, the use of a random split does not introduce substantial bias or compromise the reliability of the results, as it allows the model to be tested under conditions that resemble real sleep patterns. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Table 2. Class distribution per split. Percentage of Wake, NREM, and REM samples in the training and validation sets across the five splits of the cross-validation. The data indicate that the class distribution remains stable across splits, suggesting that the random split does not introduce substantial bias. https://doi.org/10.1371/journal.pone.0323689.t002 Recognizing the challenge posed by class imbalance in the dataset, the class weighting technique was applied, where weights are assigned to each class inversely proportional to their frequency in the dataset. This approach ensures that minority classes contribute more significantly to the loss function during training, preventing the model from being biased toward the majority class. By adjusting the importance of each class in this way, class weighting helps mitigate the imbalance effect, leading to a more equitable and representative training process. The class weights w c were calculated as the inverse of the class frequencies, normalized by the total number of samples: (9) where N is the total number of samples, C is the number of classes, n c is the number of samples in class c. For binary cross-entropy, which is used for sleep/wake classification, the weighted loss is computed as: (10) where y i the true label (0 or 1) for sample i, p i is the predicted probability for the positive class, w pos and w neg are the weights for the positive and negative classes, respectively. For categorical cross-entropy, which is used for sleep stages classification (three classes), the weighted loss is computed as: (11) where y i,c is a binary indicator (0 or 1) for whether class c is the correct classification for sample i, p i,c is the predicted probability for class c, w c is the weight assigned to class c.

[END]
---
[1] Url: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0323689

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/