STQS: Interpretable multi-modal Spatial-Temporal-seQuential model for automatic Sleep scoring

Shreyasi Pathak*, Changqing Lu, Sunil Belur Nagaraj, Michel J.A.M. van Putten, Christin Seifert

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

34 Citations (Scopus)
129 Downloads (Pure)


Sleep scoring is an important step for the detection of sleep disorders and usually performed by visual analysis. Since manual sleep scoring is time consuming, machine-learning based approaches have been proposed. Though efficient, these algorithms are black-box in nature and difficult to interpret by clinicians. In this paper, we propose a deep learning architecture for multi-modal sleep scoring, investigate the model's decision making process, and compare the model's reasoning with the annotation guidelines in the AASM manual. Our architecture, called STQS, uses convolutional neural networks (CNN) to automatically extract spatio-temporal features from 3 modalities (EEG, EOG and EMG), a bidirectional long short-term memory (Bi-LSTM) to extract sequential information, and residual connections to combine spatio-temporal and sequential features. We evaluated our model on two large datasets, obtaining an accuracy of 85% and 77% and a macro F1 score of 79% and 73% on SHHS and an in-house dataset, respectively. We further quantify the contribution of various architectural components and conclude that adding LSTM layers improves performance over a spatio-temporal CNN, while adding residual connections does not. Our interpretability results show that the output of the model is well aligned with AASM guidelines, and therefore, the model's decisions correspond to domain knowledge. We also compare multi-modal models and single-channel models and suggest that future research should focus on improving multi-modal models.
Original languageEnglish
Article number102038
Number of pages16
JournalArtificial intelligence in medicine
Early online date27 Feb 2021
Publication statusPublished - 1 Apr 2021


  • UT-Hybrid-D
  • Sleep stage annotation
  • Deep learning
  • EEG, EOG, EMG signals
  • Post-hoc interpretability
  • Explainable AI
  • Sleep scoring


Dive into the research topics of 'STQS: Interpretable multi-modal Spatial-Temporal-seQuential model for automatic Sleep scoring'. Together they form a unique fingerprint.

Cite this