SleepSMC: Ubiquitous Sleep Staging via Supervised Multimodal Coordination

Thank you for recognizing and supporting the writing and experimental work in our paper! We understand that your main concerns are related to methodological innovation and some experimental details. Below, we address these points:

1. Why was "Supervised Contrastive Learning" not used as a baseline? What is the motivation for contrastive learning?

   The "Supervised Contrastive Learning" method is built on contrastive learning between augmented and original data samples, aiming to enhance general representation capabilities. In contrast, our contrastive learning approach focuses more on inter-modal coordination and alignment, with the goal of improving single-modality robustness and performance in an end-to-end manner.

   To evaluate the effect of data augmentation and provide a meaningful comparison, we supplemented our experiments with the recent method you suggested: MERL (Liu et al., ICML2024). This method incorporates both inter-modal and intra-modal contrastive learning. Specifically, inter-modal alignment facilitates information transfer, while intra-modal contrastive learning leverages data augmentation to enhance representation and robustness. From the experimental results, our method still demonstrates significant superiority.

2. Should methods for handling missing modalities in other fields also be compared?

   Thank you for your suggestion. ShaSpec (CVPR 2023) and DrFuse (AAAI 2024) are excellent works, and we will cite and compare them in the revised paper's related work section. Additionally, we supplemented comparisons with DrFuse (AAAI 2024) under all scenarios across three datasets. The experimental results show that our method still outperforms these approaches.(The table has been combined with Question 5 below.)

3. What is the significance of single-modality available? Why not consider a set of modalities?

   In sleep monitoring, multimodal devices are often complex, difficult to wear, and significantly disrupt sleep experiences, posing substantial obstacles to the monitoring process. Such devices are not suitable for ubiquitous scenarios (e.g., home settings) and may even decrease monitoring accuracy. We are developing a single-modality device aimed at enabling anyone, even without specialized medical knowledge, to perform comfortable, convenient, and accurate sleep monitoring through simple contact.

   Therefore, our method primarily focuses on single-modality testing scenarios. Thank you for pointing this out. We have added this limitation in the "Discussion and Limitation" section of the revised paper.

4. Do you expect high uncertainty at the beginning of training? Any experiments on dynamic weight adjustment? Experiments with alternative metrics?

   1. In fact, we do not expect high uncertainty, as it is undesirable. Ideally, uncertainty should be low for perfect modality coordination learning. Our goal is to enable the model to overcome situations of high uncertainty and large differences, thereby improving robustness.

   2. All weights in our method are dynamically computed and adjusted, adapting at each step during training.

   3. We conducted experiments using the uncertainty variance metric, directly applying the negative value of formula (5), $-r_i^{u_a}$. However, as explained in formula (6), "the exponential function introduces a smooth, continuous inverse scaling, where weights decrease more sharply for high uncertainty and more gradually for low uncertainty, reflecting the varying tolerance of the model." This approach achieves better performance. Details are shown in below:

6. Why are uncertainty estimation methods mentioned in related work not applicable to the target scenario?

   Thank you for pointing this out. We will elaborate on this in the revised manuscript. These methods focus on effective multimodal fusion, including retaining useful modality information and removing irrelevant data. However, they do not address single-modality application scenarios, which are our primary concern. Their methods require all modalities present during the testing phase, and retaining only one modality significantly degrades accuracy. In contrast, our method improves the performance of individual modalities through coordinated learning, enabling robust single-modality applications in ubiquitous scenarios.

7. The performance improvement of the uncertainty-based weighting module is smaller than that of the contrastive learning module. How is randomness mitigated?

   Thank you for highlighting this. To mitigate randomness, we have taken the following steps:

   • Experimentation: As described in Section 5.2 of the original paper, we conducted extensive experiments, and all reported results are aggregated from five-fold cross-validation.

   • Metrics: To account for the impact of randomness, we introduced the Kappa metric. Kappa is a robust metric designed to measure agreement between predictions and ground truth while accounting for agreement due to chance. The Kappa results clearly illustrate the contribution of the uncertainty-based weighting module.

   • Writing: Thank you for pointing this out. We acknowledge that the role of the uncertainty-based weighting module is smaller than that of the contrastive learning module. Our original language may have been misleading, implying that most of the performance gain came from the uncertainty-based module. We have revised this statement to: "The Supervised Modality-level Contrastive Coordination module plays a more significant role in both scenarios. Meanwhile, the Uncertainty-based Feature Weighting module demonstrates relatively enhanced performance in unimodal compared to multimodal scenarios." Thank you again for your thoughtful feedback, which has improved our paper.

8. Fluctuations in Figure 3c?

   The robustness gap between EMG and the other two modalities is significant, resulting in less fluctuation for EMG compared to EEG and EOG. For EEG and EOG, their robustness is closer, leading to relative changes in their rankings. Additionally, robustness gaps always exist, and our goal is not to eliminate them but to demonstrate the significance of our method through these gaps. Our method dynamically leverages these differences to achieve better information transfer.

9. Minor writing suggestions?

   Thank you for your meticulous review, which has made our paper better! We have incorporated your suggestions into the revised manuscript and have thoroughly checked the entire paper for any writing issues.

5. Can unsupervised contrastive learning methods mentioned in the paper be fairly compared?

   Thank you for your suggestion. We supplemented our comparisons with the method by Liu et al. (2024) across all scenarios on three datasets. To ensure a fair comparison, we adapted the method into a supervised, end-to-end approach while retaining its core modality-level and intra-modal contrastive learning components.

• Multimodal scenario:

• Unimodal scenario + ISRUC-S3 dataset:

• Unimodal scenario + the other datasets: Please refer to Section A.3 of the revised manuscript for the full results and analysis.

Reference
[1] Liu Y, Jia Z. Bstt: A bayesian spatial-temporal transformer for sleep staging[C]//The Eleventh International Conference on Learning Representations. 2023.
[2] Phan H, Chén O Y, Tran M C, et al. XSleepNet: Multi-view sequential model for automatic sleep staging[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(9): 5903-5915.
[3] Yao W, Yin K, Cheung W K, et al. DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(15): 16416-16424.
[4] Liu C, Wan Z, Ouyang C, et al. Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement[C]//Forty-first International Conference on Machine Learning, 2024.

The authors have addressed my comments, including comparison with more recent baselines, clarification of the results analysis, and the experiment setting. I appreciate the authors' efforts in making the necessary adjustments. I believe the quality of paper has improved, and I have increased my score accordingly.

Dear reviewer,

Thank you for your time and effort in reviewing our paper, and thank you for your recognition and positive feedback on our paper. Your support has brought us great encouragement.

We are really happy to address all your concerns, and we are very grateful for your help and encouragement, which makes our paper better and better. If you have further questions, we will be happy to receive your feedback.

Thank you again for your support and valuable insights!

Best wishes

Thank you for recognizing and supporting the practical significance and writing quality of our paper! We understand that your main concerns are related to the methodological innovation and some experimental details. Below, we address these points:

1. This paper does not introduce a new concept, but only combines and applies existing methods?

   Our method incorporates supervised modality-level contrastive coordination and uncertainty-based weighting. The former introduces label information during modality alignment to enhance accuracy, aligning not only modality instances at the same time but also across different times, fully learning the category-relevant temporal information. The latter facilitates dynamic filtering of information during transmission, significantly improving the model's robustness.

   The combination of these two innovations effectively achieves single-modality sleep staging in ubiquitous scenarios. Therefore, while our work demonstrates innovation and practical significance in its application, it also includes substantial methodological novelty.

2. Why not adopt a sequence-to-sequence paradigm? Can this method adapt to a sequence-to-sequence paradigm?

   1. Many existing methods are not based on a sequence-to-sequence paradigm but still achieve excellent results. For example, several recent methods:

      [1] Jia Z, Wang H, Liu Y, et al. Mutual Distillation Extracting Spatial-temporal Knowledge for Lightweight Multi-channel Sleep Stage Classification[C]//Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024: 1279-1289.
      [2] Thapa R, He B, Kjaer M R, et al. SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals[C]//Forty-first International Conference on Machine Learning. 2024.
      [3] Zhu H, Zhou W, Fu C, et al. Masksleepnet: A cross-modality adaptation neural network for heterogeneous signals processing in sleep staging[J]. IEEE Journal of Biomedical and Health Informatics, 2023, 27(5): 2353-2364.

   2. Although our method is not specifically designed as a sequence-to-sequence paradigm, it inherently learns sequence-level information during the supervised contrastive coordination process. Our approach not only aligns sequences at the same time point but also sequences of the same class across different time points. This approach considers sequence-level contextual consistency and effectively utilizes class label information to handle situations such as abrupt class transitions in sleep staging tasks.

      For instance, during transitions between two sleep stages, traditional sequence-to-sequence methods might mistakenly classify adjacent sequences as belonging to the same stage. Our method addresses this issue effectively. Clearly, our approach is adaptable to a sequence-to-sequence paradigm.

3. The selected baselines are outdated. Why choose ISRUC-S3 instead of S1?

   Thank you for your suggestion. The three methods you mentioned are highly influential in the sleep staging domain. We will include these three papers in the revised manuscript's related work section and provide comparisons. Additionally, we supplemented the experiments with five recent methods (including BSTT and XSleepNet), and the results under two scenarios demonstrate that our method still achieves the best performance.

   Regarding our choice of the smaller S3 subset instead of S1, there are two main considerations:

   • Dataset scale: Although the SleepEDF dataset only includes 78 subjects, its total data volume is much larger than that of S1 and is often considered a large-scale dataset. We have already validated our method on SleepEDF.
   • Scenario: S3 and S1 originate from the same source. Our method focuses on multimodal robustness in ubiquitous and low-resource scenarios. Thus, we prioritized validating performance on low-resource, small-scale datasets like S3.

• Multimodal scenario:

• Unimodal scenario + ISRUC-S3 dataset:

• Unimodal scenario + the other datasets: Please refer to Section A.3 of the revised manuscript for the full results and analysis.

Reference
[1] Liu Y, Jia Z. Bstt: A bayesian spatial-temporal transformer for sleep staging[C]//The Eleventh International Conference on Learning Representations. 2023.
[2] Phan H, Chén O Y, Tran M C, et al. XSleepNet: Multi-view sequential model for automatic sleep staging[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(9): 5903-5915.
[3] Yao W, Yin K, Cheung W K, et al. DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(15): 16416-16424.
[4] Liu C, Wan Z, Ouyang C, et al. Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement[C]//Forty-first International Conference on Machine Learning, 2024.

4. Supplement and proof not detailed enough?

   Thank you for your feedback, which has improved the readability of our paper. In the revised version, we have further supplemented the relevant definitions and proofs for greater clarity and completeness.

We sincerely apologize for any confusion caused. In fact, our experimental setup differs significantly from that of the original paper. The primary reason for this is that the experimental setting in the original paper [1] is not entirely standardized.

From the publicly available code of the original paper [1], it can be observed that the authors only divided the dataset into a training set and a test set. Specifically, they trained the model on the training set and reported the test set results for the best-performing epoch. Similarly, in another work, MSTGCN [2], the same author of [1] used a comparable setting, as seen in their released code.

However, from a methodological perspective, this approach is not strictly standardized. To better avoid test set data leakage and evaluate the cross-subject generalization performance of the model, we introduced a validation set by randomly splitting 20% of the training set. The model was then trained using the remaining 80% of the training data, and the best-performing model on the validation set was saved for evaluation on the test set.

This approach inevitably reduced the amount of data available for training (to 80%) and ensured that the test set was not directly used for model selection. To maintain fairness, we applied this standardized experimental setup consistently across all our experiments.

This explains why we did not directly cite the results reported in the original paper. We hope this addresses your concerns.

[1] Liu Y, Jia Z. Bstt: A bayesian spatial-temporal transformer for sleep staging[C]//The Eleventh International Conference on Learning Representations. 2023.

[2] Jia Z, Lin Y, Wang J, et al. Multi-view spatial-temporal graph convolutional networks with domain generalization for sleep stage classification[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 1977-1986.

Dear reviewer:

According to your suggestion, we added experiments on the ISRUC-S1 dataset with four comparison methods in two scenarios. Thank you for your suggestion, which make our paper better. If you have any additional suggestions or feedback, we would be truly grateful to hear them.

1. Multimodal scenario:

2. Unimodal scenario + ISRUC-S1 dataset:

Best wishes

Dear Reviewer HHyg,

We would like to express our sincere gratitude for your careful review and thoughtful comments, which have greatly helped improve our paper. We deeply appreciate the time and effort you have put into providing such valuable feedback.

As the discussion period is drawing to a close, we humbly hope to earn your support. We have made every effort to address your concerns, and we sincerely welcome any further questions or suggestions you may have. Your insights are truly invaluable to us, and we would be grateful for any additional guidance you can offer.

Thank you again for your time and consideration.

Thank you for recognizing and supporting our experimental work and writing! We understand that your main concerns are related to the motivations behind our method and some implementation details. Below, we address them in detail.

1. The motivation is not clear? Where is the novelty of the method?

Thank you for your recommendation. We have carefully reviewed the CoRe-Sleep and TSEDSleepNet methods you mentioned, which are excellent sleep staging approaches. However, our method is significantly different from theirs, and we will compare and cite it in the revised paper. Details are as follows:

• Motivation: Our method aims to address the problem of multimodal training with single-modality testing, significantly improving single-modality performance during the testing phase. This makes it well-suited for ubiquitous applications, such as home sleep monitoring in ubiquitous scenarios. In contrast, CoRe-Sleep and TSEDSleepNet focus on achieving better multimodal fusion. CoRe-Sleep emphasizes robust fusion, while TSEDSleepNet focuses on class imbalance and temporal modeling capabilities.

• Methodological Novelty: We introduce supervised contrastive learning for modality-level coordinated and associated learning, effectively addressing the inconsistency between modalities during training and testing phases. Additionally, we propose uncertainty-based weighting to facilitate the selection and transmission of high-quality information during coordination. While CoRe-Sleep also performs modality alignment, it relies on unsupervised alignment and does not fully leverage class label information or optimize the alignment process. On the other hand, the primary innovations in TSEDSleepNet lie in its model structure and its loss function designed for class imbalance.

• Experiments: To further demonstrate the effectiveness of our method, we have included comparisons with four state-of-the-art methods, covering approaches for multimodal missing and sleep staging methods. On three datasets, under both multimodal and unimodal scenarios, our method consistently achieves the best performance.

1. Multimodal scenario:

2. Unimodal scenario + ISRUC-S3 dataset:

3. Unimodal scenario + the other datasets: Please refer to Section A.3 of the revised manuscript for the full results and analysis.

Reference
[1] Liu Y, Jia Z. Bstt: A bayesian spatial-temporal transformer for sleep staging[C]//The Eleventh International Conference on Learning Representations. 2023.
[2] Phan H, Chén O Y, Tran M C, et al. XSleepNet: Multi-view sequential model for automatic sleep staging[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(9): 5903-5915.
[3] Yao W, Yin K, Cheung W K, et al. DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(15): 16416-16424.
[4] Liu C, Wan Z, Ouyang C, et al. Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement[C]//Forty-first International Conference on Machine Learning, 2024.

2. What is the computational complexity?

   Our model uses only a CNN-based structure, its computational complexity is low overall, as detailed in the table below. In particular, the computational complexity during the single-modality testing phase is extremely low, which strongly supports its deployment in ubiquitous single-modality scenarios.

   Scenario	Training phase	Testing phase
   Multimodal	33.0M FLOPs & 1.3M param	16.5M FLOPs & 0.6M param
   Unimodal	26.1M FLOPs & 1.0M param	6.9M FLOPs & 0.2M param

3. How equation 10 become optimized?

   Equation 10 defines the overall objective function for optimizing SleepSMC. It integrates a supervised contrastive loss $𝐿_{Con}$ to align cross-modal features and a classification loss $𝐿_{cls}$ to ensure accurate sleep stage predictions. The $𝐿_{Con}$ stabilizes the training process by controlling the contribution of auxiliary modalities.

   These losses are jointly optimized through a sum. The joint optimization is performed using stochastic gradient descent (SGD) with the Adam optimizer, which efficiently handles the gradients of both losses.

Our method indeed enables real-time sleep monitoring. Within the context of supervised contrastive learning, contrastive learning is performed during the training phase. The goal is to enhance the learning of modality-specific, class-relevant information through multimodal data. This approach significantly improves the model's performance in single-modality testing scenarios. In practical applications, users only need to wear a simple single-modality device, such as an ear-EEG device. The model can process the collected single-modality data to accurately classify sleep stages. By analyzing sleep cycles and structures, effective sleep monitoring becomes straightforward.

Feasibility of Real-Time Monitoring:

As discussed in Part 3/n, the computational complexity of single-modality model inference is extremely low, requiring only 6.9M FLOPs. Even on a mid-range ARM CPU (capable of executing 5 GFLOPs per second), the inference time is approximately 1.38 milliseconds, while a typical clinical sleep segment spans 30 seconds. Since the inference time is orders of magnitude smaller than the sample duration, real-time monitoring is highly feasible.

Let me know if you have further questions!

Dear reviewer,

Thank you for your time and effort in reviewing our paper, and thank you for your recognition and positive feedback on our paper.

We have provided comprehensive responses to your concerns through detailed explanations and thorough experimental validations. In particular, we have explained the real-time monitoring issue you mentioned.

We hope that we have addressed your concerns. If you have any further questions, we would be happy to receive your feedback.

Thank you again for your support and valuable insights!

Best wishes

Thank you for your comprehensive recognition and support of our work! We understand that your main concerns are related to certain experimental details and writing aspects of the paper. We address them below:

1. A concise list of symbols to better clarify notation?

   We have added such a list in Section A.1 of the revised appendix. Thank you for your suggestion, which has improved our paper!

2. Solid contribution in sleep staging but without significant breakthroughs. Could it yield greater benefits in other multimodal learning problems?

   Achieving a 2-3% improvement in the sleep staging field is already a substantial contribution. Furthermore, our paper primarily focuses on enhancing sleep staging comfort in ubiquitous scenarios. However, our method does indeed have the potential to be applied in other fields. Thank you for pointing this out—we have included this possibility in the "Discussion and Limitation" section of the revised appendix.

3. Cross-subject experiments

   As described in the experimental settings of Section 5.2, all our experiments adopt a cross-subject setup, considering that physiological signal models are typically designed to generalize in such scenarios. In all experiments, the training and testing sets are cross-subject, and we report aggregate results over five different splits. We apologize for any confusion and have highlighted the cross-subject setup in Section 5.2.

4. Could the model handle dynamic shifts in modality reliability, and do the authors envision practical deployments for health monitoring or similar applications?

   Yes, the model can handle dynamic shifts in modality reliability. During training, the reliability of each modality is re-evaluated at every step, and relevant metrics are recalculated. Our method ultimately enhances the overall robustness of the model, which allows it to adapt to reliability changes during the testing phase to a certain extent. What a coincidence! We have indeed envisioned practical deployments for health monitoring. We are currently developing a wearable single-modality sleep monitoring device and plan to deploy this model in such applications.

Dear Reviewer,

Thank you for taking the time and effort to review our paper, and thank you for your recognition of our paper.

We have provided a comprehensive response to your concerns through detailed explanations.

We hope we have addressed your concerns. If you have any further questions, we would be delighted to hear your feedback.

Thank you again for your support and valuable insights!

Best regards