Representational Knowledge Distillation Across Wearable Biosignals

Re W2

We thank the reviewer for raising this important point. While we are not sure what SOTA methods the reviewer is referring to for unsupervised representational knowledge distillation across wearable biosignals, we respectfully want to mention that our original submission already had several comparisons versus competitive baseline methods including “accelerometer-based” prediction methods that the reviewer mentions: 1) self-supervised accelerometer encoders trained with contrastive learning and masked auto encoding, with two different architectures of Transformer and EfficientNet (now even more evidence in Figure 2, Table 2, Appendix Tables 9, 10, 11, 12, 13), 2) supervised accelerometer encoders (now even more evidence in Figure 2, Appendix Tables 9, 10, 11), 3) simultaneous multi-modal contrastive learning of PPG and accelerometer (as an alternative multi-modal method, Table 3). This is while we never claimed that the message of our paper was proposing a novel method but rather the application of transferring knowledge from PPG to accelerometer and we used our 2-stage distillation method (Fig. 1) as a means to achieve this. We want to further clarify:

• Our original submission already included comparison with simultaneous multi-modal contrastive learning as a multi-modal method to bind cross-modal representations: To the best of our knowledge, there’s no prior work on cross-modal representational knowledge distillation in wearable biosignals, particularly PPG to accelerometer in a dataset at such scale. However, there’s prior work on multi-modal contrastive learning of biosignals, for which, we had a comparison in our original manuscript and demonstrated that our method achieves a significantly higher accuracy (Table 3). In addition, in our experiments, we had tried techniques similar to self-distillation (e.g., where the embeddings are turned into a probability distribution, such as DINO [1]), and the knowledge is transferred via a cross-entropy loss, and despite various tuning attempts, we observed representational collapse in that technique. Moreover, while we have this comparison, the message of our manuscript is the efficacy and application of knowledge distillation from PPG to accelerometer on such large-scale dataset, and not necessarily the exact means to do it.
• Our original submission already included various competitive baselines for accelerometer-based prediction methods: The reviewer comment about “[lack] of comparisons with accelerometer-based prediction methods” is inaccurate. We originally had detailed comparisons across various available labeled data regimes compared to 1) self-supervised accelerometer encoders trained with contrastive learning and masked auto encoding, with two different architectures of Transformer and EfficientNet (now even more evidence in Figure 2, Table 2, Appendix Tables 9, 10, 11, 12, 13), 2) supervised accelerometer encoders (now even more evidence in Figure 2, Appendix Tables 9, 10, 11). This is in contrast with the reviewer’s comment about “[lack] of comparisons with accelerometer-based prediction methods”; we have covered a diverse set of ways to predict downstream targets from accelerometer with unsupervised (1) and supervised methods (2).
• Our manuscript contribution is not directly the methodology but demonstrating the application: As mentioned above in the previous weakenss, our goal is to demonstrate the efficacy and application of transferring knowledge from a PPG encoder to an accelerometer encoder, and creating one accelerometer encoder that is predictive of a wide array of downstream health targets. While we have already provided several comparisons, we respectfully do not believe additional comparisons are required to support the paper’s message because our main message is not about the method itself.

[1] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9650-9660).

Summary

We thank the reviewer for reviewing our manuscript and pointing out a set of important modifications. We have responded to the reviewer’s comments below (W: Weakness, Q: Question), and have addressed them accordingly in the uploaded revision of the paper. We are looking forward to the reviewer’s revised assessment of our manuscript given the improvements suggested by all the reviewers, which we summarized in the "Summary of revision" on top.

There are already too many cross-modal distillation methods to reduce model parameters and improve the accuracy of small models. For example: [1] Cross-modal Knowledge Distillation for Vision-to-Sensor Action Recognition. [2] Capturing causality and bias in human action recognition. [3] Brain-Machine Machine Coupled Learning Method for Facial Emotion Recognition [4] Emotionkd: a cross-modal knowledge distillation framework for emotion recognition based on physiological signals [5] Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition [6] Distilling privileged multimodal information for expression recognition using optimal transport

In terms of experiments, I understand that only one large-scale data set is used for pre-training. But since it is representation learning, why not use data from multiple different tasks for downstream tasks? For example, fall detection, action recognition, emotion recognition, etc.

There are already too many cross-modal distillation methods to reduce model parameters and improve the accuracy of small models. For example: [1] Cross-modal Knowledge Distillation for Vision-to-Sensor Action Recognition. [2] Capturing causality and bias in human action recognition. [3] Brain-Machine Machine Coupled Learning Method for Facial Emotion Recognition [4] Emotionkd: a cross-modal knowledge distillation framework for emotion recognition based on physiological signals [5] Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition [6] Distilling privileged multimodal information for expression recognition using optimal transport

Dear reviewer,

None of these papers are directly related to our work. Our work focuses on improving health applications via cross-modal distillation across wearable biosignals, where both high-fidelity and low-fidelity modalities are physiological signals collected from wearable devices. Examples of wearable biosignals are ECG, PPG, IMU/Accelerometer, etc that are commonly collected from consumer wearable devices such Apple Watch, Fitbit, Oura ring, Whoop, etc. In addition, as we mention multiple times in the paper, we model accelerometer during low-motion periods where it captures minute fluctuation of the body which are a direct result of the heart function, which is distinct from activity recognition from gross motion. For example, we say “Also, our work focuses on accelerometer during low-motion periods where it captures minute cardiovascular signals such as the ballistocardiogram, which is distinct from modeling slower changes in accelerometer during gross motion for health and fitness (Hallgrimsson et al., 2018; Ni et al.,2019; Spathis et al.,2021).”. Therefore, using accelerometer for applications such as activity recognition or fall detection is not relevant to our work. For health applications and at low-motion periods, we show that a single accelerometer foundation model is predictive of 51 health conditions, which has never been shown before. Below is a brief summary of the references you mentioned, none of them: 1) model teacher and student modalities that are both wearable biosignals, 2) focus on health/diagnosis applications.

[1]: Video to motion accelerometer for activity recognition

[2]: (Not cross-modal) Motion accelerometer for fall detection

[3]: (Not knowledge distillation) Multi-modal image-EEG for emotion recognition

[4]: EEG to Galvanic Skin Response for emotion recognition

[5]: Image (visual frames) to audio/EDA/EMG for expression recognition

[6]: Same as 5

In terms of experiments, I understand that only one large-scale data set is used for pre-training. But since it is representation learning, why not use data from multiple different tasks for downstream tasks? For example, fall detection, action recognition, emotion recognition, etc.

As we mentioned above, activity recognition/fall detection is not relevant to our work as we model accelerometer during low-motion periods where it captures minute fluctuation of the body which are a direct result of the heart function. However, we indeed included data from 51 downstream tasks for health applications in our paper (please take a look at Figure 2, Table 2, Appendix Table 12), and several of them are related to mental health / mood. We list them here in case you missed them but please take a look at our “summary of revision” and revised paper:

Downstream targets/tasks in our paper are: 1) Heart rate, 2) Heart rate variability (SDNN, RMSSD), 3) Age, 4) Biological Sex, 5) BMI, 6) ACE-inhibitors meds, 7) Active alcohol use, 8) Active smoking, 9) Afib, 10) Allergy, 11) Anti-anxiety meds, 12) Anti-psychotics meds, 13) Anticoagulants meds, 14) Antidepressants meds, 15) Antiplatelets meds, 16) Anxiety, 17) Artery disease, 18) Arthritis, 19) Asthma, 20) Beta-blockers meds, 21) Blood pressure, 22) Blood pressure meds, 23) Calcium-channel blockers meds, 24) Cancer, 25) Chemotherapy, 26) Cholesterol, 27) Chronic bronchitis, 28) Depression, 29) Diabetes, 30) Diuretics meds, 31) Heart attack, 32) Heart disease, 33) Heart failure, 34) Heart rhythm, 35) Hip/Knee, 36) Kidney, 37) Liver, 38) Lower back, 39) Neck disorder, 40) Neuropathy, 41) Opioid painkillers, 42) Osteoporosis, 43) Pacemaker, 44) Painkillers, 45) Sleep apnea, 46) Sleep meds, 47) Stroke or TIA, 48) Thyroid, 49) Urinary, 50) Vision, 51) Hearing.

Dear reviewer,

Hope this message finds you well and thanks again for your feedback regarding our paper. Just a friendly reminder that we are approaching the end of the discussion period.

We, along with other reviewers, sincerely believe that our paper has been significantly strengthened since your pre-rebuttal evaluation. We would greatly appreciate it if you could take a moment to review our response, reassess our revised manuscript from all aspects in light of the rebuttal, and consider updating your score to reflect the improvements. We look forward to hearing from you. Thank you!

Summary

We thank the reviewer for their kind mention of our manuscript strengths and pointing out a set of important modifications. We have responded to the reviewer’s comments below (W: Weakness, Q: Question), and have addressed them accordingly in the uploaded revision of the paper. We believe the reviewer’s comments have positively improved our manuscript. We are looking forward to the reviewer’s revised assessment of our manuscript given the improvements suggested by all the reviewers, which we summarized in the "Summary of revision" on top.

Re W1 (part 1)

We would like to thank the reviewer for raising this important point and we agree with the reviewer that a more diverse set of evaluations sheds more light on the generalizability of the transferred knowledge and our distilled accelerometer encoders. To this end and in line with the reviewer’s suggestion, we have added 49 new downstream targets: age, body mass index (BMI) and biological sex prediction as well as 46 health targets derived from AHMS survey questions (see last item below). We would like to emphasize that:

• We originally used HR/HRV as example downstream targets because of their significance in wearable devices due to their information about an individual’s of health status, training loads and stress level, and because wearable devices typically predict HR/HRV frequently throughout the day: We originally used HR/HRV as our downstream targets due to the importance of predicting them frequently using wearable devices throughout the day and in various environments, which matched well with our motivation of creating powerful encoders for low-fidelity but easy-to-collect biosignals such as accelerometer that are available on all wearables (see our Introduction). These targets, particularly HRV (SDNN/RMSSD) are widely used in wearable devices [1], and are known to be indicative of health status [2], training load in athletes [3] and stress levels [4]. We have now clarified further significance of HR/HRV in the manuscript: “We perform linear probing for predicting heart rate (HR), and two popular measures of heart rate variability: standard deviation of normal-to-normal intervals (SDNN) and root mean square of successive differences (RMSSD). These targets are chosen because they are widely used in wearable devices (Natarajan et al., 2020) and are indicative of health status (Shaffer & Ginsberg, 2017), training load in athletes (Plews et al., 2013) and stress levels (Kim et al., 2018).”. We want to clarify that, for estimating HRV, traditional algorithms operate by first estimating locations of peaks corresponding to individual heart beats, followed by generating statistics over inter-beat intervals [2]. While this may be lot easier to achieve with high-fidelity modalities (such as ECG or PPG), it is an extremely challenging problem with wearable accelerometer signals. We believe that to be able to estimate HRV metrics, the model has to develop some understanding of morphological features corresponding to the heart beat in time domain. This is very different from capturing frequency information that is captured in the heart rate.
• Our alignment analysis was agnostic to downstream targets and we observed near perfect alignment between accelerometer and PPG embeddings, i.e., our accelerometer encoder can capture PPG-like embeddings irrespective of the downstream target: In addition, we had performed an alignment analysis in an unsupervised manner, demonstrating that we can match accelerometer embeddings to their correct PPG embedding pair with 99.2% top-1 accuracy. This analysis is agnostic to the downstream target and holistically estimates how well these embeddings are aligned irrespective of the downstream target. If the distilled accelerometer embeddings are aligned with the PPG embeddings, it means we can expect improved performance for distilled accelerometer embeddings (as the low-fidelity modality) for any generic downstream target due to alignment to PPG (as the high-fidelity modality).

I appreciate the softened language around the claims of the paper, as well as the increased amount of downstream targets, showing the increased utility of the embedding alignment. While the scientific usefulness of this work is (in my opinion) quite compelling, I think that the primary concern of the other reviewers is rooted in that ICLR may not be be the best venue to show this off, due to the limited machine learning methodological advancements, benchmarks, reproducibility, and very high specificity in it's application.

Nevertheless, I believe that this is interesting work that could be accepted into ICLR due to the strong empirical results shown, so I have adjusted my score accordingly from weak reject (5) to a weak accept (6).

Re W3

This is an important point. We respectfully want to clarify that the reviewer that “Out of three comparisons, two of them are random guessing and random weight models.” is inaccurate. We respectfully want to mention that our original submission already had comparisons versus: 1) self-supervised accelerometer encoders trained with contrastive learning and masked auto encoding, with two different architectures of Transformer and EfficientNet (now even more evidence in Figure 2, Table 2, Appendix Tables 9, 10, 11, 12, 13), 2) supervised accelerometer encoders (now even more evidence in Figure 2, Appendix Tables 9, 10, 11), 3) simultaneous multi-modal contrastive learning of PPG and accelerometer (as an alternative multi-modal method, Table 3). This is while we never claimed that the message of our paper was proposing a novel method but rather the application of transferring knowledge from PPG to accelerometer and we used our 2-stage distillation method (Fig. 1) as a means to achieve this. Also, we originally only had the random models for the alignment analysis purely for visualization purposes (which we now moved to the new Appendix Figure 4). In addition, we also would like to respectfully emphasize that we originally put “random guessing” in only one of our tables (Table 1) to provide the chance-level performance (not baseline) so it would be clear how much above chance performance the other numbers are. Therefore, we kept that row and changed to “Chance-level performance” for more transparency in the new Table 1, and Appendix Table 8.

Regarding the reviewer’s comment “I would expect at least two more methods from signal conversion (low to high) to show and support the claim that the proposed framework is better than previous methods in non-private datasets.”, as mentioned above our message is to demonstrate the efficacy and application of transferring knowledge from a PPG encoder to an accelerometer encoder, and creating one accelerometer encoder that is predictive of a wide array of downstream targets. While we have already provided several comparisons, we respectfully do not believe additional comparisons are required to support the paper’s message because our main message is not about the specific method itself but the application.

Re W4

We would like to thank the reviewer for this suggestion. We will change the title for the camera ready submission by explicitly mentioning accelerometer, but during the review period we keep it as to avoid confusion for the AC and other reviewers.

Re Q1

We would like to thank the reviewer for raising this important point. We agree with the reviewer that root mean squared error (RMSE) and Pearson Correlation (Pearson’s R) are important metrics and provide additional information regarding HR/HRV estimations versus ground truth. To address this, we have now reported these metrics, in addition to our original mean absolute error metric for HR/HRV predictions in new Appendix Tables 9, 10 and 11. We would like to emphasize that the same order between the models still holds for HR/HRV predictions with these new evaluation metrics, i.e., distilled accelerometer encoders are still better than supervised and uni-modal accelerometer encoders, and all our conclusions remain the same. We believe these new metrics suggested by the reviewer provide additional information about our evaluations.

Re W1 "The main idea and the specific components of the paper are not novel. Obtaining a better representation from a low-fidelity signal using a high-fidelity one is already explored [1]." (part 1)

We would like to thank the reviewer for this important point. We have now discussed the mentioned manuscript in various locations in our manuscript (see last item below) but we believe the reference mentioned by the reviewer [1], while interesting and of great importance, does not negatively impact the contributions of our work due to major differences. In summary, [1] presents a cycleGAN to reconstruct ECG from PPG, and the authors show that heart rate calculated directly from the reconstructed ECG from PPG provides a good estimate of the ground truth heart rate. We would like to emphasize that:

• We respectfully believe our methodology, our application, our dataset, and most importantly our empirical findings significantly differ from [1] (and any prior work), and our manuscript provides transformative evidence of using accelerometer for diverse health applications/diagnoses to the wearable research community.
  • Methodology: Methodology used in our work is distinct from [1]. We investigate transferring representational knowledge (compact embeddings) from an already trained embedding encoder for PPG to another embedding encoder for accelerometer, which falls under knowledge distillation [2, 3]. This is while the prior work investigates cross-modal reconstruction of ECG from PPG in the output domain with a cycleGAN. In our original submission’s title/abstract/text, we have frequently emphasized that our work is “representational knowledge distillation”, which is a narrow variation of knowledge distillation referring to transferring embeddings/representations from a network to another network [2], and we disagree that [1] fits in this category, nor this prior work has investigated creating generalist/foundation embedding models like we have. Also, in details, almost all training and modeling choices significantly differ between the two works.
  • Application: We have repeatedly mentioned in our original submission that our goal is to build models for low-cost power-efficient biosignals that are available in any wearable without requiring any optical sensors, which is why we used accelerometer. This is a transformative application/finding for wearable research and industry to use accelerometer for a wide array of health/diagnosis applications. The mentioned paper by the reviewer [1] (and any prior work), does not show this, and in fact we disagree with the reviewer that [1] has a “low-fidelity” biosignal and we believe PPG is high-fidelity and predictive of a wide array of health targets, in many cases better than ECG [4].
  • Dataset: Prior work [1] uses 4 public datasets that in total have 125 participants. Our dataset includes 172,318 participants with the data collected under natural daily life conditions covering 4 years of data. This is 1378x larger in terms of number of participants. We hope that the reviewer, like we do, appreciates the challenges of modeling large-scale datasets and the generalizability of the findings in our manuscript to a broad range of demographics, which is important for health applications.
  • Findings: In our revised manuscript, we show that a single distilled accelerometer encoder is readily (without fine-tuning) predictive of heart rate, heart rate variability, age, body mass index (BMI), biological sex, as well as 46 other health related targets spanning targets for underlying health conditions, use of medications and lifestyle habits. This is all done by only linear probing of the embeddings of single accelerometer encoder, which means a single compact embedding vector is predictive of all these targets. We respectfully believe, this is significantly different than the mentioned work by the reviewer which only looks at heart rate by counting the number of peaks from the reconstructed ECG. In addition, we have done several ablation studies for sweeping the number of labels, compressing the model sizes, different architecture, teacher pre-training framework, and etc. These are all distinct from prior work.

[1] Sarkar, Pritam, and Ali Etemad. "Cardiogan: Attentive generative adversarial network with dual discriminators for synthesis of ecg from ppg." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 1. 2021.

[2] Hinton, G. (2015). Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531.

[3] Tian, Y., Krishnan, D., & Isola, P. (2019). Contrastive representation distillation. arXiv preprint arXiv:1910.10699.

[4] Abbaspourazad, S., Elachqar, O., Miller, A. C., Emrani, S., Nallasamy, U., & Shapiro, I. (2023). Large-scale training of foundation models for wearable biosignals. arXiv preprint arXiv:2312.05409.

Summary

We appreciate the reviewer for detailing our manuscript’s strengths, and their thoughtful suggestions. We have responded to the reviewer’s comment below, and have addressed them accordingly in the uploaded revision of the paper. We believe the reviewer’s comments and suggestions have significantly strengthened our manuscript. We are looking forward to the reviewer’s revised assessment of our manuscript given the improvements suggested by all the reviewers, which we summarized in the "Summary of revision" on top.

I thank the authors for their detailed response and their efforts.

While the novelty of this work is a secondary concern, the evaluation methodology and underlying motivation raise significant questions.

First, sampling signals exclusively during low-motion periods appears to inherently bias the dataset. By excluding segments where subjects are running, walking, or climbing stairs, the dataset is probably limited to resting HR/HRV values. This narrow scope restricts the generalizability of the evaluation and risks overfitting to the specific conditions of the data.

Second, the authors suggest that Ballistocardiogram (BCG) signals, detectable by the accelerometer during low motion, are informative [3,4]. While I agree with this, it raises an important question: if these BCG signals are present, why not use a signal processing pipeline to directly extract HR/HRV values? What specifically justifies deploying a large (2M-5M-parameter for a wearable device) neural network instead of a more efficient and interpretable method? This also contradicts the authors' stated focus on low-power, resource-constrained settings. If signal processing pipelines fail—which is doubtful given the low-motion sampling—why are such methods not included as baseline comparisons?

Third, while I understand that downstream tasks (e.g., medication use prediction) may not be feasible with signal processing alone, traditional machine learning methods (e.g., random forests, SVMs) using simple features (mean, power, median, HR/HRV from signal processing) should be provided as a baseline. Including these with confidence intervals would highlight the relative improvement offered by the neural network model. The comparisons provided (e.g., Table 12) fail to establish the superiority of the proposed method because no traditional ML baselines with simple features are included (or a mean predictor). Furthermore, without reporting confidence intervals, it is difficult to assess whether the observed differences in metrics—some of which are very close—are significant.

Given the limitations of the current evaluation methodology and the contradictory motivation of the manuscript, the discussion of novelty remains difficult and unnecessary at this stage. Therefore, I will maintain my original rating.

1. Prediction of demographic variables, age, biological sex, and BMI. Please note that the chance-level performance (mean predictor that the reviewer refers to) for age is 11.06, BMI is 5.68, and chance-level performance for biological sex (ROC AUC) is 0.5. The gap between our method and the baseline the reviewer suggests is extremely larger compared to the gap between the baseline and the mean predictor, which is an indication of how performant our models are.

2. Prediction of other health targets. Similar to above, our embeddings are significantly better than the baseline the reviewer suggests.

Third, while I understand that downstream tasks (e.g., medication use prediction) may not be feasible with signal processing alone, traditional machine learning methods (e.g., random forests, SVMs) using simple features (mean, power, median, HR/HRV from signal processing) should be provided as a baseline. Including these with confidence intervals would highlight the relative improvement offered by the neural network model. The comparisons provided (e.g., Table 12) fail to establish the superiority of the proposed method because no traditional ML baselines with simple features are included (or a mean predictor). Furthermore, without reporting confidence intervals, it is difficult to assess whether the observed differences in metrics—some of which are very close—are significant.

As we mentioned above and as the reviewer agrees, there’s no well-known “signal processing pipeline” for these targets, our work is the first of its own demonstrating accelerometer is predictive of all these demographics variables and health targets with such high accuracy.

However, to still address the reviewer’s concern, we used even a more accurate measure of HR/HRV as if we hypothetically had an accurate “signal processing pipeline” to measure HR/HRV from accelerometer, and showed that still we cannot predict the health targets accurately. To this end, as baseline feature streams, we used HR/HRV from the values logged on Apple Watch (HR/HRV from PPG [1], which are better than accelerometer) + some derivative accelerometer features (amplitude / power) to predict our demographic variables and health targets. It is worth mentioning that for these baseline feature streams, we included their mean/median/std, so 3 values per feature stream as suggested by the reviewer. Also, we trained the predictor model from this final feature vector to the target using 1) linear regression, 2) random forest, and report the better performance of the two. As expected, we demonstrate that our distilled accelerometer encoder embeddings are significantly and by a large margin better in predicting these targets. The number of held out participants in these evaluations is very large (~17k unseen participants) so even marginal differences are significant, but for the camera ready version we will add significance p-values for more transparency. These results rule out the hypothetical scenario the reviewer raises, and further validate how performant our models are.

[1] Using Apple Watch to measure heart rate, calorimetry, and activity: https://www.apple.com/health/pdf/Heart_Rate_Calorimetry_Activity_on_Apple_Watch_November_2024.pdf

Second, the authors suggest that Ballistocardiogram (BCG) signals, detectable by the accelerometer during low motion, are informative [3,4]. While I agree with this, it raises an important question: if these BCG signals are present, why not use a signal processing pipeline to directly extract HR/HRV values? What specifically justifies deploying a large (2M-5M-parameter for a wearable device) neural network instead of a more efficient and interpretable method? This also contradicts the authors' stated focus on low-power, resource-constrained settings. If signal processing pipelines fail—which is doubtful given the low-motion sampling—why are such methods not included as baseline comparisons?

There is no contradiction here, we show that a single accelerometer encoder is simultaneously predictive of 51 health targets, which in fact is a power-efficient solution. For estimating HRV, traditional algorithms for PPG/ECG operate by first estimating locations of peaks corresponding to individual heart beats, followed by generating statistics over inter-beat intervals [1]. While this may be lot easier to achieve with high-fidelity modalities (such as ECG or PPG), it is an extremely challenging problem with wrist-based accelerometer signals because peak detection methods do not work for wrist-based accelerometer like they do for PPG/ECG. An example of how different the signals are in PPG and accelerometer are depicted in Figure 1 — there are no clear peaks in accelerometer as for PPG. This is to be expected — accelerometer picks up extremely minute movements, and the overall signal-to-noise ratio to find the specific movements corresponding to heartbeats from such data is many orders of magnitude weaker than PPG during even low-motion segments.

In addition, there is no prior work demonstrating prediction of all these health conditions and HR/HRV simultaneously using accelerometer, not only with the “signal processing pipelines” the reviewer refer to without providing references, but also with any other modeling framework. In spite of that, we already have comparisons with supervised algorithms for accelerometer for HR/HRV and showed that our models are better not only at the full data availability regime, but also with only a limited amount of labels. Given the large volume of data available to us, it should be clear that a flexible enough neural network trained via direct supervision should greatly outperform simple hand-crafted features from a “signal processing pipeline” — and even then, we see that our supervised accelerometer models are still worse than our proposed distilled accelerometer embeddings for predicting HR and HRV.

With that said, we are also demonstrating that a single accelerometer encoder is predictive of all these other targets, a much more efficient approach than building 51 separate “signal processing pipelines” (or supervised encoders) for each target, likely requiring extensive manual tuning in order to get it to work (if it is even possible at all).

This work is the first work demonstrating 1) accelerometer-derived signals are predictive of all these health conditions — this is an important scientific finding, and 2) a single accelerometer encoder can predict all these targets simultaneously with one embedding vector. Our results are indeed in line with our motivations because accelerometer is 1) power efficient to collect, 2) is available in almost all wearables, particularly those without optical sensors for PPG, 3) a single accelerometer encoder can simultaneously predict 51 targets.

[1] F. Shaffer and J. P. Ginsberg, “An Overview of Heart Rate Variability Metrics and Norms,” Front. Public Health, vol. 5, p. 258, Sep. 2017, doi: 10.3389/fpubh.2017.00258.

Dear reviewer,

Hope this message finds you well. We are thankful for your suggestions to improve the quality of our manuscript. We have now added statistical significance p-values to the Appendix Table 12 to demonstrate that our distilled accelerometer encoders are statistically better than the baseline acceleroemter encoders, as you suggested.

With this change, we have now carefully addressed your questions and concerns from your original and follow up feedback, and incorporated them appropriately in the revised manuscript. We would be grateful if you could consider reassessing our manuscript and raising your score. We believe, like other reviewers do, our manuscript has improved significantly thanks to your and other reviewers' suggestions. Thank you!

Furthermore, without reporting confidence intervals [in Appendix Table 12], it is difficult to assess whether the observed differences in metrics—some of which are very close—are significant.

We have now added statistical significance p-values to the Appendix Table 12 as you suggested. As we mentioned above, the number of data points in our comparisons is very high (~17K) so even marginal differences are likely significant. To demonstrate this, we have now reported statistical significance p-value for "Accel-KD via PPG-MAE" being better than "Accel-CL" (as the best uni-modal accelerometer encoder). We observed in 44/46 comparisons, the distilled accelerometer encoder had statistically better performance. We added additional text to the Appendix Table 12 with statistical significance comparison p-values: "Asterisks indicate statistical significance for the comparison between the best distilled accelerometer encoder (“Accel-KD via PPG-MAE”) versus the best uni-modal accelerometer encoder (“Accel-CL”). For statistical significance, we calculated 200 bootstrapped ROC AUC values for each evaluation, and then performed one-sided Wilcoxon Rank-Sum test to compute the p-value of the statistical comparis. We report ”***” for P < 5e−4, ”**” for 5e−4 ≤ P < 5e−3, ”*” for 5e−3 ≤ P < 5e−2 and ”n.s.” for P ≥ 5e−2, where P is the p-value of the comparison.".

Please do not get me wrong, I am not trying to overshadow the efforts put into the paper. I am simply asking the questions I would consider as a reader, as future readers might also wonder why these points were not raised by the reviewer.

We appreciate your time and energy for giving us valuable feedback, and helping us improve our manuscript.

While I largely agree with Reviewer 3EkY’s feedback on limited methodological advancements, lack of benchmarks, reproducibility issues, and narrow application focus, I found the baseline comparisons for HR/HRV tasks particularly weak. The additional experiments introduced during the discussion phase raised further questions regarding the evaluation and soundness of the paper. Therefore, I am trying to clarify them.

Thanks for raising this point. We respectfully request you to evaluate our manuscript from all aspects. Overall, We believe our manuscript has breadth of information for practitioners and researchers working with wearable biosignals, and even if we don't introduce novel components (like a new loss function), we believe our approach to the problem (2-stage masked autoencoding and then distillation across modalities for an unsupervised end-to-end method), the implementation details, and empirical findings have several novelties and are valuable for the community. Even [R-3EkY] that is being referred to, has mentioned several strengths of our work in their initial and final evaluation, including "Although cross-modality knowledge distillation has been studied before, applying this onto biosignals is a novel and interesting problem." and "scientific usefulness of this work is (in my opinion) quite compelling", and has given us an above acceptance score.

We understand your initial concerns but we did our best to address all of them, and we respectfully believe our manuscript has improved significantly over the course of the rebuttal period from your first evaluation; we genuinely request you to evaluate the current state of our manuscript from all aspects with a degree of openness and for a fair evaluation, thank you!

Dear reviewer,

We would like to thank you for investing the time to review our manuscript and we are more than happy to answer your questions. Please find our responses below:

I agree with the authors that accelerometer-based HR/HRV measurement is prone to errors and challenging. However, wasn’t the primary motivation of this research to address those challenges and obtain HR/HRV from IMUs? Even the introduction emphasizes this goal.

This is an important point to clarify. In our manuscript's Introduction/Abstract, we never say our primary goal is to predict HR/HRV but if there's a specific point that has caused the reviewer to say that, we will be more than happy to address in the camera ready version (the window to update the pdf has ended). In turn, we say our motivation is "developing digital biomarkers from any wearable device with lower-fidelity biosignals, and help individuals track their health more frequently and conveniently", and in line with that, we demonstrate that a single accelerometer embedding model can capture rich embeddings that are readily predictive of a variety of targets (a very efficient solution) -- with all the new evidence, only 2 out of 51 targets we present in our revised manuscript are HR/HRV. Also, our method can work for any other downstream targets, we provided evidence with the example labels that were available to us, but future work can look into other downstream targets as well.

My previous concern was about the signal processing approaches in this direction. A literature search reveals several works that have successfully used wrist-based accelerometers to obtain HR/HRV [1-3], with some even reconstructing pulse waves for direct comparison to R peaks in ECG. These studies show that beats can be detected while excluding motion artifacts, and recent methods demonstrate strong performance. I wonder why the authors did not compare their results with these approaches. While I understand that these methods may not address health-related targets, considering the HR/HRV results presented in the manuscript, a future reader might see this as a fair question. Without such comparisons, the claim that "a flexible neural network trained via direct supervision should greatly outperform simple hand-crafted features from a signal processing pipeline" does not appear to be scientifically justified.

This is very important to clarify. Given our response in the previous round (see "Response to [R-GDg8] (Part {12,13,14}/n)", we provided strong evidence that even if there was an accurate estimation of HR/HRV from accelerometer, we would still not be able to predict all the other 49 targets including demographics variables, health conditions, use of medications and lifestyle habits (even the reviewer agrees with this in their previous response, that's why they asked for the additional evidence which we provided). In fact, the gap of performance is dramatically large which represents that our embeddings extract significantly richer information that just HR/HRV. Given all the evidence we have provided, we believe an additional signal processing baseline for HR/HRV is out of scope for our manuscript, and future work can look into that (we can add this as a discussion to the camera ready version). As a side note, as we mentioned in our prior responses, our work covers low-motion periods throughout daily living, which is significantly more challenging than just modeling overnight accelerometer data the reviewer has provided references for. As an extra side note, we respectfully want to re-emphasize that our motivation was building efficient solutions for predicting health targets from available biosignals on all wearables such as accelerometer, and we demonstrate that a single accelerometer encoder is predictive of all these other targets. We believe this is a much more efficient approach than building separate signal processing pipelines (or supervised encoders) for each target -- in fact, we originally provided supervised evaluations to contextualize our label efficiency analysis and not as an alternative efficient solution.

Dear Reviewer,

We hope this message finds you well. Thank you once again for your feedback on our paper and for your thoughtful comments throughout the discussion period.

We have done our best to address your questions and incorporate your initial and follow-up feedback into the manuscript. We sincerely believe that the manuscript has improved significantly as a result.

As we approach the end of the discussion period, we would greatly appreciate it if you could take a moment to reassess our manuscript from all aspects in light of the rebuttal, and we would be grateful if you could consider updating your score to reflect the improvements since your pre-rebuttal evaluation/score.

Thank you for your time and support, and we look forward to hearing from you.

Summary

We appreciate the reviewer’s feedback, their kind words about the strengths of our work, and their careful examination of our manuscript. We have responded to the reviewer’s comments below (W: Weakness, Q: Question), and have addressed them accordingly in the uploaded revision of the paper. We believe the reviewer’s comments have positively improved our manuscript. We are looking forward to the reviewer’s updated assessment of our manuscript given the improvements suggested by all the reviewers, which we summarized in the "Summary of revision" on top.

Re W1 & W2 & Q1

We would like to thank the reviewer for raising this point. We want to clarify that to the best of our knowledge, our work is the first that demonstrates unsupervised knowledge distillation of representations across wearable biosignals, particularly so in a dataset at the scale of what we have used in our manuscript. We are not familiar with any prior work on representational knowledge distillation across wearable biosignals, but if the reviewer is aware of such work, we would appreciate any pointers they have so that we can update our related work and be sure to better contextualize our contributions. That said, we totally agree with the reviewer to further clarify our contributions to avoid misunderstandings, therefore to more precisely state our main contribution, we have now softened the language used in our abstract, contribution list and related work in Introduction:

• Abstract: “While multi-modal modeling and cross-modal reconstruction of biosignals have been done before, here, we demonstrate that we can distill representational knowledge across biosignals with different levels of fidelity, i.e., from PPG to accelerometer, using 20million minutes of unlabeled data collected from ∼172K participants in the Apple Heart and Movement Study under informed consent”.
• Contribution list in Introduction: “1) Representational knowledge distillation across wearable biosignals in a large-scale dataset: While multi-modal modeling and cross-modal reconstruction of biosignals have been done before, we study representational knowledge distillation across wearable biosignals using a dataset at such large scale with 20M minutes of multi-modal sensor data from ∼172K participants.”.
• Related work: “All in all, while multi-modal modeling and cross-modal reconstruction of biosignals have been done before, our work studies unsupervised representational knowledge distillation across wearable biosignals using large-scale data collected from wearable consumer devices to develop a single encoder for accelerometer that is readily predictive of a wide array of downstream health targets.”

Re Q2

We would like to thank the reviewer for pointing this out. The analysis for contrastive learning and EfficientNet was done to provide an ablation to the generalizability of the approach to both the architecture and teacher pre-training in one shot, and originally we didn’t separate these two factors, and we agree that it made it difficult to infer the effects of each factor. While this analysis did not directly affect the conclusions in our paper (it was an ablation), to alleviate the concerns and isolate the effects of the two factors, we now repeated the analysis for 1) teacher pre-training with contrastive learning, 2) uni-modal accelerometer pre-training with contrastive learning, 3) distilling the Teacher encoder from 1 mentioned here to student accelerometer encoder, where the encoder architectures are Transformers, and we provided these numbers in the new Appendix Table 13. Overall, we observed no meaningful and qualitative difference in contrastive learning using EfficientNet and Transformers, and all of our original conclusions remain intact. Note that we now have the combinations of (Transformer, MAE & MAE $\rightarrow$ KD), (Transformer, CL & CL $\rightarrow$ KD), (EfficientNet, CL & CL $\rightarrow$ KD) in the paper. The 4th combination (EfficientNet, MAE) is not practically doable because masked autoencoding, as known in [1], is specific to Transformer architecture due to MAE’s components such as masking and dropping.

[1]: He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16000-16009).

Re Q3

This is a great point. We have now added new Appendix Table 14 providing evidence for this selection by comparing downstream performance of these two positive pair selections for targets such as heart rate and heart rate variability. As shown, the pre-trained teacher PPG encoder with segment-level positive pairs has significantly better prediction performance.

Summary

We appreciate the reviewer’s feedback, their kind words about the strengths of our work, and their careful examination of our manuscript. We have responded to the reviewer’s comments below (W: Weakness, Q: Question), and have addressed them accordingly in the uploaded revision of the paper. We believe the reviewer’s comments have positively improved our manuscript. We are looking forward to the reviewer’s revised assessment of our manuscript given the improvements suggested by all the reviewers, which we summarized in the "Summary of revision" on top.

Regarding W1 & Q1

We would like to thank the reviewer about this comment and pointing out these references. Our original submission already included various competitive baselines for accelerometer-based prediction methods: We originally had detailed comparisons across various available labeled data regimes compared to 1) self-supervised accelerometer encoders trained with contrastive learning and masked auto encoding, with two different architectures, 2) supervised accelerometer encoders, which are in line with the reviewer’s comment about “models that directly use acceleration as input and heart rate as output”, as they cover a diverse set of ways to predict downstream targets from accelerometer with unsupervised (1) and supervised methods (2).

Regarding the mentioned references, the general theme across the references [1-3] is modeling the dynamical relationship between gross body motion and the user's heart rate. Activity metrics from these prior works include steps, speed, sleep, or accelerometer statistics at a very slow sampling rate (0.067Hz in [2]) which capture very slow changes in activity/movement. The modeling is done using sparse inputs (one sample every few seconds) over longer time scales (typically up to a few hours) and can include periods of high motion. While such models can effectively learn a measure of cardiovascular activity and fitness, the heart rate estimates are just approximations since the modeling inputs do not directly contain any (heart) beat-level information. In contrast, our paper learns a representation of the accelerometer signal over a short time scale of 60s using the raw signals at 64 Hz during periods of low-motion and at rest. This allows the model to examine the minute fluctuation of the body which are a direct result of the heart function. This approach is commonly referred to as Ballistocardiography [4, 5], and an example of the associated low-amplitude features can be observed in the periodic fluctuations shown in the sample of accelerometer signals depicted in Figure 1. Since this captures signatures of individual heart beats, we expect the heart rate estimates during the downstream evaluation to be of much higher quality compared to [1-3], albeit only limited to low motion scenarios. We have now added these references to the Introduction and Discussion section of the manuscript:

• Introduction: "Also, our work focuses on accelerometer during low-motion periods where it captures minute cardiovascular signals such as the ballistocardiogram, which is distinct from modeling slower changes in accelerometer during gross motion for health and fitness (Hallgrimsson et al., 2018; Ni et al.,2019; Spathis et al.,2021)."
• Discussion: “Our work primarily focused on accelerometer signals during low-motion and sedentary periods (Section 4.1), where accelerometer captures ballistocardiogram and therefore minute cardiovascular-related information (Inan et al., 2015; Kim et al., 2016). Future work can investigate models that take slower-scale activity metrics on wearable devices such as steps, speed, sleep, and slow changes in accelerometer, as well as minute changes in accelerometer at sedentary settings to improve the performance for downstream targets (Hallgrimsson et al., 2018; Spathis et al., 2021; Ni et al., 2019).”.

[1] Hallgrímsson, Haraldur T., et al. "Learning individualized cardiovascular responses from large-scale wearable sensors data." arXiv preprint arXiv:1812.01696 (2018).

[2] Spathis, Dimitris, et al. "Self-supervised transfer learning of physiological representations from free-living wearable data." Proceedings of the Conference on Health, Inference, and Learning. 2021.

[3] Ni, Jianmo, Larry Muhlstein, and Julian McAuley. "Modeling heart rate and activity data for personalized fitness recommendation." The World Wide Web Conference. 2019.

[4] H.-G. Kim, E.-J. Cheon, D.-S. Bai, Y. H. Lee, and B.-H. Koo, “Stress and Heart Rate Variability: A Meta-Analysis and Review of the Literature,” Psychiatry Investig, vol. 15, no. 3, pp. 235–245, Mar. 2018, doi: 10.30773/pi.2017.08.17.

[5] Inan, O. T., Migeotte, P. F., Park, K. S., Etemadi, M., Tavakolian, K., Casanella, R., ... & Di Rienzo, M. (2014). Ballistocardiography and seismocardiography: A review of recent advances. IEEE journal of biomedical and health informatics, 19(4), 1414-1427.

• While three reviewers are very positive about novelty and impact of our approach and findings: “This paper presents a novel approach to learning representations from wearable sensor data” [R-AiqZ], “The paper addresses a significant real-world problem with direct applications in wearable health technology.” [R-hLgx] and “Although cross-modality knowledge distillation has been studied before, applying this onto biosignals is a novel and interesting problem.” [R-3EkY], the other 2 reviewers raise concerns about novelty, where only one of them [R-GDg8] explicitly mentions a related work [1] in terms of these concerns for the novelty. However, we respectfully believe our methodology, our application, our dataset, and most importantly our empirical findings significantly differ from [1] (and any prior work), and our manuscript provides transformative evidence of using accelerometer for diverse health applications/diagnoses to the wearable research community.
  • Methodology: Methodology used in our work is distinct from [1]. We investigate transferring representational knowledge (compact embeddings) from an already trained embedding encoder for PPG to another embedding encoder for accelerometer, which falls under knowledge distillation [2, 3]. This is while the prior work investigates cross-modal reconstruction of ECG from PPG in the output domain with a cycleGAN. In our original submission’s title/abstract/text, we have frequently emphasized that our work is “representational knowledge distillation”, which is a narrow variation of knowledge distillation referring to transferring embeddings/representations from a network to another network [2], and we disagree that [1] fits in this category, nor this prior work has investigated creating generalist/foundation embedding models like we have. Also, in details, almost all training and modeling choices significantly differ between the two works.
  • Application: We have repeatedly mentioned in our original submission that our goal is to build models for low-cost power-efficient biosignals that are available in any wearable without requiring any optical sensors, which is why we used accelerometer. This is a transformative application/finding for wearable research and industry to use accelerometer for a wide array of health/diagnosis applications. The mentioned paper by the reviewer [1] (and any prior work), does not show this, and in fact we disagree with the reviewer that [1] has a “low-fidelity” biosignal and we believe PPG is high-fidelity and predictive of a wide array of health targets, in many cases better than ECG [4].
  • Dataset: Prior work [1] uses 4 public datasets that in total have 125 participants. Our dataset includes 172,318 participants with the data collected under natural daily life conditions covering 4 years of data. This is 1378x larger in terms of number of participants. We hope that the reviewers, like we do, appreciate the challenges of modeling large-scale datasets and the generalizability of the findings in our manuscript to a broad range of demographics, which is important for health applications.
  • Findings: In our revised manuscript, we show that a single distilled accelerometer encoder is readily (without fine-tuning) predictive of heart rate, heart rate variability, age, body mass index (BMI), biological sex, as well as 46 other health related targets spanning targets for underlying health conditions, the use of medications and lifestyle habits. This is all done by only linear probing of the embeddings of single accelerometer encoder, which means a single compact embedding vector is predictive of all these targets. We respectfully believe, this is significantly different than the mentioned work by the reviewer which only looks at heart rate by calculating it directly from the reconstructed ECG. In addition, we have done several ablation studies for sweeping the number of labels, compressing the model sizes, different architecture, teacher pre-training framework, and etc. These are all distinct from prior work.

[1] Sarkar, Pritam, and Ali Etemad. "Cardiogan: Attentive generative adversarial network with dual discriminators for synthesis of ecg from ppg." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 1. 2021.

[2] Hinton, G. (2015). Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531.

[3] Tian, Y., Krishnan, D., & Isola, P. (2019). Contrastive representation distillation. arXiv preprint arXiv:1910.10699.

[4] Abbaspourazad, S., Elachqar, O., Miller, A. C., Emrani, S., Nallasamy, U., & Shapiro, I. (2023). Large-scale training of foundation models for wearable biosignals. arXiv preprint arXiv:2312.05409.

• Modified the language about our contributions: There were few concerns about the language used in the manuscript for our contributions. To more precisely state our main contribution, we have now softened the language used in our abstract, contribution list and related work in Introduction, for example we list our contribution as: “While multi-modal modeling and cross-modal reconstruction of biosignals have been done before, we study representational knowledge distillation across wearable biosignals using a dataset at such large scale with 20M minutes of multi-modal sensor data from ∼172K participants.”.

• Provided additional evidence and ablations: We also provided additional evidence about:

  • Root mean squared error and Pearson’s R metrics for heart rate and heart rate variability predictions in new Appendix Tables 9, 10, and 11, and showed that our conclusions were unchanged.
  • Our positive pair selection in contrastive learning in new Appendix Table 14.
  • Why we did not scale our model sizes beyond 6.3M parameters in new Appendix Table 16.
  • The effect of number of PPG channels in teacher pre-training in new Appendix Table 15.

• Modified text and additional references: We also modified and added text in various sections of the manuscript, and provided additional references, as suggested by reviewers throughout the manuscript.

• Modified the organization of the manuscript: we slightly modified the organization of the manuscript to fit the manuscript within the 10-page limit while incorporating the new changes.

While the above are the changes we made to the manuscript, we would like to clarify few points regarding concerns raised about novelty and comparisons:

• While a couple of reviewers claim that: “The baseline comparison is also extremely limited. Out of three comparisons, two of them are random guessing and random weight models.” [R-GDg8] and “The paper lacks comparisons with state-of-the-art (SOTA) cross-modal distillation methods, as well as accelerometer-based prediction methods” [R-9r1K], we respectfully want to mention that our original submission already had comparisons versus: 1) self-supervised accelerometer encoders trained with contrastive learning and masked auto encoding, with two different architectures of Transformer and EfficientNet (now even more evidence in Figure 2, Table 2, Appendix Tables 9, 10, 11, 12, 13), 2) supervised accelerometer encoders (now even more evidence in Figure 2, Appendix Tables 9, 10, 11), 3) simultaneous multi-modal contrastive learning of PPG and accelerometer (as an alternative multi-modal method, Table 3). This is while we never claimed that the message of our paper was proposing a novel method but rather the application of transferring knowledge from PPG to accelerometer and we used our 2-stage distillation method (Figure 1) as a means to achieve this.
• We agree that the individual modeling components used in our paper are not themselves novel in the deep learning domain, and we never claimed we created new individual components (e.g. masked autoencoding, contrastive learning, knowledge distillation, knowledge distillation across pre-trained models) — in fact, even in our original manuscript we said: “We combine and adopt techniques inspired by uni-modal and multi-modal pre-training frameworks from other domains of deep learning to create a fully unsupervised distillation framework for biosignal time-series, which is particularly crucial for health applications where labeled data is limited”. That said, we respectfully believe figuring out how best to combine and adopt existing techniques in order to solve real-world problems is extremely non-trivial, particularly in such large-scale datasets and novel applications for accelerometer signals. We are not aware of any prior work that does the modeling similar to how we do for wearable biosignals.
• We believe our technical implementation, solution to the problem and empirical findings are valuable to researchers and engineers in research and industry labs working with data from wearable devices. Practical applications of how to combine such techniques are non-trivial, and we would like to respectfully state that according to ICLR reviewer guidelines (https://iclr.cc/Conferences/2025/ReviewerGuide), “Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc).”, and we believe our manuscript does fit well in this category, which is why we originally submitted our manuscript to the “Applications” track.