Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions

Thank you for the positive feedback and helpful comments towards improving our work. We focus on responding to the major themes of your comments:

Measuring Difficulty of Downstream Tasks

We agree that it is important to contextualize the difficulty of the tasks. To overcome this, we included baseline statistics of the input data and an existing PPG foundation model as a strong baseline to contextualize the performance of WBM. The performance of these models shows that no task is particularly easy or hard, but the strong performance of WBM relative to these baselines helps show the efficacy of the proposed model.

Per your suggestion, we will include the case/non-case distribution for all binary outcome labels in the camera ready version of the paper to improve our evaluation report. However, importantly, prevalence does not immediately define the difficulty of the downstream task.

Using Highly Sparse Input Modalities

This is a key characteristic of an observational study such as AHMS collected under real-world conditions where these variables are less often logged due to their nature and behavior of the participants. It is true that given the unique nature of these variables, it is harder for the model to leverage their temporal dynamics and changes, but our goal here is to model the data as-is but use state-of-the-art architectures such as state space models and Transformers that can leverage any potential temporal dynamics or inter-variable relationships. We experimented with tokenizers such as “Tuple“ tokenizer that does not do any form of imputation (therefore no constant input for sparse variables); we observed degraded performance with the ”tuple“ tokenizer in our hyperparameter search experiment in Appendix Tables 8 & 9.

We do find that the learned embeddings retain information for several variables that are natively sampled at a weekly frequency (e.g. six minute walk distance and walking steadiness score, see Table 10 in the Appendix). However, as you point out, there are some variables that we do not capture well in our embeddings. This could be due to their low prevalence or the use of the contrastive loss for training, which we will discuss further in the camera ready. As you suggest, we could remove such variables for computational efficiency, and we will add a line explaining this to the camera ready version of the paper.

Quality Control of Inputs and Labels

For input behavioral data, we do not do a per-modality or per-variable quality control and we use the data as is. However, for turning these behavioral data into week-level segments for training WBM, we perform the following cleaning steps: 1) z-score each variable then clip any outliers to [-5,5], 2) drop weeks whose number of variables are in bottom 5% percentile of all weeks, 3) drop weeks that have less than 5 days of data, 4) drop subjects with less than 5 weeks of usable data, or who were enrolled in the study for less than 90 days.

For segment-level labels of baseline history and medication, you bring up an important point in that we are working with self-reported labels. In the camera ready version, we will discuss the caveat of parsing the labels from self-reported surveys (although for some of our evaluations we do use more rigorous lab measurements, eg, diabetes).

Finally, we clarify the quality control for defining sleep labels. To obtain sleep labels, a participant must wear the watch overnight in order to get sleep metrics. We further limited to weeks where 5/7 days in the week had sleep metrics, meaning the watch was on overnight. This means that the PPG was recorded overnight as well, ensuring a fair comparison between both techniques.

We will be sure to clarify all of these points in the camera ready version.

Making Model Weights Open Source

Unfortunately the model weights cannot be shared due to the specifics of the informed consent for participants in the study. We will provide all necessary details in the camera ready and will put a note for interested parties to reach out to the authors for more details. However, for completeness, we will include a model card with the exact details of model size, number of layers, and other necessary training parameters in the camera ready.

Thank you for your positive feedback and suggestions aimed towards improving our work!

Contextualizing Comparisons between WBM and Baseline/PPG

We appreciate your feedback on tempering our claims.

First, we clarify the WBM vs baseline comparison. The subject-level tasks in Figure 3 are intentionally simple, as they aggregate a subject’s full history. Basic aggregate feature statistics and demographics may perform well, explaining the small median improvement of WBM. However, WBM significantly outperforms the baseline in a few key cases (e.g. smoking status and anti-psychotics usage). Its real strength lies in the more difficult time-varying tasks, where it consistently surpasses the baseline in detecting changes in health state on all tasks.

Next, we emphasize that WBM and PPG are complementary. WBM excels in some tasks (e.g., sleep duration and infection), while PPG is stronger in others (e.g., diabetes). However, combining both achieves the best subject-level performance in 42/47 tasks (with a majority being significant), and in all but 1 of the segment-level tasks (where it’s within margin of error). Behavior data should complement, not replace, sensor data when building prediction models from wearables.

We will edit the language in the camera ready based on your feedback and our response above to better clarify our contributions.

Combining WBM and PPG Representations

We apologize that this was unclear. We will clarify in the camera ready that we combined WBM and PPG embeddings by concatenating the two 256D embedding vectors into one 512D embedding vector. There are many better ways to build multimodal representations using fusion techniques either at the input or representation level that we did not explore. We will add a discussion of these as future work in the camera ready.

Relation to Broader Wearable Community

Thanks for raising this point. We will improve the discussion by connecting our work with the broader space of digital health and wearables, and mention the potential clinical impact such wearables-based health predictions might have in the future if safely deployed at-scale.

Label Leakage in Downstream Tasks

This is a subtle but important point, as label leakage is a major challenge in building foundation models, and you are correct that a small number of our input variables (e.g. basal energy) rely on age and sex as inputs. However, we clarify that age and sex prediction are not meant to showcase the value of our model. We view these tasks as sanity checks that our model is able to encode information that we already expect should be partially available. We will make this caveat more clear in the camera ready. We will also emphasize the importance of the other tasks, especially the segment-level tasks as mentioned above. Label leakage should not be a major concern for the 55 other downstream tasks, as none of those labels are used as part of the input variables.

Computational Cost of Training and Inference

The final WBM was the result of 6 epochs of training which took 16 hours of training time on 8 A100 GPUs. The learned model can quickly perform inference, and embeddings can be used easily across many tasks. We will add these details in the camera ready.

Robustness to non-Apple devices and other wearing patterns

You bring up an important point about generalizing to non-Apple devices and other wearing patterns. During training, we opted to remove weeks with low wear time, so we expect performance will degrade when applied to participant weeks with limited wear time. Evaluations on non-Apple devices is not possible, as much of the data can only be collected on Apple devices (particularly behavioral features derived via proprietary algorithms). However, our training details and insights provide a useful framework for others to train models on other wearables. We will discuss these limitations in the camera ready.

Typos and Writing Suggestions

We appreciate the feedback on typos, writing and citation improvements; we will fix these in the camera ready. We clarify that “1-layer multi-layer perceptron” means there is one hidden layer in addition to the input/output layers.

Miscellaneous Responses

Thank you for bringing up important topics regarding interpretability, selection bias, and evaluation across different demographic groups. Due to space constraints, we point you to responses to R1 and R2 respectively for these topics.

We will also clarify in the camera ready version that the choice to fit models weekly vs. at a participant-level was made based on the task. For time-varying tasks where we make predictions at every week, we fit models at a weekly level. For static targets, we fit models at a participant level.

Thank you for your positive feedback and useful comments for enhancing our work. We respond to specific suggestions below:

Interpretability of WBM

Interpretability of foundation models is an active area of research that remains extremely important. Unfortunately, it remains non-trivial to understand how input features affect the learned representation in order to ascertain feature importance for any given downstream task. As you suggest, one technique might involve independently perturbing each input sensor to understand its effect on the learned representation. However, understanding the correct way to perturb these irregular data in a meaningful and scalable way remains an important open problem. We will discuss the importance of interpretability and some potential next steps in the camera ready version of the paper.

Sleep Metrics and IMU

When evaluating using sleep metric labels, our goal was to showcase one example where we expect behavior data to be much more predictive than PPG. Sleep metrics are only estimated when a subject wears their watch overnight, ensuring that we have some amount of PPG overnight alongside the other behavior data fields. In general, a passive measurement of PPG is attempted roughly every 2 hours for most subjects, and the measurement is only retained if the subject is sufficiently quiescent ensuring that the PPG data has low noise.

The sleep metrics on the Apple Watch are derived only from a continuous stream of IMU (3-axis accelerometer) during a sleep session, and PPG/behavior is not used. Processing such continuous IMU streams involves the use of complex data pipelines that were not available to us; the volume of such data would make it impossible to scale to using most days and subjects from across the study. Therefore, including such IMU data in our modeling was out of scope for our study. However, given that sleep labels are derived from IMU, we expect markedly stronger predictions if we include IMU in the input of our models.

PPG prediction of deep sleep & "PPG will not provide the same holistic view of an individual’s week" phrasing

It is generally the case that total sleep duration, sleep efficiency, and deep sleep in particular decrease with age. Since PPG (collected roughly every 2 hours - see above) contains strong age-related signals, we would expect that it should be able to leverage such information to make decent predictions about average sleep metrics for an individual. Note that the baseline model (which explicitly includes demographics) also performs better on deep sleep prediction, suggesting that demographics plays a more important role. We will rephrase this to “PPG does not provide as comprehensive a view of an individual’s week, since it is only measured a few times each day”. We will also add additional clarification and caveats around the sleep analyses.

Improvements on Contrastive Learning Framework

We agree that the contrastive framework could be improved upon. We explored the use of a masked auto encoder approach, but found that this resulted in poor performance (see Appendix A.5.3). We hypothesize that this may be due to the high degree of noise and irregularity in the behavior data, making complete reconstruction of the input an overly challenging task that leads to representations that do not generalize well to new tasks. We will expand on this hypothesis further in the camera ready, as well as discuss other techniques that future work could consider adapting to this type of data to improve upon our framework such as joint-embedding predictive architectures (JEPA). Even though our contrastive learning approach is not set up to capture intra-subject changes, empirically based on our results it still has some ability to do so.

Clarifying “Human Behavior” Throughout

Thank you for finding this unclear sentence, we will rephrase in the camera ready. We agree that the use of the term “behavior” may be confusing — we will be sure to carefully go through the manuscript and only use behavior in the intended use-case (i.e., when discussing behavior data) and avoid overloading the term.

Details on Statistical Significance and Bootstrap Performance

Thank you for the great suggestion for including bootstrap CIs and p-values in the manuscript, we will add these in the camera ready format. We will also clarify how we calculate bootstrap confidence intervals: we resample the test set 1,000 times and recompute performance metrics on each resampled test set for each method. The confidence intervals and p-values are then computed empirically on this bootstrapped set of performance metrics.

Thanks for your positive comments and constructive feedback to help us improve this work! We focus on responding to the major themes of your comments:

Choice of Mamba-2 and comparison to SOTA deep learning models:

This is an important point to clarify. As stated in Sections 4.2 and 4.3 and Appendix Section A.5.1, we compared the Mamba-2 model architecture with two alternative architectures: Self-attention Transformer, and Rotary Transformer. These two are among the most commonly used state-of-the-art Transformer architectures in various domains. In addition, for a fair comparison, we did a full grid search over these 3 architectures, and over 3 different tokenizers, as well as sweeping other hyperparameters. Within our hyperparameter search experiment, we observed that Mamba-2 with the TST tokenizer generally outperformed the other alternatives, including the 2 Transformers architectures. Refer to Table 9 in Appendix Section A.5.2 for a full comparison of how often TST+Mamba-2 achieved the best performance compared to the other 8 architecture+tokenizer combinations.

Inclusion of more baseline models would improve the rigor of the evaluation:

We have included comparisons of WBM with baseline architectures/tokenizers, and a competitive PPG baseline. In addition, we have discussed a baseline comparison with respect to other pre-training methods such as masked autoencoding (Narayanswamy 2024) in Appendix A.5.3, as well as a simple baseline of turning behavioral data to its mean and standard deviation statistics. However, we do agree with that including more baselines can improve our evaluations, and we will discuss this as a caveat in the camera ready version. We welcome any feedback on specific baselines that you feel would particularly strengthen our work.

Potential demographic biases and performance in demographic subgroups:

We agree that characterizing demographic biases are essential for health applications. The Apple Heart and Movement Study has its own limitation and biases (Truslow et al. 2024), for example, the dataset is more biased towards a younger male population. However, given the large scale of this study compared to other studies, our models are still trained and evaluated on a large cohort including participants from diverse demographics. In the camera ready version of the paper, we will include distributions of demographics statistics from our pre-training data. We will also add to the appendix the performance of our models within demographic subgroups on a representative subset of the full set of tasks considered, focusing on tasks where the combination of WBM+PPG performs best. Specifically, we will show demographic subgroup performance on a representative set of targets: heart failure, active smoker, and calcium-channel blocker baseline tasks, as well as on the pregnancy and infection tasks. We will add some discussion around potential fairness concerns and demographic biases to the camera ready version of the paper, noting that a complete fairness investigation into our final models was out of scope for this work.