SensorLM: Learning the Language of Wearable Sensors

Thank you very much for acknowledging the novelty and the contributions of our work! We are glad that you found the paper comprehensive and valuable with strong empirical results. In the following, we address your concerns in detail. We hope that these will further clarify our work and lead to a favorable increase of the score.

W1: The quality of the pretraining data relies on heuristically generated captions, which may introduce noise or bias that affects the downstream alignment performance.

We appreciate the reviewer’s concern about potential noise or bias introduced by rule‑based captions, particularly for the statistical and structural components. The semantic captions, however, are derived from self‑reported user events. Our hierarchical scheme therefore blends human‑provided information with programmatically extracted patterns, reducing reliance on any single heuristic. In addition, every caption is paraphrased through diverse templates generated by LLMs (Appendix A.2), which broadens linguistic variety and better reflects how users might describe sensor traces. This strategy delivers scalability that manual annotation cannot match, allowing us to assemble a large-scale sensor-language corpus.

Empirically, our results support that SensorLM remains robust despite possible noise, achieving strong alignment performance across diverse downstream tasks. Section 5.3 and Appendix C.5 analyze the contribution of each caption type and training strategy, offering further insight into alignment dynamics.

We believe this trade-off between caption realism and scalability is a practical and necessary step toward enabling large-scale sensor-language learning. In future work, incorporating more naturalistic language data (e.g., user descriptions or clinical notes) could further improve alignment quality and enable more model capabilities. We will add more discussion on the trade-off of rule-based captioning and human annotations in the revised manuscript.

W2 & Q1: The model is currently evaluated only on wearable sensor data, and its applicability to other time-series domains—such as industrial or medical sensors—remains untested.

Thank you for the thoughtful comment. While our current evaluation focuses on wearable sensor data, the SensorLM framework is designed to be modality-agnostic and generalizable to other time-series domains. The captioning pipeline for deriving statistical and structural patterns from raw signals, and the ability to incorporate new semantic information via captions, is not specific to wearable devices. Our pretraining objectives and modular architecture can also be readily adapted to structured inputs from industrial or medical sensors. We see this as a promising direction for future work and believe that SensorLM lays the foundation for scalable sensor-language learning across diverse domains. We will add these discussions in the revised manuscript.

Q2: What is the computational overhead of SensorLM at inference time compared to conventional sensor models, especially in edge or real-time environments?

SensorLM demonstrates competitive inference time overhead compared to conventional baselines. For zero-shot inference, the combined sensor and text encoders require 122 GFLOPs. In few-shot or linear probing scenarios, SensorLM's sensor encoder, at 92 GFLOPs, offers comparable efficiency to the SSL baselines (105 GFLOPs), maintaining a total cost around 100 GFLOPs. The complete caption generation process requires 155 GFLOPs; however, this cost is justified by the model's distinctive interpretive capabilities.

Challenges for deployment in highly constrained environments are similar to those of baseline SSL methods, but these can be mitigated through established optimization strategies like model compression or hybrid on-device/cloud inference.

We will include a detailed discussion of these considerations in the revised manuscript.

Q3: Can the hierarchical captioning pipeline be adapted or extended to support human-written descriptions or other forms of supervision beyond rule-based heuristics?

Yes, the hierarchical captioning pipeline is modular and can be adapted to incorporate human-written descriptions, clinical notes, or expert annotations in place of or alongside existing captions. Its design supports seamless integration of diverse textual inputs. While large-scale expert annotations are challenging to obtain, our caption generation pipeline and pretraining framework is readily adaptable should such supervised data become available. As a first step, our rule-based approach makes scalable sensor-language pretraining possible.

We will add a discussion section on the adaptability to human-written or other forms of supervision in the revised manuscript.

Q4: How robust is the model when some sensor modalities are missing, noisy, or partially corrupted at inference time?

Thank you for raising this point. Our model is trained to be robust to missing or partially corrupted modalities. During pretraining we encounter natural sensor missingness and apply imputation strategies following established practice  [ref 6, ref 20], which allows the model to learn such patterns.

To further understand the effect of missingness on the performance, we conduct additional experiments on (1) subgroup analysis stratified by percentage of missingness, and (2) comparisons when certain sensor modalities are entirely absent.

As shown in the tables above, while missing modalities and increased missingness ratio lead to slight performance degradation, the model maintains strong robustness and generalization across tasks.

For noisy or corrupted inputs we likewise anticipate some degradation, as is typical for most models; however, consistent with the empirical scaling laws reported in Appendix C.1, SensorLM benefits from large-scale, diverse training data that may help mitigate this effect.

We will report these results and add a robustness discussion in the revised manuscript.

We hope the above responses have addressed all concerns from the reviewer adequately. We would also like to update the reviewer on our commitment to open‑sourcing SensorLM:

• Dataset. Under our IRB‑approved protocol, we have obtained informed consent to release a de‑identified version of the downstream dataset, including hypertension, anxiety, age, and BMI labels. Access will be limited to researchers with verified academic affiliations.
• Pretrained model. We will release SensorLM checkpoints trained on these data together with example code for inference and fine-tuning.
• Code. We will open-source the complete caption-generation pipeline to facilitate reuse by the broader research community.

Once again, thanks for your time and constructive feedback. We would really appreciate it if the reviewer could consider raising their scores and championing the paper.

Thank you for your thoughtful feedback, your comments have helped us strengthen the evidence for our approach. However, we believe that there are several important misunderstandings which we would like to clarify and address point-to-point:

W1: The main contribution of this work is… it is critical to publish the dataset.

W5: The paper would be more suitable for the dataset track instead of the research track.

We respectfully disagree with this assessment. We would like to clarify that SensorLM’s key contribution extends well beyond a “sensor‑text dataset”, addressing the full pipeline of sensor‑language pretraining with several advances:

1. Defining paired language for sensor data where none naturally exist. We introduce a scalable hierarchical caption framework, enabling alignment learning at scale, resulting in the largest sensor‑language corpus to date, orders of magnitude larger than prior studies (Table 1).
2. Determining what makes for effective captions. Contrary to the common assumption [ref 36], we show that relying solely on high‑level semantic descriptions is sub‑optimal across tasks (see W2 responses).
3. Establishing empirical scaling laws for sensor‑language alignment. We conduct the first large‑scale study of sensor-language alignment and demonstrate clear scaling laws (also noted by Reviewer wWho).
4. Providing a modular and generic pre‑training framework. The framework makes it straightforward to attach, remove, or re‑weight objectives. This modularity allows systematic ablations, fast integration of new components, and reveals design trade‑offs unexplored in prior image-text studies (see W3 responses).

These methodological, empirical, and conceptual advances go well beyond dataset construction and align with the expectations of the main track. Importantly, they align with the NeurIPS reviewer guideline that

”originality does not necessarily require introducing an entirely new method but can come from new insights, novel combinations, or improved understanding of existing approaches”

Open‑source commitment. In addition, we would also like to update the reviewer on our commitment to open‑sourcing SensorLM:

• Dataset. Under our IRB‑approved protocol, we have obtained informed consent to release a de‑identified version of the downstream dataset (including hypertension, anxiety, age, and BMI labels) to researchers with verified academic affiliations.
• Pretrained model. We will release SensorLM checkpoints trained on these data with example code for inference and fine-tuning.
• Code. We will open-source the complete caption-generation pipeline to benefit the broader research community.

We hope these clarified contributions, along with our open-sourcing commitment, encourage the reviewer to reassess our paper’s novelty and significance.

W2: The quality of the dataset is not studied…how the hierarchical captions impact the performance.

Thanks for the suggestion. We would like to clarify that the dataset quality has been carefully benchmarked with established SSL methods.

First, SSL baselines benchmarked under the few-shot protocol in Sec. 5.1 (Appendix Table 19, Fig. 9) show that models pretrained with SensorLM and hierarchical captions consistently surpass SSL baselines trained on the same data.

In addition, we adapt UniMTS [ref 36], a foundation model capable of zero-shot classification by using a CLIP objective and semantic activity caption, to our dataset (row 1 below) and examine how our hierarchical captioning strategies (rows 2-3 below) affect performance:

The table shows that adding structural captions improves zero‑shot activity recognition, whereas statistical captions benefits metabolic‑health tasks, consistent with our ablation study (Table 4, Sec. 5.3). These findings confirm both dataset quality and the effectiveness of the hierarchical caption strategy.

We will incorporate the additional baseline and analyses into the revised manuscript.

W3: The novelty of the SensorLM model…benefits of tuning the loss weights $\lambda$.

We respectfully disagree with the claim. Our contribution extends beyond architectural design (see our responses to W1) and addresses the full pipeline of sensor‑language pretraining, offering new insights into aligning wearables with natural language.

While our architecture skeleton follows established image-text paradigms, our work introduces a modular and generic pretraining framework that makes it straightforward to attach, remove, or re‑weight objectives. This modularity allows systematic ablations, rapid integration of new components, and reveals design tradeoffs unexplored in prior image-text studies.

To support this claim, we study two extensions that would be less practical without such modular design.

(1) Loss-weight exploration. Per reviewer suggestion, we sweep the relative weights of the contrastive ($\lambda_{con}$) and captioning ($\lambda_{cap}$) losses beyond fixed settings by CLIP ($\lambda_{con}=0$) and CoCa ($\lambda_{con}=\lambda_{cap}=1$). We find clear task‑specific trade‑offs: a 1:2 ratio boosts zero‑shot activity recognition, whereas 5:1 yields the best metabolic‑health prediction. These results show that careful tuning, enabled by our framework, produces measurable gains.

(2) Sensor-reconstruction head. We append either an auto‑encoder (AE) or vector‑quantized auto‑encoder (VQ‑AE) decoder to the sensor encoder, so that training jointly optimizes contrastive alignment, captioning, and signal reconstruction​. Adding the reconstruction loss improves health‑prediction metrics with only a modest drop in activity recognition. The VQ‑AE variant achieves the best overall trade‑off:

These extensions, enabled by the modular framework, show that it supports rapid experimentation and yields concrete performance benefits. Again, this is consistent with the NeurIPS reviewer guideline:

”originality does not necessarily require introducing an entirely new method when the work enables new insights or capabilities"

We hope the clarifications and new results clarify the significance of our contribution. We will incorporate the discussions and results into the manuscript.

W4: In Tables 2 and 3, the results of the baselines are nearly random… fine-tuned using the proposed dataset first, then make inferences and comparisons.

Thank you for the comment. We acknowledge that the LLM baselines in Tables 2 and 3 were not originally designed for high-dimensional time-series reasoning.

To address the reviewer’s concern, we fine-tuned Gemini 2.0 Flash on our dataset using supervised activity labels, following a standard supervised fine-tuning (SFT) procedure. We show zero‑shot balanced accuracy under the same protocol as Table 2:

The table confirms that fine‑tuning yields only marginal gains over the vanilla Gemini, and still falls well short of SensorLM, despite Gemini 2.0 Flash being substantially larger.

These results highlight two persistent limitations of (fine‑tuned) LLMs:

1. Due to intrinsic context‑length constraints, LLM requires downsampling to interpret raw sensor data, limiting their ability to capture fine‑grained dynamics;
2. SFT on supervised activity labels reduces instruction‑following ability of LLMs in other tasks, leading to limited zero‑shot generalization.

We will add the fine‑tuned results in the manuscript and include UniMTS (see our response to W2) as a stronger zero-shot baseline for a more thorough comparison.

Q1: What does it mean by "zero-shot" in your context?...collected separately from your training set?

Thanks for the question. By “zero‑shot” we adopt the convention used in CLIP and CoCa: the model is evaluated exactly as trained, with no task‑specific tuning or calibration. Our test cohorts are participant‑independent and entirely disjoint from the pretraining pool.

Additionally, for categorical tasks (Table 2) and unseen activity classes (Fig. 6), we present label names never observed during pretraining; the model must map them to sensor patterns through the learnt sensor-language representation.

We hope these clarifications and our open-sourcing commitment encourage the reviewer to reassess the contributions of the paper. We are more than happy to answer more questions. If the discussions and results successfully addressed all your concerns, please consider raising your score.

Thank you for your thoughtful and constructive feedback! We are happy to learn that you found the experiments extensive and the paper well-structured. Below, we provide additional results and clarifications, which we hope will encourage the reviewer to increase the score.

W1: All downstream tasks rely on a fixed 26 features... multiple sensors each supply their own 26-D vectors.

Thank you for raising this. We provide further clarifications and results as follows.

First, we use 26 features (Table 6, Appendix A.1) as a generalizable superset derived from common smartwatch sensors (ACC, PPG, EDA, etc.) across > 30 commercial devices. Our pretraining set covers 18 device models (e.g., Google Pixel Watch 1-3, Fitbit Sense 1-2, Charge 4-6, Versa 1-4, Inspire 2-3), while the metabolic downstream set has an additional 17 devices (e.g., Fitbit Flex, One, Zip, Charge HR, Charge 3). The superior performance across devices/populations (Fig. 5, Table 19) validates the generalizability of derived features, which reflects real-world deployment across hardware platforms.

In addition, not all devices offer the complete set of 26 features. In both pre‑training and downstream sets, sensor missingness arises naturally (e.g., many devices like Fitbit Versa or Charge lack EDA sensors). For instance, the Metabolic set has on average 11 ($\pm$ 5) sensor features completely absent; in such cases we imputed values using the population mean computed from pre‑training data, following established practice [ref 6, ref 20].

Consequently, we report comparisons when certain sensor modalities are unavailable:

This confirms that SensorLM remains robust to missing sensor modalities (results in the main paper included all missingness scenarios).

While we acknowledge that the current feature set may not be exhaustive, we would like to emphasize that the captioning pipeline and modeling framework are not hard-coded to these 26 features. The framework is designed for extensibility: new features can be incorporated by extending the text description without architectural modification. While our feature set already aligns with sensors on 30+ commercial devices, device-specific models could also be trained and their embeddings/predictions can be aggregated at inference time [ref 1, ref 33].

To summarize, we believe the current design is extensive and representative for sensor‑language modeling, and we hope it will serve as a sound basis for future work. We will add the above results and discussions on sensor feature dependency in the revised manuscript.

W2: Zero-shot evaluation is compared only with Gemini/Gemma .. making claims of superiority less convincing.

Thanks for pointing out these related works. We would like to offer clarifications and results as follows.

First, among the cited methods, Moment [1] and Chronos [2] target zero‑shot forecasting, not classification or retrieval. Mantis [3] offers zero-shot embedding extraction similar to our SSL baselines. All three require training a classifier for downstream evaluation, placing them in the few‑shot rather than actual zero‑shot setting (i.e., no tuning or adaptation).

UniMTS [ref 36] is the closest approach of comparison. We discussed it in our related work and now we directly compare with it. Specifically, it employs a CLIP [ref 21] objective and treats activity descriptions as text input, corresponding to the semantic caption in our framework. We have adapted and trained UniMTS under our setup and report the results below:

The table confirms that SensorLM outperforms UniMTS across zero-shot tasks. We note that the benefits stem from (1) our hierarchical caption design, where relying solely on semantic captions is sub‑optimal across tasks (Table 4, Sec. 5.3), and (2) our modularized training framework, which enables systematic comparison of objectives (CoCa proves more effective than CLIP; Table 5, Sec. 5.3).

We will incorporate this additional baseline and extend the discussion of related works as suggested by the reviewer in the revised manuscript.

W3: Training and test sets come from the same user cohort ... demonstrate genuine robustness.

We believe there is a misunderstanding here by the reviewer. We would like to clarify that all tasks in our paper employ a leave-person-out evaluation protocol, as specified by the train and test cohorts in Table 7 (Appendix).  Our statement that the data are “from the same population” in Sec. 5 refers only to shared recruitment criteria and overall distribution, not to any overlap of participants between training and test sets.  We will revise this wording to eliminate ambiguity and emphasize our leave-person-out evaluation.

On a related note, in addition to the default leave‑person‑out evaluation, we also assess cross‑device generalization on the metabolic tasks by comparing in-domain (N=83,243) and out-of-domain (N=6,943) devices (as discussed in W1).  Following the protocol in Sec. 5.1, we report few-shot classification and regression results:

As shown, the model maintains strong OOD performance: while “BMI” and “Hypertension” show minor variation, “Anxiety” and “Age” exhibit slight improvements under OOD settings.

We believe the clarified leave‑person‑out protocol and the results on OOD devices address the concern and demonstrate SensorLM’s robustness. We will incorporate the analysis and update the manuscript accordingly.

W4: The architecture largely reuses established image-text paradigms (ITC, CoCa) with minimal conceptual change.

We appreciate the comment and would like to clarify our architectural contribution. Although the architecture skeleton follows established image-text paradigms, our work introduces a modular and generic pre‑training framework that makes it straightforward to attach, remove, or re‑weight objectives. This modularity allows systematic ablations, enables rapid integration of new components, and reveals design trade‑offs not explored in prior image-text studies.

To support this claim, we study two extensions that would be less practical without such modular design.

(1) Loss-weight exploration. We sweep the relative weights of the contrastive ($\lambda_{con}$) and captioning ($\lambda_{cap}$) losses beyond fixed settings by CLIP ($\lambda_{con}=0$) and CoCa ($\lambda_{con}=\lambda_{cap}=1$). We find clear task‑specific trade‑offs: a 1:2 balance improves zero‑shot activity recognition, whereas 5:1 yields the best metabolic‑health prediction. These results indicate that careful tuning, enabled by our framework, yields measurable gains:

(2) Sensor-reconstruction head. We append either an auto‑encoder (AE) or vector‑quantized auto‑encoder (VQ‑AE) decoder to the sensor encoder so that training jointly optimizes contrastive alignment, caption generation, and signal reconstruction. Adding the reconstruction loss improves health‑prediction metrics with only a modest drop in activity recognition. The VQ‑AE variant achieves the best overall trade‑off:

These straightforward extensions, enabled by the flexibility of the modular framework, demonstrate its support for rapid experimentation and yield concrete performance benefits. This is consistent with the NeurIPS reviewer guideline:

”originality does not necessarily require introducing an entirely new method when the work enables new insights or capabilities"

We hope these clarifications and new results highlight the significance of our contribution. We will incorporate the above discussions and results into the manuscript.

References:

1. Goswami et al. "Moment: A family of open time-series foundation models." ICML 2024.
2. Ansari et al. "Chronos: Learning the language of time series." TMLR 2024.
3. Feofanov et al. "Mantis: Lightweight calibrated foundation model for user-friendly time series classification." arXiv (2025).

We hope the above responses have addressed all concerns from the reviewer adequately. We would also like to update the reviewer on our commitment to open‑sourcing SensorLM:

• Dataset. Under our IRB‑approved protocol, we have obtained informed consent to release a de‑identified version of the downstream dataset, including hypertension, anxiety, age, and BMI labels. Access will be limited to researchers with verified academic affiliations.
• Pretrained model. We will release SensorLM checkpoints trained on these data together with example code for inference and fine-tuning.
• Code. We will open-source the complete caption-generation pipeline to facilitate reuse by the broader research community.

Once again, thanks for your time and constructive feedback. We would really appreciate it if the reviewer could consider raising their scores and championing the paper.

I appreciate the effort that the authors put into this rebuttal. In the rebuttal, the authors promised to release the downstream dataset, pretrained model and the codes. It's a pity that the full dataset cannot be released (which I totally understand), as the power of this work is the large-scale captioned sensor data. As demonstrated in the experiments, scaling plays a big role in this work (even though the gains beyond 12 million hours diminish). With such large-scale data, it is possible to leverage a relatively simple model architecture to achieve good downstream performance.

Thank you for your supportive remarks! We are delighted that the experiments and analyses were interesting and thorough to you. Below, we address your concerns in detail. We hope that these will further clarify our work and lead to a favorable increase of the score.

W1 & Q1: Can the proposed pipeline or pretrained model generalize to out-of-domain data?

Thank you for your constructive comment. We agree that out-of-domain generalization is crucial. First, we would like to clarify that we trained and tested our models with a diverse set of device types (more than 30). Our pretraining set included 18 smartwatch and fitness-tracker models (e.g., Google Pixel Watch 1-3, Fitbit Sense 1-2, Charge 4-6, Versa 1-4, Inspire 2-3). The activity downstream dataset used the same devices, while the metabolic downstream dataset has an additional 17 device types (e.g., Fitbit Flex, One, Zip, Charge HR, Charge 3). A complete device distribution will be provided in the Appendix.

As a result, we are able to demonstrate generalizability on the metabolic test set by comparing performance on in-domain (N=83,243) and out-of-domain (N=6,943) devices. We follow the setting in the main paper (Sec. 5.1) and report classification (AUROC) and regression (MAE) results:

As the table confirms, the model exhibits sound OOD generalization: while “BMI” and “Hypertension” predictions show slight variation, “Anxiety” and “Age” predictions may improve under OOD conditions.

Moreover, although training a foundation model on every possible device is impractical, our approach mitigates this challenge by operating on minutely features (e.g., heart rate, step count) derived from raw sensor data, which are inherently more consistent across devices. To support this, we present the distribution of “heart rate” measurements across representative devices:

This confirms the robustness of the derived features across hardwares. We will add the results to the revised manuscript.

W2 & Q2: Does the modality alignment strategy weaken modality-specific representations?

Thanks for the question. We provide further discussions and clarifications as follows.

Existing work in the image-language domain indicates that straightforward vision-language alignment objectives, when trained on large datasets, yield vision‑only encoders with SOTA performance [ref 21, ref 33]. Following the empirical scaling behavior observed for sensor-text data (see Sec. 5.2), we expect that the same simple objective, applied at scale, will produce strong sensor‑only encoders.

Our experiments support this: Fig. 5 and Appendix C.2 compare embeddings learned by our sensor-language model with sensor-only baselines, showing that our model outperforms across tasks and data sizes. This suggests that the alignment strategy preserves modality‑specific information relative to unimodal objectives.

To summarize, we believe the current design should be extensive and representative for sensor-language modeling research. We hope it will serve as a sound basis and inspire future work. We certainly welcome any further specific suggestions regarding possible extensions.

W3 & Q3: Can SensorLM handle scenarios with missing modalities or varying modality combinations?

Thanks for pointing this out, and we apologize if it wasn’t clear. In fact, in both our pretraining and downstream datasets, sensor missingness arises naturally (e.g., devices like Fitbit Versa or Charge lack EDA sensors). We have performed thorough preprocessing for data missingness following established practices [ref 6, ref 20]: When an entire sensor modality is absent, we impute values with the population mean computed from the pretraining data. For temporal gaps within a sensor stream (e.g., dropouts), we apply linear interpolation across gaps and forward/backward filling at sequence boundaries.

Consequently, SensorLM is trained on data with up to 80% missingness. It learns to model missingness patterns and shows robustness across scenarios.

To further address the reviewer’s concern about performance under missingness, we conducted additional experiments on (1) subgroup analysis stratified by percentage of missingness, and (2) comparisons when certain sensor modalities are entirely absent.

First, we observe only mild degradation as missingness increases from <40% to 60–80%:

Second, when specific sensor modalities (e.g., EDA or HRV) are entirely absent, we note a moderate AUROC decrease (0.02–0.09) relative to the full‑modality setting:

These findings confirm that, although performance degrades slightly under missingness, SensorLM remains robust and flexible with incomplete sensor data. All results reported in the paper are aggregated over the entire test set, which already includes these missing‑data scenarios. We will incorporate the new analyses in the revised manuscript.

W4 & Q4: Could the authors provide more details on the design and implementation of the hierarchical caption?

We appreciate the reviewer’s interest. We would like to clarify that definitions of the statistical, structural, and semantic levels are presented in Fig. 1B and Fig. 3, with rewrite templates in Appendix A.2. We provided additional representative examples in Fig. 4 and Figs. 10, 11.

When levels are combined, captions are concatenated in the sequence: (semantic, structural, statistical). Their interaction and individual contributions are studied in Table 4 and Table 23 (Appendix), where results indicate that leveraging both shared and modality-specific information improves downstream performance.

For completeness, we will include a full example in the Appendix. We welcome any further questions regarding specific elements of the caption hierarchy.

W5: Given the large number of participants involved … practical guidance on data annotation would improve reproducibility and usability.

Thank you for the suggestion. We would like to refer the reviewer to details in Sec. 3.2 and Sec. 5.

For semantic captions and downstream task labels, all event-level information (including user activity, mood, disease diagnoses, demographic data) is self-reported by the user. This constitutes our "hard" labeling. Sensor-derived statistics and structural patterns are computed programmatically from raw signals. We apply sliding windows and fit linear regression within each window to categorize trends (e.g., increasing, decreasing) based on slope thresholds. We also use peak detection algorithms (scipy.signal.find_peaks) to identify spikes and drops.

This design ensures the approach is both scalable and reproducible. Structural and statistical captions can be derived from any comparable sensor data, and the semantic component can be extended to other self-reported or externally annotated information.

We fully agree with the reviewer that this pipeline can benefit the broader research community. To further support adoptions, we will open source the captioning code and provide practical documentation in the final release.

W6 & Q5: Is the proposed approach extendable to unaligned multimodal data?

Thanks for raising this. We would like to further discuss several aspects.

First, for the input features, we compute minutely features from raw sensor signals, which reduces misalignment issues when aggregating data from diverse devices and sensors [ref 6, ref 20]. The OOD performance reported in W1 & Q1 supports its robustness.

In addition, when data from certain sensor channels or intervals are missing, we apply imputation and related preprocessing procedures established in prior work. The results reported in W3 & Q3 confirms that SensorLM generalizes well under these conditions.

Beyond these measures, SensorLM is readily extendable to handle more irregular data. This can be achieved by integrating established methods (e.g., appending a missingness mask or using alternative tokenization strategies [1, 2]) within the sensor encoder, which are orthogonal to our main framework and can be seamlessly incorporated.

We will add the above analyses and discussions in the revised manuscript.

References:

1. Beebe-Wang et al. "PAITS: pretraining and augmentation for irregularly-sampled time series." arXiv preprint (2023).
2. Erturk et al. "Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions." ICML 2025.

We hope the above responses have addressed all concerns from the reviewer adequately. We would also like to update the reviewer on our commitment to open‑sourcing SensorLM:

• Dataset. Under our IRB‑approved protocol, we have obtained informed consent to release a de‑identified version of the downstream dataset, including hypertension, anxiety, age, and BMI labels. Access will be limited to researchers with verified academic affiliations.
• Pretrained model. We will release SensorLM checkpoints trained on these data together with example code for inference and fine-tuning.
• Code. We will open-source the complete caption-generation pipeline to facilitate reuse by the broader research community.

Once again, thanks for your time and constructive feedback. We would really appreciate it if the reviewer could consider raising their scores and championing the paper.