UniMTS: Unified Pre-training for Motion Time Series

We sincerely thank the reviewer for the valuable feedback and for recognizing our idea, promising results and clear presentation. We have addressed the comments as below.

[1] Importance of synthesized data

We add one experiment by incorporating Capture-24 (one of the largest real data from free-living environments) into pre-training. We randomly sample it to 10% of our synthetic dataset size to mitigate any bias due to its larger size. Table 2 in the general response PDF shows performance gains mainly in datasets with wrist-worn devices (e.g., WISDM, Wharf) and simpler activities (e.g., UCI-HAR), as Capture-24 was collected with wrist-worn devices and limited activity descriptions. However, overall performance slightly declines due to its single device focus and poor label description quality. This highlights the significance of our synthesized data, which has all joint locations and detailed text descriptions mostly absent in current real-world datasets, facilitating generalization across diverse activities and devices.

[2-3] More related works

We appreciate the additional references (pre-training motion time series [1-3], random rotations [4-6]) and will make sure to include them in the final version. We are also open to including any further related works as suggested by the reviewer.

[4] Rotation

Yes, they share the same rotation. Conversations with leading sensor vendors like STM, Invensense and Bosch confirm that acc and gyro are typically calibrated to have negligible misalignment factors, so we can use the same rotation.

[5] Data normalization

We normalize the data to ensure consistency in unit measurements, e.g., standardizing accelerations to $m/s^2$. We will make this clear in the final version.

[6] Data generation

We acknowledge that there exist simulators that help generate motion time series. However, unlike previous works, we jointly synthesize data across the entire body, and model their correlations via graph encoder for better location generalization. We also align time series with text semantics for activity generalization, presenting the first unified pre-training framework.

[7] Zero-shot generalization

ImageBind pre-trained on Ego4D data with only head-mounted devices cannot generalize to most activities involving different device placements. This highlights the importance of synthesized data covering all joints. UniMTS and ImageBind predict unseen activities by matching time series embeddings with text label candidates, selecting the one with the highest similarity score as shown in Figure 3 of the paper.

[8] Zero-shot setting

Baselines are trained on their original pre-training corpus (e.g., ImageBind, IMU2CLIP on Ego4D) without fine-tuning on synthetic data. We add one experiment by continually training IMU2CLIP on our synthetic dataset as shown in Table 1 of the general response PDF. IMU2CLIP improves after training on the synthetic data, which again highlights the importance of our synthesized data pre-training. Yet, it still underperforms UniMTS, as our graph encoder better models cross-device spatio-temporal relationship.

[9] Documents

iOS uses inertial force and Android uses accelerating force. We noted such sign discrepancy when collecting data using devices from both OS platforms. This is also confirmed in the Sensor Logger App manual. Rebuttal policy excludes external links, and we will include them in the final version.

[10] Domain adaptation (DA)

Existing DA methods mostly assume identical label names and activity counts between source and target, such as cross-user and cross-dataset DA only for common classes [7-9]. We aim for a more challenging scenario with varying label names across datasets, which existing DA generally cannot address. We will include DA methods as references and make this clear in the final version.

[11] Zero-shot performance

Real-world motion time series classification in a zero-shot manner is extremely challenging, yet we’ve shown a 342.3% improvement over the best baseline. The score also averages ~60% when we retrieve the top-2 activities. Moreover, fine-tuning on 1-shot/10-shot per class further enhances the model to ~60%/~75% for most datasets.

[12] Cross-dataset experiment

The zero-shot experiment shows how our pipeline generalizes across datasets. It works the same by substituting the synthetic data with others like Capture-24 (see Response [1]). We use synthetic data for its high quality with all-joint coverage and detailed descriptions.

[13] Full-shot baseline

Our compared baselines like GPT4TS (NeurIPS'23), SHARE (CIKM'23), ImageBind (ICCV'23), IMU2CLIP (EMNLP'23) are all SOTA general motion time series classification models. There are more recent papers but they mostly focus on sub-domains such as low-resource [1] or federated learning [10] scenarios instead of general classification.

Reference

[1] Generalizable low-resource activity recognition with diverse and discriminative representation learning, KDD 2023

[2] What makes good contrastive learning on small-scale wearable-based tasks? KDD 2022

[3] Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications, SenSys 2021

[4] Device orientation independent human activity recognition model for patient monitoring based on triaxial acceleration

[5] Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks

[6] Practically Adopting Human Activity Recognition, MobiCom 2023

[7] SWL-Adapt: An unsupervised domain adaptation model with sample weight learning for cross-user wearable human activity recognition, AAAI 2023

[8] Semantic-discriminative mixup for generalizable sensor-based cross-domain activity recognition, IMWUT 2022

[9] CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised Pretraining, IMWUT 2024

[10] Flames2graph: An interpretable federated multivariate time series classification framework, KDD 2023

We greatly appreciate the reviewer for all the valuable feedback and for recognizing our technical contributions, novelty of our method, thoroughness of our experiments and the clarity of our paper presentation. We have addressed the comments as below.

[1] Baselines trained on BioBank

We thank the reviewer for mentioning these related works. We will make sure to include them in the final version. We have conducted experiments to compare UniMTS with the most recent work of Yuan et al. 2024 which is pre-trained on the BioBank dataset. We would like to emphasize that

a. The BioBank dataset is limited to signals collected from wrist-worn devices, offering less diversity compared to our synthetic dataset that covers joint locations across the entire body.

b. Yuan et al. 2024 is pre-trained through self-supervised learning and does not involve text semantics aligning, so it cannot be applied for zero-shot recognition as UniMTS does. This highlights the advantages and generalizability of UniMTS.

We compare the few-shot fine-tuning and full-shot performances of UniMTS vs Yuan et al. 2024 respectively in Figure 1 and Table 3 of the attached PDF in the general response. UniMTS performs consistently better in both few-shot and full-shot scenarios, demonstrating the effectiveness of our pre-training pipeline.

[2] Add Capture-24 in pre-training

We have conducted experiments by incorporating Capture-24 dataset into pre-training. We randomly sample it to 10% of our synthetic dataset size, in order to mitigate any bias due to its larger size. We report the zero-shot performance of UniMTS after pre-training with Capture-24 in Table 2 of the attached PDF in the general response.

Notably, performance improvements are primarily observed in downstream datasets that utilize devices attached near the wrist (e.g., WISDM, Wharf) and datasets with simpler activities (e.g., UCI-HAR), which correlates with Capture-24’s collection procedure with wrist-worn devices and limited activity descriptions. However, there is a slight decline in overall performance, attributed mainly to the dataset's focus on a single device location and the poorer quality of label descriptions.

These findings highlight the significance of our synthetic data pre-training, which has high-quality all-joint coverage and detailed, diverse text descriptions mostly absent in current real-world datasets, facilitating generalization across varied activities and devices. We also note a similar trend when increasing the proportion of Capture-24 in our pre-training corpus.

[3] New references

We thank the reviewer for pointing to these related works. We will ensure to add these references in the final version.

[4] Data format during fine-tuning and sampling frequency

The input data are of shape [batch size, sequence length, number of joints, number of feature channels] during fine-tuning. We assign data to specific joint locations according to where the downstream devices are attached, and assign zeros to the remaining joint locations. The sampling frequency is 20 Hz, consistent with the HumanML3D dataset, which is enough for activity recognition tasks.

[5] Release of pre-trained model weights

We will release both pre-trained model weights and all the code upon acceptance of the paper.

Thank you very much for the helpful feedback.

Your evals make sense. I fear that the issue with existing benchmark is that it is very easy to overfit on tiny test set (data with a few subjects) with a large model. So I wouldn't necessarily trust the test results even though they conform with existing literature for consistency sake.

This work is of high quality and has high impact to the field of ubiquitous computing and moderate impact on several other fields including pre-training using synthetic data and multi-modal learning.

I thoroughly enjoyed reading this paper. Looking forward to seeing the final version this manuscript and the open release of the model!

Thank you for all the additional experiments that provided pre-training based on synthetic data compared to biobank pre-training. I just have two more comments and questions:

1. In your Figure 1 comparing BioBankSSL and UniMTS, the few-shot few-tuning performance might be too good to be true, at least in the first instance. How are you doing the train/test split? Currently, the axis is the number of samples from each activity class. If your train/test split is not subject-wise, then the test result is biased.

2. When you incorporated real data for pre-training (wrist-worn data), your evals on wrist datasets improved despite the evals on other datasets decreased. This is telling me that realistic data is still better than synthetic data if available. I wouldn't expect a model pre-trained on the wrist to work well with other placements, for example. It may be worth discussing the value of realistic and synthetic data in different use cases. Given this observation, I was surprised to find that your BioBankSSL experiments showed that synthetic pre-training on wrist data did better.

Overall, human activity recognition has poor benchmark datasets, often extremely limited in scale (n<50 participants). I wouldn't hold the authors accountable for addressing the benchmark issue for the community. It will take time, but once this paper is accepted and externally validated by other researchers, we can better test the generability of the proposed framework.

You have done a great job addressing most of my concerns, and I have adjusted my score from 6->7.

Obviously, it would be great if you could reply to my new queries above :D

We express our gratitude for the reviewer’s constructive feedback, and for recognizing the soundness of our technical claims, the scale and performance of our experiments, as well as the clarity of our paper’s presentation. We have addressed each of the concerns as below.

[1] Contribution of the paper

a. We propose the first unified pre-training framework for motion time series. While large-scale pre-training is common in vision and audio domains, it is challenging to develop similar pre-trained models given the diversity of motion time series data due to variations in device locations, device mounting orientations, and human activity types across different datasets. Existing works mostly train and test their models on a single type of datasets with limited generalizability. Our proposed UniMTS framework for the first time addresses all these challenges and generalizes to diverse downstream datasets.

b. UniMTS generalizes across datasets with various device locations. Real-world motion time series datasets typically focus on different device locations, and no single real-world dataset provides comprehensive location coverage. To address this gap, we propose to synthesize motion time series from motion skeletons with all-joint coverage. By utilizing a graph encoder, we effectively model the spatio-temporal correlations across these joints, enhancing the generalizability of UniMTS across diverse device locations.

c. UniMTS also generalizes across device orientations. The device mounting orientations could significantly affect the downstream performance. To ensure the model remains robust regardless of how the devices are oriented, we propose to use rotation-invariant augmentation techniques during pre-training.

d. Additionally, UniMTS generalizes across different activity types. We align motion time series with text semantics through contrastive learning, enabling classification beyond predefined class labels and supporting arbitrary activity types. To improve the diversity and generalization of activity type representations, we further use text augmentations from LLMs.

We would like to highlight that utilizing LLMs is not our main contribution. Our main contribution is the novel pre-training framework for motion time series, which represents a significant advancement unseen in existing literature. LLMs are just one of the techniques we use to augment text descriptions for enhanced pre-training activity diversity. In addition to this technique, we have proposed various methods, as outlined above, to improve generalization across multiple aspects.

[2] Which LLM is used

We use GPT-3.5 (“gpt-3.5-turbo”) as discussed in Section 4.1 in the paper. We will make sure to mention this in earlier sections in the final version.

[3] Paper scope

Our work is of significant relevance to various key topics that have a wide audience in NeurIPS, including time series analysis, machine learning for healthcare, pre-training, physics-based simulation, and motion modeling. One of the key challenges in these communities is to improve model’s generalizability across diverse datasets. Our contribution falls under this area by designing the first unified pre-training procedure for motion time series, which successfully generalizes to various device locations, device orientations and activities. We believe our insights are valuable to the community and align well with the scope of NeurIPS.

[4] Domain adaptation and continual learning methods

Existing domain adaptation methods mostly assume that source and target datasets share the same label names and have the same number of activities, such as cross-user domain adaptation and cross-dataset domain adaptation only for those common classes [1-3]. We aim for a more generic yet challenging generalization scenario where pre-training and downstream datasets share different label names. Therefore, existing domain adaptation methods generally cannot be applied to address the activity generalization challenge that our work seeks to overcome. Moreover, continual learning [4] typically involves training on the new activities with the risk of catastrophic forgetting. In contrast, our model is able to generalize to new activities in a zero-shot fashion. We will include domain adaptation and continual learning methods as references and make this clear in the final version.

Reference

[1] Hu, R., Chen, L., Miao, S., & Tang, X, SWL-Adapt: An unsupervised domain adaptation model with sample weight learning for cross-user wearable human activity recognition, AAAI 2023.

[2] Lu, W., Wang, J., Chen, Y., Pan, S. J., Hu, C., & Qin, X, Semantic-discriminative mixup for generalizable sensor-based cross-domain activity recognition, IMWUT 2022.

[3] Hong, Z., Li, Z., Zhong, S., Lyu, W., Wang, H., Ding, Y., He, T. and Zhang, D., CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised Pretraining, IMWUT 2024.

[4] Jha, S., Schiemer, M., Zambonelli, F. and Ye, J., Continual learning in sensor-based human activity recognition: An empirical benchmark analysis, Information Sciences, 2021.

We sincerely thank the reviewer for the valuable comments and for recognizing the effectiveness and clarity of our work. We have addressed each comment as below.

[1] How to obtain text descriptions

We use the original text descriptions in the motion skeleton dataset HumanML3D. As detailed in Appendix A.1 of our paper, there is an average of 3 textual descriptions paired with each motion skeleton sequence in the HumanML3D dataset, resulting in a total of 44,970 textual descriptions with a vocabulary size of 5,371. Some example text descriptions are “the standing person kicks with their left foot before going back to their original stance”, “a man lifts something on his left and places it down on his right”, “a person walks slowly forward then toward the left hand side and stands facing that direction”.

[2] Usage of LLM

We apply LLM to generate paraphrases of original text descriptions in the HumanML3D dataset to further increase the diversity. This allows UniMTS to better learn the text semantics and generalize to varied label descriptions in the downstream datasets. For instance, consider the initial description of a skeleton motion: “a person falls to the ground in a sitting motion and then pops back up in a standing position”. From this, the LLM generates three distinct paraphrases: “a person drops into a seated position before quickly rising to their feet”, “a man crouches down and quickly jumps up”, and “someone squats low and then springs up”. These variations substantially broaden the range of text descriptions, helping the model in generalizing across various label names in the downstream datasets.

[3] Space and time complexities compared with baselines

a. For space complexity, our graph encoder contains only 4.94M parameters, which is significantly smaller compared with the 18.69M used in the IMU encoder of the best existing baseline ImageBind.

b. For time complexity, our pre-training takes a one-time 32 hours. Our fine-tuning is also efficient. On one example dataset of UCI-HAR, full-shot fine-tuning of UniMTS takes ~1.3 minutes to converge while it takes ~9.8 minutes for ImageBind to converge. Moreover, we have run a power estimate assuming 0.1Hz cadence (i.e., 10-second window size), and it takes ~22.64 mW to run the whole graph model on an eNPU (embedded Neural Processing Unit), which is much smaller than ImageBind IMU encoder’s power consumption of ~702 mW. Therefore, our model is efficient for real-world applications and suitable to be deployed on edge devices.

[4] Our proposed method can improve the efficiency of existing methods

a. As we discussed in Response [3], UniMTS is more efficient than baselines in terms of both time and space complexities, and converges faster during fine-tuning.

b. Our synthetic data pre-training also provides an effective initialization which further improves the fine-tuning efficiency. For example, fine-tuning on downstream datasets takes on average 10 epochs to converge. Moreover, such benefits are model-agnostic and we can also improve the fine-tuning efficiency of baseline models when they are pre-trained on our synthetic data.

We sincerely thank the reviewer for the insightful feedback. We compare the theoretical space and time complexities with the best-performing baseline ImageBind as follows. Since UniMTS and ImageBind share similar text encoders, our analysis mostly focuses on their respective IMU encoders. Let $B$, $L$, and $F$ represent the batch size, sequence length, and feature dimension, respectively. Additionally, let $V$ and $V'$ denote the total number of joints (with $V=22$ for the HumanML3D dataset) and the number of joints with attached devices in downstream datasets ($V' \leq V$). We note that the skeleton graph is sparse and the number of edges is $E=V-1$.

For UniMTS, the time complexity is $\mathcal{O}(BVLF^2 + BVLF)$ and the space complexity is $\mathcal{O}(BVLF + F^2 + V)$. For ImageBind, the time complexity is $\mathcal{O}(BV'LF^2 + BV' L^2F)$ and the space complexity is $\mathcal{O}(BV'LF + F^2 + BV'L^2)$. We observe that UniMTS does not contain the $L^2$ terms present in ImageBind (i.e., complexity grows linearly rather than quadratically with respect to sequence length), making it more efficient in terms of both time and space complexities.

We sincerely thank the reviewer for the valuable feedback, and for recognizing the novelty of our approach and the promising results from our experiments. We have addressed each of the comments as below.

[1] Quality of generated IMU data

UniMTS is aimed for action recognition, so the sampling frequency of 20Hz is sufficient. We leave other tasks such as inertial navigation as future work as we discussed in the “Limitation and Future Work” section in the paper. Our current simulation process has incorporated IMU drift as offset vectors. We will make this clear in the final version. Spike noises are relatively uncommon to occur over a long period of observation window (e.g., a few seconds), especially considering that modern wearable devices have built–in low pass filters, and thus have minimal impact on downstream classification. Therefore, we have chosen to incorporate the more common Gaussian noise during simulation, as specified in Equation 4 in the paper. We will leave exploring other types of noises as future work.

[2] Efficiency of graph encoder

Our graph encoder contains only 4.94M parameters, which is significantly smaller compared with the 18.69M used in the IMU encoder of the best existing baseline ImageBind. On one example dataset of UCI-HAR, full-shot fine-tuning of UniMTS takes ~1.3 minutes to converge while it takes ~9.8 minutes for ImageBind to converge. Moreover, we have run a power estimate assuming 0.1Hz cadence (i.e., 10-second window size), and it takes ~22.64 mW to run the whole graph model on an eNPU (embedded Neural Processing Unit), which is much smaller than ImageBind IMU encoder’s power consumption of ~702 mW. Therefore, our model is efficient for real-world applications and suitable to be deployed on edge devices.

We sincerely thank the reviewer for the insightful feedback, and for recognizing the novelty, experimental rigor, and clarity of our presentation. We have addressed the comments as below.

[1] Ego4D and Ego-Exo4D benchmark

a. We acknowledge that these are important action recognition benchmarks; however, we would like to highlight that these two benchmarks are mainly aimed for multimodal action recognition by incorporating video and audio modalities [1,2]. The motion time series in these datasets are collected from head-mounted devices, and recognizing activities based purely on head movement is a very challenging task. Therefore, only very few papers target this task, such as IMU2CLIP. For example, IMU2CLIP reports an F1 score of only 31.89 for a 4-class action recognition task.

b. We choose Ego4D as an example benchmark for comparison with IMU2CLIP. It’s important to note that models on the Ego4D benchmark are already pre-trained on the Ego4D dataset, but our UniMTS has never seen the Ego4D dataset during pre-training. Therefore, for a fair comparison, we split the Ego4D dataset into a 80% training set and a 20% test set, continue pre-training of UniMTS on the training set, and compare with IMU2CLIP pre-trained on the same training set. We report both accuracy and F1 in the following table. As the original Ego4D dataset only contains text descriptions without label IDs, for the test set, we derive class labels by extracting stem verbs from these original text descriptions, and choose top 10, top 20, top 50 activities for evaluation. During evaluation, for both UniMTS and IMU2CLIP, we compute the similarity of signal embeddings and embeddings of each label candidate. The activity with the highest similarity score is predicted as the label.

We observe that UniMTS also consistently outperforms the current state-of-the-art method on the Ego4D benchmark, demonstrating the effectiveness and generalizability of our pre-training framework.

c. We will explore combining visual features with inertial signals for action recognition as future work. We believe that inertial signals will help the visual features to learn implicit physical constraints, potentially leading to better generalization. We will also include these benchmarks as references in the final version.

[2] Generalizability and potential edge cases

As we discussed in the “Limitation and Future Work” section in the paper, the simulated signals are for body joints, but real signals might be collected near – rather than directly on – the body joints. For example, sensors on smartwatches collect data near the wrist, not on the wrist joint itself. We plan to incorporate random offset vectors to better simulate real-world signal variations near joints. Other potential edge cases include missing data and multi-device synchronization for real-time motion time series classification, which we plan to address by simulating missing data across time and devices during pre-training.

Reference

[1] Tan, S., Nagarajan, T. and Grauman, K., Egodistill: Egocentric head motion distillation for efficient video understanding, NeurIPS 2023.

[2] Liu, M., Ma, L., Somasundaram, K., Li, Y., Grauman, K., Rehg, J.M. and Li, C., Egocentric activity recognition and localization on a 3d map, ECCV 2022.