X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing

Q8: How does the number of the layers in cross-modal transformers impact the experimental results?

Answer: We appreciate the suggestion and added the ablation of the number of the layers (depth) in cross-modal transformers on the XRF55 HAR task. The default depth in our experiments is 1 and the depth we choose in the comparison experiment is 4. The results shown below indicate that adding the number of the layers in the cross-modal transformer will not yield better performance.

Q9: Could the equal attention across all modalities lead to a negative impact in tasks that primarily rely on a specific modality, especially if some modalities contain less informative data?

Answer: The cross-attention module we design would not lead to negative impact in certain tasks, which could be proven in the Table 4 & 5 that compare the X-Fusion results with Transformer-only results across all modality combinations.

The goal of our method is not to enforce a rigid balance across all modalities but to mitigate the risk of over-reliance on dominant modalities, which could result in the loss of valuable information from weaker modalities.

Our cross-attention fusion module achieves this by reintegrating overlooked modality-specific information into the cross-modal embeddings, thereby encouraging the model to also learn how to extract features from weak modalities during the modality invariant training process.

This process does not diminish the model's ability to prioritize and extract features from more informative modalities, which is achieved by the cross-modal transformer structure. Instead, it encourages the model to learn richer and more complementary cross-modal features, ultimately improving its performance across diverse scenarios. This balanced approach ensures robustness and adaptability without compromising the model's effectiveness in tasks that heavily rely on specific modalities.

Q10: Computing all modalities in each iteration increases complexity, especially in multi-modal and long-sequence modeling, posing challenges for real-time applications and practical deployment.

Answer: Actually our inference complexity is only related to the number of modalities we use in a given scenario. For example, the X-Fi is trained on 5 modalities, but in the inference only the radar is used. Then only the radar encoder, cross-attention module, and key-value generator are activated so the complexity is still acceptable.

The training complexity is not increased with the number of modality increased in each iteration, because for each iteration, we train X-Fi on a specific number of modalities and discard other modalities. This mimics the situation when some modalities are missing, enhancing the robustness of X-Fi and controlling the complexity for each iteration. Once the X-Fi model is trained, the model will only compute the corresponding parts for the modalities utilized in the specific application scenario. However, we admit that more iterations might be demanded for the convergence of X-Fi when more modalities are considered.

Reference:

[1] Ma, Mengmeng, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. "Are multimodal transformers robust to missing modality?." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18177-18186. 2022.

[2] Memmel, Marius, Roman Bachmann, and Amir Zamir. "Modality-invariant Visual Odometry for Embodied Vision." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21549-21559. 2023.

[3] Dong, Hao, Ismail Nejjar, Han Sun, Eleni Chatzi, and Olga Fink. "SimMMDG: A simple and effective framework for multi-modal domain generalization." Advances in Neural Information Processing Systems 36 (2023): 78674-78695.

Q5: Are there any ablation studies on the components of the proposed method, including the ablation study of the cross-attention module?

Answer: The ablation study of the number of blocks on the X-Fusion stacked block is shown below.

We find that one X-Fusion underfits on rgb modality and the results on rgb data is much worse than the other two variants' results. The results indicate that one X-Fusion block is not enough to effectively capture modality-specific features of each modality and at least two stacked X-Fusion blocks are required to achieve robust performance.

However, we find no significant difference in performance between using two stacked blocks and four stacked blocks. Thus, increasing the number of stacked blocks further does not enhance performance, as the model converges to a stable optimal minimum.

Q6: Why is the X-Fi LiDAR single input result not as good as the results in MM-Fi, where Lidar achieves good performance?

Answer: We appreciate the question and conducted investigations on the problem. The reason is that we are carrying out modality invariant training on multiple modalities. Different modalities vary in their fitting complexity, and some may remain underfitted with the same number of training samples.

For example, modalities like WiFi and LiDAR often require additional training samples to achieve an adequately fitted state. We could add more LiDAR data samples in the training process by adjusting the hyperparameters of the modality existence list to refine the model performance. For example, when we experimented on MM-Fi HAR task, we first set all modality existence probability to 80% and the model performed very poorly when facing a single LiDAR input. When we increased the LiDAR probability to 90% and reduced other modalities probabilities to 50%, the model performance improved a lot. After that, we further fine-tune the model by incorporating more LiDAR samples, and the single LiDAR result continues to improve. The comparison is illustrated in the table below.

The performance of X-Fi on single LiDAR input continuously improves alongside adding more LiDAR training samples, as highlighted in the table above. This proves that we could adjust the hyperparameters of the modality existence list to add more specified training samples and refine the model performance.

Q7: To ensure a fair comparison in Tab. 4, the number of parameters in the transformer and X-Fusion block should be equalized.

Answer: We appreciate the suggestion. For fair comparison, we added the number of parameters for both the VO Transformer block and the X-Fusion block below.

We would emphasize that the number of active parameters for the VO Transformer block is fixed regardless of the number of modalities in the input combination. However, X-Fi has modality-specific structures that will be only activated when the corresponding modality is in the input and thus the number of parameters will be different for different numbers of modalities in the input combination. From the comparison, we can conclude that the superior performance achieved by X-Fusion is not achieved by only introducing more parameters.

Q4: Are there any empirical evidence to substantiate the modality-specific features extracted into the K-V pairs. Some additional constraints are necessary. For instance, SimMMDG [3] leverages distance loss between modality-specific and modality-shared features.

Answer: We believe this suggestion is quite interesting so we added an experiment. In order to make sure that the modality-specific features are extracted into the K-V pairs and to align all modality-specific features and cross-modal embedding in a shared embedding space, we followed the method proposed in SimMMDG [3] to compute the average distance loss between each modality-specific features and cross-modal embedding. This distance loss was optimized jointly with the task loss during each training iteration.

However, we found the model became hard to converge during training and it is easy to have gradient exploration issues. The training process is illustrated in appendix Figure 8.

To tackle this issue, we tried to adjust the weighting parameter of the distance loss in the joint loss and found that the training became a bit more stable. However, there still exists a sudden gradient exploration issue after 13 training iterations.

In conclusion, we realize that additional constraints between modality-specific features and cross-modal embedding is necessary to ensure effective modality-specific features extraction into the K-V pairs and could enhance model performance. However, further experiments and extensive hyperparameter tuning are required to determine how to introduce such constraints without compromising the stability of modality-invariant training.

Q5: Are there any ablation studies on the components of the proposed method, including the ablation study of the cross-attention module?

Answer: We would first apologize for not clearly explaining the transformer-based fusion comparison in section 4.2 and revised the section for better clarity. In fact, the transformer-based fusion we conducted could serve as an ablation study to evaluate the contribution of the cross-attention module. The transformer-based fusion is a reproduction of the SOTA modality-invariant method, VO Transformer [2], which removes the cross-attention module used for feature injection and only remains a cross-modal Transformer for feature fusion. The comparison results shown in Table 4 and Table 5 demonstrate the essential and effectiveness of the cross-attention module. We have added the references here for better clarification.

We also conducted further ablation studies on the key-value generators, the number of iterations on a X-Fusion iterative block, and the number of blocks of the X-Fusion stacked blocks. The results shown below are added to the appendix in the paper.

The ablation study of key-value generators proves that utilizing key-value generators to obtain key-value pairs is more efficient than directly using initial modality features as k-v.

The ablation study of the number of iterations on a X-Fusion iterative block is shown below.

We find that the performance with 4 iterations is better than the performance with 2 iterations across all modality combinations. The results indicate that when the number of iterations is set to 2, the model finds difficulties in digging and preserving features from weak modalities. We need to set the number of iterations on the X-Fusion block to at least 4 to repeatedly inject information from weaker modalities, enabling the model to more robustly fit all participating modalities during training.

We sincerely appreciate the reviewer rbmV for the detailed review, insightful questions and constructive comments. We are glad that the reviewer acknowledges “the meaningful attempt” we made and “the promising approach” we proposed. Here we answer all the questions and add extensive experiments for clarification. We hope the responses can address the concerns. In addition, we have submitted a revised manuscript with an appendix where we mark all the suggested tables, figures, and analytics in magenta color.

Q1: The section of problem definition is missing and the clear details about the inputs and outputs are crucial for reader to understand

Answer: We appreciate the suggestion and incorporated the problem definition of HPE and HAR tasks in section 3 line 151- 158, along with a detailed description of input formats in Section 4.1.

The definition is:

Assume that input $X$ comprises a random combination of modalities within a specific range, including any number and type of modalities. Our objective is to learn a unified representation $z_{cm}$ that integrates modality-specific features from $X$ by using a modality-invariant foundation model $\mathcal{F}$. This representation can be mapped to various down-stream human sensing task spaces $t_{i}$ by using corresponding task-specific functions $g_{t_{i}}$, such that the task-specified loss function $L_{t_{i}}(g_{t_{i}}(\mathcal{F}(X)); y_{t_{i}})$ is minimized. In this paper, the $y_{t_{i}}$ refers to 3D human joints for HPE task and human activity classes for HAR task. Thus we propose X-Fi, a modality-invariant human sensing framework designed to map the arbitrary multimodal input $X$ to any task-specific target domain $t_{i}$.

For each modality, the input format remains consistent across all tasks. For instance, image frames are used in both HPE and HAR.

Q2: There are some undefined notations, such as in line 172 and in Eq. 4

Answer: We appreciate the corrections. We revised and unified all the notations in both the figures and the text throughout the paper.

Q3: Why encourage paying equal attention across modalities in subsection 3.2.2 when dominant modalities provide more valuable human sensing information and equal weighting could lead to the inclusion of irrelevant information?

Answer: We appreciate the suggestion. The motivation of our cross-attention module is that, previous studies [1] (CVPR 2022) & [2] (CVPR 2023) have shown if one modality disproportionately influences the model, the system will fail to generalize when certain modalities are distorted or missing.

In [1], they reported the observation that ‘the Transformer models degrade dramatically with modal-incomplete data’. They trained a ViT on ‘full’ sets of image and text modalities and tested only with unimodal input. The result on MM-IMDb dataset reveals that the performance drops by 14.3% when missing image, and drops 36.7% when missing text.

Similarly in [2], the trained modality-invariant transformer model marginally depends on the presence of depth modality. ‘Dropping RGB barely decreases performance, while dropping Depth causes the localization to fail more drastically’, as they described.

Thus, balancing contributions allows the model to fully leverage complementary information across modalities and is essential to ensure robust and equitable performance across different scenarios.

Dear reviewer rbmV,

As the rebuttal deadline is approaching in 2 days, could you please have a look at our response? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 6113

Dear reviewer rbmV,

As the rebuttal deadline is approaching within a day, could you please have a look at our response? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 6113

Q3: Would the authors consider conducting further experiments with other cross-modal foundational models (e.g., Meta-Transformer, ImageBind) as baselines to provide a more comprehensive comparison?

Answer: To better demonstrate the significance of our proposed X-Fi. We added further experiments with ImmFusion [3]. We first trained the model using the modality combination from the original paper (I+D+R) and then adapted the method to include all modalities (I+D+R+L+W). The results are shown below and indicate the superior performance of our proposed X-Fi compared to ImmFusion.

Q4: Is it possible to include additional benchmark datasets with IMU data for HAR task?

Answer: Yes, we conducted initial experiments on mRI dataset [6] that includes IMU data.

Our current motivation is centered on passive sensing from a single perspective in order for application scenarios such as autonomous driving and robotics sensing. Thus, all the modalities we utilized are based on passive sensing and the datasets we used are based on these sensors. But our method is easy to be expanded to active perception sensors by incorporating more modality-specific feature tokens into the modality invariant training process.

In the experiment, we used protocol 1 and setting 1 utilized in the original paper and the results shown below demonstrate the superior performance of our proposed X-Fi compared with the baseline.

Reference:

[1] Wu, Jiannan, Yi Jiang, Bin Yan, Huchuan Lu, Zehuan Yuan, and Ping Luo. "UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces." arXiv preprint arXiv:2312.15715 (2023).

[2] Memmel, Marius, Roman Bachmann, and Amir Zamir. "Modality-invariant Visual Odometry for Embodied Vision." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21549-21559. 2023.

[3] Chen, Anjun, Xiangyu Wang, Kun Shi, Shaohao Zhu, Bin Fang, Yingfeng Chen, Jiming Chen, Yuchi Huo, and Qi Ye. "Immfusion: Robust mmwave-rgb fusion for 3d human body reconstruction in all weather conditions." In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2752-2758. IEEE, 2023.

[4] Zhang, Yiyuan, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. "Meta-transformer: A unified framework for multimodal learning." arXiv preprint arXiv:2307.10802 (2023).

[5] Yang, Jianfei, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. "Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing." Advances in Neural Information Processing Systems 36 (2024).

[6] An, Sizhe, Yin Li, and Umit Ogras. "mri: Multi-modal 3d human pose estimation dataset using mmwave, rgb-d, and inertial sensors." Advances in Neural Information Processing Systems 35 (2022): 27414-27426.

We sincerely appreciate the reviewer u1Rk for the insightful questions and constructive comments. We are glad that the reviewer appreciates that “the problem we address is important” and “the paper is clearly written and well-structured”. We address the lack of comparisons to prior methods and add experiments on other SOTA methods to prove the effectiveness of our proposed X-Fi. We also address the lack of evaluating X-Fi on datasets containing IMU sensor and add experiments on the mRI dataset [6] to prove the proposed X-Fi could be expanded to wearable sensors. In addition, we have submitted a revised manuscript with an appendix where we mark all the suggested tables, figures, and analytics in magenta color.

Q1: Add the following citation and briefly investigate the related cross-modal / modality-invariant foundational models.

Answer: We appreciate the suggestion and conduct literature reviews on the recommended papers. The discussion about these approaches and the corresponding citations are included in the related work section 2.2.

Q2: Can the authors provide additional details on the distinguished contribution points of the X-fusion block compared to existing cross-attention fusion approaches?

Answer: Actually cross-attention module design is based on existing cross-attention fusion works. It is a revamped module yet it does not denote our main novelty. The novelty of this paper is to address a practical problem in the human sensing research community: in the real-world sensing scenario, we always demand a single model that is trained only once and then used with different modality combinations of inputs for different applications, without the need for retraining multiple models.

Several gaps are filled by the proposed X-Fi framework, including handling arbitrary number of input modalities, capturing and preserving unique characteristics of each modality, retaining each modality’s distinct features during multimodal fusion, and designing a unified training strategy to enable modality invariant human sensing. Cross-attention module is incorporated specifically to retain each modality’s distinct features during multimodal fusion. Therefore, we slightly modified the UniFusion module in the work UniRef++ [1] and validated its effectiveness on two large human sensing datasets, demonstrating performance improvements.

Q3: Would the authors consider conducting further experiments with other cross-modal foundational models (e.g., Meta-Transformer, ImageBind) as baselines to provide a more comprehensive comparison?

Answer: We appreciate the suggestion. We added experiments with Meta-Transformer [4] to compare our proposed X-Fi with SOTA multimodal fusion approaches that are based on fixed modality combinations. The experiments were conducted on the MM-Fi dataset [5] for the HPE task.

In experiments, we used the Meta-Transformer-B16 encoder as the backbone. Initially, we fine-tuned the pretrained encoder weights (from the original repository) and trained a regression head on all modalities (I+D+R+L+W). Next, we froze the encoder weights and fine-tuned the regression head on I+D+L modalities, obtaining the Meta-Transformer-B16F result. Finally, we fine-tuned both the encoder and the regression head on I+D+L modalities to produce the Meta-Transformer-B16T result.The results shown below demonstrate X-Fi’s superior performance compared to meta-transformer.

Additionally, we have already compared our method with the VO Transformer [2] in the original manuscript Section 4.3. The transformer based fusion method is the VO Transformer structure, and we have added the references here for better clarification. The experiment results have been listed in Table 4 and Table 5 and show our proposed X-Fi perform better on all modality combinations.

Dear reviewer u1Rk,

As the rebuttal deadline is approaching, could you please have a look at our response? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 6113

Dear reviewer u1Rk,

As the rebuttal deadline is approaching in 2 days, could you please have a look at our response? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 6113

Dear reviewer u1Rk,

Thank you for acknowledging that our rebuttal has mostly addressed your initial concerns. We greatly appreciate your engagement and constructive comments.

Currently three reviewers have responded that the rebuttal has addressed their concerns and may increase the score, (while two reviewers have not responded yet). It is believed that the revised manuscript has been signficantly improved by address your concerns and adding many experimental results and analysis. If so, we kindly request if you could consider to increase the rating of our submission, as the current score still remains below the borderline.

Thank you so much for your time and consideration.

Best regards,

Authors of Submission 6113

We sincerely appreciate your helpful suggestions! Thanks for your time and effort!

Q4: I suggest having a joint loss that includes loss for each modality and the X-Fusion output.

Answer: We believe this suggestion is quite interesting so we added an experiment. In order to align each modality-specific features and cross-modal embedding, we followed the method proposed in SimMMDG [3] to compute the average distance loss between each modality-specific features and cross-modal embedding. This distance loss was optimized jointly with the task loss during each training iteration.

However, we found the model became hard to converge during training and it is easy to have gradient explosion issues. The training process is illustrated in appendix Figure 8.

To tackle these issues, we tried to adjust the weighting parameter of the distance loss in the joint loss and found that the training became a bit more stable. However, there still exists a sudden gradient explosion issue after 13 training iterations.

In conclusion, we realize that additional loss between modality-specific features and cross-modal embedding is necessary to align each modality features to a shared embedding space and could enhance model performance. However, further experiments and extensive hyperparameter tuning are required to determine how to introduce such alignment without compromising the stability of modality-invariant training.

Q5: I suggest an ablation study on the number of stacked blocks in X-Fi block.

Answer: We appreciate the suggestion and conducted an ablation study on the number of stacked blocks on MM-Fi HAR task.

We find that one X-Fusion underfits on rgb modality and the results on rgb data is much worse than the other two variants' results. The results indicate that one X-Fusion block is not enough to effectively capture modality-specific features of each modality and at least two stacked X-Fusion blocks are required to achieve robust performance.

However, we find no significant difference in performance between using two stacked blocks and four stacked blocks. Thus, increasing the number of stacked blocks further does not enhance performance, as the model converges to a stable optimal minimum.

Q6: I am expecting an experiment which gradually adds new modality each time.

Answer: We believe this suggestion is quite interesting so we added an experiment. We first trained an X-Fi model from scratch with RGB and Depth modalities. Then we added mmWave modality and fine-tuned the model based on the pretrained parameters. The results are added in appendix Table 9 and shown below.

The results show similar performance after adding mmWave modality and fine-tuning the pretrained model. This indicates that X-Fi could be easily expanded to newly introduced modalities without harming the performance on the existing modalities.

Reference:

[1] Yang, Jianfei, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. "Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing." Advances in Neural Information Processing Systems 36 (2024).

[2] Wang, Fei, Yizhe Lv, Mengdie Zhu, Han Ding, and Jinsong Han. "XRF55: A Radio Frequency Dataset for Human Indoor Action Analysis." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, no. 1 (2024): 1-34.

[3] Memmel, Marius, Roman Bachmann, and Amir Zamir. "Modality-invariant Visual Odometry for Embodied Vision." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21549-21559. 2023.

[4] Chen, Anjun, Xiangyu Wang, Kun Shi, Shaohao Zhu, Bin Fang, Yingfeng Chen, Jiming Chen, Yuchi Huo, and Qi Ye. "Immfusion: Robust mmwave-rgb fusion for 3d human body reconstruction in all weather conditions." In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2752-2758. IEEE, 2023.

[5] Zhang, Yiyuan, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. "Meta-transformer: A unified framework for multimodal learning." arXiv preprint arXiv:2307.10802 (2023).

We sincerely appreciate the reviewer Rmx6 for the insightful questions and constructive comments. We are glad that the reviewer acknowledges "the simple yet effective framework" we proposed. We address the lack of comparisons to prior methods and add experiments on other SOTA methods to prove the effectiveness of our proposed X-Fi. We also add experiments as suggested to boost the performance of X-Fi. In addition, we have submitted a revised manuscript with an appendix where we mark all the suggested tables, figures, and analytics in magenta color.

Q1: The paper needs polishing and revision.

Answer: We appreciate the suggestions. We revised the paper carefully and made corrections to polish the paper.

Q2: Are there any comparisons against the SOTA method? Why only compare with baseline?

Answer: The two baselines we adopted are originally proposed in the MM-Fi [1] and the XRF55 [2] dataset papers. And these results are the SOTA results so far on these datasets after searching through all the multimodal fusion research works based on these datasets.

For modality invariant SOTA method VO Transformer [1], we have already compared our method with VO Transformer in the original manuscript Section 4.3. The transformer based fusion method is the VO Transformer structure, and we have added the references here for better clarification. The experiment results have been listed in Table 4 and Table 5.

As suggested, to better demonstrate the significance of our proposed X-Fi, we expand the ImmFusion method by concatenating multimodal features and expand the VO Transformer method by feeding all modalities’ features into a single transformer structure, which are then compared to our method on the benchmark.

To better demonstrate the significance of our proposed X-Fi. We added experiments with ImmFusion [4] and Meta-Transformer [5] to compare our proposed X-Fi with SOTA multimodal fusion approaches that are based on fixed modality combinations. The experiments were conducted on the MM-Fi dataset [1] for the HPE task.

For ImmFusion [4], we first trained the model using the modality combination from the original paper (I+D+R) and then adapted the method to include all modalities (I+D+R+L+W). The results are shown below and indicate the superior performance of our proposed X-Fi compared to ImmFusion.

For Meta-Transformer [5], we used the Meta-Transformer-B16 encoder as the backbone to carry out experiments. The results shown below demonstrate X-Fi’s superior performance.

Q3: What is the dataset setting used in the experiments?

Answer: We described the dataset details and added the dataset split settings descriptions in section 4.1 line 353 - 354 . The split setting we used for MM-Fi is S1 Random Split and the original split setting for XRF55, as outlined in their respective papers.

Dear Reviewer Rmx6,

As the rebuttal deadline is approaching, could you please have a look at our response? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 6113

Dear reviewer Rmx6,

As the rebuttal deadline is approaching in 2 days, could you please have a look at our response? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 6113

Dear reviewer Rmx6,

Thank you for acknowledging that our rebuttal has mostly addressed your initial concerns. We greatly appreciate your engagement and constructive comments.

Currently three reviewers have responded that the rebuttal has addressed their concerns and may increase the score, (while two reviewers have not responded yet). It is believed that the revised manuscript has been signficantly improved by address your concerns and adding many experimental results and analysis. If so, we kindly request if you could increase the rating of our submission, marking the significance of our revisions according to your valuable suggestions.

Thank you so much for your time and consideration.

Best regards,

Authors of Submission 6113

Reference:

[1] Ma, Mengmeng, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. "Are multimodal transformers robust to missing modality?." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18177-18186. 2022.

[2] Memmel, Marius, Roman Bachmann, and Amir Zamir. "Modality-invariant Visual Odometry for Embodied Vision." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21549-21559. 2023.

[3] Wang, Fei, Yizhe Lv, Mengdie Zhu, Han Ding, and Jinsong Han. "XRF55: A Radio Frequency Dataset for Human Indoor Action Analysis." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, no. 1 (2024): 1-34.

[4] Dai, Shenghong, Shiqi Jiang, Yifan Yang, Ting Cao, Mo Li, Suman Banerjee, and Lili Qiu. "Advancing Multi-Modal Sensing Through Expandable Modality Alignment." arXiv preprint arXiv:2407.17777 (2024).

[5] Yang, Jianfei, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. "Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing." Advances in Neural Information Processing Systems 36 (2024).

[6] Chen, Anjun, Xiangyu Wang, Kun Shi, Shaohao Zhu, Bin Fang, Yingfeng Chen, Jiming Chen, Yuchi Huo, and Qi Ye. "Immfusion: Robust mmwave-rgb fusion for 3d human body reconstruction in all weather conditions." In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2752-2758. IEEE, 2023.

[7] Zhang, Yiyuan, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. "Meta-transformer: A unified framework for multimodal learning." arXiv preprint arXiv:2307.10802 (2023).

[8] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

[9] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

Q12: Why is the cross-modal embedding residual added back in the cross-modal transformer?

Answer: The cross-modal embedding is residual added back to the cross-modal transformer in order to retain cross-modal features. This residual connection is designed by considering that the cross attention fusion modules for each modality will cause the information loss of the previously learned cross-modal embedding, therefore downgrading the fusion efficiency. Such design will retain previously learned cross-modal features as much as possible while injecting the modality-specific information in the cross attention modules.

Q13: Why do we need k-v generators rather than directly use initial modality features as k-v?

Answer: As discussed in the section 3.2.2, the purpose of k-v generators is to preserve and refine the modality-specific information that will be injected back to the fusion process in the cross attention module. To prove that utilizing k-v generators to obtain key-value pairs is more efficient than directly using initial modality features as k-v, we conducted the ablation study of k-v generators on XRF55 HAR task in order to The results shown in the table below supports the efficiency of our k-v generator design and added to the appendix Table 9.

Q14: In Tables 1 and 2 I would suggest labeling the reported metrics either in the column header or caption (it sounds like MSE for HPE and Accuracy for HAR?).

Answer: For HPE, MPJPE and PA-MPJPE are the metrics evaluated and Table 1. has listed the two metrics in the column header in the original manuscript. For HAR, we did miss the description of the metric utilized and we have added the ‘accuracy’ term in the caption to make clear of the metric.

Q15: Can you circle on the images where you are observing that RGB captures better leg movements and depth is better for the arm in Figure 3? Also, any thoughts on why this would be the case?

Answer: We zoomed in the areas where we observed that RGB captures better leg movements and depth is better for the arm, the picture after enlargement is added in the appendix Figure 5.

To seek the reason for such observation, we explored the characteristics of the rgb data and the depth data in MM-Fi. We found that due to the arm's movement typically being faster than the legs, combined with the insufficient frame rate of the camera, the arms in the RGB samples are often blurry, while the legs are very clear. This results in poor RGB-based estimation performance for the arms. In contrast, in depth images, the arm contours are very clear, but the legs almost blend into the background, making their contours indistinct. Such inherent flaws in the data leads to poor depth-based estimation performance for the legs. We also included two images samples from rgb and depth in the appendix Figure 6.

Q16: Are there other works that perform multimodal HPE or HAR on the MM_Fi and XRF55 datasets that you can compare to?

Answer: We appreciate the suggestion and we went through all the papers citing MM-Fi and XRF55 datasets. To date, only one paper [4] used the MMFi and XRF55 datasets for multimodal HAR. But it focuses solely on modality alignment to improve single-modality results rather than conducting multi-modal fusion. Their model achieved 43.90% accuracy on the single LiDAR modality in the MM-Fi dataset, whereas our model reached 52.7%. On the XRF55 dataset's mmWave modality, their approach achieved 50.30%, while ours reached 83.9%.

Beside the research works based on the MM-Fi and XRF55 datasets, we also actively looked for multimodal fusion methods and carried out experiments on the dataset in order to strengthen our X-Fi results. The comparisons are added in the appendix Table 5 and shown below.

Q6: Why do you claim the approach is a ‘foundation model’ and how do you define it?

Answer: We appreciate the suggestion. X-Fi partially aligns with the foundation model concept by enabling modality-invariant representation learning. In current LLM research, a foundation model refers to a large-scale, pre-trained model that serves as a versatile base and could learn unified representations across diverse inputs [8][9]. They are capable of fine-tuning or adapting to a wide range of downstream tasks across domains.

The foundation model we refer to is one that, given a set of modalities, the model requires only a single round of modality-invariant training and could handle versatile modality combinations during inference without the need for retraining or fine-tuning. X-Fi enables modality invariant representation learning, allowing it to process diverse multimodal combinations across different scenarios without the need for retraining on specific modality configurations, which partially aligns with the foundation model concept.

However, we acknowledge that X-Fi has some limitations compared to existing LLM foundation models, as it requires larger-scale data from a wider range of scenarios for modality-invariant training to fully achieve the capabilities of a foundation model. We added foundation model related descriptions in the section 2.2 line 138-143 to clarify why our proposed X-Fi could be regarded as a foundation model.

Q7: Could the author add the discussion of multimodal missing modalities literature review?

Answer: We appreciate the suggestion and conducted literature reviews on the recommended papers. The discussion about these approaches and the corresponding citations are included in the related work section 2.2.

Q8: Are decision-level approaches restricted to fixed modality combinations? Since they can average or vote on outputs, they might easily handle adding or dropping modalities.

Answer: We agree with the reviewer and the decision-level fusion methods are not restricted to fixed modality combinations. We have added corresponding discussions in Section 2.2. However, the performance could be hindered by less informative modalities. For instance, due to the low resolution of WiFi, WiFi-based HPE results are worse than the other modalities and simply averaging the outputs of other modalities with WiFi can degrade the performance.

Q9: Why is the discussion in the last paragraph citing LLM methods and a visual odometry method in robotics very relevant?

Answer: This is because some novel multimodal LLM methods inspired our human sensing multimodal fusion design as they are proposed for cross modal fusion of image-text, embodied sensor-text, and RF sensors-text. Also, suggested by reviewer u1Rk , we cited some more relevant LLM works. The visual odometry method in robotics [2] introduced the concept of modality-invariant methods, which is one of our key concepts in the paper.

Q10: Could the author label the arrows and boxes in figure 2 with the notation? what does $n_{f}$ and $d_{f}$ refer to? Is $\theta_{E}$ Just the concatenation of $\theta_{F}$ and $\theta_{LP}$ in equation(1)? Is positional encoding shown in Equation (1)?

Answer: We appreciate the suggestion. We have refined the notations and have added notations into Figure 2.

The dimension of $n_{f}$ and $d_{f}$ we chose is 32 and 512, respectively, and we have added the description in section 4.1.

The $\theta_{E}$ is the concatenation of $\theta_{F}$ and $\theta_{LP}$ since the feature encoders we designed consist of feature extractors and linear projectors. The positional encoding is not shown in Equation 1 because the equation is to describe the components within the feature encoders.

Q11: Should the author rename the “cross-attention transformer” to “cross-attention module”? Could the author provide detailed diagrams of the KV generators and the cross-attention modules?

Answer: We very much appreciate the suggestion and agree that “Cross-attention module” is indeed a better term to describe the designed cross-attention part. The cross-attention part we designed does not contain the QKV projection part and thus it couldn’t be named as “cross-attention transformer”. We revised all the “cross-attention transformer” terms to “cross-attention module” in the paper. We also added detailed diagrams of the KV generators and the cross-attention modules in the appendix Figure 7.

We sincerely appreciate the reviewer meFd for the insightful questions and constructive comments. We are glad that the reviewer acknowledges that “the need for a flexible human sensing model is well motivated” and “modality training is appreciated”. Here we answer all the questions and add extensive experiments for clarification. We hope the responses can address the concerns. In addition, we have submitted a revised manuscript with an appendix where we mark all the suggested tables, figures, and analytics in magenta color.

Q1: Are there any previous works or experiment results that support the statement that the attention mechanism was weighing some modalities more than others?

Answer: Yes. [1] (CVPR 2022) reported the observation that ‘the Transformer models degrade dramatically with modal-incomplete data’. They trained a ViT on ‘full’ sets of image and text modalities and tested only with unimodal input. The result on MM-IMDb dataset reveals that the performance drops by 14.3% when missing image, and drops 36.7% when missing text. Similarly in [2] (CVPR 2023), the trained modality-invariant transformer model marginally depends on the presence of depth modality. ‘Dropping RGB barely decreases performance, while dropping Depth causes the localization to fail more drastically’, as they described.

We admit that we didn’t clearly state the motivation of our X-Fusion design in section 3.2.2 and thus we modified the corresponding description. Additionally, we conducted a detailed literature review about the phenomenon that ‘the attention mechanism was weighing some modalities more than others’ and found the following works report the issue.

Q2: How did you determine the final probabilities of every modality in modality-invariant training procedures?

Answer: We started from setting equal probabilities for every modality, but then we found some modalities are under-fitted by the model. After evaluating the modality characteristics, we adjusted the corresponding modality probabilities to improve the performance.

For the results presented in Table 3, we first evaluated the baseline results from the XRF55 dataset paper [3] and observed that the RFID modality performance is significantly worse than the other two, which motivated our first setting that emphasized RFID. However, after evaluating the outcomes with this setting, we noticed the model was underfitted on the WiFi modality. Consequently, we gradually increased the occurrence probability of WiFi to improve its performance.

Similarly, for the MM-Fi dataset experiments, we first assessed the characteristics of all modalities and tested the model performance with equal modality occurrence probabilities (set to 0.5). Given the large performance gap between the baseline and model results for LiDAR, we decided to incorporate more LiDAR samples into the training process to address this disparity.

Q3: In Tables 1 and 2, was X-Fi trained once and tested on various modality combinations, while baselines were retrained for each combination?

Answer: Yes, X-Fi was only trained once with our proposed modality-invariant training strategy and tested on various modality combinations. And the baselines were trained on each combination separately, i.e. we trained 62 independent baseline models on MM-Fi dataset to form baseline results shown in Table 1. We emphasized the corresponding descriptions in section 4.1 and 4.2.

Q4: Is Figure 1 trying to show that X-Fi can replace a bunch of modality-specific models and doesn't need to be retrained for each combination of modalities?

Answer: Yes, the purpose of Figure 1. is to show that X-Fi could provide one unified modality-invariant solution to all potential modality combinations. The left image shows that existing human sensing solutions are mostly designed on fixed modality combinations and we need to train different models with different modality combinations respectively for various application scenarios. In order to emphasize that X-Fi doesn’t need to be retrained for each combination, we modified the diagram correspondingly.

Q5: For Figure 2, why are the last 3 modalities with dotted lines? Can you label the dark gray background boxes (which module is the X-Fusion part)? Can you enlarge the font size in the diagram? What does $N$ stand for in the arrow loop?

Answer: The last 3 modalities with dotted lines represent those potential modalities that are not included in the given scenario, or we can call them ‘inactive’ modalities. However, those modalities could be utilized in other scenarios and the corresponding modality-specific structures will be activated

We appreciate your suggestions on the diagram design and we labeled the dark gray background boxes, enlarged the text font size, and added more descriptions about the diagram to the caption in the paper.

The $N$ in the arrow loop stands for the number of iterations on the X-Fusion block.

Dear reviewer meFd,

As the rebuttal deadline is approaching in 2 days, could you please have a look at our response? We are looking forward to your reply!

Feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 6113

Dear reviewer meFd,

As the rebuttal deadline approaches in less than a day, we kindly ask if you could take a moment to review our response. We are encouraged that four reviewers have acknowledged that our responses address most of their concerns, with three either raising or expressing their intention to raise their ratings.

We are sincerely looking forward to your reply and feel free to let us know if you have any other concerns. Thanks for your time and effort!

Best Regards,

Authors of Submission 6113

I'm impressed with the author's rebuttal. I think they addressed most of my concerns well. However, I'm still wary of the strength of the overall work. In particular, addressing missing modalities is not very novel, and although other works may not use the exact terminology "modality-invariant" model, I do believe this has been investigated before in many applications, also using attention mechanisms. I am also still not convinced this model provides the sort of general reasoning a foundation model is supposed to represent and still believe some of the notation/formulation presented is slightly difficult to follow.

Nonetheless, I think this iterative application of the cross-modal transformer with modality-specific and fused info is unique and interesting. To my understanding, this work's contribution is sufficient and relevant to ICLR. Thus, I will increase my score to marginally above acceptance.

Q1: Why do we claim modality invariant human sensing as one of our key contributions, since conceptually approaches proposed in [1] and [2] could in theory be adapted? Comparisons with prior multi-modal fusion approaches like [2] should be considered to better highlight the strengths of the proposed method?

Answer: For VO Transformer [1], we have already compared our method with VO Transformer in the original manuscript Section 4.3. The transformer based fusion method is the VO Transformer structure, and we have added the references here for better clarification. The experiment results have been listed in Table 4 and Table 5.

Additionally, we also added the comparison with another SOTA multimodal fusion method Meta-Transformer [4], which is trained and evaluated on fixed modality combinations. We used the Meta-Transformer-B16 encoder as the backbone to carry out experiments. The results shown below demonstrate X-Fi’s superior performance.

Q2: Are there any additional baselines, such as comparing recent single-modality methods to demonstrate the advantages of multi-modal fusion. The experiments miss important baselines.

Answer: As discussed in Table 1 and 2 of the original manuscript, we have already included results of the single modality with latest methods as the backbone. Moreover, the detailed description of the selected single modality backbones are illustrated in appendix A.1. The comparisons between single modality results and X-Fi multimodal fusion results demonstrate the superior multimodal fusion ability of the proposed Xi-Fi. For example, the image plus mmWave radar multimodal result shown in line 331 surpasses both the image and mmWave radar single-modal baseline results in line 325 and line 328 by 26.6% and 37.2% in terms of MPJPE, respectively. Due to page limitations, we had to put the settings and detailed descriptions in the appendix Section A.1, which may have led to misunderstandings. We revised and emphasized corresponding descriptions in the main text.

Q3: What is the number of iterations needed for the X-fusion block in experiments? How to determine the number of iterations for the X-fusion block?

Answer: The default iteration is 4 as illustrated in our experiment setting. As suggested, we added the missing descriptions about the number of iterations for the X-fusion block to the paper in line 381. To illustrate how the number of iterations affects the model performance and to figure out how to determine the number, we carry out a new experiment with different the number of iterations on XRF55 dataset for HAR task. The result and the corresponding discussion shown below are added into the appendix A.4 and Table 8.

When the number of iterations is set to 2, the model still cannot generate good results on some hard modalities such as WiFi and RF. In these modalities, the resolution is low so it’s harder to learn modality-specific features. To further explore if the default 4 iterations are sufficient, we increased the number of iterations to 5. However, the model can only generate similar results as the iteration to be 4. In summary, we currently set the iteration to be 4 which has been sufficient, but we believe precisely tuning this hyperparameter can further achieve better results.

Q4: It will be interesting to compare average pooling vs. cross-attention with learned queries for dimension reduction. Need to justify why average pooling is designed in the cross-modal transformer.

Answer: It is a quite interesting idea and we added an experiment. We do realize the cross-attention with learned queries might be a good solution for dimension reduction, which have recently been proved to be effective in the Vision-Language Model paper (Q-Former) [5] and vision soft prompt [6]. To this end, we conducted comparison experiments by replacing the average pooling based dimension reduction with the cross-attention based dimension reduction. However, we find it hard to make the cross-attention based dimension reduction converge during modality invariant multimodal training this week. The training process is illustrated in appendix Figure 8.

Possible reasons are (1) our datasets include a large amount of modality pairs, making it hard to utilize the existing feature encoders to achieve modality alignment, (2) the scale of multimodal data pairs is not as large as that of image-text pairs, making it hard to conduct large-scale encoder pretraining. Thus we keep average pooling in our current design, which is a simple solution for dimension reduction for now.

Utilizing cross-attention with learned queries for dimension reduction is a relatively independent novelty for multimodal fusion research. In the future, we will adopt the two-stage training strategy in vision-language models, starting with large-scale pre-training to develop strong encoders for tight correlation among modality features, followed by training the subsequent components that include utilizing cross-attention for dimensionality reduction.

Q5: What is the definition of $n_{f}$ in L229 and what is the number used in the experiment?

Answer: The $n_{f}$ represents the number of feature tokens for each modality feature and the number used in the experiment is 32. We have added the missing descriptions of $n_f$ in line 161 and 375.

Reference:

[1] Memmel, Marius, Roman Bachmann, and Amir Zamir. "Modality-invariant Visual Odometry for Embodied Vision." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21549-21559. 2023.

[2] Chen, Anjun, Xiangyu Wang, Kun Shi, Shaohao Zhu, Bin Fang, Yingfeng Chen, Jiming Chen, Yuchi Huo, and Qi Ye. "Immfusion: Robust mmwave-rgb fusion for 3d human body reconstruction in all weather conditions." In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2752-2758. IEEE, 2023.

[3] Yang, Jianfei, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. "Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing." Advances in Neural Information Processing Systems 36 (2024).

[4] Zhang, Yiyuan, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. "Meta-transformer: A unified framework for multimodal learning." arXiv preprint arXiv:2307.10802 (2023).

[5] Li, Junnan, Dongxu Li, Silvio Savarese, and Steven Hoi. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." In International conference on machine learning, pp. 19730-19742. PMLR, 2023.

[6] Zhou, Kaiyang, Jingkang Yang, Chen Change Loy, and Ziwei Liu. "Learning to prompt for vision-language models." International Journal of Computer Vision 130, no. 9 (2022): 2337-2348.

We appreciate the reviewer H5r3 for the constructive comments. We are glad that the reviewer acknowledges "the paper addresses an interesting and important problem". Based on the suggestions, we mainly address the lack of comparisons to prior methods and add experiments on other SOTA multimodal fusion methods to prove the significance of our proposed X-Fi. In addition, we have submitted a revised manuscript with an appendix in which we add all the suggested tables, figures, and analytics in magenta color.

Q1: Why do we claim modality invariant human sensing as one of our key contributions, since conceptually approaches proposed in [1] and [2] could in theory be adapted? Comparisons with prior multi-modal fusion approaches like [2] should be considered to better highlight the strengths of the proposed method?

Answer: We claim it as one of our key contributions since modality-invariant human sensing is a new task in human perception, i.e., human activity recognition (HAR) and human pose estimation (HPE). As discussed in the manuscript, though some previous works also studied modality-invariant [1] or multimodal fusion [2] tasks, they differ significantly from ours regarding the task, model architecture, training strategy, and experiments. In this answer, we support our argument with both conceptual comparison and supplementary experiments. We first compare the conceptual difference between our work and [1][2] in the table below.

In summary, our proposed X-Fi (1) deals with a more difficult task with 5 different modalities and flexible inputs, (2) proposes a novel structure that allows flexible inputs with possibly missing modalities (instead of vanilla transformer), (3) presents a novel training strategy that drops whole modalities for training, and (4) conducts extensive experiments on 2 datasets with 31 modalities combinations, demonstrating the significant improvement.

As suggested, to better demonstrate the significance of our proposed X-Fi, we expand the ImmFusion method by concatenating multimodal features and expand the VO Transformer method by feeding all modalities’ features into a single transformer structure, which are then compared to our method on the benchmark.

For ImmFusion [2] on HPE task, we compare it on the original setting (image+point cloud, I+D+R), and on all modality settings (I+D+R+L+W). The results are shown below and indicate the superior performance of our proposed X-Fi.