Source-free Cross-modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Data

1. The most pressing reservation regarding this paper pertains to its novelty. The presented research appears to be an incremental advancement, largely built on the foundations set by Ahmed et al. in the domain of source-free cross-modal knowledge transfer. Its central contribution seems limited to the introduction of a translation module, potentially underscoring a lack of significant technical advancement.

2. While the authors duly note the implications of the translation network—namely, increased parameters and extended training durations—the evaluation predominantly hinges on accuracy metrics. Given the similarity of the task to that of Ahmed et al., a holistic assessment encompassing accuracy, memory, and time overhead during training would provide a more equitable comparison between the baseline and the current approach.

Firstly, I have reservations about the proposed task as it appears to be a forced combination of depth maps, infrared images, RGB, and source-free domain adaptation. I am uncertain about the motivation behind considering such a scenario, as it seems like an arbitrary merging of two distinct tasks.

Regarding the technical approach, I find some aspects to be incremental. Specifically, the proposed translation scheme has been extensively used in the context of source-free domain adaptation [1][2][3]. Additionally, the use of additional TI (unrelated) data for distillation in cross-modal distillation and similar approaches in semi-supervised learning have been widely explored. It seems that the paper does not yield particularly novel or interesting conclusions in the context of this task. Furthermore, there is a lack of discussion on how the cross-modal representations are better suited for source-free domain adaptation.

The paper lacks comparisons with recent source-free domain adaptation papers [1][2][3][4][5][6], which is essential for evaluating the proposed method's performance against state-of-the-art approaches in this field.

There are some writing issues and details that need attention. For example, the arrows in Table 3 are not aligned properly, which affects the overall readability and clarity.

[1] Generative Alignment of Posterior Probabilities for Source-free Domain Adaptation, WACV2023

[2] Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation, IJCAI2021

[3] Source-Free Domain Adaptation via Distribution Estimation, CVPR2022

[4] Guiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation, CVPR2023

[5] Class Relationship Embedded Learning for Source-Free Unsupervised Domain Adaptation, CVPR2023

[6] Attracting and dispersing: A simple approach for source-free domain adaptation, NeurIPS2022

Major concerns:

1. The weird setting. Srouce-Free Domain Adaptation (SFDA) is a similar setting where only a pre-trained model and unlabeled target data are available. However, this setting, i.e., cross-modal knowledge transfer, even assumes that extra source-target data pairs are accessible, which is quite weird. If we obtain the predictions of the pre-trained source model over source RGB images, we almost get source labels because the predictions are expected to be relatively accurate. Therefore, the definition of the setting is questionable.

2. The detailed definition of TR and TI data are unclear, which makes me confused when going through the paper. Does the TR data refer to source image-label pairs? And does the TI data refer to source-target pairs? A clear definition is strongly encouraged at the beginning of the paper.

3. Inconsistency in context. In the related work section, the authors claim that "data generation-based methods may generate noisy source-like images". However, an image translation engine is used in the TGMB module, which makes it inconsistent. How to ensure the adopted image translation model is not noisy?

4. Missing ablations. In Table 4, the result of $L_{rec} + L_{IM}$ is missing. In Table 6 and Table 7, the ablations of $\alpha_{im}$ and $\beta_{self}$ are not conducted over a clean baseline (i.e., $\alpha_d$ and $\beta_f$ are not 0 in the table).

Minor problems/questions:

5. Is $T(\cdot)$ necessary? What will happen if we simply copy the single-channel depth map into three-channel RGB images?

6. Similar motivation appears in [A], where [A] adopts an image translation engine that transfers source images to the target style to learn domain-invariant feature representations. The source-target image pairs used in [A] are similar to the TI data.

I am looking forward to an open dialog towards my concerns, especially about the setting. I am willing to raise my rating if all of my concerns are well addressed.

References

[A] H. Wang et al. Pulling Target to Source: A New Perspective on Domain Adaptive Semantic Segmentation. arXiv preprint arXiv:2305.13752, 2023.

1. One potential weakness of the paper is the lack of an end-to-end framework. The current approach is non-end-to-end, which may limit its practicality and ease of implementation. Developing an end-to-end framework would enhance the usability and efficiency of the proposed method.
2. Another weakness is the reliance on the availability of task-irrelevant data. While the paper acknowledges the potential of task-irrelevant data, it does not address the practical challenges of obtaining such data in real-world scenarios. Providing insights or suggestions on how to acquire or utilize task-irrelevant data more effectively would strengthen the applicability of the proposed framework.
3. Additionally, the paper could benefit from a more thorough discussion of the limitations and potential drawbacks of the proposed method. While some limitations are mentioned, such as the reliance on the source model for guiding the translation process, a more comprehensive analysis of the limitations and their impact on performance would provide a more balanced assessment of the approach.
4. Lastly, the paper could provide more detailed explanations and justifications for the design choices and hyperparameter settings. This would help readers understand the rationale behind the decisions made and provide insights into potential variations or improvements to the proposed method. Addressing these weaknesses by developing an end-to-end framework, providing practical solutions for obtaining task-irrelevant data, conducting a more comprehensive analysis of limitations, and offering detailed explanations for design choices would enhance the overall strength and impact of the paper.

1. The loss functions in this paper are not well-designed or clearly explained. For example, in equation 1, the RGB reconstruction loss is set as a naïve Mean Square Error loss, while there are some other reconstruction losses more useful that can be used in the RGB reconstruction task. Using the MSE loss alone may cause the reconstructed image not continuous with the adjacent pixels, which may not make the translation net well-trained, and it may further lead to the network not working to its true capacity.

2. The framework in this paper may not convinced to leverage the TI-paired data to their full potential. In the TGMB module, the translation net aims to convert single-channel depth/infrared data into three-channel source-like RGB images, however, it seems too naïve to construct it with an FC layer, a BN layer, and a Conv layer, which may directly cause the failure of the reconstruction process. As Figure 4 in the Appendix shows most images are not well reconstructed. Still in the TGMB module, the Fs feature extractor’s parameters are fixed all the time, however, it may constrain the generalization ability of the translated TR source data to retrieve their better features. The five classifiers in TGMB and TGKT modules also have the same problem, it is not promised that all these fixed modules will perform to their best ability while not finetuned with the translated new domain data.

3. On some key issues, the paper does not achieve high scores compared with the related works and also lacks an in-depth analysis of the experimental results. First, on SUN RGB-D dataset experiments, the proposed method performs worse than the SOCKET method in K-v2 to K-v1, Real to K-v1, K-v1 to K-v2, Xtion to Real, Xtion to Xtion according to Table 2. It is not proper to say that the proposed method outperforms all other current works while neglecting these lower scores. The only analysis of the performance drop of the K-v1 to K-v2 task just accuses the less reliable mutual information about the TR task for some tasks, and there is no further comprehensive analysis in supplement material about this problem.

1. Regarding the related work on Cross-Modal Distillation, most of the reviewed papers are 2-3 years old. It would strengthen the literature review to include some more recent works in this area.
2. The experiments are conducted on relatively small datasets with several hundreds to two thousand samples. Does this limit the generalization ability of the method? Please discuss the performance on larger-scale datasets.
3. The paper mentions that adopting mutual information loss in TGMB can guide the translation process. However, the estimation of mutual information relies on the source model's predictions, which could be unreliable without access to real source data. Did you evaluate the quality of the source model's predictions on the translated data? Please discuss the impact of noisy predictions.
4. The authors just evaluate the proposed method on the classfication tasks, across RGB->Depth/RGB->NIR. Did you validate the approach on other tasks like segmentation or detection? Testing on more diverse tasks would better demonstrate generalizability.
5. While ablation studies validate individual components, the interactions between different losses in TGMB and TGKT are not analyzed. For instance, how does the balance between L_D and L_IM impact translation quality? A more thorough analysis would provide better insights.