MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification

Q1: This paper appears to be the only one among all compared methods that adopts CLIP as the backbone. It would be helpful to clarify whether this choice contributes additional performance gains.

A1: Thank you for your suggestion. To validate that CLIP contributes additional performance gains, we conduct experiments on our server to compare Top-ReID with our method, both using a CLIP-based ViT backbone. As shown in Table A, our method outperforms Top-ReID by approximately 11.1 % in mAP and 11.3 % in R-1. These results confirm the superior performance of our approach under identical settings and with the same backbone.

Table A: Performance comparison on CLIP-based ViT.

Q2: The discussion on KDL may be somewhat insufficient. While the authors have dedicated adequate space to describing how KDL is implemented, they have not clearly articulated how it contributes to the model’s performance or why it is designed in its current form. I encourage the authors to elaborate on the underlying design rationale of KDL in their rebuttal.

A2: Thank you for pointing out this issue. We design the KDL objective to enforce a knowledge gap between specific and shared features. Without KDL, the model may collapse into relying only on shared features. We therefore ensure that combining specific and shared features yields higher retrieval performance than using either feature alone. To achieve this, we apply the formulation in lines 222–225, which imposes distance constraints in the feature space. Table 4 validates our approach: as shown in Table A, adding KDL improves mAP by 2.0% and Rank-1 by 1.5% across multiple scenarios. In subsequent versions, we provide more detailed explanations. We have also revised this section accordingly to make it clearer.

Table B: Ablation study.

Q3: I believe that using a classifier to explicitly distinguish between different modalities is a simple yet effective strategy. However, it appears that the authors did not adopt this approach in the current work. I would suggest including a comparison to assess whether incorporating such a strategy could lead to further performance improvements.

A3: Thank you for your suggestion. Actually, in our ablation study, we perform relevant experiments, but our inadequate description of the baseline may lead to misunderstanding. Specifically, in Table 4(a), configuration “1” corresponds to using a single classifier for all modalities, while configuration “3” corresponds to assigning a separate classifier to each modality. The ablation results show that modality-specific classifiers effectively improve performance, boosting mean mAP by 13.4% and mean R-1 by 13.7%.

Table C: Comparison of different classifiers.

Q4: In Table 2, the performance of TOP-ReID appears to be mistakenly highlighted in bold.

A4: Thank you for your suggestion. We will make the corresponding changes in future versions.

Q1: Lack of a Clear Classification Framework in Related Work: The related work lacks a clear technical taxonomy. The current version lists existing methods but does not organize them by core challenges (e.g., Conventional multi-modal methods,ViT-based, missing-modality methods). It also lacks analysis of each method's key ideas and limitations.

A1: Thank you for the valuable feedback. We agree that the introduction and related work section would benefit from a clearer technical taxonomy and deeper analysis. We have revised these sections following your suggestions to categorize existing methods by core challenges and include a discussion of their key ideas and limitations.

Q2: Redundant or Delayed Definition of Technical Terms (e.g., ViT): Several technical terms (e.g., ViT) are either repeated or introduced only after they have already been used multiple times.

A2: Thank you for the comments. We have revised the manuscript to ensure that all technical terms are clearly defined and used.

Q3: Binary Availability Mask Explanation is Overly Verbose: The explanation of the binary availability mask spans over 0.5 pages. The logic of zero-padding can be simplified or moved to the supplementary material. The current version distracts from the main technical innovation..

A3: Thank you for the suggestion. We have accordingly reduced the content of the related section following your suggestion.

Q4: Knowledge Discrepancy Loss Definition (Line 224) is Unclear: The structure of the knowledge discrepancy loss is hard to interpret. Anyway, this is a good paper with a new task and reasonable solution, although there are some minor flaws in the details.

A4: Thank you for pointing out this issue. We design the KDL objective to enforce a knowledge gap between specific and shared features. Without KDL, the model may degrade into relying only on shared features. We therefore ensure that combining specific and shared features yields higher retrieval performance than using either feature alone. To achieve this, we apply the formulation in lines 222–225, which imposes distance constraints in the feature space based on the mentioned relation. Table 4 validates our approach: as shown in Table A, adding KDL improves mAP by 2.0% and Rank-1 by 1.5% across multiple scenarios. We have also revised this section accordingly to make it clearer.

Table A: Ablation study.

Q1: The any-to-any setting is highly related to both modality-matched object re-identification and cross-modality object re-identification. However, this work only considers the modality-matched side. Therefore, comparison and discussion with cross-modality methods will also be useful.

A1: Thank you for your feedback, and we incorporated additional discussions on cross-modality object re-identification. In this experiment, we evaluate the DEEN [1] algorithm on the RGBNT201 dataset under the R-to-N scenario. As shown in Table A, our results demonstrate that even with specialized training for R-to-N, DEEN achieves an average precision 3.5% lower than our method. Notably, our model exhibits stronger generalization capability across diverse application scenarios.

Table A: Comparison of DEEN and Our Method in R-to-N Scenario.

[1] Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification. CVPR 2023.

Q2: During the retrieval phase. From the paper, the author considers the shared representations equally, whether the shared feature from the same modality. From the visualization results, the shared feature.

A2: Thanks for your valuable suggestion. We have evaluated the performance under different weight combinations of specific features (w_sp) and shared features (w_sh), and showed the result below. The experimental results indicate that adjusting the weights does indeed lead to a slight improvement. Specifically, assigning a higher weight to specific features benefits retrieval in modality-matched scenarios, while a higher weight to shared features helps in modality-mismatched retrieval. We will include a detailed discussion on parameter settings in a future version to update the relevant experimental sections.

Table B: Analysis of different weights.

Q3: Figure 3 lacks the visualization result of the baseline method. It will make audience hard to get the improvement of MDL. The same issue applies to Table 5.

A3: Thank you for the suggestion. The baseline method does not employ MDL, so it does not separate specific and shared features and only produces fused features. Therefore, we omit this step from the visualization. Due to rebuttal interface constraints, we cannot upload images here, but we are willing to include this visualization in the revised manuscript.

Q4: It would be helpful to clarify how the 4 testing scenarios under modality-mismatched conditions were selected.

A4: NIR is the second most widely used modality in surveillance systems after RGB, and RGB-to-NIR research continues to attract significant interest. Therefore, we evaluate our method in four R-to-N-based application scenarios.

Thanks for your constructive feedback. Below, we respond to your key concerns point by point.

Q1: The comparison to state-of-the-art methods seems potentially unfair due to the use of a "modality availability mask" during inference. This mask provides explicit information about the modality scenario for both the query and gallery, an advantage not available to the baseline methods. This calls into question whether the reported performance gain stems from the novelty of the modality-decoupling approach itself, or simply from leveraging this additional information. To provide a fairer assessment, the paper should compare MDReID against an alternative strategy that also utilizes this mask. For instance, one could train an ensemble of separate "expert" models, each specialized for a specific modality combination (e.g., an R-to-N model, an RT-to-NT model). At inference time, the availability mask could then be used to select and deploy the appropriate expert model for the given scenario. A comparison against such a baseline would better isolate the true contribution of the unified MDReID framework over a simpler, scenario-specific approach.

A1: Thank you for your feedback. The use of the modality availability mask is justified, since explicit information about the modality scenario is defaulted as available information and used by all methods. For example, the Complementary Reconstruction Module (CRM) in Top-ReID introduces dense token-level reconstruction constraints to reduce the distribution gap across different image spectra. As a result, it predicts unavailable modality information by leveraging available modalities, representing the current state-of-the-art approach. To this end, TOP-ReID needs to know the explicit information about the modality scenario, and then it can apply the specific generator for reconstruction (e.g., R->T). Meanwhile, following your suggestion, we train four expert models for the RT-to-NT, RT-to-N, R-to-N, and R-to-NT scenarios. Table A shows that our method outperforms these experts by 9.2% on average in mAP and 8.9% in R-1. Furthermore, our single-model approach adapts to all four scenarios, offering greater flexibility. Although Top-ReID also handles multiple scenarios, our method surpasses it by 3.4% in average mAP and 1.1% in average R-1.

Table A: Performance analysis of expert model, Top-ReID, and Ours.

Q2: While the paper compares computational costs in terms of parameters and FLOPs, it omits the retrieval cost (i.e., search time), which is a critical metric for ReID systems. The proposed similarity calculation is more complex than a standard feature dot product. Therefore, a comparison of the actual search time against baselines is necessary to fully evaluate the practical efficiency of the proposed method in large-scale gallery settings.

A2: Following your suggestion, we further evaluate the total feature generation and retrieval times of different models on the RGBNT201 test set under the RNT-to-RNT, RT-to-NT, and R-to-N scenarios. Theoretically, our method reduces retrieval cost compared to Top-ReID. For instance, in the RT-to-NT scenario, our approach only matches T-to-T (specific features) and R-to-T (shared features), whereas Top-ReID first generates the missing modality features before performing a full RNT-to-RNT matching. Table B confirms this, showing that our method incurs no additional time and generates features more rapidly.

Table B: Performance analysis under different methods.

Q3: To provide stronger intuition for the proposed feature decoupling, a more detailed ablation study is needed. The paper should analyze the individual contributions of the modality-shared and modality-specific features. For instance, what is the performance when using only the shared features or only the specific features during inference? This analysis should be conducted for both modality-matched and mismatched scenarios to clearly demonstrate the respective roles of the decoupled components. Also, the ablation studies of these individual features with different loss functions would be also helpful.

A3: Thank you for your suggestion. In our model, both shared and modality-specific features are essential. In the RT-to-NT task, using only shared features overlooks the unique characteristics of the T modality, while using only modality-specific features prevents valid R-to-N matching. To quantify this, we evaluate RT-to-NT performance under three settings: shared features only, modality-specific features only, and both combined. As shown in Table C, combining shared and modality-specific features yields the best performance.

Table C: Performance analysis under different features.

Q4: (minor) Notations throughout the method is confusing and lacks rigor.

A4: Thank you for the suggestion. We have revised the notations throughout the manuscript to improve readability in recent days. If you have any recommended parts, please provide them in the discussion. We are willing to further clarify them.