Unbiased Prototype Consistency Learning for Multi-Modal and Multi-Task Object Re-Identification

Response to Reviewer NDQB

Thank you for your detailed review and helpful suggestions. Your feedback is highly appreciated and has played an important role in enhancing the presentation and technical depth of our work. We will release our code to provide the implementation details.

Weaknesses & Questions

W 1) : Effect of Image Resizing on Performance

A1 : We normalize images from different categories to a resolution of 256×128 and use a shared tokenizer for processing. In theory, employing separate tokenizers for pedestrians and vehicles while sharing a common encoder could potentially lead to better performance. However, the imbalance in data distribution across different categories may introduce optimization difficulties. Furthermore, our experimental results reveal that resizing vehicle images to the same resolution as pedestrian images has a relatively minor effect on retrieval performance. Specifically, we conduct experiments with DeMo on the RGBNT100 dataset with different input resolutions (256×128 and 128×256). The results of mAP/R1 under 256×128 and 128×256 are 86.0/97.2 and 86.7/97.7 . There exists only a negligible performance difference of around 0.7%/0.5% , which can be attributed to the fact that most original images have widths smaller than 128 pixels, resulting in minimal data compression or loss during resizing.

W 2) : Analysis of Temperature Hyperparameter

A2 : The temperature hyperparameters used in Equations (10) and (16) are set to the same value. We have supplemented ablation studies on the temperature hyperparameter. The detailed experimental results are shown in the table below. As observed, our method demonstrates its effectiveness across different temperature settings.

*Table. **Ablation study of temperature hyperparameters $\tau$

W 3) : Computation Details of Prototypes

A3 : At the beginning of each training epoch, we recalculate the prototype corresponding to each identity, which is then used in the subsequent training process. Thank you for pointing out this omission, we will provide a detailed explanation of this computation procedure in the manuscript.

W 4) : Textual Errors

A4 :Thank you for your suggestion. We will revise and correct the textual errors in the the final version.

Response to Reviewer U2xk

Thank you for your time and thoughtful review. We appreciate your positive feedback on our contributions, writing clarity, and experimental design. We will release our code to provide greater clarity and transparency regarding the implementation details. We hope our responses below address your concerns and lead to a favorable reassessment.

Weaknesses & Questions

Q 1) : Comparison with Traditional Cross-modal Methods

A1 : The cross-modal method DEEN achieves Rank-1 accuracies of 22.41, 12.77, and 8.78 on R-N, R-T, and N-T on the RGBNT201 dataset, respectively. Under more challenging settings involving multiple retrieval modalities and increased category interference, our method achieves corresponding accuracies of 23.21, 14.59 and 14.47. It can be observed that our method consistently outperforms DEEN all three scenarios.

Q 2): Imbalance in the Number of Identities and Images

A2 : We acknowledge that there exists a significant discrepancy in the number of identities and the number of images per identity between the RGBNT201 and RGBNT100 datasets. However, the primary focus of this work is to address the challenges arising from heterogeneous retrieval modalities and category variations, rather than applying any strategy to balance the distribution of identities or image counts. That said, we agree that a unified ReID model should be capable of handling such imbalanced scenarios in real-world applications. Therefore, we plan to explore strategies for mitigating these sample imbalance issues in our future work.

W 1) & W 4) : Dataset Selection and Configuration

A3 : 1） Scarcity of Multimodal ReID Datasets. Your suggestion aligns well with the limitations we discussed in Section 5 of our paper. Indeed, incorporating additional object categories or environmental contexts would further demonstrate the effectiveness and generalizability of our proposed method. However, existing multi-modal ReID datasets are still relatively limited, mostly constrained to pedestrian and vehicle categories. In future work, we will actively monitor the release of new multi-modal datasets covering other categories, and we also plan to construct our own datasets for these categories. These will be integrated into our framework to support more comprehensive and in-depth studies.

2） Selection of Datasets for Cross-Domain Generalization. Given the similarity in data collection environments between RGBNT201 and RGBNT100, we conduct the main experiments in our paper on the combined RGBNT201 + RGBNT100 dataset to validate the effectiveness of our method in both cross-modal and cross-category scenarios. In contrast, MSVR310 is collected under a different setting, and thus we use it separately for cross-domain generalization experiments to evaluate the generalization ability of the model trained on RGBNT201 + RGBNT100.

W 2) & W 3) : Textual Errors and Pseudocode

A4 : Thank you for your suggestion. We will revise the textual errors and include the corresponding pseudocode in the supplementary materials of the final version to facilitate reproduction and further development by other researchers.

Response to Reviewer aGKs

We greatly appreciate the reviewer’s time and effort in evaluating our paper. We are grateful for your thoughtful feedback, which has guided us in refining our manuscript. We will release our code to better illustrate the details.

Weaknesses & Questions

Q 1) : Construction of the Prototype Memory Bank

A1 : We recalculate the prototype for each identity at the start of every epoch. Although this introduces additional computational overhead, the prototypes encapsulate both class-level and identity-discriminative information, which significantly enhances retrieval accuracy across various modalities.

Q 2) : Ablation Study of Loss Weight Hyperparameters

A2 : We conduct ablation studies to evaluate the impact of different loss weight configurations for α and β. The detailed results are presented in the table below. As shown, our proposed method consistently achieves strong and stable performance across a wide range of loss weight settings, demonstrating its robustness to hyperparameter variations.

Table. Ablation study of loss-weight hyperparameters $\alpha$ and $\beta$

Thanks for the author's response. My concerns are well solved. I have also checked other reviewers' comments and found no more concerns. Therefore, I will keep my score.

Response to Reviewer wjAM

We sincerely thank you for dedicating your time to review our work and for your constructive feedback. We hope this rebuttal will help clarify our contributions and encourage a favorable reevaluation. We will also improve writing , clarity, correct typos, and release our code to better show the details.

Weaknesses & Questions

W 1) & Q 2) : Motivation and Necessity of M³T

A1 :We thank the reviewer for the insightful comment. Real-world surveillance often involves diverse object types and multiple sensing modalities. While PEFT-based approaches like ICPL-ReID achieve competitive results with lower training costs, they are still limited to single-objective tasks and require separate models for different scenarios, resulting in redundant and expensive deployment. We propose the M³T task, which aims to address multiple retrieval modalities and identity categories within a unified model UPCL. This approach not only reduces deployment overhead but also maintains strong retrieval performance across diverse scenarios. In future work, we will explore how to leverage PEFT strategies to enhance the efficiency and adaptability of the M³T task.

Q 1) & Q 3): Divergence between Cross-modal and Multi-modal Features

A2 :Thank you for the insightful question. In the cross-modal embedding space, most existing cross-modal loss functions primarily aim to enforce modality-invariant identity representations across different modalities. However, they often neglect the preservation of modality-specific identity cues. As a result, the substantial gap observed in the fused multi-modal space does not originate from the fusion strategy itself (e.g., concatenation), but rather from the loss of modality-specific identity information during cross-modal learning, which in turn causes divergence within each modality.

To further illustrate this characteristic, we conduct a feature comparison experiment between DeMO and our proposed UPCL framework. Specifically, for two image pairs of the same identity from the RGBNT201 test set, we extracted features using both DeMo and UPCL, and computed pairwise feature distances across all modalities. The sample feature distances are respectively shown in the two tables below. Values enclosed in parentheses in the second table represent the changes relative to the corresponding figures in the first table. The results show that for the same-identity samples A and B, while the cross-modal feature distance decreases, both the intra-modal and fused multi-modal feature distances increase. This phenomenon suggests that the variation in the fused feature space stems from the loss of modality-specific identity information, rather than from the fusion strategy itself. Our method addresses identity-level and category-level consistency by mitigating the loss of modality-specific information during cross-modal learning, while simultaneously enhancing modality-specific and modality-relevant discriminative cues, thereby improving the model’s performance across diverse retrieval scenarios.

W 2) : Prototype-based Innovations for the M³T Task.

A3 : 1）Conventional VI-ReID Methods in M³T. Conventional VI-ReID cross-modal methods are not readily extendable to multi-modal retrieval scenarios and often suffer significant performance drops under more complex modality combinations. For example, we evaluated the VI-ReID-specific method DEEN on the RGBNT201 dataset. While DEEN achieves mAP/R1 scores of 22.27/22.41 and 16.51/12.77 under R-N and R-T settings respectively, its performance drastically declines to only 10.05/8.78 under the N-T setting.

2）Multimodal ReID Methods in M³T. Similarly, existing multi-modal methods struggle when faced with diverse retrieval categories. For instance, although the DeMO algorithm achieves strong performance on the standalone RGBNT201 dataset (79.0/82.3 in mAP/R1), its accuracy drops markedly to 64.4/63.8 under the M³T task setting, where multiple modalities and categories are jointly considered.

3）Innovations of Our Method. Existing cross-modal and multi-modal methods are typically designed with fixed modality characteristics, limiting their scalability to another modalities. Our prototype-based approach possesses greater flexibility, enabling seamless extension to a wider range of modalities and identity categories. Moreover, our prototype-based method achieves mAP/R1 scores of 22.33/23.21 (R-N), 16.77/14.59 (R-T), 17.93/14.47 (N-T), and 64.91/67.12 (multi-modal) under the M³T task setting. These results clearly surpass those of existing cross-modal and multi-modal methods. In summary, our method achieves superior experimental results within a simple framework, while demonstrating strong portability and scalability, collectively validating the innovation of our approach.

W 3）& Q 4) : The Innovation and Effectiveness of Components in UPCL

A4 : 1）Discrepancies across Different Prototype Types Eq. (7) aggregates features from all modalities of the same identity, but tends to be dominated by one modality. Equation (8) computes a modality-specific prototype for each individual modality, but these prototypes tend to be biased toward the characteristics of their respective modalities. To mitigate this, Equation (9) performs an aggregation over modality-specific prototypes, balancing the identity-related information from each modality. This generates a robust modality-unbiased prototype that better captures the consistent semantic representation of the corresponding identity. In addition, within CPCR, we perform clustering over modality-unbiased prototypes belonging to the same category, thereby producing in a category-consistent prototype that encapsulates class-level semantic consistency.

2）Clarification on the Role of Contrastive Loss. Based on the identity-level and the category-level consistenc semantics in the above two types of prototypes, UPCL utilizes a contrastive loss to promote more compact feature distributions for the same identity across different modalities. Compared with the baseline, UPCL achieves average mAP/R1 improvements of 6.90/9.91 on RGBNT201 and 2.70/5.72 on RGBNT100 across all retrieval modality combinations（six cross-modal and one multimodal testing scenarios. These results clearly demonstrate the effectiveness of the proposed method.

Q 5) : Single-Model Performance and Extended Retrieval Scenarioss

A5 : 1） The Upper Bound of Single-model Performance. We evaluated DEEN and DeMO on the single RGBNT201 dataset for cross-modal and multi-modal retrieval tasks, respectively. DEEN achieves mAP/R1 scores of 22.17/22.41, 16.51/12.77, and 10.05/8.78 on R-N, R-T, and N-T retrieval tasks. DeMO achieves 79.0/82.3 in the multi-modal retrieval setting. However, DeMO suffers a significant performance drop under the M³T setting, yielding 64.35/63.76（-14.65/-18.54） under the multi-modal retrieval setting. In comparison, our method achieves superior performance of 64.91/67.12, demonstrating better generalization and robustness in complex multi-modal, multi-object scenarios.

2）Extended Retrieval Scenarioss. Theoretically, there are 7 modality types (3 uni-modal, 3 bi-modal, 1 tri-modal), resulting in 49 total retrieval scenarios. Due to the limited number of samples in existing multi-modal ReID datasets and the lack of a standardized cross-modal evaluation metric, we focus on commonly-used retrieval setups for fair comparison. Your suggestion is highly valuable, and we plan to develop multimodal ReID datasets of new categories and explore the full spectrum of modality combinations in future work.

Q 6) : Analysis of Performance Differences Among Multimodal Methods

A6 : 1) Performance Improvement of Baseline. Our baseline includes additional cross-modal alignment losses (all shown in Eq. 5), which directly enhance consistency across modalities. This is not present in most CLIP-based methods, hence the performance improvement.
2) Comparison between DeMo and PromptMA. DeMo designs modality-specific experts for fusion, which enhances multi-modal integration but does not explicitly optimize for cross-modal consistency. In contrast, PromptMA introduces cross-modal interactions at the prompt level, which significantly boosts cross-modal retrieval performance. Thus, the performance gap between PromptMA and DeMo reflects the difference in design focus—fusion versus alignment.

Thanks for your feedback. I will revisit the authors’ response and give it further deliberation before adjusting the score.

1. Since the proposed task integrates multiple datasets, it remains unclear whether the overall training time is superior to existing single-task methods. Moreover, the use of prototypes may introduce substantial computational overhead. We encourage authors to report training time and inference efficiency compared with standard baselines.

2. Can a more generic input resolution, such as 224×224, be adopted to accommodate both pedestrians and vehicles, instead of the fixed 256×128 or 128×256 sizes?

3. A more comprehensive evaluation is needed. For example, if larger-scale datasets such as Market-1501-MM and WMVeID863 are included, will the proposed method still perform well or suffer a performance drop?