Learning Commonality, Divergence and Variety for Unsupervised Visible-Infrared Person Re-identification

Weaknesses

W1: From Table 2, it is evident that the method is based on a relatively strong baseline. Does the paper use different baselines, like ADCA, still achieve good results?

A1: We acknowledge the importance of evaluating our method with different baselines. As shown in following Table, we have included results with two additional baselines, ADCA and MMM. Our method, when combined with these baselines, still achieves significant improvements. For instance, ADCA's Rank-1 accuracy improves from 45.5% to 52.5% and mAP from 42.7% to 48.5% when integrated with our PCLHD. Similarly, MMM's Rank-1 accuracy improves from 61.6% to 65.9% and mAP from 57.9% to 61.8%, and the gain is remarkable by 4.3% and 3.9%. These results demonstrate the effectiveness of our approach across different baselines.

W2: The paper introduces a momentum backbone in the PCLHD method but does not fully explain its advantages.

A2: In unsupervised learning, noise is inevitable, and dynamic prototypes can introduce a certain degree of instability. The momentum encoder mirrors the structure of the online encoder and is updated by the accumulated weights of the online encoder. Our method is resistant to sudden fluctuations or noisy updates, leading to more consistent and robust feature representations.

W3: Cross-modality associations are especially important for USVI-ReID, what is the association strategy used in this paper? Some details need to be supplemented for clarity.

A3: In USVI-ReID methods, the process is typically divided into two stages. The first stage involves intra-modality identity alignment learning, while the second stage focuses on cross-modality associations. Our strategy emphasizes obtaining accurate intra-modality alignment suitable for the second stage. As shown in Figure 3c, our cross-modality association accuracy is significantly higher than existing unsupervised methods. We utilize progressive contrastive learning, including centroid and hard prototype contrastive learning, to achieve this. For inter-modality association, we build on the well-aligned intra-modality identities and use existing methods like PGM and MMM to handle the association. This ensures effective cross-modality matching by leveraging the strengths of both intra- and inter-modality learning.

Questions

Q1: What insights does this method offer for other multimodal tasks? How can the approach be adapted to enhance performance in different multimodal scenarios?

A1: Our method offers valuable insights for multimodal tasks by demonstrating the effectiveness of progressive contrastive learning with hard and dynamic prototypes. When dealing with multimodal feature alignment and representation, it may be beneficial to make the intra-modality feature alignment and then make the inter-modality feature alignment. By mining divergence and variety through these prototypes, our method enhances clustering quality and feature discrimination, which is crucial for aligning cross-modal features. For instance, in image-text matching tasks, hard and dynamic prototypes help capture divergence and variety in visual and textual data, leading to better alignment between the two modalities.

Thank you so much for acknowledging the strengths of our method again. We have carefully considered your constructive comments and here are the answers to your concerns.

C1: Do I understand correctly that you do not plan to include a limitation section? If that is the case, I strongly urge you to consider adding such a section.

A1: We greatly value the reviewers' feedback and will be adding a limitation section as shown in the general response to the manuscript. We believe this addition will provide a more complete understanding of our work.

C2: In Eq. 19, Does $v_j$ refer to the query image? Could you please clarify this here? Besides, $v_q$ should be $v_j$ in line 180.

A2: Thank you for your detailed feedback. in Eq. 19, $v_j$ denotes the query image. Besides, We will also correc $y_q$ to$y_j$ in line 180.

C3: Please note that there is a typo in line 175.

A3: Thank you for pointing out the typo. We will thoroughly proofread the manuscript and correct typos to the best of our efforts.

Weaknesses

W1: While introducing hard prototype contrastive learning can capture divergences, selecting hard samples may lead to model instability during the early training stages. Although progressive contrastive learning strategy is introduced to mitigate the above problem, balancing the impact of hard samples remains a challenge.

A1: Our ablation experiments show that even starting with HPCL and DPCL without using PCL, the method achieves the Rank-1 accuracy of 63.7%, only with the 0.7% decrease. This is because CPCL is involved in training, helping the model maintain stability when learning divergence and variety. Additionally, our experiments demonstrate that method is not highly sensitive to the $E_{CPCL}$ hyperparameter. For more details, please refer to Q3 for Reviewer mgXA.

W2: The strategy for selecting dynamic prototypes might require more complex implementation and higher computational resources on larger datasets.

A2: We acknowledge that dynamic prototype learning requires more complex implementation and higher computational resources during training. However, it is important to note that during the inference stage, this process is not needed. The complexity and computational requirements are the same as other unsupervised methods, with no additional overhead introduced.

W3: Although the paper mentions that the code will be released after acceptance, it currently lacks detailed instructions for reproducibility. The accuracy of pseudo-labels directly impacts the model training. Some parameters in DBSCAN are very important and should be analyzed further.

A3: For fair comparisons, we follow the experimental details provided by ADCA and PGM for all other aspects. The detailed parameters are as follows: We use Adam optimizer to train the model with weight decay 5e-4. The initial learning rate is 3.5e-3 and decays 10 times every 20 epochs. The maximum distance for DBSCAN is set to 0.6 on SYSU-MM01 and 0.3 on RegDB. The minimal number for two datasets is set to 4 during clustering. We will add these details to the supplementary materials to enhance reproducibility.

Questions

Q1: Is $L_{CPCL}$ involved in the optimization process during the second stage?

A1: $L_{CPCL}$ does participate in the optimization of the second stage. As indicated by the background color in Figure 2, $L_{CPCL}$ covers both stages.

Q2: Is the comparison fair with GUR without camera information?

A2: GUR shows two results, with and without camera information, with the latter being lower. All the unsupervised methods in Table 1, including ours, do not use camera information. To ensure a fair comparison with all methods, we present the results of GUR without camera information, not with the intention of diminishing GUR’s results. GUR vs. our method achieves 63.5% vs. 65.9% Rank-1 accuracy with camera information, and our method outperforms GUR’s best result by 2.4%. Moreover, GUR vs. our method achieves 73.9% vs. 89.6% Rank-1 accuracy on RegDB (V2T), resulting in a remarkable gain of 15.7%.

Dear Reviewers,
Thanks very much for your time and valuable comments. We understand you're busy. But as the window for responding and paper revision is closing, would you mind checking our response and confirm whether you have any further questions? We are very glad to provide answers and revisions to your further questions.
Best regards and thanks,
Authors of # 1045

Q1: The use of the farthest sample as a method for representing diversity in the HPCL (as described in Equations 6 and 7) relies on the assumption that distance is an adequate measure of sample spread within a cluster. Given that distance can be a weak indicator and variance might offer a better representation of statistical distribution, did the authors explore alternative methods for representing diversity, such as variance or other metrics? If so, please discuss the findings.

A1: We chose the farthest sample distance measure to represent diversity due to its simplicity and effectiveness. Distance measure is a common approach in machine learning and person re-identification, particularly in subspace learning. Thank you for your suggestion, we also explored using variance as an alternative measure for diversity. Our experiments indicate that using variance resulted in 2.8% reduction in Rank-1 compared to the our HPCL, as shown in following table. The reason for this is that our HPCL focuses on the extreme points in the feature space, which helps the model focus on the divergent samples, rather than simply minimizing intra-class variance.

Q2: The DPCL section is not clearly articulated. Specifically, the concept of the query and its application in equation 19 are obscure and require further clarification.

A2: We apologize for the lack of clarity in the DPCL section. The query in this context refers to the feature representation of the input image that is being compared to the dynamic prototype. We followed the definition of the query as used in MoCo, PGM, and GUR. We will revise the manuscript to make this explanation clearer.

Q3: How does the performance of the proposed solution vary with different values of the E_CPCL constant? Was any ablation study conducted to investigate this aspect? An examination of the impact of varying E_CPCL on the solution's effectiveness would provide deeper insights into the robustness and adaptability of the proposed method. Such analysis would be crucial for validating the consistency and reliability of the results.

A3: We understand the concern regarding the constant $E_{CPCL}$ being set to 50. We have now conducted ablation studies to justify this choice. Our experiments tested various values for $E_{CPCL}$, and we found that 50 is not the best choice. The results showed that 40 epochs provided the best performance. We will include these ablation study results in the revised manuscript to support our choice of $E_{CPCL}$.

Q4: Although it is common practice to integrate the proposed methodology into unsupervised solutions, I believe that knowing the number of classes in advance and setting the number of clusters accordingly constitutes a form of supervision. While it is less intensive than labeling every sample with its respective class, it is still a form of supervision. In my opinion, these methods should be categorized under "unlabeled samples" rather than "unsupervised" learning.

A4: We appreciate your insight regarding the classification of our method. However, our pseudo-labels relies on the DBSCAN clustering algorithm, which does not require prior knowledge of the number of classes. For instance, while the actual number of classes in the dataset is 395, our clustering results in 384 clusters. This slight variation demonstrates that our method can effectively approximate the true number of classes without supervision.

Thank you once again for providing feedback on our paper and for your willingness to consider improving the score.

C1: I still have doubts about using distance in HPCL since in many Re-ID solution there is significant effort also in metric learning showing that distance is not capturing all the characteristics of the feature space.
A1: We agree with your observation that distance is not a perfect method for capturing all the characteristics of the feature space. However, the key idea of HPCL is to focus the network on divergence using distance-based hard prototype learning, rather than on measurement methods. We have used the simple distance measure to verify that focusing on divergence is important. We also acknowledge that there are more complex ways to measure divergence, and we believe that using more advanced metrics could improve our approach.

C2: Concerning the $E_{CPCL}$ the authors confirm the necessity of ablation on such a parameters since 40 is a better value. What is missing is a comment on why such a parameters has such an impact and with such characteristics.
A2: The parameter $E_{CPCL}$ is crucial in determining the timing for focusing on divergence and variety. When $E_{CPCL}$ is set to 20, the hard prototypes are primarily composed of noisy samples. However, when $E_{CPCL}$ is set to 40, these prototypes are more likely to consist of clear hard samples. For instance, at the 20th epoch, the ARI score for IR image pseudo-label accuracy is only 0.68, but it improves to 0.89 at the 40th epoch, which demonstrates how $E_{CPCL}$ affects the quality and composition of prototypes. When $E_{CPCL}$ is set beyond 40, the network has already somewhat converged, which reduces the impact of proposed method.

Weaknesses

W1: L215 states that each batch contains 16 person IDs and 16 samples per person. Is this a confusion and it should be based on the assignments from the clustering? Or are these actually person IDs from the annotations?

A1: We apologize for the confusion. The statement in L215 indeed refers to the selection of samples based on the assignments from the clustering, not annotations. We followed the same experimental setup as existing unsupervised VI-ReID methods such as ADCA, PGM, and GUR. We will clarify this in the revised version.

W2: The novelty isn't that great, the base method PGM covers most of the approach and the real novelty lies in a slight modification to the contrastive learning setup for a part of the training.

A2: PGM solves the USL-VI-ReID by creating modality-specific memories and minimizing contrastive loss across the query images and centroid prototypes. The limitation of PGM is that it only focuses on the commonality among features, which causes the pseudo-labels generated by the clusters to be unreliable, as shown in Figure 2c. Just like a normal distribution, which requires both the mean and the variance to accurately reflect the data distribution, HPCL and DPCL are designed to focus both divergence and variety. Importantly, we address issues not only present in PGM but also in most unsupervised VI-ReID methods, which overlook aspects of divergence and variety. As shown in the following table, our method significantly improves the performance of other methods. Specifically, when combined with PCLHD, MMM's Rank-1 improves from 61.6% to 65.9%. Additionally, we need to correct a misunderstanding: our momentum update is inspired by MoCo, not dynamic prototype learning. MoCo focuses on contrastive learning using a queue and momentum encoder. It does not involve any dynamic prototype learning.

W3: Why is there no discussion about unsupervised learning in general, specifically with a focus on contrastive learning and its modifications?

A3: We agree that the related work can be strengthened to include more discussion on unsupervised learning in general. We will revise the related work in the future version.

W4: While the performance boost in general is quite good, figure 2a does raise the question how stable it is w.r.t to some of the hyperparameters.

A4: Hyper-parameter $\lambda$ is a weighting parameter to trade-off $L_{HPCL}$ and $L_{DPCL}$. We conducted extensive experiments with various values of $\lambda$ (e.g., 0.1, 0.25, etc.) and found that $\lambda = 0.5$ provided the best results, as demonstrated in Figure 2a.

W5: Overall the discussion of the results is often a bit meaningless and just describes the values in the tables, partially even in a bit of a weird fashion.

A5: We acknowledge that the discussion of the results is insufficient. We admit there is still a large gap between our method and SOTA fully-supervised methods on SYSU-MM01. Nonetheless, fully supervised methods have drawbacks, such as high annotation costs. Additionally, our method can enhance the performance of popular methods like ADCA, PGM, and MMM. For instance, combining PCLHD with MMM improves its Rank-1 from 61.6% to 65.9% and mAP from 57.9% to 61.8%, and the gain is remarkable by 4.3% and 3.9%. The improvement indicates the benefits of focusing on divergence and variety in this field, and demonstrates that our approach is not only effective on its own but also beneficial when integrated with other methods.

Questions

Q1: Why are the "Baseline" results in table 2 so different from the PGM results?

A1: The official paper may have contained typos, as we obtain better results on indoor search using the official code. The better performance on indoor search is normal because indoor environments are relatively simple, stable lighting and fewer background distractions, making matching easier. Most methods in Table 1 also show better results for indoor search compared to all search.

Q2: Why did you duplicate almost every single equation just for visible and infrared?

A2: Thanks for your valuable comments, we will revise this in the future version.

Q3: In figure 2c, it very much confused me to see a line between the data points since the different methods are in no way connected.

A3: Thank you for your suggestion, we will replace it with a bar chart to improve clarity.

Q4: Is figure 3 generated based on training of test images?

A4: The t-SNE map in Fig.3 is plotted by 10 randomly chosen identities of test images on SYSU-MM01.

Limitations

L1: I find it somewhat lazy to not put some effort into coming up with a meaningful discussion of what still needs to be improved.

A1: Our method relies on the DBSCAN to generate pseudo-labels. For extremely large-scale datasets, DBSCAN might not perform well, thereby affecting the overall method's performance. Besides, if the data has a large intra-class discrepancy, it might impact the results, and we plan to address this in future research by hierarchical clustering.

L2: This is actually outrageous: "There is no societal impact of the work performed."

A2: Thank you for your thorough review and detailed feedback. We apologize for the misunderstanding. As you rightly pointed out, performing this kind of research can significantly impact the effectiveness of surveillance. We mistakenly interpreted the Broader Impacts section as referring only to negative societal impacts, hence our response of [NA]. We will review and correct the Paper Checklist accordingly.

We sincerely thank you once again for your valuable comments. We have carefully considered your concerns and have provided our responses below.

C1: Could you actually elaborate on W3? You plan to add additional related work, but maybe give a list of the papers you plan to discuss here? As is, I can only hope you will come up with a meaningful discussion and that doesn't feel very convincing.

A1: Thank you for your follow-up question. We plan to additional related work about unsupervised learning in general, specifically with a focus on contrastive learning and its modifications. Specifically, we will discuss the following related works:
Contrastive learning has achieved remarkable advancements in visual representation learning by pulling positive samples closer together and pushing negative samples further apart in feature space[1,2,6]. In contrastive learning, positive and negative samples play a crucial role and have a direct impact on the model's performance. Therefore, some data augmentation methods[2,3] have been proposed to augment positive pairs to learn discriminative features. For negative pairs, which have received less attention in contrastive learning, MoCo[1,4,5] series construct a memory bank to maintain numerous negative samples for increasing model’s performance. However, these methods define samples from different instances as negative, which is not feasible for ReID. In ReID datasets, many samples have the same identity, causing many positive samples to be wrongly labeled as negative.
[1]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
[2]Xuyang Zhao, Tianqi Du, Yisen Wang, Jun Yao, and Weiran Huang. ArCL: enhancing contrastive learning with augmentation-robust representations. ICLR, 2023.
[3]Demirel B U, Holz C. Finding order in chaos: A novel data augmentation method for time series in contrastive learning. NeurIPS, 2023.
[4]Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint, 2020.
[5]Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. ICCV, 2021.
[6]Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi. Vision-language pre-training with triple contrastive learning. CVPR, 2022.
If you have any more relevant works in mind, we would greatly appreciate your suggestions.

C2: Also, as another reviewer highlighted, will you be adding a limitations section to the paper?
A2: We greatly value the reviewers' feedback and will be adding a limitation section as shown in the general response to the manuscript. We believe this addition will provide a more complete understanding of our work.

C3: You have promised to add quite some things to the final paper and I'm wondering how all of this will fit and which parts will be sacrificed and moved to the supplementary to make all of this fit.
A3:To accommodate the additional content, we plan to move the discussion of Unsupervised Single-Modality Person ReID in the related work section, as well as Figure 2a and its related discussion, to the supplementary material. This will allow us to maintain the core content of the paper while ensuring that all important details are still accessible.

C4: In another comment you wrote "For instance, in image-text matching tasks, hard and dynamic prototypes help capture divergence and variety in visual and textual data, leading to better alignment between the two modalities." Is this some idea that you came up with, or is there an actual paper that does this?
A4: This idea, to the best of our knowledge, has not appeared in any existing papers. It is an original concept that we plan to explore further as we expand this work.

Dear Reviewer edeS,

We sincerely appreciate your insightful review, as it plays a pivotal role in enhancing the quality of our manuscript. Furthermore, we have carefully considered your feedback to refine our paper further. As the discussion phase comes to an end, we want to ensure that we have thoroughly addressed all of your concerns. If there are any additional points of concern that you believe would contribute to the improvement of our manuscript, please let us know. We genuinely appreciate your expertise and the dedicated time you have devoted to reviewing our work.

Warmest regards,

Authors