Empowering Visible-Infrared Person Re-Identification with Large Foundation Models

We appreciate the positive feedback regarding the clear architecture figure, feasibility, and soundness of our method. We also acknowledge and appreciate the constructive criticisms for improving certain aspects of our paper writing.

Q1: Explain the discrepancy of the misaligned results of YYDS in Tables 3 and 4 with those reported in the original paper.

A1: We primarily explored the performance of existing models on the proposed automatically expanded tri-modality datasets, at the mainstream image resolution of 144*288, and under the newly proposed task setting. Consequently, YYDS was tested on our data and setting, yielding dissimilar results from the original paper. Extensive experiments demonstrate that our method performs better and is more robust under the proposed task.

Q2: The main content lacks a description of the data generation process.

A2: Thanks for the suggestion, we will add an introduction about the generator fine-tuning process for text generation in the main text to ensure the coherence and readability.

Q3: The task setting should be introduced in the main text.

A3: Thanks for the suggestion. We will add a detailed explanation of the task setting to the introduction.

Q4: The font size in Figure 2 and the tables need adjustment. Several writing errors in the paper need correction.

A4: We will correct the noted writing errors and adjust the font size in Figure 2 and the tables to enhance readability.

Q5: Authors are encouraged to add descriptions of limitations to their papers.

A5: The limitations were initially detailed in Appendix D.

• The quality of generated text can indeed affect model performance, particularly when the original images' (for generation) quality or generators' capabilities are suboptimal.
• However, even on challenging LLCM and lower-resolution RegDB, with generated descriptions not completely accurate, our method still achieves improved performance. This demonstrated the robustness of our method against inaccuracies in descriptions.
• To provide more valuable insights to the community, we will also add a discussion of potential ways to improve the quality of generated text, like progressive generation strategy and images augmentation for fine-funing VI-ReID specialized description generators.

We are grateful for the positive recognition of the soundness, clearly presentation of our framework and also appreciate your detailed comments aimed at improving our paper writing. We believe our revisions will address your suggestions. Thank you for the valuable feedback for guiding these improvements.

Q1: How testing settings align with "real-world conditions"?

A1: Humans perceive objects as visible images, and eyewitness descriptions are based on these perceptions. These descriptions, rich in information complementary to infrared modalities, serve as auxiliary clues for retrieval. Given the variability in eyewitness descriptions of the same target, our task setting allows any description of visible images of the same identity to be used with infrared features for retrieval, mimicking the varied visual perceptions of human eyewitnesses.

Q2: Some key elements in the appendix should be included in the main text.

A2: Thanks for the suggestion, we will add details of the proposed task setting and an introduction about the generation process into the main text to ensure the coherence and readability.

Q3: The introduction is somewhat lengthy and verbose, making the method appear repetitive.

A3: We will streamline the introduction as suggested to reduce redundancy and focus more on the key ideas and innovations of our approach

Q4: There are some grammatical errors, and the tenses are inconsistent.

A4: We will make revisions to correct all grammatical errors and ensure the consistency of simple present tense throughout the document.

Q5: The structure of the paper requires adjustments, particularly in refining details within specific methods. Additionally, the insights should be clarified and made more understandable.

A5: We will adjust the structure of the paper and add detailed motivation, rationale, and insights into the confusing parts of the methodology, ensuring that the details and insights of our method are clearly communicated and easily understandable.

We are grateful for your recognition of the adequate experiments, the soundness and competitive performance of our work. We also appreciate the constructive comments on the motivation, rationale, and content distribution balance, which is valuable for improving our paper writing.

Q1: Motivation and advantages of each module.

A1: For better presentation, we adjust the framework as shown in Figure 1 (the submitted rebuttal pdf). LLM augmentation is now in "Text Generation" and elaboration of the proposed method is included in rebuttal pdf for better understanding. The motivation and advantages of each module are as follows:

• Text Generation. Existing methods rely on fixed manual annotations for training, incurring labor and time costs and is sensitive to text variation. Our method uses fine-tuned language vision models to automatically generate text and employs LLM random rephrasing to create dynamic descriptions for training, enhancing our framework's robustness against text variation.
• Incremental Fine-tuning Strategy (IFS) includes Fusion Module and Modality Joint Learning to integrate text into the infrared modality for improved cross-modal retrieval. Existing methods require prior-information like pre-defined color vocabularies for text-visual complmentary information alignment, complicated architecture with much additional parameters for information extraction and fusion, causing potential information loss. Our method creates fusion features at feature level by arithmetic operations and fine-tunes LVM with end-to-end ReID loss to jointly align semantics from all modalities without prior-information, mitigating the fusion-visible discrepancy and achieving more accurate cross-modal retrieval.
• Modality Ensemble Retrieving (MER). Features of different modalities focusing on distinct information, motivated by this, we fully uses of features from all query modalities to form ensemble representations, boosting retrieval accuracy and robustness against challenging retrieval cases.

Q2: Comparison of existing text-assisted ReID methods like YYDS & and this paper's method.

A2:

YYDS:

• Manually collected descriptions for images.
• Prior-information for text-visual complementary information alignment, like pre-defined color vocabularies; complicated architecture with much additional parameters for information extraction and fusion, causing potential information loss.
• Fixed auxiliary descriptions for training, sensitivity to text variation.

Our framework:

• Automatically generated text from visible and infrared images.
• Feature level fusion module without additional parameter. End-to-end ReID loss fine-tunes LVM, guiding semantic alignment across all modalities without prior-information, significantly mitigating fusion-vision discrepancy and achieving more acurrate cross-modal retrieval.
• Employs LLM random rephrasing to create dynamic descriptions for training, improving the robustness against textual variation.

Q3: Motivation behind the new testing setting.

A3: Humans perceive objects as visible images, and eyewitness descriptions are based on these perceptions. These descriptions, rich in information complementary to infrared modalities, serve as auxiliary clues for retrieval. Given the variability in eyewitness descriptions of the same target, our task setting allows any description of visible images of the same identity to be used with infrared features for retrieval, mimicking the varied visual perceptions of human eyewitnesses.

Q4: Imbalance of content distribution among several modules.

A4: For better content distribution balance, we will re-organize the content about task definition, text-generation, and add more details of each module.

Q5: Challenges unresolved by current text-assisted VI-ReID methodologies. How the proposed approach addresses these issues comprehensively?

A5:

• Sensitive to the text variation. Our method utilizes LLM based random rephrasing to create dynamic text for training, significantly enhancing the robustness against text variation.
• Struggling text-infrared information integration and fusion-visible features alignment. By fine-tuning LVM with end-to-end ReID loss to align semantics across all modalities, we simultaneouly achieve text-vision alignment for better infrared compensation and fusion-visible alignment for more accurate retrieval.

Q6: The quality of generated text descriptions may affect the retrieval performance.

A6:

• The quality of generated text affects model performance when original images' (for text generation) quality or generators' capabilities are suboptimal.
• However, even on challenging LLCM and lower-resolution RegDB with generated descriptions not completely accurate, our method still achieves improved performance. This demonstrated the robustness of our method against inaccuracies in descriptions.

Q7: Didn't introduce limitations.

A7: We introduced the limitation about impact of text quality in Appendix D. More detailed discussion and direction for future improvements will be added in the revision.

Q8: Using formula to express voting scores.

A8: The scores are difined as:

$$score = f_{ensemble} \cdot f^I _{rgb} = (f^I _{ir} + f^T _{rgb} + f^{fusion} _{rgb})/3 \cdot f^I _{rgb} = ([f^I _{ir}, f^T _{rgb}, f^{fusion} _{rgb}] \cdot [f^I _{rgb}, f^I _{rgb}, f^I _{rgb}])/3$$

The score equivalently represents the cosine similarity between the visible feature and a concatenated infrared-text-fusion feature, increasing the feature dimension. This higher-dimensional space increases the distance between identities, enhancing identity discrimination.

Q9: There are some typos in this paper.

A9: We will revise all mentioned and other found typos as suggested.

We are grateful for your positive recognition of the detailed tables, clear figures, and the soundness of our framework. We also appreciate your constructive comments and will revise and clarify the suggested points to improve the quality of the paper writing.

Q1: Some content in the appendix should be moved to the main text to enhance readability.

A1: As suggested, we will add details of the proposed task setting and an introduction about the generation process into the main text to ensure the coherence and readability.

Q2: Compared with the previous text enhancement method YYDS, the advantages of this paper should be further explained.

A2:

YYDS

• Manually collected descriptions for images.
• Prior-information for text-visual complementary information alignment, like pre-defined color vocabulary dictionary; complicated architecture with much additional parameters for information extraction and fusion, resulting potential information loss.
• Fixed auxiliary descriptions for training, resulting sensitivity to text variation.

Our framework

• Automatically generated text from visible and infrared images.
• Feature level fusion module without additional parameter. End-to-end ReID loss fine-tunes LVM, guiding semantic alignment across all modalities without prior-information, significantly mitigating fusion-vision discrepancy and achieve more acurrate cross-modal retrieval.
• Employs LLM random rephrasing to create dynamic descriptions for training, improve the robustness against textual variation.

Q3: There are some spelling errors in the paper (line 131).

A3: We will conduct a thorough review and the correction of spelling errors throughout the document to ensure professionalism and clarity.

Q4: Varied comparison settings in Table 4, especially the differences in results for YYDS in Tri-SYSU-MM01 vs. Tri-RegDB and Tri-LLCM.

A4: We will standardize the experimental setups for YYDS across all tests to use the "I+T->R" configuration. Both YYDS and our method employ joint text and infrared sample retrieval, whereas other methods solely use infrared queries.

Q5: Briefly discuss the reliance on the accuracy of text information generation, which may pose a limitation.

A5:

• The quality of generated text can indeed affect model performance, particularly when the original images' (for generation) quality or generators' capabilities are suboptimal.
• However, even on challenging LLCM and lower-resolution RegDB, with generated descriptions not completely accurate, our method still achieves improved performance. This demonstrated the robustness of our method against inaccuracies in descriptions.

We appreciate your recognition of the soundness, competitive performance, and adequate experiments of our method. Our method is a novel exploration of applying foundation models to down-stream data-intensive multimodal tasks. It uses LVM-generated text to enrich infrared representations and employs an end-to-end ReID loss to fine-tune the LVM text encoder, minimizing discrepancies between visible and fusion features thereby achieving competitive performance across three expanded VI-ReID datasets. We are grateful for feedback on the clarity and logical coherence of our presentation. In response, we will refine our manuscript to highlight contributions, correct misleading references and improve readability, especially in Section 3.2 and fusion process. We have adjusted the framework, moving LLM augmentation to "Text Generation," and added elaborations of each module, as detailed in Figure 1 of the submitted rebuttal PDF.

Q1: Suggestions for clear presentation of Section 3.2, especially fusion process.

A: Incremental Fine-tuning Strategy (IFS) includes Fusion Module and Modality Joint Learning, integrating text into the infrared modality for improved cross-modal retrieval.

Fusion Module. As shown in Figure 2 from the rebuttal material, the fusion process is defined as:

$$f^I_{rgb} = f^I_{ir} + f^I_{comp} = f^I_{ir} + (f^I_{rgb} - f^I_{ir}) \approx f^I_{ir} + (f^T_{rgb} - f^T_{ir}) = f^I_{ir} + f^T_{comp} \triangleq f^{fusion}_{rgb}$$

Where $f^I_{ir}$ and $f^I_{rgb}$ are infrared and visible features; $f^T_{ir}$ and $f^T_{rgb}$ are text features of visible and infrared image; $f^T_{comp}$ and $f^I_{comp}$ are text and visual complementary information for infrared modality, respectively.

The visible feature $f^I_{rgb}$ decomposes into the infrared feature $f^I_{ir}$ and its complementary feature $f^I_{comp}= f^I_{rgb} - f^I_{ir}$, and similarly, $f^T_{comp} = f^T_{rgb} - f^T_{ir}$. Using foundation model's basic text-visual alignment capability, visible text features $f^T_{rgb}$ and infrared text features $f^T_{ir}$ are roughly equivalent to visible $f^I_{rgb}$ and infrared $f^I_{ir}$ features respectively. Thus, $f^T_{comp}$ is roughly equal to $f^I_{comp}$. We can create fusion features by adding $f^T_{comp}$ to $f^I_{ir}$, roughly equivalent to visible features.

Modality Joint Learning. Freezing visual encoders, we fine-tune a LVM text encoder with ReID loss to jointly align semantics across all modalities. The loss includes a cross-entropy loss $L_{id}$ and a weighted regularized triplet loss $L_{wrt}$, defined as:

$$L_{total} = L_{id}(f^*) + L_{wrt}(f^*) \quad f^* \in \\{f^ T _ {rgb},f ^ I _ {rgb},f^ I_{ir},f^ {fusion}_{rgb}\\}$$

The fusion feature consists of the combination of frozen infrared features and text complementary features. With IFS, we can further align text and visual complementary features, thus we can further optimize fusion features, reducing fusion-visible discrepancy and improving cross-modal retrieval accuracy.

Q2: Misleading reference of Section A that does not exist within the document.

A2: The reference is detailed in Appendix A, about the fine-tuning process of two modality-specific text generators. We will add explanation of this process in main text and correspondingly correct the reference to ensure better coherence.

Q3: How the text data has been aligned with image modality in pre-train? Which text?

A3:

CLIP text encoder possesses text-image alignment capability benefited from pre-training on the large-scale image-text pairs (WebImageText) via contrastive learning:

$$\mathcal{L} = -\sum_{i=1}^{N} \left[ \log \frac{\exp(v_i ^T \cdot t_i) / \tau)}{\sum_{j=1}^{N} \exp(v _ i ^T \cdot t_j) / \tau)} \right]$$

Where $v_i$ and $t_i$ are the $i$-th image-text features pair in a batch, $\tau$ is the temperature parameter, and $N$ denotes the number of image-text pairs in the batch. Using this basic capability of CLIP, visible text and infrared text features are roughly aligned with respective visual features, which is further aligned adapting to VI-ReID task in following optimization.

Q4: Is N_{sum} different from N_i? If so, how? If not, why using this notation?

A4: They are equal because each fusion feature is created based on the corresponding infrared feature. We use these two notations to distinguish the features of different modalities more clearly, so the subscript same as their belonging modality is used for the number of features of all modalities.

Q5: Difference between MES and MER in ablation.

A5: MES (Modality Ensemble Searching) and MER (Modality Ensemble Retrieving) describe the same module in Section 3.3. We will unify all names of this module to MER.

Q6: Strategy for improving the quality of generated text.

A6: To improve generated text quality, we can apply several strategies during the generator fine-tuning process (Appendix A):

Better data for Generator Fine-tuning: During generator fine-tuning process, we can use augmentations (e.g., flipping, brightness adjustments) for more diverse images, enhancing text generation against varying image quality.

Stronger Generative LVM: Use advanced LVMs with larger parameters thereby capturing more useful visual information in images to generate textual descriptions with better quality.

Progressive Generation Strategy: Fine-tune generators on images with attribute annotations to focus more on fine-grained attributes rather than sentence modeling, then use LLMs to reorganize them into high-quality descriptive sentences.

Q7: How enhanced text quality boosts system effectiveness?

A7: During training, high-quality text allows the model to learn better text-vision correspondences, maximizing the utilization of complementary information to create fusion features. During retrieval, high-quality text provides accurate information for infrared compensation, enabling more accurate cross-modal retrieval.