LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification

Thanks a lot for reviewing our paper and giving us valuable suggestions. We will answer the questions one by one.

Q1: Visual hallucination in VLM captioning.

Thanks for your insightful comments. We acknowledge that GPT-like vision-language models (VLMs) are prone to hallucinations, particularly when generating fine-grained descriptions. Leveraging existing techniques, such as contrastive decoding [1], in data construction could further improve the accuracy and reliability of our descriptions. We will explore these enhancements in future work.

[1] Leng S, Zhang H, Chen G, et al. Mitigating object hallucinations in large vision-language models through visual contrastive decoding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13872-13882.

Q2: Explanation of coarse-to-fine captioning.

In interactive ReID, we expect the witness to provide additional fine-grained discriminative information in the subsequent dialogue rounds. This information should not merely repeat what has already been stated. Therefore, we use the coarse description as a basis for generating the follow-up description.

An alternative approach could be to generate a fine-grained description first and then remove the overlapping content with the initial description. However, our generation strategy simplifies this process of decomposing the description, making it more efficient and structured.

Q3: Revision of figure 3

(i) Thanks for your advice, we will change it to "Descriptions from Witness".

(ii) The term "clue" refers to the embeddings of the context description $z_t$. We use this term to indicate that the candidates are selected based on the current "clue" we have at hand.

We apologize for any confusion caused by this terminology and will revise the figure in the next version.

Q4: Long text encoding of CLIP model

As mentioned in Appendix C, we interpolate the text positional embeddings from 77 to 192 and finetune fine-grained captions in Interactive-PEDES to obtain the Retriever. If the total text length exceeds 192 tokens, we truncate it accordingly. In practice, since the initial description has a maximum length of 48 tokens, the average length per turn in a five-round dialogue is (192-48)/5=28.8. This is sufficient for the current interaction setup, and we can also further interpolate it to longer encoding lengths.

Q5: Details about the question model functions when the interactive framework is integrated with existing text-based methods.

Please refer to our response to Q1 from reviewer fi2Z.

Q6: Re-organize section 3.2 (dataset construction)

Thanks for your valuable suggestions. We will revise this section to improve the clarity.

Q7: Abbreviation of interactive person re-identification.

Thank you for pointing this out. We will update the abbreviation to "Inter-ReID" in the new version.

We appreciate the insightful questions and will address your concerns in the following.

Q1: Comparison of inference time and ablation study on the number of candidates.

We conduct these experiments on NVIDIA RTX 3090 GPUs. The inference time for the 5-round question generation of LLaVA-ReID and other baseline methods are shown in the table below:

Our multi-image questioner does not add significant computational overhead and achieves the best performance.

The retrieval results for different numbers of candidate images are shown in the table below:

The results indicate that setting $k=4$ provides the best balance between retrieval performance and computational efficiency.

In our previous experiments, the reported BRI was computed using a logarithm with base 2. According to the recently released code of BRI[1], we have corrected this by using the natural logarithm (base $e$). While this change slightly affects the absolute values of BRI, the relative ranking and overall conclusions remain unchanged.

Q2: Introduction of Gumbel subset sampling.

Thanks. To enable the selector to explore more possible combinations during the training stage, we employ a differentiable random sample strategy, namely, Gumbel-top-$k$ relaxations. This approach introduces randomness to increase the diversity of candidates while ensuring that the NLL loss gradient can be properly backpropagated for effective optimization.

Q3: BRI metric interpretation.

BRI (Best Log Rank Integral) assesses the system’s ability to refine retrieval results over multiple interaction rounds. It measures the average area under the log-rank curve of all rounds. A lower BRI value indicates a more efficient and effective refinement process. We will make this clear in the next version.

Q4: Input of Answerer in section 5.3.

The T-ReID datasets only provide caption-image pairs of individuals, we turn to the multi-modal LLM to simulate the users. Specifically, we use LLaVA-OneVision-Qwen2-7B-ov as the Answerer. It provides the follow-up descriptions, looking at the image of the target person.

Q5: Ambiguous figure of qualitative results.

The 4 images of candidates in the figure are the selected candidates and the rank represents the similarity ranking of the target person within the gallery after current round. We will improve the clarity of Figures 7 and 8 in the next version.

Thank you for acknowledging the novelty of our method. We will answer the questions one by one.

Q1: Details about the ensemble.

In our integration of LLaVA-ReID with the existing T-ReID framework, we use the human-annotated captions in the T-ReID datasets as the initial queries. We first encode an initial query $T$ using T-ReID model and compute the matching score between $T$ and gallery images:

$$\operatorname{score}(I_i|T)=\operatorname{sim}(\phi^{base}_t(T), \phi^{base}_v(I_i)),$$

where $\phi_t^{base}$ and $\phi_v^{base}$ are text and visual encoder of T-ReID model, $I_i$ is the $i$-th image in gallery, and $\operatorname{sim}(\cdot, \cdot)$ denotes cosine similarity. Next, we perform 5 rounds of interaction and encode the dialogue history $D=\{T, A_1,\dots, A_t\}$ using our Retriever, then compute the matching score:

$$\text{score}(I_i|D)=\operatorname{sim}(\phi_t(D), \phi_v(I_i)).$$

Then we re-rank the matching scores by simply averaging two scores:

\text{final\\_score}_i=\frac{1}{2}(\text{score}(I_i|T)+\text{score}(I_i|D)),

and the final score is used to compute the retrieval metrics.

It’s important to note that the T-ReID dataset only provides caption-image pairs for persons. To simulate user interactions, we turn to a multi-modal LLM. Specifically, we use LLaVA-OneVision-Qwen2-7B-ov as the Answerer, which generates follow-up descriptions based on the image of the target person.

Q2: Preprocessing of dialogue.

For dialogue preprocessing, we remove prefixes such as "Yes," "No," and multiple-choice options like "A)" and "B)" from the answers. The remaining responses are then directly concatenated and fed into the text encoder. While our current approach is straightforward, we believe that post-hoc techniques, such as dialogue rewriting, could be a promising strategy to enhance the system's ability to generalize to real-world, open-form responses.

Thanks for your comments. We will answer your questions one by one in the following.

Q1: Can you provide further analysis or experiments to assess the computational efficiency of your candidate selection module compared to baseline methods? (As well as the question generation time)

We conduct the experiment on the computation time of different select strategies, including top-k, k-means, and our selector. The experiment is performed on NVIDIA RTX 3090 GPUs, and we report the total selection time over five rounds in the table below:

Our lightweight transformer-based selector achieves the highest retrieval accuracy while ensuring no significant computational overhead compared to other methods.

We also provide the inference time for 5-round question generation of LLaVA-ReID and other baseline methods are shown below:

Our multi-image questioner does not add significant computational overhead, indicating its suitability for real-time applications.

Q2: How do you define the parameter $k$ in the Gumbel-top-k relaxation within your selection strategy?

The number of candidate images plays a crucial role in providing rich visual context to the Questioner. If too few candidates are given, the Questioner may not receive sufficient fine-grained information between persons. Conversely, an excessive number of images can overwhelm the MLLM’s capacity, introducing redundancy and extra computational costs. To further analyze the impact of $k$, we present retrieval results for different numbers of candidate images in the table below:

The results indicate that setting $k=4$ provides the best balance between retrieval performance and computational efficiency.

In our previous experiments, the reported BRI was computed using a logarithm with base 2. According to the recent released code of BRI [1], we have corrected this by using the natural logarithm (base $e$). While this change slightly affects the absolute values of BRI, the relative ranking and overall conclusions remain unchanged.

Q3: In the LLaVA-ReID framework, does the conversation get embedded into the LLaVA module before the candidate images are appended?

As depicted in Equation 3, candidate images are inserted before the dialogue history, in accordance with the default configuration of LLaVA. For improved visual clarity, we modify the input order in Figure 3.

Q4: In Figure 3, could you explain the meaning of the term “clue”?

The term "clue" refers to the embeddings of the context description $z_t$. We use this term to indicate that the candidates are selected based on the current "clue" we have at hand. We apologize for any confusion caused by this terminology and will revise it in the next version.

[1] Lee S, Yu S, Park J, et al. Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approach[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 791-809.