Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach

W5 and Q1: Implementation detail of majority voting and release of code & data.

Thanks for the question. Here we provide a pseudo code version of the implementation of majority voting. We plan to release the full code and data upon paper acceptance.

function majority_voting_aggregation(similarity_matrices):
    Input: List of similarity matrices [num_query, 1, num_target]
    Output: Aggregated performance metrics

    Initialize voted_sim_sort as an empty list

    For each query:
        Initialize rank_list and sim_sort_list as empty
        For each similarity matrix:
            Get similarity scores for the query
            Sort scores in descending order and save to sim_sort_list
            Compute the query's rank and save to rank_list

        Find the most frequent rank in rank_list
        Select the corresponding sorted similarities and add to voted_sim_sort

    Concatenate results in voted_sim_sort, sort, and extract ranks
    Return computed metrics from the ranks

We thank the reviewer for reviewing our paper and providing valuable suggestion, which can definitely help this paper more clear and stronger. We will include these feedback into our revised paper. We sincerely hope the response can help resolve the reviewer's concerns. Thanks.

Q2: How might the proposed framework perform in real-world, large-scale text-video retrieval systems?

Thanks for the interesting yet practical question. The training part can increase the generalization capability. In real world, with large-scale datasets, the model can become stronger. As for testing (deployment) in real-world scenario:

1. High retrieval accuracy. As suggested by text-enrichment in retrieval phase, even without re-training the retrieval model, by only manipulating the text query, we can get a substantial performance gain.
2. Efficiency. As we analyzed in the Appendix (Table 9), in real-world applications with large scale videos and queries, our method appears to be less efficient due to the involvement of LLM. However, this limitation can still be addressed. On the one hand, reducing inference cost of LLM is widely studied in industry, which can be transferred into this model. On the other hand, even without LLM, our model still shows decent performance with vanilla NLP toolkit, such as NLTK.

We thank the reviewer for this constructive question. We will include a dedicated section to discuss the real-world application potential and challenges.

Q3: It is better to distinguish this work from existing works that also use VLM and/or LLM.

1. High-level Design and Data-Centric Perspective. Our framework emphasizes a data-centric methodology, leveraging VLM and/or LLM primarily for data curation to address information asymmetry. Unlike prior research, which often uses these models to enhance retrieval performance directly, our work aims to underscore the critical role of data quality and the impact of data-driven strategies in this domain.
2. Testing Phase Enrichment. Existing work rarely consider the testing phase enrichment. We present a holistic solution of text query enrichment in testing phase, including query generation, query selection, and result aggregation.
3. Dependency of LLM. As we discussed in W1, the work does not have strong dependency on LLM. Using more cheap toolkits, e.g., NLTK can still harness the query selection and aggregation mechanism, boosting the performance.
4. Novelty of the Query Selection Mechanism. We emphasize that our Query Selection mechanism introduces a rarely explored area within the domain. This component is critical, as it significantly impacts performance and data efficiency.
5. Analytical Insights Beyond the Method. Beyond the technical contributions, we believe our analytical findings add substantial value to the field. For instance, the Oracle query concept demonstrates the substantial untapped potential of improved text representations, paving the way for future research in this area.

We hope these clarifications address your concerns and enhance the overall understanding of our contributions. Thank you for the opportunity to improve our work.

Dear reviewer 38bZ,

We sincerely thank you for reviewing our paper, providing constructive suggestions, and raising the score.

1. We thank the reviewer for this valuable suggestion regarding the paper writing. We will further polish the paper to highlight the contribution on the overall model structure aspect.

2. We thank the reviewer for the constructive suggestion. Designing alternative algorithms as comparisons can definitely better demonstrate the effectiveness of FQS algorithm. We show the extended experiment results in the Table below.

Firstly, we experiment with the suggested "minimum similarity threshold", denoted as FQS (s=) in the Table, where the value of "s" is the similarity threshold. In implementation, when iteratively selecting queries, we add an additional condition that requires the selected queries to meet the minimum similarity. If no query meets this condition, we will use the original user query. The results show that when setting the threshold to 0.5 or 0.75, the performance is the same as without the condition, suggesting that most of the queries are distributed in a closely relevant range. When further increasing the threshold to 0.85 or 0.95, we find that the retrieval performance starts to decrease. It is due to that higher similarity threshold will limit the diversity of the selected queries and constrain the retrieval performance.

Secondly, we implement the suggested "minimize the minimum distance" version, which is denoted as NQS (Nearest Query Sampling) in the Table. Evaluation results show that this approach will also restrict the diversity of the selected queries, thereby hurting the retrieval performance.

These experiments can further verify the effectiveness of the proposed FQS algorithm. We greatly thank the reviewer for this valuable and constructive feedback. We will include this experiment in the cam-ready paper.

3. Thanks for the kind reminder. We have updated the paper to include more comparisons in Table 2 and Table 3.

In the cam-ready version, we will further involve your suggestions to make the paper stronger. Thanks!

Best,

Authors of Submission862

W1: The captions and queries are generated by VLM and LLM, which may contain hallucination information. It is better to provide evidence on the reliability of using these models.

• Thanks for this constructive question. Indeed, VLM/LLM generated contents inevitably contain hallucinatory information. We have the following designs to address this potential issue.
  • Text-enrichment in training. In this phase, the data are mainly generated by VLM via captioning. We use these data to "pre-train" the model. The model are later trained on the standard dataset with human annotated captions. This two-stage design makes the training procedure more robust to the noise in stage-1, as stage-2 can help rectify the noisy part with more reliable data.
  • Text-enrichment in testing. In this phase, the data are mainly generated by LLM. To ensure the potential hallucination not harm the retrieval performance, we have three designs: 1) a carefully curated prompt that instructs the LLM to follow the semantic meaning of the user query; 2) a novel query selection mechanism that selects reliable yet diverse queries from the generated ones, its effectiveness is validated in the ablation study (Table 7); 3) a majority voting aggregation strategy that filters out the unreliable results. The three designs establish a hierarchy to remove hallucination while preserving useful data.
• Apart from the designs, we have the following evidences to prove this aspect.
  • First, the overall performance gain of our method has effectively proved that these generated data can be used to significantly improve the performance, while the potential hallucination can be blocked or removed.
  • The choice of LLM plays an important role in this issue, as stronger LLMs tend to generated more reliable contents, while weaker LLMs are more likely to generate hallucination. In Sec. 4.2.4 and Table 8, we compare a strong LLM GPT-4 and a weak LLM Phi-3.5, the result shows that 1) Phi-3.5 indeed generates more hallucinatory content than GPT-4; 2) Our method is robust to the choice of LLM, using Phi-3.5 can still achieve comparable performance to GPT-4. The result further indicates that our method is robust to hallucination.

W2: It is better to prove the performance gain is not from the accumulation of more computation, as VLM and LLM are involved in the proposed method.

• We thank the reviewer for pointing out this problem. We acknowledge that computation cost is an important aspect for a model, especially for model-centric methods with novel architecture designs. However, we would like to emphasize that our method, as a data-centric approach, focus more on the effect of data. Both VLM and LLM are simply used as data generation tools.
• Specifically, for VLM, it is used to generate training data, which is already an affordable choice compared to human annotation. The cost of such data generation are rarely taken into consideration of computation cost [1], as training is a one-pass process.
• For LLM, it is used to enrich text queries during testing phase. We agree that this needs to be investigated, since the LLM computation is used at each time of inference. We thank the reviewer for raising this valuable question. We analyze the computation and inference time introduced by the data text enrichment of test phase in the table below.

In this table, we use 1k videos from the MSR-VTT dataset as the test set and measure the computation cost and time efficiency for using one user query to retrieve the target video. We observe that:

• For each user query, involving LLM as the data generation tool can significantly boost the performance, but also increase the compute and time, ranging from 31.72% to 81.02%.
• Although GPT-4 is supposed to be a much larger model than Phi-3.5, its API time latency is much smaller. This gap reveals that there is large room for optimizing the time latency of Phi-3.5 in practical applications.
• As our main target is to investigate and reveal the potential of data, but not the data generation tool itself, we also explore using NLTK toolkit as a (almost) free lunch to generate data. It introduces negligible compute and time latency, showing a considerable performance gain compared to baseline, while still worse than LLMs.

We thank the reviewer for raising the valuable suggestion. We will include the discussion into the revised paper.

[1] Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?

Dear reviewer Tkyv,

We sincerely appreciate your efforts on reviewing our paper and raising the score.

Regarding your concern about hallucination of VLMs and LLMs, existing experiment results show that our method is robust to such noisy data. We acknowledge that hallucination is still a potential limitation that may affect the performance. In the updated paper, we have added a discussion about this limitation in the Appendix.

In the cam-ready version, we will further involve your suggestions to make the paper stronger. Thanks.

Best,

Authors of Submission862