IVCR-200K: A Large-Scale Benchmark for Interactive Video Corpus Retrieval

The paper has several weaknesses that will need to be tackled before being accepted. Most of them are related to clarity of the task, and dataset, missing references, and clear technical flaws.

1. The paper claims that the 80.8% of the users would prefer interactive search functionality. This is a very strong claim especially with the little information provided on how this number was found. The paper mentions and survey on 500 users, but did not provide the survey. Depending on the nature of the questions and the population the results can be interpreted in different manners. I think the authors need to rephrase of clarify accurately the meaning of this study.

2. The paper does not cite a few important works very related to the topic. First, the ActivityNet dataset was proposed by Caba et al [A], not Krishna, the one referenced in the paper is ActicityNet Captions which is an extension of the original ActivityNet, please give proper attribution to the previous fundamental works in the field. The paper has lots of connection with [B], since the latter one introduced the task of single video moment retrieval in a video corpus. However, there was no mentioned of this paper whatsoever.

3. The paper utilizes automated method to create and refine dataset annotations. They claim that there is a refinement process afterwards but do not describe how is this one performed. Additionally, the multi-turn setting is not explained clearly in the paper. What kind of multi-turn scenarios there are? There are so many way to interact with a system in a multi-turn manner. One could look for videos and refine the search based on the results, look for videos then look for moments in the video, look for moments and then refine the search, etc. The authors mentioned a few of these examples but did not explain how was the benchmark built to cover them. Without this context, it is really hard to understand the results presented in the multi-turn setting in table 4.

4. The authors claim to be measuring BLEU-4 and GPT-4 score metrics on this task. However, I don't understand what kind of answers are the authors evaluating what is the ground-truth for these answers? Is it simply the caption of the video? This requires clarification.

5. The provided baselines make sense for the video and moment retrieval tasks. However, the use of a Multimodal LLM for the fine-grained part of the task (video moment retrieval) seems to be not appropiate. First of all, the MLLM can only process a limited number of frames, which limits the resolution of the system to regress accurate time-stamps. This fact can be seen in table 2, where a method that was state-of-the-art in video-language grounding 4 years ago (2D-TAN) outperforms the proposed method by more that 40 absolute points.

6. The training of the MLLM has one major technical flaw. The paper states that they use the MLLM for re-ranking the top retrieved videos given by the retrieval system. They do it by training the MLLM with three losses, one of them being a cross-entropy loss shown in equation 2. This cross-entropy loss is flawed since it is applied to the video indices in each retrieved set, where the model is expected to identify the "correct" video among the top-k candidates. This setup treats each re-ranking instance as a unique classification problem over a dynamically changing set of "classes" (the retrieved videos), with each new top-k set effectively re-shuffling the class labels. Cross-entropy relies on fixed, stable classes to guide learning, and without this consistency, the model faces moving targets that prevent it from generalizing any meaningful re-ranking ability. A more suitable approach would involve using ranking-specific losses that do not depend on changing class labels but instead focus on optimizing the order of relevance among retrieved videos, such as pairwise or list-wise ranking losses. Given that there was no ablation of this losses, its hard to diagnose how much is this loss affecting the performance of the system. However, if my understanding is correct, this loss is not the way to train a re-ranking system.

7. The conclusions on table 4 are not generalizable to the benchmark, they only talk about the limitations of the current method on performing fine-grained localization. The paper could have proposed a two-stage method using a state-of-the-art retrieval system + a state-of-the-art moment retrieval system and evaluate if under the same setting as table 4. Since this setting is the main selling point of the paper. However, the paper only evaluates the proposed system despite the weaknesses and limitations.

[A] Caba Heilbron, F., Escorcia, V., Ghanem, B., & Carlos Niebles, J. (2015). Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 961-970).

[B] Escorcia, Victor, et al. "Finding moments in video collections using natural language." arXiv preprint arXiv:1907.12763 (2019).

W1. As indicated in Table 2, InterLLaVA's performance in video moment retrieval may not be state-of-the-art, revealing some weaknesses in the model. W2. During the training of InterLLaVA, questions are divided stage by stage, which may hinder its ability to handle complex or "jumping" questions. This could explain the model's subpar performance in direct video moment retrieval.

1. It was good that the authors used a survey to motivate the interactive retrieval task. However, I found it quite difficult to find any details of what the survey contained. I read the description given in Appendix A, which describes the scale of the survey but not the survey questions. Without a detailed description of survey questions and responses statistics, it is quite difficult to assess the validity of claims made in the introduction. For example, in lines L048-L051, the authors say “our questionnaire indicate that interactive demands exhibit intricate behavioral patterns.”

2. To support the claim that “users desire ‘multi-turn interaction’ with systems” (L044), the authors cite the number of rounds of interaction in ShareGPT conversation dataset as being “remarkably high at 7.27.” (L048). However, my understanding is that ShareGPT primarily contains data from text-based chat dialogues, rather than multi-turn video retrieval. As such, it doesn’t seem like particularly strong evidence that users want multi-turn behavior in the video setting. A caveat here: unfortunately, the dataset has been taken offline so I’m basing my claim that ShareGPT is primarily text dialogues from my memory - please correct me if I’m mistaken. The authors could potentially address this by providing additional evidence that is more specific to the multi-turn video retrieval setting, or by presenting arguments for why text-based interactions are meaningful evidence for their claims.

3. Table 1 contains a comparison with prior work. In the “Real interaction” column, only the proposed dataset (ICVR-200K) is ticked. However, if I understand correctly, the dialogue is mostly generated by GPT-4 (using the pipeline shown in Figure 3). I say mostly, because in L427, the authors write “while most dialogues consist of concatenated single-round exchanges, we also gather a limited number of multi-turn dialogues from real users.” I understand that there is a human expert review process, but I would not describe this data pipeline data as “Real interaction”, given the heavy role played by GPT-4. The proposed pipeline design also makes it somewhat unsurprising that the average length of questions and answers in IVCR-200K is much longer than AVSD (as discussed in L269). (I would expect GPT-4-generated text to have this property.) It would be appreciated if the authors could clarify how their dataset meets their definition of "Real interaction"?

4. I found several parts of the paper quite difficult to follow. To give a few concrete examples: (4.1) When defining the “interactive” task in L099-L123, the third component of the definition is “Real interaction.” which is described as follows: “The pioneers create simulated environments to generate interactive data (Ma & Ngo, 2022), but we emphasize that only truly understanding users can optimize a better search experience.” I’m not sure what “truly understanding users” means here? Is this a technical claim or an aspiration? How does it relate to the property of “Real interaction.” If I should interpret it to mean “using real user data”, would this not imply that real user data should be collected? (My understanding, pointed out in weakness 3, and reflecting Figure 3 of the paper, is that the multi-turn dialogues are mostly generated by GPT-4). (4.2) In figure 1, it would be helpful to have a caption explain the takeaways. I understand that (a), (b) and (c) each illustrate a different task, but is there some significance to the red sad faces and happy green faces? Some possible interpretations I had were (i) The face is happy for (c) because this is the formulation proposed by the authors; (ii) The face is happy for (c) because this is what the survey suggested that users prefer; (iii) The face is sad for (a) and (b) because the retrieval result quality is lower as a consequence of failing to use the history of previous queries. If the authors could clarify this, it would be appreciated. (4.3) In the caption for Table 2, it says “bold represents optimal performance.” However, every entry in the “InterLLaVA (Ours)” row is bolded, even when it is not the best performing. For instance, under the R@1 IoU=0.5 metric, this method lags behind several of the Moment Retrieval baselines. Are these baselines comparable? If so, shouldn't 2D-TAN be bold here?