Temporal Sentence Grounding with Relevance Feedback in Videos

Dear Reviewer P4nK,

Thank you for your comprehensive and positive review of our work. We appreciate your insights and suggestions for further improvement. Below, we address your concerns point by point.

Q1: The paper should illustrate bad examples in the visualization section, and discuss the limitations of the proposed method.

A1: Following the reviewer's suggestion, we have included two bad examples in the attached PDF file of the global response. In the first example of the query text "A person starts sneezing", our model incorrectly judges the relevance feedback due to the lack of audio cues, which are crucial for identifying the action of sneezing. In the second example, the query text is "A person opening a laptop," but the video segment actually shows the person closing a laptop. Our model misinterprets the temporal sequence of actions, mistaking the closing action for the opening action. These examples demonstrate that our proposed model struggles with handling audio-related actions and temproal-sensitive content. However, such limitation can be alleviated by integrating audio features and temporal modeling in the video representation moudel.

Q2: The proposed relation-aware segment predictor is key for TSG-RF. Could it be adapted for traditional TSG methods to enable them to work for TSG-RF?

A2: Of course, we can adapte our main components to traditional TSG methods to enable them to work for TSG-RF. Note that, to effectively address the TSG-RF task, it is essential to use not only the Relation-aware Segment Predictor (RaSP) but also the Multi-granularity Relevance Discriminator (MgRD). These components work together to assess relevance at multiple granularities and provide accurate segment grounding. Following the reviewer's suggestion, we adapt them into an existing traditional TSG model EAMAT, which is the best performer among our four compared models. As shown in the following table, EAMAT with our devised RaSP and MgRD outperfomes the original EAMAT and the enhanced EAMAT++ with an extra relevance discriminator. The results demonstrate the adaptability and effectiveness of our proposed modules for enhancing traditional TSG methods to work for the TSG-RF task.

Q3: As the proposed method requires additional relevance feedback, I wonder how the extra computational workload would be included.

A3: To measure model's computational workload, we use GFLOPs and parameter count. By adding relevance feedback capability to our model, the GFLOPs and parameter count increase from 0.077 to 0.081 G and 1.21 to 1.27 M , respectively, with a relatively small increase of 5.1% and 4.9%. This negligible increase is due to our multi-task framework, where the additional relevance feedback module shares the feature extraction modules with the segment grounding module. This efficient design ensures that the overall computational workload remains low while enhancing the model with relevance feedback capability.

We will incorporate all the above in the final version. Thank you.

Dear Reviewer eoKJ,

Thank you for your detailed and positive review of our work. We appreciate your insights and suggestions for further improvement. Below, we address your concerns point by point.

Q1: The weaknesses of this paper include a lack of sufficient baselines for comparison with the RaTSG network. For instance, the paper does not investigate the use of Large Multimodal Models (LMMs) to assess the relevance between the query text and video, which could offer a more robust benchmark for evaluating the proposed method. Incorporating such comparisons would enhance the overall evaluation of the RaTSG network.

A1: While we acknowledge the importance of comparing our method with large multimodal models (LMMs), it is important to note that current state-of-the-art LMMs, such as Llava and GPT-4, do not yet have the capability for tackling the video-based task. Instead, we choose a large-scale pre-trained vision-language model, i.e., CLIP, to assess the relevance between the query text and video. Specifically, CLIP is utilzed to measure the cosine similarity score between the query text and each frames of a video, and the final relevance is obtained by aggregating the similarity scores over all frames. We use two aggregate methods: averaging all scores (CLIP-Avg) and averaging on top-5 scores (CLIP-Top5). Then we jointly use CLIP and EAMAT that is the best performer among our four compared models, to conduct relevance feedback and grouding. The reulsts on Charades-RF are summarized in the following table. CLIP-based models achieve low accuracy, demonstating its poor ability for relevance prediction. We attribute it to the fact that CLIP is pretrained without addtional training, thus suffering from domain shift issues on the traget dataset.

Q2: The paper also lacks adequate qualitative analysis. Including more qualitative examples of both successful and unsuccessful cases would provide deeper insights into the strengths and limitations of the RaTSG network.

A2: To discuss the limitations of the work, we have included two unsuccessful examples in the attached PDF file of the global response. In the first example of the query text "A person starts sneezing", our model incorrectly judges the relevance feedback due to the lack of audio cues, which are crucial for identifying the action of sneezing. In the second example, the query text is "A person opening a laptop," but the video segment actually shows the person closing a laptop. Our model misinterprets the temporal sequence of actions, mistaking the closing action for the opening action. These examples demonstrate that our proposed model struggles with handling audio-related actions and temproal-sensitive content. However, such limitation can be alleviated by integrating audio features and temporal modeling in the video representation moudel. For the successful examples and corresponding discussion, we illustrate them in Figure 4 and Section 4.4.

We will incorporate all the above in the final version. Thank you.

Dear Reviewer dpih,

Thanks for your detailed review and constructive feedback. We appreciate your insights and would like to address your concerns point by point.

Q1: discussion about highlight detection

A1: We thank the reviewer for sharing the insight. Actually, our RF task is quite different from highlight detection. Specifically, highlight detection mainly aims to identify the most interesting or important segments within a video based on a given natural language query, focusing on segments that are salient or engaging. This task is similar to the traditional temporal sentence grounding, and existing highlight detection methods still naively assume that there are always highlights or interesting segments in the video. In contrast, our RF setting is specifically designed to handle cases where no relevant segments are found. This involves providing relevance feedback to indicate when no relevant segments exist, which is essential for practical applications where not all queries have corresponding relevant segments. We believe that our RF setting can also be plug-and-play to be applied to the highlight moment detection tasks. We will discuss the highlight detection in the related work.

Q2: The method is cumbersome, featuring several incremental designs.

A2: Our proposed framework is specifically designed to handle the challenges posed by our new introduced TSG-RF task, incorporating several new key modules to ensure effective performance (please see the global response). Moreover, our RaTSG framework is a multi-task model that achieves high efficiency with relatively low complexity. As shown in Table 1 of our paper, our model has fewer parameters compared to other baseline models, indicating that our approach is not cumbersome.

Q3: The experiments utilize outdated datasets. Newer datasets, such as QV-Highlights, should be considered.

A3: As shown in the table below, we make a statistics on the number of top-tier conferrence papers (NeurIPS、CVPR、ICCV、ECCV、AAAI、ACM MM) using the three datasets over the past three years, finding that Charades-STA and ActivityNet Captions are more popular than QV-Highlights. Though including QV-Highlights would enhance our work, we believe that the extensive experiments on two widely-used TSG datasets are convincing. Moreover, we try to use QV-Highlights, but we could not complete dataset downloading, reconstruction, and running experiments with multiple baselines within the limited rebuttal time. We leave the use of QV-Highlights for future work.

Q4: The baselines are no up to date which do not include sota methods. such as UMT, UniVTG, QD-DETR.

A4: We would like to clarify that our paper does include comparisons with recent state-of-the-art methods:

• The following table summarizes the traditional TSG performance comparison between the suggested UMT, UniVTG, QD-DERT and our compared ADPN on Charades-STA. It shows that our chosen ADPN is almost the SOTA method, performing much better than the other methods, especially in terms of R1@0.7. Therefore, we argue that our compared method is not outdated.

• Additionally, following the reviewer's suggestion, we have included a comparison with UniVTG and QD-DETR. Note that we did not compare with UMT considering its performance is much worse than the others (please see the above table). The table below summarizes the performance on the Charades-RF dataset. Our proposed model still outperforms the UniVTG and QD-DETR counterparts. We will update Table 1 by including UniVTG and QD-DETR and revise the corresponding discussion accordingly in the revison.

Q5: The Relevance feedback in Video Moment Retrieval is a trivial research question. Either we could generalize this problem to general video-related tasks.

A5: We respectfully disagree with the assertion that relevance feedback in video moment retrieval is a trivial research question. Actually, the TSG-RF task addresses a significant gap in the field of temporal sentence grounding by introducing a scenario where no relevant segments might exist in the video. This is a critical aspect that current methods do not adequately address. The relevance feedback mechanism in our work is crucial for several reasons:

• More Practical: In real-world applications, not every query will have a relevant segment in the video. Providing accurate feedback on the absence of relevant segments is essential for practical usability.

• Enhanced Accuracy: The feedback mechanism improves the precision and reliability of the grounding process by dynamically adjusting based on the presence or absence of relevant segments.

• Potential for Generalization: Our framework has the potential to be generalized to broader video grounding tasks, including spatial and temporal grounding, as well as general video understanding tasks. The principles of relevance feedback can be adapted to various contexts, enhancing the robustness and versatility of grounding models across different tasks.

We will incorporate all the above in the final version (as specified in our response). Please reconsider our paper. Thank you.

Dear Reviewer 3Qyt,

Thank you for your thorough review and valuable feedback. We appreciate your insights and would like to address your concerns point by point.

Q1: The work claimed to incorporate foreground/background info. However, it was simply adding a binary classification, such binary foreground classification for temporal grounding was already explored in [1]. The video part is done by weighted sum, which is also a standard way of aggregating frame features.

A1: In our work, we make the first attempt to fill the critical gap in traditional temporal sentence grounding tasks that assume the existence of relevant segments in the corresponding video but fails to provid feedback on non-existence relevant content. In addition to proposing a novel TSG-RF task and a new framework, we claim that we have also developed novel technical designs for the pipeline.

In particular, as for the foreground/background information learning, our foreground-background detection is not simply implemented with a binary classification. Although the naive binary classification head has been investigated and utilized to detect the foreground clips in previous work [1], it is simply/directly applied to the raw clip-level features for classification. Such design is coarse and fail to capture the long-term dependency in the complex videos. In contrast, we propose a more effective multi-granularity relevance discriminator. By employing both frame-level and video-level relevance discriminators, our dual-level multi-granularity relevance discriminator allows us to effectively balance fine-grained and coarse-grained relevance assessments, thereby enhancing the overall robustness and applicability of our model.

Additionally, while the weighted sum operation is indeed a standard operation in video analysis, it is effective enough to handle the frame-wise contexts aggregation for determining the query-related video-level content. Moreover, aggregating frame features is not the focus of our work, and any feature aggregation operation can be theoretically used in our proposed framework. Note that our main contribution is proposing a novel TSG-RF task and a new framework that incorporates relevance feedback capability.

Q2: In Table 1, the author explained why the performance is 50%. Why are the other rows all have 71.94% of accuracy and 81.6% accuracy?

A2: In Table 1, the performance of traditional TSG methods, such as VSLNet, shows 50% accuracy because our newly constructed test set has an equal ratio (1:1) of samples with and without grounding results (i.e., query-relevant segments exist or do not exist in the corresponding video). Since the traditional TSG methods assume that all samples have grounding results, they lead to a relevance prediction accuracy of 50%.

For the enhanced versions of the traditional models (i.e., VSLNet++, ADPN++, SeqPAN++, and EAMAT++) , we incorporate an additional relevance discriminator to their models. This relevance discriminator is trained separately and then directly combined with the baseline methods, endowing them with the same ability to discriminate relevance (Please refer to Appendix B.1 for more implementation details). This additional relevance discriminator achieves the 71.94% and 81.6% accuracy on Charades-RF and ActivityNet-RF, respectively. It is worth noting that as all enhanced models share the same relevance discriminator, they achive the same accuracy on the same dataset.

Q3: Although this work proposes a new RF setting, it will be interesting to know how the model performs in a standard setting to see if there is a performance gap.

A3: Actually we have conducted experimetns with the standard setting in Appendix B.3 of our paper. Below is the Table from the appendix, summarizing the performance comparison in the context of the TSG task. Although our proposed RaTSG is not consistently the highest performer, it demonstrates competitive results across various metrics. It is important to note that our model is specifically designed for the TSG-RF task, which includes the challenge of handling samples without grounding results. This specialized focus may slightly affect its performance on the standard TSG task, yet RaTSG remains highly competitive. This demonstrates the robustness and versatility of our approach, indicating its strong potential in more complex real-world scenarios.

We will incorporate all the above in the final version. Please reconsider our paper. Thank you.