We thank the reviewer for the positive feedback on our benchmark. We reply to each concern below:

W1: Good question. The primary challenge in the current VMR community is that existing datasets like ActivityNet-Caption, Charades, and TACoS are designed for short videos with simple actions. However, these datasets have several limitations:

1. Scale and Complexity: These datasets contain relatively few videos, and the events or actions within the videos are often singular and straightforward. This simplicity limits their ability to simulate the complexities and diversity of generalized video moment retrieval (GVMR) tasks, which require handling everyday, dynamic, and varied scenarios.
2. Lack of Progress on Standard Benchmarks: Despite years of research, these datasets have been extensively studied, and models like BAN-APR [1] (stops at 43% for IoU@0.7 metric for years) have achieved strong performance. However, there have been limited breakthroughs in performance on these benchmarks in recent years. This stagnation highlights the need for a larger, semantically rich dataset to drive further advancements in GVMR research.

While the MAD dataset offers semantic richness, it primarily consists of curated movie scenes. This focus on narrative-driven and meticulously designed settings makes it more suitable for tasks like causal inference in storytelling, which, while relevant, is only a subset of the broader VMR research community's needs. Additionally, ethical concerns surrounding the use of movies further complicate its applicability.

In contrast, the YFCC100M dataset offers significant advantages:

• Scale and Diversity: It contains a massive number of videos, with varied lengths and rich semantic information, covering a wide range of motions, events, and everyday scenarios.
• Generalization: Its diversity in content makes it an ideal choice for developing and evaluating models capable of handling generalized video moment retrieval in real-world scenarios.

These characteristics make YFCC100M the preferred choice for our work, as it aligns closely with the need for a large-scale, semantically diverse dataset to advance GVMR research.

W2: We appreciate the observation. While the average number of segments corresponding to a query is less than 2, this figure includes no-target cases, which lowers the overall average. For queries that do have multiple targets, the actual average number of segments is 3. As shown in Figures 4, 5, and 6 in the Appendix—particularly Figures 4(a) and 4(b)—our dataset features a relatively balanced proportion of multi-target queries and includes a meaningful number of no-target queries. These figures provide a more comprehensive perspective on the dataset’s structure and highlight the challenges it introduces compared to existing datasets.

W3: In Table 2, the metrics mAP@K, N-acc, and T-acc are specifically designed to evaluate performance in multi-target and non-target scenarios:

• mAP@K measures the model’s ability to handle multi-query grounding. A segment is considered grounded if its IoU with a ground truth segment exceeds a threshold. The average precision (AP) for a single query is calculated as the ratio of the number of grounded segments to the total number of ground truth segments, reflecting the model’s effectiveness in multi-target scenarios.

• N-acc evaluates whether a non-target segment is correctly classified as a non-target sample.

• T-acc measures whether a segment labeled with a correct query is accurately grounded as a target sample.

Our model achieves SOTA performance across all these metrics compared to open-source solutions, demonstrating its robustness and effectiveness in both multi-target and non-target scenarios.

We appreciate the reviewer's positive feedback on our model, training pipeline, comparisons with baselines, and its versatility on related tasks. We address your concerns below:

W1:

Firstly, we compare our dataset with other mainstream datasets for the video moment retrieval (VMR) task. As shown in Table 1, our dataset contains 9,957 videos, which is on par with other datasets (ranging from 127 to 14,926 videos). Furthermore, it includes 153,191 queries, surpassing most VMR datasets (which typically contain fewer than 100,000 queries, except for MAD). Notably, none of the existing datasets support complex query types, such as multi-target or no-target queries simultaneously. Additionally, we conducted a query length comparison with other VMR datasets, as illustrated in Figure 5(b) in the Appendix. Our dataset features a much larger proportion of longer queries compared to TACoS_2DTAN, Charades-STA, and Activity-Net Caption. Specifically, most of our queries range from 10 to 40 words in length, whereas the majority of queries in other datasets are within 30 words.

Secondly, we have provided train, validation, and test splits for the NExT-VMR dataset. Figure 4 in the Appendix illustrates the distribution of queries across these splits, with 107,589 queries in the training set, 11,808 in the validation set, and 33,794 in the test set. Additionally, we also provide query annotation distributions for the train, validation and test sets in Figure 6. These distributions share similar characteristics, showing moments widely spread across the upper triangle, despite many being concentrated in the upper-left region.

W2: We apologize for the confusion regarding Figure 3. In the updated figure, the different colors represent distinct, non-overlapping segments with different ground truth labels that are to be grounded in the video. These colors visually differentiate segments corresponding to specific queries. To address the issue, we have revised and clarified the figure in the updated PDF, ensuring it provides a more accurate and detailed representation of our model.

Q1: Yes, we have implemented all publicly available methods as baselines for multi-target prediction and adapted them to include non-target detection evaluation. As shown in Table 2, while these methods demonstrate a certain level of performance, our specially designed modules offer notable improvements, showcasing the effectiveness of our approach.

To further evaluate performance on standard datasets, we implemented two one-to-one baseline models. The results indicate that while these models perform adequately on single-target queries, their performance on multi-target metrics is notably lower compared to specifically designed multi-target models like BCANet. This highlights the necessity of specialized designs for multi-target video moment retrieval tasks.

Q2: Yes, our model is specifically designed to support querying multiple related clips in a video using a single query. It effectively handles one-to-one, one-to-many, and even one-to-none query scenarios. This functionality aligns with our core motivation to address diverse grounding tasks, making the model well-suited for retrieving and grounding multiple related clips for a single query.

Q3: Good catch! We specifically chose 9,957 videos from YFCC100M based on the following considerations:

• Appropriate Duration: The videos in our dataset have an average duration of 35.54 seconds, striking a balance between capturing meaningful information and avoiding unnecessary computational complexity. This makes the dataset well-suited for video-related tasks.
• Diversity and Scale: We carefully removed videos with repetitive scenes to enhance diversity and ensure that each video contributes unique content. This approach maintains the complexity of the dataset while increasing its variety. As shown in Table 1, the resulting dataset is also comparable in scale to other widely-used Video Moment Retrieval (VMR) datasets, making it robust enough for comprehensive experiments and fair comparisons with existing benchmarks.
• Future Expansion Plans: While this version of the dataset meets current research needs, we plan to expand it further in the future to provide the community with an even larger and more diverse resource for advancing video understanding tasks.

By carefully selecting videos based on these criteria, we ensured the dataset aligns well with the requirements of our experiments while maintaining high quality and relevance. We hope this clarifies your concerns.

Q4: Yes, we will release our dataset and code once accepted.

We thank the reviewer for their positive feedback on our task, model, and results. We address the weaknesses below.

W1: Yes, we have addressed your concerns in the subsequent sections of the paper. The enhancements in the model's efficiency, interpretative prowess, and precision are demonstrated and analyzed in our experiments. Specifically:

• Interpretative prowess and precision refer to the model's robustness in understanding the query, which is primarily reflected in metrics such as IoU and mAP. These results are detailed in the main results table, Table 2.
• Efficiency is further analyzed Table 7. on component-wise evaluations, where we break down the inference times of key modules to highlight the model's performance in retrieval speed.

For comprehensive details, please refer to Table 2 and the accompanying discussions.

W2: You may find more detailed statistics in Appendix: Figure 4 about the distribution of (a) query numbers relevant to N clips and (b) Query Type; Figure 5 about (a) Statistics Distribution for Query Number and Each Corresponding Segments of Ground Truth Timestamps in Log Scale. (b) Distribution for Query Length with comparison with other datasets; Figure 6 about Moment annotation distributions of NExT-VMR dataset, where “Start” and “End” axes represent the normalized start and end time points, respectively. Additionaly, the average video duration length is 35.54 seconds.

W3: Yes, we evaluated BCANet on traditional VMR datasets, Charades and TACoS, comparing it against all publicly available multi-prediction models. BCANet outperformed these models in one-to-one VMR tasks, as shown in Table 1, highlighting its strong adaptability and superior performance in this context.

Table 1: Performance Comparison on Charades and TACoS Datasets

We thank the reviewer for the positive comments on our task and our extensive experiments. We address the concern below:

W1: The method you mentioned from the paper “Temporally Grounding Language Queries in Videos by Contextual Boundary-Aware Prediction” [1] utilizes clip-level features to predict boundary features through binary classification, as defined by the loss function:

This approach enhances the model’s prediction by conducting boundary-aware binary metric learning directly on the features before grounding MLPs.

In contrast, our method introduces a novel approach by applying contrastive learning at the video-language fusion stage. We leverage contrastive learning to ensure that our decoder queries, which are fused representations of language and video inputs, have a higher affinity (measured by smaller distances in the contrastive feature space) to the time spans of interest (SoI). This is achieved by incorporating boundary information—obtained through the average pooling of features within the boundaries—into each mini-batch during training.

Our design offers several key advantages:

1. Uniform Awareness Across Different SoIs: The decoder queries become uniformly attentive to various SoIs, enhancing the model’s ability to accurately focus on relevant temporal segments.

2. Robustness to No-Target Queries: In situations where there is no SoI (queries without a target ground truth), our decoder queries appropriately refrain from responding to any features, thereby improving the model’s robustness and reducing false positives.

By integrating boundary awareness through contrastive learning at the fusion stage, our method differs from previous works that focus on boundary prediction at the feature level. This fusion-stage approach not only improves model performance but also effectively handles scenarios that traditional boundary prediction methods may not address as efficiently.

Based on your feedback, we have added a technical comparison with the method [1] you mentioned in the revised version of our paper in sec. 4.2.

[1] J. Wang, L. Ma, and W. Jiang, “Temporally grounding language queries in videos by contextual boundary-aware prediction,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 7, pp. 12168–12175, 2020.

Data Source Transparency

"Data Source Transparency: In your response, you mention that the media files were directly crawled from Flickr under their respective Creative Commons licenses. However, the paper explicitly states that the dataset is derived from YFCC100M, without clarifying that the media files were obtained from Flickr. This omission creates ambiguity for readers regarding the actual source of the data."

We have already clarified the data source in our previous responses. To reiterate, all raw video files used in our research were directly crawled from Flickr under their respective Creative Commons licenses. This has now been explicitly stated in both the revised PDF version of the manuscript and in our earlier explanations.

Compliance with the Webscope License Agreement

"The paper explicitly states that the annotations (query statements) used in your work rely on YFCC100M. If your annotations are derived from YFCC100M metadata, this would invoke the Webscope License Agreement, which prohibits redistribution of modified datasets or annotations without explicit permission. If this is the case, please provide evidence of permission or authorization from Yahoo for such use."

We would like to clarify that no annotations from the YFCC100M dataset were used in our work. The annotations in our study were created through a rigorous process with great efforts, which has been explained in detail in Figure 2 and the Appendix of our paper. Thus, there was no reliance on YFCC100M metadata for the creation of our query statements.

To ensure absolute transparency and address the concerns raised, we would like to emphasize the following:

1. All raw data (video content) used in our research was independently crawled from Flickr under their respective Creative Commons licenses.
2. To further address the ambiguity as you mentioned, we could have removed all mentions of YFCC100M in our paper. However, we believe this is not the ideal solution, as YFCC100M is a foundational and highly respected dataset that has significantly contributed to multimedia research. Removing references to it would undermine its value and importance to the community.

Dear Yinhang,

We acknowledge YFCC100M as a well-designed and impactful dataset, widely referenced and built upon by other works such as [1], [2], and [3]. We believe it is appropriate and respectful to credit YFCC100M for its contributions to the field.

If you are one of the original authors of YFCC100M, we would welcome the opportunity to engage in a direct and constructive discussion to address any concerns and ensure full compliance with the authors' intentions. If, however, you are not affiliated with the authorship of YFCC100M, we kindly request that these concerns be evaluated with consideration of the detailed clarifications provided above.

References

[1] FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation. WACV 2021.

[2] Towards Open-ended Visual Quality Comparison. ECCV 2024.

[3] Panoptic Video Scene Graph Generation. CVPR 2023.

Thank you for your detailed response. I have two key concerns that I would like to highlight:

1. Data Source Transparency: In your response, you mention that the media files were directly crawled from Flickr under their respective Creative Commons licenses. However, the paper explicitly states that the dataset is derived from YFCC100M, without clarifying that the media files were obtained from Flickr. This omission creates ambiguity for readers regarding the actual source of the data.

2. Compliance with the Webscope License Agreement: The paper explicitly clarify the annotations (Query statements) used in your work rely on YFCC100M. If your annotations are derived from YFCC100M metadata, this would invoke the Webscope License Agreement, which prohibits redistribution of modified datasets or annotations without explicit permission. If this is the case, please provide evidence of permission or authorization from Yahoo for such use.

Addressing these points will help ensure transparency and compliance with licensing terms. Thank you for your attention, and I look forward to your response.

Dear Luo Yinhang,

We’d like to make following explanation:

The relation tuples and corresponding ground-truth timestamps were not derived from the YFCC100M metadata. In fact, such information does not exist in the metadata. In the previous statement, "extract" refers to our manual annotation process, where we watched the videos directly. This process involved human observation to identify basic action units within the videos and annotate the query bases and their corresponding time segments. To address this, we have updated the wording in the revised PDF to more precisely describe our contributions and eliminate ambiguity.