When One Moment Isn't Enough: Multi-Moment Retrieval with Cross-Moment Interactions

We appreciate the acknowledgement of the rigorously verified dataset and the innovative IoU-guided post-verification module. We address the concerns below:

W1: Thank you for pointing this out. We will make a correction in the revised version, along with a careful review of similar formatting issues.

W2: Thank you for your detailed comments regarding Figure 3.

1. Dummy token placement: Although this is just a schematic illustration of the model, we will revise the position of the dummy token concatenation to reduce confusion.

2. Key/Value construction in ACA: The Adaptive Cross-Attention (ACA) [1] module’s internal attention structure is different from the classic cross-attention structure, especially in the key/value construction. Our ACA module adopts the exact same design and code implementation from CG‑DETR[1]/FlashVTG[2]. Here we clarify the exact behaviour of this module.

Formal definition

The attention weights and fused features are

We denote the input video clips $V$ as a query matrix $p_Q(\mathbf V) \in \mathbb{R}^{L_v \times d}$, the key matrix as $p_K([\mathbf Q,\tilde{\mathbf D}]) \in \mathbb{R}^{(L_q + L_d) \times d}$ formed by concatenating the text features $\mathbf Q$ with dummy tokens $\tilde{\mathbf D}$, and the value matrix as $p_V(\mathbf Q) \in \mathbb{R}^{L_q \times d}$ using only the text features. The functions $p_Q(\cdot)$, $p_K(\cdot)$, and $p_V(\cdot)$ are learnable linear projections that map input features into the query, key, and value spaces, respectively.

Code‑level detail

After computing attention on the full key (text + dummy), we mask out the dummy slice of $V$:

# crossattention.py (line 385)
attn_output = torch.bmm(
    attn_w[:, :, num_dummies:],   # weights over text positions
    v[:, num_dummies:, :]         # value = text tokens only
)

Hence, dummy tokens steer the attention via the key but never inject their own content into the output—mitigating the mismatch between limited query scope and untrimmed video semantics while keeping the fused feature purely textual.

Revision actions

1. Figure 3 will be redrawn so the Q/K/V flow and dummy masking are explicit.
2. We will add the above equations and code pointer to the appendix for transparency.

We appreciate the reviewer’s careful reading, which helps us improve the paper’s clarity.

W3: Thank you for noting that our current draft gives only a cursory treatment of the dummy encoder. We would like to clarify its role and provenance, and we will add a detailed discussion to the supplementary material:

Dummy Encoder

• Design: an L-layer Transformer encoder (L = args.dummy_layers) with 8-head self-attention, PReLU activation, and an optional final LayerNorm.

• Input: [N_dummy learnable tokens] + [text tokens], each with positional embeddings; standard padding mask applied.

• Mechanism: dummy tokens attend to the full sentence, aggregating its global semantics. → Output: B × N_dummy × D, compact text queries.

• Function. During Adaptive Cross‑Attention, video clips (queries) can allocate weight either to true text tokens or to these dummy positions. Because the final value comes solely from text tokens, the dummy tokens function as a controllable “attention sink”: they absorb weight from video clips that are semantically irrelevant to the query, preventing spurious video‑text alignments while preserving global context modelling.

• Provenance. The dummy encoder is inherited unchanged from previous works (e.g., CG‑DETR[1], FlashVTG[2]) and is not claimed as part of our own contribution.

Q1: We appreciate the reviewer’s concern regarding potential overlap among Top‑k predictions, which is an important aspect for multi-moment retrieval.

1. Theoretical Guarantee — Dual Redundancy Mitigation

• Temporal NMS Upper Bound. During inference, temporal NMS with IoU threshold 0.7 is applied to all candidate segments. By definition, any two retained segments satisfy IoU ≤ 0.7, effectively limiting high-overlap cases.

• Post-Verification (PV) Consistency Constraint. The PV module re-scores segments with tIoU supervision and contrastive representation loss $\mathcal{L}_{\text{repr}}$, which penalizes semantically redundant segments, promoting diversity at the objective level.

Conclusion: PV module suppresses redundancy via learning signals, while temporal NMS further provides an upper-bound on overlap, together ensuring diverse Top‑k outputs.

2. Empirical Evidence — Improved Segment Diversity

We report average pairwise IoU among Top‑k predictions on the QV-M² val set:

These consistent gains indicate that the second and third predictions better align with different ground-truth segments. If all segments were highly overlapping, mIoU@2 and mIoU@3 would not increase simultaneously.

[1] Moon, W., Hyun, S., Lee, S., & Heo, J. P. (2023). Correlation-guided query-dependency calibration for video temporal grounding. arXiv preprint arXiv:2311.08835.

[2] Cao, Z., Zhang, B., Du, H., Yu, X., Li, X., & Wang, S. (2025, February). Flashvtg: Feature layering and adaptive score handling network for video temporal grounding. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 9226-9236). IEEE.

We thank the reviewer for noting the task’s novelty and our strong empirical results. We address the concerns below:

W1: We compared our model with the state-of-the-art temporal-aware video large language models in a zero-shot setting.

We selected Chat-UniVi [1], a CVPR 2024 Highlight paper, as our baseline model. It can encode up to 100 frames per video, giving the temporal coverage MMR needs, whereas others like VideoLLaVA [2] and Video-ChatGPT [3] handle only around 8 frames, which is insufficient for reliable multi-moment retrieval. Below we provide the exact prompt and the corresponding zero-shot results.

This is a {duration:.2f} second video clip. 
Provide single or multiple time intervals within the range [0, {duration:.2f}], 
formatted as '[a, b]', that corresponds to the segment of the video best matching the query: {query}.
Respond with only the numeric interval.

The zero-shot evaluation results are provided below, highlighting the challenges posed by QV-M².

W2, L1: Expanding the scale of MMR datasets is indeed crucial for driving further progress.

Based on our annotation experience with QV-M², we see several promising directions to scale up:

1. Synthetic Multi-Moment Construction from other Datasets: By transplanting existing ground-truth moments from other datasets (e.g., TACoS, Charades-STA) into unrelated regions of the same video, we can simulate multi-moment scenarios at scale. While this approach may introduce artifacts, it offers a practical way to generate large-scale pseudo-MMR data for pretraining.

2. Leveraging Video Captioning Datasets: Since the most resource intensive aspect of MR dataset construction lies in timestamp annotation. Video captioning datasets are already temporally aligned with captions. These captions could be converted into SMR style annotations, and LLMs may be used to group semantically related but temporally disjoint segments to simulate multi-moment supervision.

3. Semi-Automatic Annotation with Vision-Language Models (VLMs): Another promising direction is to leverage VLMs to generate candidate temporal segments for a query, followed by human verification. This would reduce manual effort while maintaining annotation quality.

We believe these directions are promising for extending MMR datasets beyond current scale and will incorporate them into the revised version of our work.

W3: Thank you for pointing this out. We will make correction in revised version, along with a careful review of similar formatting issues.

[1] Jin, P., Takanobu, R., Zhang, W., Cao, X., & Yuan, L. (2024). Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13700-13710).

[2] Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., & Yuan, L. (2024, November). Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 5971-5984).

[3] Maaz, M., Rasheed, H., Khan, S., & Khan, F. (2024, August). Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 12585-12602).

We appreciate the recognition of the new QV-M² dataset and FlashMMR’s state-of-the-art performance. We address the concerns below:

W1: Thank you for pointing out this omission. Here we provide the Train/Val/Test split size.

QV‑M² is an incremental dataset, and therefore use the full set of annotations in our experiments, including the original QVHighlights labels. The exact split statistics are now included in the revised manuscript:

W2: We stated the IoU threshold for mR@k in line 178 of the paper. And we will add the implementation details in supplementary.

In our implementation, we average results over multiple thresholds: [0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95]. Here, we give the definition of Mean Recall@k:

where $\tau$ is the IoU threshold for determining whether a prediction is considered a match to a ground truth, $k \in \{1,2,3\}$ denotes the rank, $\mathbf{1}[\cdot]$ is the indicator function, $\mathcal{Q}$ is the set of all queries, and $\mathcal{G}(q)$ represents the ground truth moments associated with query $q$.

W3, Q1, Q4: To better understand the dataset itself, we present more details of QV‑M² through the following comparisons and illustrative examples. And we will add it in the revised version.

Key differences between QV‑M² and QVHighlights. Here is the primary differences between the two datasets across different dimensions.

Annotation example. Although our main goal was to obtain one‑to‑many labels, we intentionally increased the semantic richness of the queries—an aspect missing from earlier MR datasets and consistent with the enhanced annotation strategy outlined above. In revised manuscript we will provide additional examples to highlight the advantages of QV‑M². Here are some representative queries from QV‑M²:

1. "Drone pans over a marina and a fountain before landing on a park bench where two women wave." (Context and temporal dependency)

2. "A woman shows the clothing before pointing the camera at the mirror." (Context and temporal dependency)

3. "A woman without sunglasses in a blue shirt talks to the camera while walking outside." (Negative description)

4. "A reporter in a yellow shirt without hat is reporting on a bushfire emergency." (Negative description)

W4: Thank you for pointing this out. We have updated it in the revised version, using lowercase p to denote the layers in the feature pyramid.

Q2: We appreciate the reviewer’s question and would like to clarify the statement.

While NExT-VMR is introduced for multi-moment retrieval (MMR), its average of only 1.51 ground-truth moments per query indicates that most queries still correspond to a single ground truth moment. Furthermore, its evaluation metrics remain unchanged from single moment retrieval (SMR) setting using R1@IoU and mAP@IoU over predicted segments, without modification to account for multiple relevant moments per query.

In contrast, QV-M² provides denser multi-moment supervision with an average of 2.5 ground truth moments per query, and supports evaluation that naturally extends standard SMR metrics without assuming a single target, enabling accurate and fair assessment of multi-moment predictions. We believe this better captures the unique challenges of MMR in both data and evaluation.

Q3: Thank you for your insightful comment. You are correct that our current evaluation process assumes oracle knowledge of the number of ground truth moments per query.

Specifically, model itself does not predict the number of relevant moments, but generate a fixed number of moment predictions (e.g., n = 10) and rank them by confidence scores, selecting the top-k predictions to do the evaluation. This design aligns with prior moment retrieval benchmarks (e.g., MomentDETR[1], QD-DETR[2]) and ensures fair comparison across models. Notably, our metric mR@k partially reflects the model’s ability to handle under-prediction, as it measures the recall of ground truth moments within the top-k predictions.

We acknowledge that this assumption limits the practical applicability of the system in open-world settings. Incorporating moment count prediction along with precision-recall-based metrics that account for over- and under-prediction, would be valuable future directions. We appreciate your recognition that our method and dataset is a step toward more realistic video understanding.

[1] Lei, J., Berg, T. L., & Bansal, M. (2021). Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34, 11846-11858

[2] Moon, W., Hyun, S., Park, S., Park, D., & Heo, J. P. (2023). Query-dependent video representation for moment retrieval and highlight detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 23023-23033).

Thank you to the authors for the detailed response. Many of my queries have been addressed, however I have a few points remaining.

For W1, it was not clear to me that this was an incremental dataset and therefore included the original QVHighlights. It would be helpful to make this clearer in the paper. To clarify, I assume the QV-M$^2$ moment statistics include the QVHighlights annotations, rather than being the statistics for just the new subset? Likewise, I assume the QV-M$^2$ results include results on QVHighlights test annotations also?

On W2, I was referring to the threshold values used in the implementation, which are not stated in the original version. Thank you for providing these here.

For W3, it is stated that there are cross-video query annotations. Are the associations to the cross-video query annotations explicitly labelled? As in, is there information within the dataset that associates the queries to moments across the multiple videos? Rather than just having the same query multiple times across the videos without an explicit association. This is something I did not understand from reading the original paper. It would be helpful to include this in the paper.

We thank the reviewer for highlighting our practical multi-moment task and the effectiveness of the PV module. We address the concerns below:

W1, Q1: We follow the bias evaluation methodology proposed in [1], particularly the analysis of temporal annotation bias in Fig. 2. We replicate similar experiments on QV-M² and our FlashMMR model.

The results show that our dataset exhibits significantly lower temporal bias compared to other commonly used benchmarks such as Charades-STA[2] and TACoS[3]. As image uploads are not supported here, we report the distribution of normalized start/end positions in the training set below. The results demonstrate that QV-M² has a much more uniform temporal distribution, highlighting its advantage in mitigating temporal bias. "Pred." refers to the predictions generated by FlashMMR. "Interval" denotes the normalized temporal interval of the video.

W2, Q2: We thank the reviewer’s question regarding the computational overhead of our proposed Post-Verification module. To clarify, we conducted thorough experiments analyzing both runtime and scalability concerning video length and the number of candidate predictions.

Computational Complexity Analysis: The theoretical complexity of the Post-Verification module are as follows:

• Inference: Time Complexity: $O(K\bar{L}d^2)$, Memory Complexity: $O(K\bar{L}d)$

• Training: Time Complexity: $O(K\bar{L}d^2 + \bar{V}^2d)$, Memory Complexity: $O(K\bar{L}d + \bar{V}^2)$

Where, $K$ is the number of candidate predictions per video, $d$ is the feature dimension, $\bar{L}$ is the average length of candidate, $\bar{V}$ is the average length of video.

Experimental Validation: We empirically validated the above theoretical analysis by varying video lengths and candidate counts. All experiments utilized a batch size of 32, a feature dimension of 256, and were performed on an NVIDIA RTX 4090 GPU with an Intel(R) Core(TM) i9-14900KF CPU.

(1) Scalability with Video Length:

These results confirm our theoretical expectation: during inference, runtime and FLOPs remain stable irrespective of video length. In contrast, during training, runtime and FLOPs increase linearly with video length due to the $\bar{V}^2d$ term. Since $\bar{V}^2 < K\bar{L}d$ in our setting, the $\bar{V}^2d$ term remains a lower-order contribution, which explains why the additional FLOPs during training are moderate.

(2) Scalability with Candidate Set Size:

These experiments demonstrate a clear linear relationship between candidate count and computational overhead in both inference and training phases, again aligning well with our complexity analysis.

In summary, these experimental results demonstrate that, under the current setting, the computational complexity of our Post-Verification module is reasonable and acceptable.

W3: The method indeed borrows several modules from previous approaches and combines them sensibly. However, these modules do not adapt well to the new MMR task. The newly proposed post verification module is a useful addition given the task setting, and is shown to work well in the ablation.

W4: We would like to clarify that our MMR scenarios are not constructed by combining existing single-moment annotations. Instead, all multi-moment labels are manually annotated from scratch. Furthermore, to ensure annotation consistency and minimize potential bias, we employed a structured quality control process following the annotation phase.

C1: Thank you for the suggestion. We will revise the abstract to emphasize that such cases already exist in previous datasets, in order to strengthen our motivation and clearly position the proposed MMR task.

C2: Yes. We will clearly state that the new annotations explicitly include temporal sequencing information in the revised version.

As mentioned in the second point of our annotation guidelines, both SMR and MMR annotations encode temporal sequencing information. For example, in the video 1tGsRVsMtlc_660.0_810.0, the query "A woman shows the clothing before pointing the camera at the mirror." contains the temporal cue "before", indicating that the event should occur prior to "pointing the camera at the mirror." If one were to simply align the visual content with the text while ignoring this temporal cue, the subsequent action would mistakenly be included, making it impossible to correctly determine the end time of the intended event.

C3, Q4: Yes. We still evaluate FlashMMR on YouCookII (an instructional-video dataset for SMR), even though newly proposed PV module is meant to address the multi-moment retrieval (MMR) task.

As a result, the single-moment retrieval (SMR) performance primarily reflects the capability of our baseline model, FlashVTG, as supported by the experimental results below. Our proposed module does not significantly affect the SMR task. Notably, videos in the YouCookII dataset are much longer than those in previously used datasets (e.g., QVHighlights and Charades-STA), while our baseline model is not well-suited for handling such long videos.

C4: Thank you for the insightful suggestion. We agree that automatic query generalization, e.g., turning "cutting tomato" into "cutting vegetables", could help build a large "silver-standard" dataset.

A practical caveat is that the generated queries from instructional datasets tend to stay at the action level, whereas mainstream MR benchmarks use event-level queries. Even so, the technique remains attractive for producing large-scale pre-training data. For the present work, we therefore focus on a manually curated, event-level benchmark and leave "silver-standard" expansion as promising future work.

C5: Thank you for the suggestion. We have added additional relevant datasets (YouCookII, COIN, and HiREST) to Table 1 in the revised version.

C6: Thank you for the valuable suggestion. We have included citations of TMLGA, DORi, and ExCL in the revised Related Work section, as they are highly relevant to our task.

These methods all contribute to moment retrieval:

• TMGLA proposes a proposal-free, end-to-end method for natural language-based temporal localization using a guided attention mechanism and soft labels to improve efficiency and accuracy.

• DORi proposes a language-conditioned graph-based method for temporal moment localization. It models human-object-activity interactions via spatio-temporal graphs and uses iterative message passing to align video content with natural language queries.

• ExCL proposes a modular framework that fuses video and text features via cross-modal LSTMs and directly predicts clip boundaries using span predictors.

Q3: Yes. We agree that hierarchical or sparse interaction mechanisms may provide a more efficient way to model semantic consistency in the post-verification stage, especially by focusing on high-confidence segments.

We adopt a GRU-based design with $L_{repr}$ to model local and global temporal consistency between predicted moments and the query. In contrast, purely hierarchical or sparse interaction mechanisms may lack fine-grained global context, and their supervision signals can become sparse. As noted in the computational complexity analysis, the design remains efficient in scenarios where $\bar{V}^2 < K\bar{L}d$. Nonetheless, we acknowledge the potential benefits of alternative interaction strategies, and we consider this an important direction for future exploration.

[1] https://arxiv.org/abs/2101.09028

[2] Tall: Temporal activity localization via language query.

[3] Grounding action descriptions in videos.