CausalVTG: Towards Robust Video Temporal Grounding via Causal Inference

We sincerely thank the reviewer for the thoughtful and detailed feedback. In response to the concerns raised, we have made the following clarifications and additions:

• We evaluated CADE on the Charades-CG dataset to show its benefit under compositional generalization and domain shift scenarios (Q1).
• We introduced two more challenging negative sampling strategies — GT-excluded and Semantic Hard negatives — and demonstrated their effectiveness on Charades-RF (Q2).
• We reported training and inference cost comparisons with major baselines, confirming CausalVTG’s computational efficiency (Q3).

We believe these clarifications and additional results address the reviewer’s concerns. If any issues remain unclear, we would be happy to provide further discussion.

Response to Q1

While CADE is conceptually important, its removal causes relatively minor degradation in performance. Could the authors provide more targeted experiments (e.g., under domain shifts or style variations) to better demonstrate its effect?

Yes, we further evaluated CADE separately on the Charades-CG dataset [1], which specifically tests novel phrase compositions and unseen words in temporal grounding tasks. These experiments demonstrate CADE's strong capability in addressing superficial co-occurrence patterns by enabling the model to generalize beyond previously observed linguistic combinations, an advantage particularly evident in compositional generalization scenarios​​.

Response to Q2

Have the authors considered generating more challenging or realistic counterfactual samples instead of random negative sampling? Would this further improve the robustness?

We thank the reviewer for the constructive suggestion. In addition to the original random negative sampling, we investigated two more challenging strategies to improve robustness: (1) GT-excluded negatives, where ground-truth segments are masked to force the model to reason over visually similar but irrelevant content; and (2) Semantic Hard negatives, where each query is paired with the most semantically similar but non-matching video in the batch.

Experiments on Charades-RF, which contains naturally ungrounded queries, confirm the benefit of both strategies: GT-excluded negatives improve performance under stricter IoU settings, while Semantic Hard negatives further strengthen overall discriminative ability by compelling the model to distinguish fine-grained semantic differences.

Response to Q3

The method depends heavily on large-scale pre-trained models, but there is no detailed discussion on computational cost or efficiency compared to existing baselines.

To evaluate the computational overhead, we trained and evaluated all models on the QVHighlights dataset using an NVIDIA A800 GPU (80GB memory) with a batch size of 64 over 50 epochs. As summarized below, CausalVTG's inference runtime is slightly longer than simpler baselines, yet remains highly competitive. Notably, prior methods such as Moment-DETR and QD-DETR originally required up to 200 epochs, and R2-Tuning utilized extensive GPU memory in training due to reversed recurrent block.

Response to Paper Formatting Concerns

The paper uses bolded phrases at the beginning of paragraphs without any punctuation; The reference formatting is inconsistent.

We thank the reviewer for carefully pointing out these formatting issues, and we will correct the bolded paragraph headers and ensure consistent reference formatting in the final version.

[1] "Compositional temporal grounding with structured variational cross-graph correspondence learning." CVPR 2022

[2] "Detecting moments and highlights in videos via natural language queries." NeurIPS 2021.

[3] "Query-dependent video representation for moment retrieval and highlight detection." CVPR 2023.

[4] "Correlation-guided query-dependency calibration for video temporal grounding." arXiv preprint arXiv:2311.08835 (2023).

[5] "R2-tuning: Efficient image-to-video transfer learning for video temporal grounding." ECCV 2024.

We appreciate the reviewer’s critical feedback and agree that additional targeted analysis is essential for strengthening the paper. In response, we conducted new experiments and provided clarifications.

• To address the need for a thorough ablation study, we added detailed experiments showing the individual and combined contributions of CADE, MSTP, QRM, and QGR, confirming that each module is complementary (W1, Q2).
• To clarify the reviewer’s concern that performance gains mainly come from MSTP, we demonstrate that MSTP improves proposal generation, while CADE uniquely mitigates spurious correlations via causal adjustment, playing an irreplaceable role (W2).
• To respond to the reviewer’s question about confounders, we explicitly define and illustrate them in the VTG context (e.g., spurious query–visual co-occurrences) and commit to clarifying these points in the main paper (Q1).

We believe these additions resolve the reviewer’s key concerns and strengthen the overall contribution of our work. If any issues remain unclear, we would be happy to provide further discussion.

Response to W1

No thorough ablation study is provided in main paper. Since the proposed CausalVTG is a combination of CADE, MSTP, QRM and QGR as shown in Table 4 in supplementary material, a detailed ablation study is needed in main paper. A base version of CausalVTG can be base model + CADE, then the integration of any one or two modules/methods given MSTP, QRM, and QGR should be included.

We thank the reviewer for highlighting this important point. In response, we have conducted a comprehensive ablation study following the reviewer’s suggestion. The results demonstrate the individual and joint contributions of CADE, MSTP, QRM, and QGR, validating their effectiveness. The ablation results demonstrate that each module is complementary and their integration is crucial to the overall effectiveness of CausalVTG. We will incorporate a detailed version of this analysis into the main paper.

Response to W2

Performance gain from CausalVTG is mainly from MSTP, which is not related to causal reasoning.

We thank the reviewer for this important point. While MSTP indeed contributes the largest raw performance gain by improving multi-scale proposal generation, CADE plays a distinct and critical role in addressing superficial co-occurrence patterns through front-door adjustment, which MSTP alone cannot resolve. To evaluate CADE’s effect, we evaluated it on the Charades-CG dataset [1], which stresses compositional generalization (novel phrase compositions and unseen words). Results show that removing CADE leads to clear drops in performance on these challenging settings, confirming that CADE is essential for enabling the model to generalize beyond spurious correlations, complementing MSTP rather than being redundant.

Response to Q1

The first claimed contribution of this paper is causal graph for video temporal grounding task. It would be great to straight forwardly show what is the confounder exactly in main paper. From common understanding of our community, confounder can be something like video background information or some template or pattern in language query. Explicitly show it in paper can enhance the paper quality.

Thank you for your valuable feedback. In our work, the primary confounder is the presence of superficial co-occurrence patterns between query phrases and visual contexts that spuriously influence grounding outcomes.

For example, in the Charades-STA training set, queries containing the word “sink” are commonly paired with actions like “wash” and “put”. However, when the query also includes “kitchen”, the dominant associated actions shift to “put” and “run”. This distributional shift—caused by an irrelevant contextual word—demonstrates a classic confounding effect: the added word jointly affects both the input representation and the prediction target, while being independent of the actual intended action.

This explains the failure case in Figure 1 of our paper, where R2-Tuning [2] succeeds on Query 1 but fails on Query 2. The presence of “kitchen” introduces a confounder that changes the model’s grounding behavior despite both queries referring to the same event.

In addition, we also consider unobservable confounders, such as visual decoration styles or recurring sentence templates, which may similarly bias grounding decisions. Our method addresses both types via a front-door adjustment mechanism, as supported by prior causal grounding work [3].

To address the reviewer’s suggestion, we will revise the main paper to more directly and concretely present these confounders.

Response to Q2

Referring to the weakness part, can you include a detailed and thorough ablation study in main paper? You can save much space by making Figure 2 as single-column

Following the reviewer's suggestion, we will resize Figure 2 to a single-column format in the main paper and incorporate the detailed ablation study from the supplemental material into the updated version.

[1] "Compositional temporal grounding with structured variational cross-graph correspondence learning." CVPR 2022.

[2] "R2-tuning: Efficient image-to-video transfer learning for video temporal grounding." ECCV 2024.

[3] "Cross-modal Causal Relation Alignment for Video Question Grounding." CVPR 2025.

We thank the reviewer for the thoughtful and constructive feedback, which helped us strengthen the paper with deeper analyses and clarifications. In response, we:

• Added causal baselines (DCM) and the other state‑of‑the‑art methods (DoRi, CG‑DETR) to our comparison tables, clearly demonstrating that our front‑door adjustment with counterfactual contrast achieves consistent and substantial gains over alternative causal strategies (Q1).
• Provided a sensitivity analysis of the K‑means cluster number (K) in CADE, reporting mean ± std metrics across multiple runs, which confirms the stability of our mediator design and supports the claimed causal robustness (Q2).

We believe these additions address the reviewer’s concerns and further highlight the robustness and contribution of our approach.

Response to Q1

Could you include at least one representative causal baseline in the comparison tables (or explain why this is not feasible) to clarify the benefit of your causal strategy?

Yes. In our revision, we have added DCM (Deconfounded Cross‑modal Matching) ​[1]– a causal method addressing dataset temporal‑annotation bias via back‑door adjustment . To further strengthen our empirical comparison, we also included DoRi ​[2] and CG‑DETR [3], which are state‑of‑the‑art methods that mitigate superficial co-occurrence patterns by reducing correlations between background and foreground.

The updated comparison demonstrates that our method significantly outperforms all baselines, particularly on the Charades-STA dataset, where the presence of multiple stylistically diverse annotations per video segment renders models more susceptible to superficial co-occurrence patterns.

Response to Q2

How sensitive is CADE to the K-means cluster number K and clustering randomness? An ablation or plot would help justify the causal robustness claim.

We performed a comprehensive sensitivity analysis on the Charades‑RF dataset by varying the number of K‑means clusters in {16, 32, 64, 128, 256, 512, 1024, 2048}. For each value, we ran multiple random seeds to compute mean ± standard deviation of metrics like R1@0.7, mIoU, and classification accuracy. Due to rebuttal constraints, only tabular results are presented here, but we commit to including detailed plots in the final version of the paper.

The experimental results indicate that a cluster number of 64 yields optimal performance. This choice provides comprehensive coverage of underlying semantic categories, effectively capturing confounders. In contrast, using too few clusters risks missing critical semantic distinctions, while employing too many introduces computational overhead and noisy representations, negatively impacting model performance.

[1] "Deconfounded video moment retrieval with causal intervention." SIGIR 2021.

[2] "DORi: Discovering object relationships for moment localization of a natural language query in a video." WACV 2021.

[3] "Correlation-guided query-dependency calibration for video temporal grounding." arXiv preprint arXiv:2311.08835 (2023).

We appreciate the reviewer’s constructive feedback, which helped us strengthen the paper. In response, we:

• Clarified the core causal assumptions and their appropriateness and limitations for VTG (Q1).
• Analyzed mediator design sensitivity and discussed alternative definitions (Q2).
• Expanded ablations to show the distinct and complementary roles of CADE, MSTP, and counterfactual learning (Q3).
• Detailed computational overhead with runtime and memory comparisons to simpler baselines (Q4).
• Provided prediction distribution analyses and committed to adding more visualizations to assess bias (Q5).
• Incorporated suggested references and scenarios, including temporal causality, multiple occurrences, and new YouCook2 results (S1,S2,S3).

We believe these additions meaningfully address the reviewer’s concerns and further highlight the novelty, rigor, and practical impact of our work.

Response to Q1

Key Causal Assumptions:

• Assumption 1 (Existence of Confounders): There exist latent confounders $Z$, such as visual stylistic cues (e.g., background contexts) and linguistic variations (e.g., phrasing habits), which simultaneously influence both the inputs $X=\\{V,Q\\}$ and the grounding outcome $Y$.
• Assumption 2 (Front-door Identifiability): There exists a set of mediator variables $M$ (semantic representations) that fully mediate the causal relationship $X→Y$. These mediators are assumed to be independent of direct influences from the latent confounders $Z$, thus satisfying the conditions required for the front-door adjustment.

Appropriateness for VTG tasks:

• Prior studies [1,2] clearly document the presence of stylistic biases within existing VTG datasets, highlighting the necessity of explicitly modeling such confounders.
• Employing semantic mediators aligns with recent causal inference methods that effectively disentangle spurious correlations in related multimodal grounding tasks[3,4]​.

Limitations and Constraints:

Our current SCM explicitly addresses stylistic variations but does not directly account for temporal location biases, which are also prevalent in VTG tasks [5,6]. In contrast, the DCM framework [5] explicitly employs back-door adjustments to mitigate temporal biases.

Response to Q2

The sensitivity of our framework to mediator design primarily stems from the choice of cluster number $K$ used in the K-means clustering of semantic mediators. To evaluate this sensitivity, we conducted experiments varying $K$, summarized in the table below.

Experimental results show that setting the number of clusters to 64 achieves the best performance, effectively balancing semantic granularity and noise. Fewer clusters may miss key distinctions, while too many introduce redundancy and degrade model stability.

Recent work, such as FDVAE (Front-Door Variational Autoencoder) [7], explores an alternative approach for front-door adjustment by learning latent mediators through variational inference instead of explicit clustering. Due to time constraints, we have not explored alternative mediator designs, but we will investigate them in future work to further improve performance and robustness.

Response to Q3

To clarify the contribution of each component, we provide detailed ablation results in Table 4 of our paper, which show that MSTP plays a crucial role in standard VTG tasks by generating proposals across multiple scales and granularities. To further support this, we include a new sensitivity analysis on the choice of temporal strides used in MSTP. This demonstrates that multi-scale temporal modeling is key to capturing varied action durations and enhances the model’s ability to localize events more precisely.

Furthermore, we evaluated the impact of Counterfactual Contrastive Learning (QRM) on the Charades-RF dataset, which includes queries not grounded in videos. Results confirm that QRM substantially improves the model's ability to discern and reject ungrounded queries.

Finally, we validated the effectiveness of CADE on the Charades-CG dataset [8], which includes novel phrase compositions and unseen words in the test split. The observed performance gains highlight CADE’s crucial role in mitigating stylistic biases and enhancing the model’s generalization to unseen linguistic patterns.While MSTP indeed contributes the largest raw performance gain, CADE plays a distinct and critical role in addressing superficial co-occurrence patterns through front-door adjustment, which MSTP alone cannot resolve.

Response to Q4

To evaluate the computational overhead, we trained and evaluated all models on the QVHighlights dataset using an NVIDIA A800 GPU (80GB memory) with a batch size of 64 over 50 epochs. As summarized below, CausalVTG's inference runtime is slightly longer than simpler baselines, yet remains highly competitive.

Response to Q5

Due to rebuttal constraints, we cannot include full visualizations. As a substitute, we provide below the normalized distributions of predicted temporal intervals on the Charades-STA test set. The first table corresponds to successful cases (IoU > 0.7) and the second table corresponds to failed cases (IoU < 0.7).We commit to including full visualizations and deeper analysis of temporal prediction distributions and feature biases in the final version.

Response to S1

We thank the reviewer for pointing out these two key references, both are highly relevant to the challenges addressed in our work. We will include and discuss them in the final version.

Response to S2

For temporal causality, we thank the reviewer for highlighting these important aspects. Regarding temporal causality, the referenced works introduce queries requiring reasoning over event dependencies rather than isolated segments. We agree this is a challenging and meaningful scenario, and we plan to explore temporal causal reasoning in future work.

For multiple occurrences, the QVHighlights dataset [1] includes annotations where queries may appear multiple times within a video. Our framework can handle such cases, as illustrated by the example visualization in the third subgraph of Figure 4 in our paper, which shows correct localization of repeated events. We will clarify this capability more explicitly in the final version.

Response to S3

We appreciate the reviewer’s suggestion to evaluate causal grounding on YouCook2.Our results demonstrate that CausalVTG outperforms prior methods by a clear margin, indicating improved modeling of causal relations.

[1] "Detecting moments and highlights in videos via natural language queries." NeurIPS 2021.

[2] "Compositional temporal grounding with structured variational cross-graph correspondence learning." CVPR 2022.

[3] "Cross-modal causal relation alignment for video question grounding." CVPR 2025.

[4] "Vision-and-language navigation via causal learning." CVPR 2024.

[5] "Deconfounded video moment retrieval with causal intervention."SIGIR 2021.

[6] "A closer look at temporal sentence grounding in videos: Dataset and metric." ACM MM 2021.

[7] "Causal inference with conditional front-door adjustment and identifiable variational autoencoder." ICLR 2024.

[8] "Compositional temporal grounding with structured variational cross-graph correspondence learning." CVPR 2022.

[9] "DORi: Discovering object relationships for moment localization of a natural language query in a video." WACV 2021.

[10] "Memory-efficient temporal moment localization in long videos." EACL 2023.

[11] "Query-dependent video representation for moment retrieval and highlight detection." CVPR 2023.

[12] "Correlation-guided query-dependency calibration for video temporal grounding." arXiv preprint arXiv:2311.08835 (2023).

[13] "R2-tuning: Efficient image-to-video transfer learning for video temporal grounding." ECCV 2024.