MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning

Thank you for your insightful comments.

We fully agree that the proposed MECD represents a novel, challenging, and significant task in video understanding.

We also appreciate the acknowledgment of the interest and clarity provided by the VGCM method.

Thanks for your suggestions and for recognizing the novelty and contribution of our work. Please see the responses to your comments.

Q1. Model's generalizability

To the best of our knowledge, our benchmark is currently the first and only one for the video causal discovery task. To further validate the generalization capabilities of our model, we evaluated the quality of output causal relations on a related and representative video reasoning task: Video Question Answering (VQA).

Specifically, during inference on the ActivityNet-QA dataset, we prompted Minigpt4-video with additional causal relations outputs alongside the standard question inputs. This paradigm facilitates the VLLMs to consider the task from a causal perspective. As shown in the table below, when prompted with these additional causal relations, the answering accuracy of Minigpt4-video improved by our VGCM surpasses other strong VLLMs like VideoChat2. These findings confirm that our model can provide accurate causal perception for videos, significantly improving performance on related video reasoning tasks.

Q2. Improve the display of Fig.1

Thanks for your advice, in the final version, we will appropriately explain the key elements in Fig.1 and the revised version of Fig.1 can be found in the attached PDF file in our rebuttal.

Q3. Additional metrics

As mentioned in our main manuscript (Sec. 2 lines 100-104 & Sec. 5.4 lines 318-321), our model can output a complete causal diagram, consequently, we can introduce Structural Hamming Distance (SHD) [1-2] as a supplementary metric. SHD measures the degree of matching between causal graphs by summing the number of incorrect causal relations. Compared to the Acc metric, it focus more on the global causal relationships for all events in each video.

In our MECD test set, the average number of causal relations in video causal graphs is 12.31. We report the average SHD for incomprehension in the following table, a lower Ave SHD value indicates better performance.

The results also indicate that for a majority of the models, the current metric, accuracy in identifying causal relations leading to the result event, is already adequate to represent their causal discovery capabilities. However, Gemini and GPT-4 exhibit a superior overall capacity for discovering complete causal relations.

Q4. Dataset & Code available

Thanks for your suggestion. We confirm that our code and dataset will be released as soon as possible.

Q5. Confounding factor

As you have thought, the confounding factor is unfavorable in causal relation discovery. For a further analysis of lines 205-226: When $e_k$ is masked for comparison, the causal relations between $e_k$'s adjacent events and the last event $e_N$ are affected, leading to a confounding of causal effects.

In lines 154-155, we conduct the Event Causality Test to compare the predictions of the two streams. Consequently, under the concept of control variables, as we want to evaluate the causal effect between the current event $e_k$ and $e_N$, we must prevent the causal effects between the adjacent events and $e_N$ from being affected by confounding factors. Confounding factors result in a redundant or missing causal effect. Therefore, we introduce causal inference to cut off the redundant causal effect between $e_{k+1}$ and $e_N$ by counterfactual intervention. Likewise, by reconstructing $e_k$'s causal effect through front-door adjustment, we introduce the distinction after reconstruction in the causal effect between $e_{k-1}$ and $e_N$ as the compensation.

Q6. Limitation: Captions dependency

Currently, with the rapid development of VLLMs, we believe that generating accurate captions for videos will be much easier.

Moreover, in our supplementary (Sec. E, lines 554-567), we have experimented with the dependency of the caption and video inputs, respectively. Specifically, we explored the effect of altering the input format of the two modalities and found that when we masked out 80% of the captions, the accuracy only dropped by 2.3%, proving that we can also conduct tests for videos without any captions.

Thanks for pointing this and we'll delve deeper into this issue in our future work.

References

[1] Knowledge transfer for causal discovery. IJAR 2022.

[2] Tuning causal discovery algorithms. PMLR 2020.

Thank you for your review and comments. We’re glad to hear that you found our task interesting and that you see our methods as insightful. Please see below for responses to your comments and questions.

Q1. Dataset Scale

We have the following points to this issue:

1. We envision MECD as a long-term, continuously maintained benchmark. We will progressively expand its scope by incorporating a wider range of video types and increasing data volumes. In the final version, we plan to include an additional validation set as we have collected and annotated more videos. Nonetheless, it's important to note that the current MECD is already effective in assessing the capabilities of LLMs for video causal discovery. As demonstrated in Tab. 1 of our main manuscript, even with a limited number of test samples, LLMs struggle to accurately identify and discover causal relations between video events.

2. In causal discovery tasks, the evaluation scale is more accurately reflected by the number of event pairs rather than just the number of video samples. This is analogous to similar NLP tasks focused on causal discovery within texts or documents. Frequently utilized datasets in this domain, such as SeRI [1-2], Causal-TimeBank [3-4], HiEve [5], and EventStoryLine [6], comprise a limited number of test samples. For instance, Causal-TimeBank, HiEve, and EventStoryLine datasets have only 95, 100, and 318 articles or documents available for testing, respectively. However, as the evaluation is conducted between event pairs within documents, these datasets contain 1,475, 2,282, 2,725, and 4,027 event pairs, respectively. Similarly, our MECD provides 3,681 event pairs during the test phase, ensuring a comprehensive and convincing evaluation.

3. Finally, as suggested by Reviewer-eXFX, we also try to incorporate additional metrics to provide a more comprehensive evaluation. For example, we newly report the Ave SHD metric for different baselines to reveal the ability of complete causal graph discovery for the whole video (cf. our response to Q3 of Reviewer-eXFX).

Q2. More comparison

Thank you for your suggestion. In the table below, we included results for Video-Llama and two image-based VLLMs (Instruct BLIP and LLaVA) for a comprehensive comparison. The results demonstrate that image-based LLaVA does not outperform its video-based counterpart Video-LLaVA in our task. This finding further confirms the challenging nature of our task and highlights the necessity of long-term temporal understanding to accurately identify causal relationships between video events.

Q3. Overview of causal concepts

Thanks for your valuable advice. We will add an overview of the causal concepts to the final version.

Q4-1. Base VideoBERT performance

We have also conducted experiments on our baseline model VideoBERT, achieving an accuracy of 60.9%. Our VGCM model (71.2%) has made a significant improvement compared to the baseline model.

Q4-2. Training and evaluation under strongly-supervised setting

For base multi-modal models VAR and Videobert, we report results from models fully trained on the MECD dataset. All LLM-based models are evaluated under a few-shot setting rather than zero-shot (cf. supplementary Sec. I, lines 600-606). Specifically, following the approach in causal discovery for NLP tasks [2, 7], three representative examples are provided during inference. We also investigated the impact of varying the number of few-shot examples, demonstrating the adequacy of our prompt (cf. supplementary Sec. G, lines 572-578).

Furthermore, as you suggested, to ensure a fairer comparison, we conducted a strongly-supervised experiment on the advanced VLLM method Video-LLaVA. Specifically, we fine-tuned its LLM and encoder components using LoRA under its official implementation on our entire MECD training set. As shown in the table below, Video-LLaVA gains a 4.6% improvement from fine-tuning on our dataset. However, it still falls behind our proposed method. Given that fine-tuning LLM-based baselines is time-consuming, we will include more results of VLLMs with strongly-supervised settings for a comprehensive comparison in our final version.

Q5. Definition of premise event

Thanks for your advice, the premise event is the event that happens earlier than the result event in the same video, and we will add the definition to the final version.

References

[1] Seri: A dataset for sub-event relation inference from an encyclopedia. NLPCC 2018.

[2] Reasoning subevent relation over heterogeneous event graph. KIS 2024.

[3] Annotating causality in the TempEval-3 corpus. EACL 2014.

[4] Learning to teach large language models logical reasoning. arXiv 2023.

[5] HiEve: A corpus for extracting event hierarchies from news stories. LREC 2014.

[6] Neural granger causality. TPAMI 2021.

[7] Is chatgpt a good causal reasoner? a comprehensive evaluation. EMNLP 2023.

Thanks for your valuable suggestions. We’re glad that you found our work interesting and novel. We address your concerns below.

Q1. Complexity of implementation

We have the following points for this question:

1. The main pipeline of our framework is clear and not overly complex in its implementation:

• Our basic pipeline (Sec. 4.1) follows a straightforward dual-branch architecture for processing visual and textual inputs. Semantic and causal information interact between two branches, culminating in causality estimation via a masked-based prediction task in both modalities.

• While the causal inference component (Sec. 4.2) may require some background knowledge to fully grasp theoretically, its practical implementation is neat and clear. In our final manuscript, we will provide pseudocode to further clarify these aspects.

• Contrary to the reviewer's concerns, our framework does not rely on numerous advanced techniques or require additional large-scale pretraining. It only requires pretrained visual and textual feature encoders, which are fundamental and readily accessible in the multi-modality understanding community.

• Our framework is also highly flexible, allowing researchers to replace or modify specific modules for targeted improvements. Options include selecting different visual/textual encoders, altering interaction methods between modalities, or adjusting the similarity evaluation criteria for the prediction task.

2. The simplicity of our implementation is further evidenced by its parameters and inference speed:

• Our model contains only 144M parameters, significantly smaller than even the lightest VLLM models (e.g., Video-LLaVA, VideoChat2), which have 7B parameters (~50 times more than ours).

• Our method is fast (0.76s/sample) in inference speed. The overall method introduces only 8.57% overheads to the VideoBERT baseline. Notably, our inference speed is 3-6 times faster than all VLLMs (as shown in Table 1). Our inference speed experiment is conducted on 1 NVIDIA A6000 GPU.

3. To facilitate replication and further research, we will release our code along with the dataset as soon as possible. This will enable other researchers to reproduce our results and build upon our work.

[Table 1. Efficiency of different models. Reported with average inference speed of each sample.]

Q2-1. Details of encoders

We do have provided more details about the architecture of encoders in our supplementary (Sec. B lines 510-511). Specifically, our encoder $\text{Enc}_v$ and $\text{Enc}_c$ are built upon VideoBERT, a joint model for video and language representation learning.

Besides, as also provided in our supplementary (Sec. B lines 501-502), the architecture of our pretrained encoder $\Phi_{pre}$ is ResNet-200, following the setting of the Dense Video Captioning task [1-2].

Q2-2. Details of pretraining

We have provided the detailed pre-training procedures in our supplementary (Sec. B, lines 501-504). For $\text{Enc}_v$ and $\text{Enc}_c$, our pre-training approach is relatively modest. We conducted a video dense captioning task on 3.1K samples from the ActivityNet Caption dataset to warm up the model. It requires a low resource footprint and time cost (2 NVIDIA A6000 GPUs for 8 hours), while leading to a significant improvement in final performance from 69.5 to 71.2.

Q2-3. Details of loss design

Overall, the causal relation loss $L_R$ provides direct supervision for output causal relations using standard Cross-Entropy loss, while $L_C$, $L_V$, and $L_S$ aim to further enhance the causal discovery capability from different aspects.

Specifically, $L_C$ and $L_V$ introduce auxiliary caption and reconstruction losses to facilitate event prediction, implemented by Label Smoothing loss and MSE loss respectively. They are incorporated because the Granger Causality Method determines whether an earlier event aids in predicting a subsequent one.

$L_S$, the similarity loss (implemented by InfoNCE), prompting causal relation discovery by comparing output causal feature similarity when a premise event is masked. The rationale behind this loss is that masking a non-causal event should result in a prediction of the result event similar to that of the unmasked stream.

In our manuscript, we have thoroughly examined the contribution of each loss term in Tab. 2 and Sec. 5.2 (lines 285-289). Furthermore, we have detailed the balance between these losses in Sec. 4.1 (lines 188-200), and outlined the specific hyperparameter settings in our supplementary (Sec. B lines 513-514).

Q2-4. Details about the training

Details about the training procedures have been provided in our supplementary (cf. Sec. B, lines 505-509).

Q2-5. Details about fair comparisons

To make a fair comparison, for multi-modal backbones VAR and VideoBERT, we reported their results after being fully trained on the MECD dataset. For all LLM-based models, we reported the results under the few-shot paradigm (cf. supplementary Sec. I, lines 601-606), where three representative examples are provided during inference, similar to the paradigm used in causal discovery in NLP tasks [3-4].

Q3. Typos

Thank you for your corrections. We will correct them in our final version.

References

[1] Visual abductive reasoning. CVPR 2022.

[2] End-to-end dense video captioning with parallel decoding. CVPR 2021.

[3] Reasoning subevent relation over heterogeneous event graph. KIS 2024.

[4] Is chatgpt a good causal reasoner? a comprehensive evaluation. EMNLP 2023.

Thanks for your great efforts and time in reviewing our paper.

1. Results of fine-tuned Video-LLaVA

Firstly, we want to state that the few-shot evaluation (In-Context Learning) for LLMs is widely recognized as a strong baseline for reasoning and causal discovery tasks [1-6], effectively reflecting their performance on downstream tasks. Therefore, comparing LLMs in a few-shot setting is a common and accepted practice in the field.

Secondly, neither GPT-4 nor Gemini-Pro offer interfaces for fine-tuning, consequently, we reported the few-shot results of LLMs and investigated the impact of varying the number of few-shot examples, demonstrating the adequacy of our prompt (cf. supplementary Sec. G, lines 572-578).

Furthermore, as you and reviewer-2Vef suggested, to ensure a fairer comparison, we conducted a strongly-supervised experiment on the open-source method Video-LLaVA. Specifically, we fine-tuned Video-LLaVA using LoRA under its official implementation on our MECD training set. As shown in the table, Video-LLaVA gains a 4.6% improvement from fine-tuning on our MECD. However, it still falls behind our proposed method. Since fine-tuning LLM-based baselines is time-consuming, we will include more results of fine-tuning VLLMs for a comprehensive comparison in our final version.

[1] Language models are few-shot learners. NeurIPS 2020.

[2] Rethinking the role of demonstrations: what makes in-context learning work? EMNLP 2022.

[3] Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. NeurIPS 2023.

[4] CLADDER: assessing causal reasoning in language models. NeurIPS 2023.

[5] Causal inference using llm-guided discovery. AAAI 2024.

[6] Open Event Causality Extraction by the Assistance of LLM in Task Annotation, Dataset, and Method. ACL 2024.

2. More details

We're happy to provide more details here to clarify your concerns. However, we need to state that these details can be also found in the supplementary in our initial submission. To further relieve your concerns about our implementation and reproducibility, we provide an anonymous GitHub project (https://anonymous.4open.science/r/NeurIPS-4887-MECD), which contains our data annotation, training and evaluation codes, and details about how we evaluate LLMs and how we fine-tuning the Video-LLaVA. Additional details can be found in the README.md file.

• Training procedure: All the experiments are conducted on 1 NVIDIA A40 GPU. We train our model for 20 epochs with a learning rate of 16e-5 about 6 hours. Our optimizer is consistent with the BertAdam optimizer, with 3 epochs of warm-up. We report the average results during all experiments under three random seeds (2023, 2024, 2025).
• Hyperparameters: $\lambda_C$, $\lambda_R$, $\lambda_V$,$\lambda_S$ are set to be $1.0, 4.0, 0.25, 0.05$.
• Encoder & Decoder architecture: Our encoder $Enc_{v}$, $Enc_{c}$, and multi-modal video decoder $Dec$ are built upon videoBERT, a joint model for video and language representation learning.
• Evaluation of LLMs: We prompt the GPT-4 with the following few-shot prompts to conduct evaluation (Details can be found in the anonymous GitHub project):

# Task: Each video consists of n events, and the text description of each event has been given correspondingly (separated by " ",). You need to judge whether the first n-1 events in the video are the cause of the last event, and the probability of the cause 0(uncausal) or 1(causal) is expressed as the output, Let's think step by step through the chain of thought.
Here are several examples of judging whether the first n-1 events in the video are the cause of the last event:
<start>
First example:
Text description of n events:
...
The probability output should be:
...

Second example:
Text description of n events:
...
The probability output should be:
...

Third example:
Text description of n events:
...
The probability output should be:
...
<end>

As a long-term maintenance benchmark, we confirm that the dataset and code will be public after being accepted.

We hope our responses can address your concerns and raise your rating of our paper.